Project — Interactive Learning
Vectorization & Embeddings
How do you measure the distance between two ideas? This page builds the answer from the ground up — turning words into numbers you can add, subtract, and compare — with live experiments at every step. No math background needed.
But is king closer to queen, or to carrot?
Here is the whole puzzle in one question. Suppose I ask you how far apart the words coffee and tea are. You have an instant, confident intuition — they're close. Coffee and gravity? Far. You did that without any numbers. But how? What were you actually measuring?
You were comparing qualities: both are hot drinks, both contain caffeine, both are things you order in a café. The more qualities two words share, the closer they feel. That's the key that unlocks everything on this page: if we can write down a word's qualities as numbers — a score for "is it a drink," a score for "is it hot," a score for a thousand subtler things — then meaning becomes a point in space, and distance between meanings becomes ordinary geometry.
Two honest questions fall out of this, and the rest of the page answers them by letting you try it, not just read about it:
1. Where do the numbers come from? Nobody hand-scores a dictionary. We'll see how machines learn the qualities on their own, purely from watching which words keep company with which. 2. How do we measure distance once we have them? Not with a ruler — with the angle between two vectors, a trick called cosine similarity.
Why the angle and not plain straight-line distance? Because a vector has two things going on: its direction (which way it points) and its length. Direction turns out to carry the meaning; length mostly reflects boring stuff like how often a word appears. Measuring the angle compares pure direction and ignores length — so "big" and "enormous" count as close even if one word is far more common than the other.
Meaning, one number at a time
Before any machine learning, let's do the thing by hand — because once you've felt a word become coordinates, the rest is just automation. Below, three sliders each measure one quality of a word. Drag them to describe a concept, and watch a vector appear. The panel tells you which real word your coordinates land nearest to.
Real embedding models do exactly this — but with hundreds of axes instead of three, and nobody labels them. The model invents whatever qualities best explain how words are used. That's the leap the next section tells the story of.
A short history in three leaps
The idea of "words as numbers" is older than deep learning. It arrives in three acts — each one fixing the thing the last one couldn't do.
The first vectors were simple: count the words. Represent a document as a long list of word counts — one slot per word in the vocabulary. TF-IDF (term frequency × inverse document frequency) refined this by down-weighting words that appear everywhere ("the," "of") and up-weighting words that are rare and therefore distinctive. It's brilliant, fast, and still powers search engines today.
But it has a fatal blind spot for meaning: every word is its own isolated slot. To TF-IDF, "cat" and "kitten" are as unrelated as "cat" and "helicopter" — they share no letters of overlap in the vector, so their distance is maximal. It can tell you which words a text contains, never what those words mean.
| word → | cat | kitten | dog | finance |
|---|---|---|---|---|
| Doc A | 0.9 | 0.0 | 0.4 | 0.0 |
| Doc B | 0.0 | 0.8 | 0.0 | 0.0 |
In 2013, a team at Google led by Tomáš Mikolov published word2vec, and it changed the field overnight. The insight is deceptively simple, and it's an old one from linguistics — "you shall know a word by the company it keeps." A word's meaning is defined by the words that surround it. "Coffee" and "tea" are similar not because anyone said so, but because they appear in nearly identical contexts: "a cup of ___," "I drink ___ in the morning."
word2vec turned that into a learning game. Slide a window across billions of sentences and play one guessing game over and over: given these neighbours, what word sits in the middle? Each guess, the model nudges its numbers to do a little better next time — millions of tiny corrections. ("Training a neural network" is just this loop, run at scale.) To win the game, the model is forced to give words that share contexts similar vectors. Nobody labels a single axis — the qualities emerge on their own, packed into a few hundred dense numbers per word.
And then someone noticed something astonishing. The learned space had structure you could do algebra on. Directions in the space carried meaning — one direction was gender, another was royalty — so you could literally add and subtract concepts:
king − man + woman ≈ queen
You can't subtract words, of course — but remember, each word is now just a list of numbers. So this is ordinary arithmetic: line up the lists and subtract them slot by slot, then add. Take "king," subtract "man" (which cancels out the maleness), add "woman" — and the resulting list of numbers lands almost exactly on the one for "queen." Meaning had become geometry you could compute with. You can run this yourself, on real word2vec vectors, in the Playground below.
word2vec has one limitation: each word gets one vector, forever. But "bank" by a river and "bank" that holds your money are different meanings sharing one spelling. The modern era — BERT, and the Transformer models that followed — fixed this by making a word's vector depend on the whole sentence around it. The same word gets a different embedding every time, tuned to how it's actually being used.
This is the direct ancestor of today's embedding models that power semantic search and Retrieval-Augmented Generation. The full list of what's current lives in the "Today" section at the bottom.
The Playground
Everything below runs on real pretrained word2vec vectors (Google News, 300 dimensions) for a curated vocabulary of 167 words — the actual numbers, computed right in your browser. Nothing is faked or hard-coded. Poke at it. Meaning is more manipulable than it looks.
This flattening (PCA) loses detail — the real space has 300 axes — but the clustering you see is real, and it was learned, never programmed.
Not every analogy lands perfectly — the vocabulary here is small and 2013-era vectors are blunt instruments. That it works at all, from nothing but counting co-occurrences, is the miracle.
Where embeddings are today
word2vec was 2013. The line of descent since has been about context, scale, and embedding whole sentences and documents rather than lone words. A working shortlist of what powers modern systems:
and meaning turned out to have a shape.
Every search that understands you, every recommendation that feels uncanny, every AI that answers from your documents — underneath, it is doing the one small thing you did by hand at the top of this page: turning words into points, and measuring what is near. The whole cathedral of modern AI is built on that single, strange, beautiful idea.