Project — Interactive Learning
BM25 & Keyword Retrieval
Semantic search understands what you mean. But sometimes you don't want the meaning — you want the exact word: a part number, an error code, a surname. This page builds BM25, the search algorithm that still runs the web, one honest fix at a time — with a live experiment for every idea.
Ask it for "SKU‑4471X" — and it hands you 4470, 4472, anything shaped like a code.
If you read the embeddings page, you saw the great modern idea: turn words into points in space so that close means similar in meaning. It's the engine behind semantic search and RAG. So why would anyone go back to matching literal words?
Because meaning-based search has a failure mode that is easy to miss: it never says "I don't know." Search for the error string "ERR_CONN_TIMED_OUT", a citation like "§ 230(c)(1)", a rare surname, a product SKU, a function name in a codebase — and an embedding model will confidently return things that are vaguely similar in vibe while sailing right past the exact match you needed. The words carry information that lives in the spelling itself, not in the cloud of meaning around them.
So this page rebuilds keyword search from its oldest form and watches it grow — through three specific, fixable problems — into BM25 (the name is an accident of history: it was the 25th "Best Matching" formula in a 1970s–90s research line at City University London). BM25 is thirty years old, embarrassingly simple, and still the default ranking function in Elasticsearch, OpenSearch, Lucene, and the search bar of half the sites you used today. Let's earn every piece of it.
Start by just counting words
Here is the most naive search imaginable, and it's where everything begins. To find documents about "python," count how many times each document says "python." More mentions, higher rank. This is called bag‑of‑words, and the count is the term frequency (TF).
It is a genuinely good instinct — a document that keeps saying your word probably is about your word. But used raw, it has two glaring bugs that anyone can feel:
Bug 1 — repetition runs away. Is a page that says "python" 40 times really 40× more relevant than one that says it once? A spammer could win every search just by pasting a word a thousand times. There should be a point of diminishing returns.
Bug 2 — common words drown out rare ones. Search "the python tutorial." The word "the" appears in every document ever written — counting it adds noise, not signal. The rare, specific word ("python") is the one that actually tells you where to look. Raw counting treats them as equals.
And a quieter third problem — long documents cheat. A 10,000‑word page will rack up more mentions of almost any word than a tight 50‑word snippet, just by being long. Length shouldn't buy relevance.
Three repairs, three knobs
Everything below runs the real BM25 math live in your browser — no faked numbers. Each widget isolates a single term of the formula so you can watch exactly what it does. Drag things. Break things. The formula will assemble itself by the end.
Assemble the whole formula
Now snap the three fixes together and you have BM25 — nothing more is hidden. For each word in the query, BM25 computes three things you already met — IDF (how rare, how much it matters), saturated TF (mentions, with a ceiling), length factor (adjusted for document size) — multiplies them, and sums across the query's words. That sum is the score. That's the entire algorithm.
Below is a tiny corpus with a fixed query — python transformers. Drag k₁ and b, then click any document to see its score fully worked out, term by term. The ranking re-sorts live. Try the challenge underneath.
Where BM25 shines — and where it goes blind
Every retrieval method is a set of trade-offs wearing a formula. BM25's strengths and its blind spots are two sides of the exact same coin: it reads the literal words and nothing else. That makes it razor-sharp on identity and stone-deaf to meaning.
- Exact tokens. Part numbers, error codes, SKUs, function names, citations — the strings whose value is the spelling itself.
- Rare & out-of-vocabulary words. A surname or acronym an embedding model never saw in training still matches perfectly, letter for letter.
- Fully interpretable. You can read why a document ranked #1, term by term — you just did it above. No black box.
- Fast & cheap. No GPU, no model, no vectors to store. An inverted index over billions of documents answers in milliseconds.
- Zero training. It works on your data the moment you index it — no fine-tuning, no embeddings to compute.
- Synonyms. "car" and "automobile" share no letters, so to BM25 they are strangers. It matches spellings, not concepts.
- Paraphrase. "How do I fix a flat tyre?" won't find a doc titled "repairing a punctured wheel." No overlapping words, no match.
- Meaning & intent. It has no idea "bank" can be a river or a vault. Every sense of a word is the same string to it.
- Vocabulary mismatch. When the user and the document use different words for the same thing, BM25 simply misses — silently.
Look at those two columns side by side and something jumps out: they are almost perfect mirror images. Everything BM25 is bad at, semantic search is good at — and the one thing embeddings fumble (exact, rare tokens) is precisely what BM25 nails. That is not a coincidence. It's the whole reason for the last section.
An addition, never a replacement
It would be easy to leave this page thinking BM25 is the "old way" and embeddings are the "new way" — that one replaces the other. That is the single most common misread of modern search, and it's backwards. In serious systems, BM25 doesn't compete with semantic search; it rides alongside it, covering the exact cases where meaning-based search is weakest.
Think of it as a safety net under semantic search, not a rival to it. When a user pastes an error code or a product ID, BM25 is the layer that guarantees the exact match surfaces instead of being smoothed into a cloud of "close enough." You are not choosing between them — you are keeping vector search and letting BM25 catch what it drops.
The word for fusing the two scores into one ranked list is hybrid search — and it earns its own page, because doing it well (score normalization, reciprocal-rank fusion, re-ranking) is a real topic in itself. This page stops one step short on purpose: understand BM25 as a complement first, and hybrid will make far more sense when you get there.
Hybrid Search — fusing the two → (coming soon)never actually left.
Thirty years on, after every leap in neural search, the fastest way to guarantee you find the exact thing you typed is still to count words, weight the rare ones, and forgive the long documents. BM25 isn't a relic that embeddings made obsolete — it's the sharp, literal partner that makes meaning-based search trustworthy. Keep them both. The best systems always have.