New here? Start this series at Day 13: Why chunk boundaries shape answer quality
- Day 13: Why chunk boundaries shape answer quality
- Day 14: Why embeddings are useful and easy to overtrust
- Day 15: Why vector stores are infrastructure, not magic memory
- Day 16: Why keyword search and semantic search both matter
- Day 17: Why hybrid search is often the practical default
- Day 18: Why relevant data can still be unauthorized data
Search error code E-4012 with semantic search and you may get a thoughtful paragraph about error handling in general.
Search why do customers stop paying with keyword search and you get nothing, because no document in your corpus contains that exact sentence.
The two failures have different causes. Keyword and semantic search are different tools for different jobs, and "which is better" is the wrong question.
What each one is actually doing
Keyword search matches the literal string, usually with statistical weighting so rare terms count for more. It is fast, precise, explainable, and completely indifferent to the fact that "cancel" and "terminate" mean the same thing to your users.
Semantic search compares meaning through embeddings. It handles paraphrase beautifully and is oddly unreliable on the exact tokens (part numbers, error codes, policy names) where a single character changes everything.
| Retrieval signal | Strong at | Weak at |
|---|---|---|
| Keyword | Exact terms, IDs, SKUs, error codes, quoted phrases, rare jargon | Paraphrase, synonyms, questions phrased in user language |
| Semantic | Intent, related concepts, natural-language questions, vocabulary mismatch | Exact identifiers, negation, recency, precise policy names |
| Reranking | Choosing the best few from many candidates | Finding anything that first-stage retrieval never returned |
That last row matters more than it looks. Reranking is often proposed as the fix for retrieval problems, and it cannot promote a document that never made it into the candidate set.
Stop debating, read your logs
The useful move here is empirical rather than architectural, and it takes about an hour.
Pull a day of real production queries. Sort them into two piles: the ones that hinge on an exact string, and the ones that describe an intent. Nearly every real query stream has substantial volume in both piles. The ratio is almost never what the team assumed in the design meeting.
Now check which pile your retrieval currently serves. Choosing a single method is choosing which of those users to quietly fail, and because a missed result looks like one unlucky answer rather than a systematic gap, you can fail them for months without noticing.
The failure is invisible per query
This is why the problem persists. When keyword-only search misses a paraphrased question, the user sees "no results" and rephrases or leaves. When semantic-only search mangles an invoice number, the user sees a plausible-but-wrong document and may not realise it is wrong.
Neither event looks like a retrieval architecture problem. Both look like one bad day. The pattern only becomes visible in aggregate, which is precisely why you have to look in aggregate.
Closing thought
Know which detective you hired, and know which cases that detective structurally cannot solve.
Tomorrow: what happens when you put both of them on the same case.
More on these topics
Deep dive · · 6 min read
Most RAG failures are faithful answers to bad evidence
Six posts on reranking, citations and decomposition, and the uncomfortable thing they turn out to have in common.
Deep dive · · 7 min read
No model can reason about evidence your retriever never returned
Six days of retrieval notes, and not one of the failures announced itself.
Explainer · · 2 min read
Test retrieval on its own or you will tune the wrong thing
Most "the model is wrong" problems are recall problems. Hybrid search, query rewriting and versioned indexes are where they get fixed.
Discussion