Why RAG is an evidence design problem, not a buzzword

RAG usually gets introduced as "letting the model look things up." Accurate, and not very useful. It describes the mechanism and skips the decision.

The decision is: what is your system allowed to answer from?

The closed-book problem

Out of the box, an LLM answers from training. Everything it absorbed, frozen at a cutoff, with no way to know which parts apply to you. Ask it about your refund policy and it will confidently produce a refund policy: a plausible average of every refund policy on the public internet.

RAG changes the exam format. Before the model writes a word, the system fetches relevant material and puts it in front of it: answer from this, not from memory.

The consequence people miss is that the model does not get smarter; it just stops guessing about things it never knew. That is a narrower claim than most RAG marketing makes, and a far more useful one.

Your quality ceiling moves

Once you add retrieval, answer quality stops being mostly a model property and becomes mostly a retrieval property.

That trade is worth making, but be clear about what you have signed up for. A weak model with excellent evidence produces a decent answer. A strong model with the wrong three paragraphs produces an articulate, well-structured, completely wrong answer, and it will look more trustworthy than the first one, because fluency scales with model quality while correctness scales with what you fed it.

Most "the model is hallucinating" escalations I have seen dissect into the model faithfully summarising the wrong pages.

Three decisions that outlive your tech stack

Your vector database will get replaced. These will not:

• What counts as a trusted source. Which documents may an answer be built on? Somebody decides this. If nobody decides explicitly, ingestion decides it accidentally, by whatever it happened to crawl.
• What a claim has to carry. If an answer asserts something material, can a reviewer trace it to a specific passage in under a minute? Design that path now; retrofitting provenance is painful.
• What happens when evidence is thin. This is the one teams skip, and it is the one users judge you on. Hedge, escalate, or say plainly that nothing solid was found, but pick, because the default is to generate anyway.

A test you can run this week

Take one real user question. Trace it end to end: what got retrieved, in what order, which passages reached the model, and which sentences in the final answer came from which passage.

One question, done properly, tells you more about your RAG system than a dashboard of aggregate scores. You are looking for the gap between what the answer asserted and what the evidence actually supported.

Closing thought

"Retrieve first, generate second" sounds like an implementation detail. It is really a statement about authority: the system's answers are bounded by a body of evidence you chose, rather than by whatever the model absorbed in training.

Almost everything else in RAG (chunking, hybrid search, reranking, citations) is a refinement of that one habit.

What is inside your AI's open book right now, and when did anyone last read it?