Why weak evidence should trigger recovery, not confidence

What does your RAG system do when retrieval comes back weak?

For most systems the honest answer is: exactly what it does when retrieval comes back strong. Between retrieval and generation there is nothing: no gate, no check, no branch. Whatever came back flows into the model, and the model does what models do, which is write something fluent with it.

Weak evidence should change behaviour.

If retrieval is thin, stale, contradictory, or unauthorised, the system should recover, ask, escalate, or refuse, not produce a confident answer anyway.

Installing the gate

Corrective RAG puts a checkpoint in that gap. Before generating, grade the evidence: is it relevant, sufficient, internally consistent, current enough?

If it passes, proceed. If it fails, take a different path: rewrite the query and search again, widen to another source, or state plainly that nothing solid was found.

The grader does not need to be expensive. It is judging relevance rather than writing prose, so a small fast model handles it well, and the latency cost is far lower than most teams expect for the reliability gained.

Two controls keep it from becoming a problem of its own:

• Cap the retries. Two corrected searches, then escalate or decline. An uncapped correction loop is a cost incident that looks reasonable at every individual step.
• Log the corrections. Which queries needed a second pass, and did it help? This is one of the most useful diagnostic streams you can collect, because it shows precisely where your first-pass retrieval is weak.

Refusal is a feature, and a free roadmap

Teams resist the "I could not find a reliable answer" path because it feels like admitting the product does not work.

Users see it differently. They forgive "I do not know" almost completely. They do not forgive being confidently misled, and they do not come back afterwards. One fabricated answer costs more trust than fifty honest refusals.

There is a second benefit that is easy to miss: track your refusal rate by topic and you have a map of the holes in your knowledge base. A cluster of refusals around a product area is the system telling you which documentation to write next. That signal only exists if the system is allowed to refuse. A pipeline that always answers has no way to report its own gaps.

The test that matters

Feed the system a question your sources genuinely answer poorly. Not a trick question, a real one that falls in a gap.

Watch what it does. If it retries, asks for clarification, or declines, the gate works. If it produces a smooth, plausible, unsupported paragraph, you have found the behaviour your users will eventually encounter without knowing it.