Why RAG needs both broad context and precise evidence

There is a structural tension in every retrieval system, and no chunk size resolves it.

Small chunks retrieve well and read badly. The embedding is focused, so search finds them precisely, but pulled out alone, a two-sentence fragment has lost the definitions, conditions, and qualifiers that gave it meaning.

Large chunks read well and retrieve badly. All the context is there, but the embedding now averages six different topics into one vague point in space, and precise queries stop matching it.

Pick either extreme and you lose something real.

Search small, answer big

Parent-child retrieval resolves the tension by refusing to use one unit for both jobs.

Index the small chunks, because that is what makes search precise. But when a child chunk matches, do not hand the model the child, hand it the parent: the section that chunk came from, with the surrounding paragraphs intact.

The match happens at street level. The evidence arrives as the whole neighbourhood.

Where this earns its keep

The clearest signal you need it: your system keeps quoting the rule and missing the exception directly beneath it.

That failure has a specific shape. Retrieval was correct. It found the passage stating the rule. The exception lived two paragraphs down, in a different chunk, which did not match the query because it never restated the rule it was modifying. The model answered faithfully from what it received and was still wrong.

For policies, contracts, technical documentation, medical and legal content (anywhere meaning is carried by conditions and carve-outs), this pattern is close to mandatory.

Three details that decide whether it works

• Size the parent sensibly. A parent spanning an entire 40-page policy defeats the purpose; you have reintroduced noise at the answer stage. Sections and subsections, not whole documents.
• Deduplicate. Three child matches inside one parent should produce one parent, not the same text three times. Skip this and you burn context window on repetition and skew the model toward over-weighting that source.
• Budget the context. Expanded parents are large. Decide how many you can afford per answer before latency and token cost decide for you.

Closing thought

Retrieve broadly, answer precisely, or rather, match precisely and answer with enough surrounding context that the match still means what it meant in the document.

The right sentence, stripped of the paragraph that qualified it, is a quote out of context, and the model has no way to tell the difference.

Which of your answers are built on passages that lost their qualifiers on the way in?