Smaller retrieval windows outperform larger ones past a threshold

A recurring finding across recent retrieval-augmented generation benchmarks is that stuffing more chunks into a model's context doesn't keep paying off the way intuition suggests. Once a retrieval pipeline crosses somewhere around twenty candidate chunks, accuracy on downstream question-answering and summarization tasks tends to plateau and then decline, even though the model technically has more relevant information available to it. The likely mechanism is that models struggle to weigh many similarly-scored passages against each other; irrelevant or only tangentially related chunks start diluting attention over the genuinely useful ones, and the model's own internal notion of "what matters here" gets noisier as the candidate pool grows. This has a very practical implication for anyone building a RAG system: it's tempting to treat retrieval as a numbers game, where casting a wider net and returning more results feels safer than being aggressive about filtering. But the data suggests the opposite investment pays off better. Teams get more mileage out of a strong reranking step that aggressively narrows a large initial retrieval set down to the handful of passages that actually matter, rather than passing a loosely filtered set of twenty-plus chunks straight to the generator and hoping the model sorts it out. In practice this means: retrieve broadly if you want recall, but rerank hard before you ever hit the context window, and treat "more context" as a cost with diminishing and eventually negative returns rather than a free lever to pull when accuracy disappoints.

Source

View on ShipDigest