FIELD NOTE / SYSTEMS NOTE
Evaluating and Improving Retrieval Quality in RAG Systems
A RAG system that retrieves the wrong document answers confidently from bad evidence—and no amount of prompt engineering fixes a retrieval failure. Retrieval quality is the part of the pipeline most teams measure last and break most often: it degrades silently when documents change, embedding models drift, or query distributions shift. This note covers how to measure retrieval precision and recall at the chunk level, diagnose the specific failure mode when retrieval goes wrong, and build the evaluation infrastructure that catches degradation before users do.
Retrieval is not search
Keyword search returns strings that match a query. Retrieval for RAG must return the evidence that lets a downstream model answer a task correctly. That distinction matters because semantic similarity can rank a paragraph about refunds above the exact table that contains the eligibility threshold. In a retail-banking assistant, a dense index scored 0.84 average cosine similarity on 2,000 questions while Recall@5 was only 0.71; the model saw plausible context, but 29% of required chunks never arrived. The team initially tuned the prompt and gained no measurable lift. Once retrieval was evaluated independently, the missing signal was obvious: account identifiers and policy codes were lexical anchors that embeddings underweighted. Treat retrieval as a measurable contract between a query and evidence, not as a black-box search box. The output of retrieval is a ranked set of chunks, each with provenance, access decisions, and a reason it was eligible. Generation quality is downstream of that set.
The retrieval failure taxonomy
Four failure modes cover most incidents I have debugged. A dense miss occurs when semantically related language is far from the query in embedding space. A sparse miss occurs when exact identifiers, product codes, or spelling variants are absent from the lexical match. A chunking failure happens when the required fact is split across boundaries or its heading is detached from the value. A preprocessing failure removes tables, OCR text, code, or metadata before indexing. They present differently: dense misses have low embedding similarity but reasonable keywords; sparse misses have high lexical overlap but weak semantic ranking; chunking failures return neighboring fragments without the answer; preprocessing failures return an apparently relevant chunk whose critical field is blank. In a healthcare retrieval audit, 400 medication questions produced 62 misses: 27 dense, 14 sparse, 13 chunk-boundary, and 8 OCR failures. Each group needed a different fix. Report the taxonomy in incident reviews so “retrieval is bad” becomes an owned engineering action.
Define what retrieved means
Write the retrieval contract before writing the evaluator. For each query class, specify the acceptable evidence unit, required fields, allowed freshness, access scope, and the maximum rank at which a correct chunk counts. A single-fact query may require one exact sentence; a multi-hop query may require two chunks whose identifiers must be joined. A policy query may treat a superseded document as incorrect even if it contains the right words. In legal operations, a contract-review assistant was judged against whole documents and reached 0.93 relevance. When the contract was rewritten as clause-level evidence with version constraints, only 0.79 of retrieved chunks satisfied the “current clause plus exceptions” contract. That lower number was useful: it exposed that retrieval returned archived schedules. Define graded relevance as well—exact support, partial support, related context, and distractor—so the evaluator can distinguish a near miss from an unusable result. Store the annotation rationale and source span, not just a binary label.
Build a retrieval eval set that reflects production
Start with the query distribution, not a benchmark downloaded from somewhere else. Sample anonymized production queries by intent, document family, language, answerability, and risk. Deduplicate near-identical questions, preserve hard negatives, and include queries whose answer is absent. Annotators should mark the minimum supporting chunks and any acceptable alternatives, with a second reviewer resolving disagreements. In SaaS support, a 1,500-question set built from only resolved tickets produced 0.96 Recall@10. Adding unresolved, versioned, and “where is this setting?” questions reduced it to 0.78 and revealed that the original score measured memorization of article titles. Keep a fixed golden slice for release comparisons and a rotating slice for drift. Include temporal pairs, spelling variants, abbreviations, tables, and multi-hop requests. For every case record query, corpus snapshot, expected chunk IDs, relevance grade, access class, and whether refusal is correct. Without those fields, a score cannot be reproduced after an index rebuild.
Measure precision, recall, and MRR at the chunk level
Recall@k asks whether at least one required chunk appears in the top k; it answers whether the retriever found evidence. Precision@k asks how much of the returned set is useful; it exposes noisy context and token waste. Mean reciprocal rank rewards placing the first supporting chunk early, which matters when context windows are short. For multi-hop cases, calculate recall over the full required set, not just one chunk. A payments team reported Recall@5 of 0.88 but discovered that the metric counted any chunk from the correct document; clause-level recall was 0.69 because the fee schedule lived in a second section. After fixing the annotation unit, MRR fell from 0.81 to 0.58, correctly showing that evidence was buried. Report metrics by query class and document family, with confidence intervals and sample counts. Do not average away critical failures: a 0.92 global score can hide 0.61 recall on sanctions rules. Preserve ranked IDs and scores for every query so a regression can be inspected rather than debated.
| Query + Retrieved Chunk | Retrieval Result |
|---|---|
| Q: “refund window for enterprise plans?” → Chunk: pricing overview paragraph | |
| Q: “does the API support batch requests?” → Chunk: exact API reference section | |
| Q: “what changed in v3.2 rate limits?” → Chunk: v3.1 changelog entry | |
| Q: “how do I revoke an OAuth token?” → Chunk: OAuth overview intro | |
| Q: “are SLAs available on Starter?” → Chunk: enterprise SLA section | |
| Q: “maximum file upload size?” → Chunk: exact limits reference table |
Diagnose the failure mode before fixing anything
The fastest path is a four-question diagnostic: was the source document eligible, was the required text present in the index, did the retriever rank it, and did context assembly keep it? A payments team first rewrote its system prompt after 118 confident wrong answers. The diagnostic showed 54 chunks missing from the index after a PDF parser update, 31 dense misses on short product codes, 22 reranker inversions, and 11 annotation errors. The actual fix restored table extraction, added BM25 fusion for identifiers, and recalibrated the reranker on 600 hard negatives. Recall@5 improved from 0.64 to 0.89 and time-to-detect fell from nine days to one ingestion check. Use a replay harness that compares raw source, parsed chunks, embedding neighbors, sparse matches, reranked output, and final context. The first wrong layer owns the fix. A prompt change cannot recover text that was never indexed; a new embedding model cannot repair an OCR field that is blank.
Chunking is a retrieval decision, not a preprocessing detail
Chunk boundaries determine what can be found and how much irrelevant text enters context. Fixed-size chunks are fast and predictable, but they split a table row from its header and separate a code sample from its error message. Semantic chunks group coherent prose, yet can merge two adjacent policy clauses or become too large for rank-sensitive retrieval. Document-aware chunking preserves headings, tables, lists, and code blocks; it works well for manuals but needs format-specific parsers. Hierarchical chunks keep parent summaries with child evidence and are effective for multi-hop questions, at the cost of more indexing and deduplication. In pharmaceutical documents, fixed 512-token chunks lost dosage units across 18% of table queries; document-aware parsing cut that to 4%. In legal contracts, semantic boundaries improved clause retrieval by 11 points but missed cross-references. In API documentation, code-aware chunks reduced “method exists but parameters absent” failures from 23% to 7%. Measure each strategy on each document type; there is no universal chunk size.
Calibrate your embedding model against your domain
A domain calibration study should be small enough to run weekly and rich enough to reveal the model’s blind spots. Build 300–500 query cases with positive chunks, hard negatives, abbreviations, identifiers, multilingual variants, and temporal language. Compare candidate models using Recall@k, MRR, score separation between positives and negatives, and latency at your corpus size. Plot nearest-neighbor examples, not only aggregates. In retail banking, a general embedding model placed “ACH return code R29” near generic payment-failure explanations; positive-negative margin was 0.04 and Recall@5 was 0.68. A domain-adapted model plus normalized code tokens raised margin to 0.17 and Recall@5 to 0.86, while p95 embedding latency increased 8 ms. The team kept the adapted model because the hard policy slice passed its gate. Fine-tune only after checking normalization, chunk text, and metadata filters; switching models will not fix malformed input. Version embeddings with the index and run a shadow comparison before re-embedding the production corpus.
Reranker calibration and the second-stage problem
A reranker can rescue a weak top-k set, but it cannot rank a missing chunk. Evaluate it independently by feeding the same candidate pool to each reranker and measuring NDCG, MRR, and critical-chunk recall. Keep candidate generation fixed so the second-stage comparison is fair. In an e-commerce catalog, dense retrieval put the exact warranty clause at rank 9; a cross-encoder moved it to rank 2 on most queries, but overfit title words pushed accessory pages above product manuals for short queries. On a 2,400-case set, reranking raised MRR from 0.56 to 0.74 overall while lowering recall for model-number queries from 0.91 to 0.83. The fix was a query-class router: lexical-heavy cases used a lighter reranker with identifier features, while natural-language questions used the cross-encoder. Calibrate score thresholds per class, inspect inversions, and measure reranker latency under concurrency. A reranker should be judged as a ranker, not credited for generation improvements it did not cause.
Hybrid search: when to add sparse retrieval
Sparse retrieval adds signal when exact terms carry meaning: policy IDs, error codes, drug names, account fields, version strings, and proper nouns. Dense retrieval is stronger for paraphrase and conceptual similarity. Fuse them only after measuring complementary misses. In an enterprise-support corpus, dense-only Recall@5 was 0.76 on 800 cases; BM25-only was 0.69, but their union reached 0.87 because each recovered different identifier-heavy failures. A naïve 50/50 score blend then fell to 0.81: BM25 scores and cosine scores were on incompatible scales, so lexical matches overwhelmed semantic evidence. Calibrate fusion on a held-out set, normalize scores, and tune weights by query class. Track duplicate candidates and context expansion; hybrid search can increase token load and hide a precision regression. Add sparse retrieval when it reduces high-value misses, not because it is fashionable. Keep a dense-only baseline in every evaluation so the extra index has to earn its operational cost.
Monitor retrieval quality after deployment
Production retrieval needs signals before answer quality visibly collapses. Maintain canary queries for every document category and run them after ingestion, parser, embedding, index, or reranker changes. Track Recall@k on labeled canaries, score distributions, top-result churn, duplicate rate, stale-document rate, and p95 retrieval latency. Cosine similarity drift is useful as an alert, not a quality metric by itself: a stable score can still point to the wrong document. A clinical knowledge system stayed on the same model while a new PDF template changed heading extraction; weekly Recall@10 fell from 0.91 to 0.73 over three weeks. A document-format canary caught the drop within 20 minutes of ingestion, before clinicians reported bad answers. Treat ingestion as a deployment event with a hold gate and rollback target. Log redacted query hashes, chunk IDs, corpus version, access decision, scores, and parser version. Avoid storing sensitive text; retain enough provenance to replay the failure safely.
A practical definition of done
A mature retrieval program has seven observable properties. First, a written contract defines correct evidence, freshness, access, graded relevance, and refusal behavior. Second, the eval set mirrors production, includes hard negatives and absent answers, and records chunk-level annotations. Third, Recall@k, Precision@k, and MRR are reported separately by query class and document type. Fourth, a replay diagnostic distinguishes dense, sparse, chunking, preprocessing, reranking, and assembly failures. Fifth, embedding and reranker changes are calibrated on held-out domain cases with latency and score-separation measurements. Sixth, hybrid search is justified by complementary recall and calibrated fusion rather than a blended-score hunch. Seventh, ingestion and index changes run canaries, drift checks, and release gates with reproducible traces. A payments or banking team should be able to answer which chunk was expected, why it was or was not returned, which version produced the result, and who owns the fix. That is retrieval quality: evidence you can measure, improve, and defend before users become the monitoring system.
Share a thought
Comments appear immediately. Email is optional and never shown.
No comments yet. Be the first to share a thought.