/ ENGINEERING SYSTEMS
Back to Field Notes

FIELD NOTE / RELIABILITY NOTE

Evaluating Faithfulness in RAG Systems

12 min readUpdated 2026-09-14

A RAG system can retrieve the right document, cite the right passage, and still produce an answer that says something the evidence never said. Faithfulness evaluation isolates that failure: given the exact context available at generation time, which claims in the answer are actually supported? The engineering challenge is turning that question into a repeatable measurement system that survives ambiguous claims, noisy retrieval, judge disagreement, model changes, and production release pressure.

Public-safe engineering note · no client or confidential implementation details
01

Start with the distinction that matters

Faithfulness is not the same as correctness. A response may be factually true because the model remembered something from pretraining, yet still be unfaithful if that fact is not supported by the context this request was allowed to use. The reverse can happen too: an answer can faithfully repeat a retrieved document that is stale or wrong. Relevance asks whether the response addresses the question; retrieval quality asks whether useful evidence was found; correctness asks whether the claim matches an accepted reference; faithfulness asks whether the answer is entailed by the supplied evidence. Keep those questions separate. Otherwise a single attractive score can hide whether the system retrieved the wrong passage, over-synthesized a good passage, or cited a source that does not support the sentence beside it.

A useful mental model is a support-policy assistant. If the retrieved policy never mentions a regional exception, a fluent answer that invents one is unfaithful even when the exception happens to be true in the wider world. The evaluator must judge the answer against the evidence this request was allowed to use.

Three metrics, three comparison surfaces
RELEVANCEDoes it answer the question?Query → AnswerCORRECTNESSIs it externally true?Answer → WorldFAITHFULNESSIs every claim supported?Context → Answer
Engineering example — Financial services: A policy assistant answered 94% of 500 test questions correctly, yet claim-level review found 31 unsupported qualifiers. The top-line accuracy hid a 6.2% faithfulness defect rate.
02

Define the evidence contract

Before choosing a metric, write down what the system is allowed to claim. Identify the authoritative corpus, access scope, freshness rules, and the expected behavior when evidence is missing or conflicting. Then represent each answer as atomic claims rather than treating a paragraph as one indivisible unit. “The policy allows refunds within 30 days, requires original condition, and excludes clearance items” is at least three claims. A claim should be small enough to label as supported, contradicted, or not verifiable against one or more retrieved chunks. Preserve the mapping between claim, source span, document version, and retrieval rank. This makes a score actionable: a failed claim can point to missing recall, weak chunking, an over-broad synthesis step, or a citation that was attached after generation rather than produced with it.

In practice, the contract should also preserve claim-to-source provenance: the exact span, document version, access decision, and retrieval rank. That receipt turns a failed score into an engineering action instead of an argument about whether the answer felt reasonable.

Engineering example — Healthcare: A benefits assistant treated a three-clause eligibility sentence as one claim. After decomposition into age, network, and prior-authorization claims, support fell from 0.96 to 0.81 because the third clause was absent.

Share a thought

Comments appear immediately. Email is optional and never shown.

Markdown & code fences supported
AUTH VIA: PUBLIC FORM

No comments yet. Be the first to share a thought.

01 / SYSTEM VIEWInspect the architecture Follow a request through retrieval, validation, and audit.02 / EVIDENCEReview the case studies See the engineering decisions and measured outcomes.03 / OPEN CHANNELStart a conversation Talk through a production GenAI problem.