RAG Audit: Retrieval Quality vs. Answer Confidence

A RAG Audit That Started With Retrieval and Ended at the Output Contract

A retrieval-augmented generation system can retrieve the right document, give a plausible answer, cite a real passage, and still fail the user’s request. Those outcomes sound contradictory only when retrieval quality, factual correctness, evidence support, attribution, and application acceptance are treated as a single measure. I wanted to inspect those boundaries separately rather than publish another comparison in which a fluent response is automatically called a successful answer.

For this RAG audit, I used a deliberately small collection of Git command documentation, two classical lexical retrievers, and one locally executed language model. The corpus was narrow enough to inspect passage-level evidence but overlapping enough to create legitimate confusion: commands such as git restore, git reset, git switch, and git worktree share words while performing materially different operations. The experiment asked whether the retrieved material contained the facts needed to answer, whether the generator used that material correctly, whether each citation supported its associated claim, and whether the output satisfied a strict machine-readable response contract.

The result was not a simple winner. One retriever recovered slightly more frozen evidence. A secondary, claim-by-claim content review found more supported answers in the same condition. Both methods correctly declined questions whose answers were absent from the document collection. Yet the application’s original schema accepted zero generated answers, even though some answer text was factually sound. This failure is the central engineering lesson of the RAG audit: evidence quality and production usability are related, but they are not interchangeable.

This is a bounded laboratory study, not a benchmark of every RAG architecture or of an entire model family. Its value lies in the observable failure types, the transparent denominators, and the decisions they imply for engineers responsible for real knowledge workflows.

If You Only Read One Thing

A trustworthy RAG system needs at least five separate gates: retrieve sufficient evidence, produce a correct answer, substantiate each claim with that evidence, abstain when support is absent, and satisfy the consuming application’s output contract. Passing four while failing one can still leave the user without a usable answer. In my experiment, 10 of 18 answerable questions in the BM25 condition and 13 of 18 in the TF-IDF cosine condition were judged supported and correct under a documented secondary content audit, but all 34 attempted answers violated the original strict response schema. The model’s useful text never became an accepted application answer.

The practical consequence is straightforward: evaluate a RAG workflow as a chain of independently observable obligations. Do not replace that chain with a single accuracy score, an attractive citation icon, or a confidence-sounding sentence.

Decision Brief: What the RAG Experiment Actually Tested

I framed the research around one question: In a small document corpus, which errors arise when retrieval evidence, answer wording, and apparent confidence are evaluated separately? The measured system was a local research pipeline. It was not connected to a production customer database, a live help desk, an autonomous agent, or an external search index.

The research question matters because a RAG failure can occur before generation, during generation, in the attribution layer, or when a parser receives the response. Those mechanisms have different fixes. Improving retrieval rankings cannot repair an invalid JSON schema. Tightening JSON validation cannot make an unsupported answer true. Increasing the context window cannot guarantee that an answer cites the correct passage.

I also limited the question set before the primary run. There were direct lookup tasks, paraphrases, comparisons between closely related commands, multi-document tasks, distractors, exact option flags, and deliberately unanswerable queries. Some absent questions asked for repository-specific state, hardware details, or an HTTP retry value that the Git manuals could not supply. This distinction made abstention measurable rather than a vague judgment about whether a response sounded cautious.

Evidence and Methodology: Freezing the Study Before the Outputs

For a RAG audit to be interpretable, the evaluation references cannot be quietly rewritten after seeing the model’s answers. I froze the ordered questions and their required evidence anchors before retrieval and scored generation. Each required unit represented a specific fact or passage needed for a complete answer. The experiment retained the original HTML manuals, corresponding upstream source material, license information, hashes, the question set, model prompts, ranking traces, raw outputs, and claim-level decisions.

The RAG design fixed the tokenizer, passage boundaries, retrieval cutoff, question order, model, decoder settings, and response format. BM25 used a conventional probabilistic lexical ranking configuration with k1 = 1.2 and b = 0.75. The alternative used TF-IDF with sublinear term frequency, smoothed inverse document frequency, and cosine normalization. Both indexed the same document identifiers and text, returned the same maximum of four passages, and broke ranking ties by stable passage ID. No dense embeddings, reranker, semantic query expansion, or cross-language retrieval were used.

In this RAG study, a passage could be relevant even if it did not contain the exact frozen wording. That became an important discovery. I therefore kept the original anchor-based retrieval scores unchanged and recorded a separate secondary assessment of independently sufficient alternative passages. I did not retroactively improve the frozen relevance labels simply because a model generated a convincing answer.

For the generation stage, the experiment used the same local Qwen model with a 4,096-token context and a 260-token output limit. Requests ran sequentially on the CPU with greedy decoding and a cleared key-value state. Method order alternated between questions to reduce simple ordering bias. Four questions were repeated in both conditions. The model saw the question and retrieved passages, not the reference answer key. Its actual outputs and latency were preserved, including malformed responses.

This setup allowed a narrow but useful causal description: the ranking method was varied while the main corpus, prompt structure, and generator were held fixed. It does not isolate every reason for a changed answer. A retriever can alter which passages arrive, their order, and the amount of conflicting context. Those changes can interact with the generator. A small observational comparison cannot reveal hidden model cognition.

Why Git manuals are a useful stress case

This corpus was intentionally more difficult than twelve unrelated fact sheets. git add --intent-to-add and git add --refresh share documentation but imply different index behavior. git switch and git worktree both interact with branches and worktrees, yet flags that override one operation’s safeguards do not automatically apply to the other. git clean -x and git clean -X differ by a single capital letter that reverses an important condition. A RAG system that matches related terms without respecting command scope can construct an answer that sounds technically fluent while being operationally wrong.

The selected documentation has another advantage: readers can check the underlying behavior against the <a href=”https://git-scm.com/docs” target=”_blank” rel=”noopener noreferrer”>official Git manuals</a>. The article does not ask readers to trust model fluency or a hidden grading service. The bounded outcome still depends on the chosen Git 2.56.0 snapshot and the stated relevance criteria; it is not a performance claim about newer Git releases.

RAG Audit Results: Retrieval, Grounding, and Format Compliance

The RAG audit produced 48 primary outputs: 24 questions under each retrieval method. Of those, 18 questions per method had an answer somewhere in the corpus; six did not. The distinction is essential. Retrieval quality is calculated over answerable questions, correct abstention over the six absent questions, and strict response acceptance over all 24 outputs per method.

BM25 returned a fully anchored set for eleven of eighteen answerable questions; TF-IDF returned twelve. Both settings exposed enough material to answer additional questions through alternative passages not captured by the frozen exact anchors. That is why I keep the raw frozen recall and the later sufficiency judgment side by side. They answer related but distinct questions.

The RAG evidence also limits any claim of superiority. TF-IDF was ahead on several metrics in this particular question set, but this is one small corpus, two lexical settings, one generator, and an assistant-assisted review. The result does not establish that TF-IDF is universally better than BM25, that lexical retrieval is better than dense retrieval, or that the observed differences would survive other question samples. In a RAG deployment, the proper next step would be to test the real target corpus and failure costs rather than copy this ranking as a default recommendation.

RAG Retrieval Failure Is Not the Same as Generation Failure

A useful RAG audit classifies not only whether an answer failed but where the failure began. If the necessary fact never enters the top-four context, the generator lacks the relevant evidence. If the fact is present and the answer is wrong, the problem lies downstream. If the answer is correct but the cited passage does not prove it, attribution failed even when factual truth happened to survive.

Consider a path-specific reset task that required distinguishing what happens to HEAD and the working tree from a nearby git restore operation. In one condition, the needed path-reset passage was retrieved and both parts were answered. In the other, the ranking supplied a generic cross-reference but not the decisive passage, and the answer omitted an important distinction. That is a retrieval-associated error: the missing evidence was visible in the preserved ranked context.

By contrast, a question about git clean -x versus git clean -X had sufficient source evidence in both conditions. The generated answers still got part of the option behavior wrong. For lowercase -x, the regular ignore rules are not simply reintroduced as the BM25 answer suggested; for uppercase -X, the tool can remove ignored files, including ignored build products, despite the other answer implying they would be left untouched. No extra top-four recall score could turn those mistaken statements into correct ones. The generator had an evidence-usage problem.

This difference matters operationally. A RAG engineer could spend days tuning retrieval chunk size while the most costly failures arise from model interpretation of short command flags. Conversely, replacing the model could leave a genuine retrieval omission untouched. The instrumentation must identify both conditions before the next engineering iteration.

Why top-four precision cannot tell the whole story

The experiment’s mean frozen precision at four was 26.39% for BM25 and 27.78% for TF-IDF. Those percentages sound low, but they reflect a deliberately strict relevant-chunk definition and a four-passage budget. An overlapping window can contain an adequate explanation without containing the specific anchor selected when the evaluation was frozen. A near-duplicate window may count as another chunk without offering new evidence. Therefore, chunk-level precision is not a substitute for question-level evidence completeness.

A clear example was the question about removing a tracked path from the index without deleting the working file. Both top-four lists omitted the frozen options anchor for git rm --cached. Yet both included a description passage that independently explained the index-only removal. The RAG answers were supported despite the frozen anchor miss. This is neither proof that recall does not matter nor a reason to alter the original metric. It shows why relevance policy and sufficient evidence need distinct labels.

Citations: A Correct Answer Can Still Have a Wrong Source

In a RAG system, citation presence is not citation accuracy. One of the sharpest RAG findings came from the question about git add --intent-to-add. The answer text correctly stated that Git records an empty index entry for the path without immediately staging its file contents. But the cited snippet was merely a command synopsis listing the option. It did not explain that behavior. The generator may have relied on its prior knowledge or on a related contextual clue; this experiment cannot determine the internal source of the knowledge. What it can determine is that the cited passage did not support the stated claim.

Another failure combined a wrong command with an incorrect citation. The prompt asked which git switch option can override a worktree-related restriction. Both model answers transferred a rule from a git worktree add context instead of using the applicable switch flag. The correct git switch option is --ignore-other-worktrees, documented separately. In the same task, a correct statement about a locked-worktree removal condition was attributed to the wrong source passage in one condition. A citation ID looked syntactically valid while referring to the wrong operation.

These counts should not be read as a universal citation error rate. They count citation occurrences, not unique documents or user sessions. In addition, the model wrote substantive rationales without dedicated per-assertion citations. The content review evaluated those claims, but the output contract did not render the rationale as a separately cited answer. The unit of analysis changes the denominator and can change the conclusion.

For a reader-facing RAG product, my preferred rule is stronger than “the answer has citations.” Each substantive factual statement should be checked against the exact displayed passage or a clearly defined set of passages. A source ID proves where text was retrieved from; it does not automatically prove what the answer says.

When a RAG Answer Is Correct but the Application Rejects It

This was the most consequential result in the study. The frozen interface expected a specific structured response. Accepted answer objects required an empty rationale field. In all 34 attempted answers across the two retrieval conditions, the model supplied a nonempty rationale. Those answers failed the strict schema, including answers that a later content review judged correct and properly supported. The remaining 14 primary outputs were abstentions; they satisfied the structural contract.

For a RAG application, the distinction is not cosmetic. In a real pipeline, a parser failure can prevent the application from displaying a response, completing a workflow, or passing an action to its next stage. It would be misleading to present the 10/18 and 13/18 supported-content results as the application’s success rate. They are secondary assessments of the preserved visible text, made after the original contract violations were detected. Under the unchanged production-style parser, the accepted-answer rate remained zero in both conditions.

The RAG engineering response is not to disable the schema quietly and declare success. I would keep the failure visible, then run a new, clearly versioned experiment to improve structured-output compliance: clarify the prompt contract, validate against a parser during development, consider constrained decoding where available, and count every fallback or repair path. A later revision should never be merged into the frozen experiment as if it had always passed.

The case also changes how I think about “RAG accuracy.” If a dashboard reports answer quality only for text that a human could interpret, it may conceal a complete integration failure. A production scorecard needs both semantic quality and strict contract acceptance. The former tells us whether statements are useful and defensible. The latter tells us whether the system can deliver them.

RAG Abstention: Refusing to Invent an Answer Is Not Always Enough

RAG abstention deserves its own test cases. Six questions in each condition were intentionally absent from the Git corpus. They included requests for the user’s local repository state or hardware capacity and an invented HTTP retry behavior. In all twelve condition-specific instances, the generator declined to claim that the documents contained an answer. That is a useful observation: the tested configuration handled these particular absent-evidence prompts appropriately.

However, there is another type of refusal. On an answerable question, the correct fact may exist in the corpus but not in the retrieved top-four passages. BM25 declined two such questions. Its refusal was defensible given the material it saw, yet the overall RAG system still failed to answer a task it could have answered with better retrieval. TF-IDF did not make those two context-limited abstentions, but it produced some incorrect answers when nearby passages suggested the wrong behavior.

I separate at least three states when auditing RAG abstentions: the answer is absent from the corpus; the answer is present in the corpus but absent from retrieved context; and the answer is present in context but the generator declines anyway. They imply different fixes. The first may call for a clearer user-facing limitation. The second suggests retrieval improvement. The third suggests generator or refusal-policy evaluation. Collapsing them into one “refusal rate” destroys useful diagnostic information.

Nor does six out of six mean a system is safe against all hallucinations. These were six chosen negative questions in a narrow technical collection, not a representative survey of adversarial prompts, multi-hop failures, user ambiguity, or malicious documentation. A RAG product needs negative controls that reflect its actual task domain.

RAG Answer Confidence: Direct Language Is Not Calibration

The title’s phrase “answer confidence” needs precision. I did not measure an internal probability that the model believed each answer was correct. No calibrated confidence estimates, token-level log probabilities tied to correctness, or probability forecasts were evaluated. Instead, I coded the surface language of factual assertions: direct statements, hedged statements, and explicit certainty claims.

All 71 evaluated answer-line assertions used direct wording under the study’s rule. Among those answer-line assertions, nine were factually incorrect and thirteen lacked full contextual support, including ambiguous cases. Across 99 atomic assertion instances when substantive rationale text was included, the audit recorded six unsupported assertions, six contradicted assertions, and two ambiguous assertions. Correct and incorrect claims alike appeared in ordinary declarative sentences.

For this RAG audit, that observation supports a narrow but important inference: confident-sounding prose did not reliably distinguish supported content from unsupported content in this sample. It does not establish a numerical confidence calibration curve. It also does not reveal whether the model internally recognized uncertainty and chose not to express it.

I therefore would not build a RAG quality gate around phrases such as “I am confident,” nor treat hedging alone as proof of reliability. Better signals are whether the source contains the required evidence, whether claims are entailed by that source, whether alternative interpretations were considered, and whether the system has a real abstention pathway.

The Four Failure Cases I Would Keep on Every RAG Evaluation Board

A practical RAG audit becomes more useful when each category maps to a recognizable engineering action. Four Git questions in this run demonstrate the distinction without requiring readers to inspect raw research files.

These examples are not attempts to embarrass a model or to prove that Git documentation is unusually difficult. They show the shape of errors that become expensive in developer tooling: a single character in an option, a wrong command namespace, or a correct fact attached to an inadequate citation. In these situations, the reliability of a RAG answer depends on much more than the nearest lexical match.

Repeatability, Prompt Controls, and Their Narrow Meaning

The RAG experiment executed 58 native model requests: 48 primary retrieval-and-answer requests, eight repeats, and two simple controls. Four questions were repeated for both methods under the same greedy settings. All eight repeat outputs matched their corresponding originals byte for byte. This demonstrates repeatability for those specific prompts and the observed local runtime; it does not mean an LLM is universally deterministic across hardware, runtimes, versions, or different prompts.

The two controls checked whether the model abstained with no evidence and whether inert untrusted passage text resembling a role instruction would be treated as content. Both returned valid abstentions in the recorded setup. That result is encouraging for those fixtures, but two controls cannot certify prompt-injection resistance. The runner had no tool-execution path, so the experiment did not test whether a compromised answer could trigger privileged actions.

I also separated the research run from setup failures. The local environment attempted alternative inference paths before using a CPU build of llama.cpp. The final measured generator was not a cloud API and did not use the earlier failed probes as scored results. The model was loaded once, and successive requests were timed while warm. This matters because including initialization in one condition and not the other would make latency comparisons meaningless.

Latency and Cost: What Was Actually Measured

The RAG ranking operations were fast relative to CPU generation under this small corpus. The median top-four retrieval time was approximately 0.362 milliseconds for BM25 and 0.267 milliseconds for TF-IDF cosine. Median generation wall time was 17.081 seconds and 16.288 seconds respectively. The longer inference timings describe this CPU-only, local, sequential configuration, not a performance benchmark for the underlying model on accelerated hardware or a production inference service.

The pipeline used an already local model, so metered API expenditure was zero. That does not make the experiment free: hardware, electricity, development time, and review effort were not measured. Token totals were actual recorded tokens for the primary requests, not model-pricing estimates. The result does not support a comparative cloud cost claim.

More importantly, the latency discussion must not distract from correctness. A slightly faster wrong response is not a useful optimization, and a slower correct response that fails parsing is still not accepted by the application. I would optimize a RAG deployment only after defining which correctness and reliability conditions must remain satisfied.

A Practical RAG Evaluation Framework for Engineering Teams

I would turn the findings into a five-gate evaluation record for each query. Keep a trace of the question, eligible evidence, retrieved chunks, output text, claim-level support, citations, parser result, and final user-visible outcome. Count each failure class separately. If a fix improves one metric while harming another, preserve that trade-off rather than rolling it into a composite green score.

A sixth question runs through all five: does the system know when to abstain? It should refuse unsupported tasks without mistaking a retrieval miss for proof that the underlying corpus lacks the answer. This is why my RAG audit keeps both corpus-relative and context-relative abstention categories.

A controlled follow-up worth running

For the next experiment, I would keep the same corpus and questions but deliberately improve only the output-format mechanism. The first target should be strict JSON answer acceptance, since it currently blocks every generated answer. A second stage could vary top-k, reranking, chunk boundaries, or a dense retriever under the same answer contract. A third could add an independent human reviewer for disputed claim-support labels.

Each stage requires its own versioned setup and result table. It would be inappropriate to retrofit an improved model output into the original run and imply the first RAG pipeline had passed from the beginning. Negative results are valid experimental outcomes; hiding them destroys the evidence required to improve the system.

What This RAG Audit Supports and What It Cannot Prove

This RAG audit supports a limited set of observations. In the frozen Git corpus, a modest retrieval difference coincided with different answer outcomes. A correct factual statement could be cited to a passage that did not prove it. Sufficient evidence could still lead to the wrong flag or a reversed mapping. Deliberately absent questions elicited appropriate abstentions. Every attempted answer failed a strict structured-output contract even though many outputs were useful to a human reviewer.

The experiment does not establish universal BM25 versus TF-IDF superiority, hallucination rates in production, model confidence calibration, effectiveness of prompt-injection defenses, or accuracy on arbitrary enterprise documents. The question set was authored for the experiment, not sampled from production traffic. One research assistant designed the references and judged semantic results; there was no independent blinded human assessment. The generator was a single quantized local model and the retrievers were lexical. The RAG corpus was one Git version, not live changing knowledge.

There is also a subtle evaluation dependency: the secondary claim-support judgments are more interpretive than the deterministic hashes and counts. The verifier reproduces the exported numbers from preserved judgments; it cannot make the original human-level interpretation independent. The right publication posture is to show those results, name the review method, give readers concrete counterexamples, and avoid pretending that a passing checksum audit validates semantic truth.

For me, the strongest lesson is not that one retrieval algorithm won by one question or that the model produced a particular number of mistakes. It is that retrieval, answer truth, evidence attribution, abstention, and schema conformance can diverge sharply inside one small RAG experiment. An engineering team that measures only one will miss failures the user can actually experience.

Key Terms for This RAG Audit

Retrieval-augmented generation (RAG): A workflow that supplies retrieved external passages to a generative model for answering questions.

Required evidence unit: A distinct piece of source information needed to answer a particular question fully.

Top-k retrieval: Selecting the k highest-ranked passages under a defined scoring method; this experiment used four.

Context support: Whether the passages actually presented to the generator justify the words of its answer.

Citation attribution: Whether a cited passage supports the particular statement to which the citation is attached.

Abstention: Explicitly withholding an answer because evidence is absent or insufficient; it can be appropriate yet still represent a missed opportunity when retrievable evidence exists elsewhere in the corpus.

Strict schema acceptance: Whether the generated response satisfies the machine-readable interface expected by the application, regardless of prose quality.

Surface confidence: How certain an answer sounds; not a direct measurement of a model’s internal probability or calibration.

Knowledge Check: Three Questions Before Shipping a RAG Product

If the retrieved context contains the correct fact but the answer is wrong, would increasing recall solve the immediate failure? Not necessarily. The decisive passage was available; the generator’s interpretation or attribution needs inspection.

If an answer is factually correct, is a citation automatically valid? No. The cited passage must independently support the specific claim. A synopsis listing an option is not proof of that option’s full documented behavior.

If a model answers correctly but violates JSON schema, should the run count as successful? Not under an application that rejects the response. Track content quality separately and report the original accepted-output outcome.

Sources, Methodology, and Update History

The corpus is based on a frozen Git 2.56.0 manual snapshot. Relevant reference materials include the Git add manual, Git clean manual, Git switch manual, and Git worktree manual. For methodological context, ARES separately evaluates context relevance, answer faithfulness, and answer relevance; this lab did not run ARES or claim its validation design.

Related research is collected in the Artificial Intelligence Articles section. The original experiment and article were prepared on November 11, 2026. All measured counts refer to that frozen study. I would update this article if the format contract is repaired or an independently reviewed evaluation is run, and report the new dataset separately rather than silently replacing the original outcomes.

Author’s note: I regard a RAG pipeline as a software product with evidence obligations, not merely a text generator with a search box. A useful laboratory result is one that makes the next engineering decision clearer—even when the most actionable finding is that an apparently capable system cannot yet deliver an accepted answer.