How I Measure Whether a RAG System Works

Part 2 of 2. Part 1 covers the architecture decisions and the production gotchas.

For a long time I had no reliable way to tell whether my RAG system's answers were right. I'd change something, try a handful of questions, and go with my gut. Debates about chunk size were settled by whoever argued hardest. When an answer came back wrong, I couldn't say whether the right passage was missing, or the model had it and ignored it.

The clearest case was guardrails. We wanted to make sure a question was actually about the product, so we added a relevance check. It blocked most of the legitimate queries along with the off-topic ones. We went back and forth on how strict it should be for a long time, with nothing to settle it except the handful of examples someone had tried. In the end we stopped blocking and just logged whatever it flagged. That was a decision made on gut feel in both directions, and it's the kind of thing a fixed set of questions would have answered in an afternoon.

The thing that changed this wasn't a better parser or a bigger model. It was an evaluation set: a fixed list of questions with known answers that I can run the system against whenever something changes. This post is how it works and what the first run showed.

Four ways an answer can be wrong

With a classifier you compare the prediction to the label and you're done. A RAG system gives you a paragraph of text, and it can be wrong in at least four different ways:

1. The wrong chunks came back, so the model never saw the answer.
2. The right chunks came back, but the model ignored them.
3. The model used the right chunks but left out something important.
4. The model used the right chunks and added things that aren't in them. Each of these has a different fix. A single "is this answer right?" score hides which one happened, so I score retrieval and generation separately.

The question set

My set has 35 questions over one family of documents. Each question has:

• a category: simple factual, table-based, cross-section synthesis, multi-hop, compliance check, image or figure, and adversarial (questions the system should decline)
• a difficulty: easy, medium or hard
• a ground-truth answer
• the sections and pages the answer should come from
• flags for whether it needs a table, an image, or more than one section The categories matter more than the count. If 195 of 200 questions are simple lookups, the average looks great and tells you nothing. A smaller set that deliberately covers the hard cases shows you where the system is weak.

The first questions came from a domain expert, plus candidates I generated from the documents and then reviewed. Once real users arrive, I'll refresh the set from actual query logs every few months. The questions people really ask are never quite the ones you'd come up with at a whiteboard.

Scoring retrieval

Since every question lists the sections the answer should come from, the standard retrieval metrics can be computed directly, with no model involved:

Precision@K = relevant chunks retrieved / K
Recall@K    = relevant chunks retrieved / all relevant chunks
MRR         = mean of 1 / (rank of the first relevant chunk)

NDCG@K adds a reward for putting the most relevant chunks near the top. Rank matters because models pay more attention to what comes first.

Recall is the one that showed my problems. If the right chunk never comes back, no amount of prompting fixes the answer. I look at it per category and act on the worst one.

Alongside those, a judge model rates the retrieved context for relevance and completeness on a 1-5 scale. That judged score is the retrieval column in the results table further down.

Scoring generation

Once retrieval is good, the answer can still go wrong. The judge rates each answer, again on 1-5, on:

• Faithfulness: how much of the answer is backed by the retrieved text. This is the one I care about most. I also report it the other way round, as a hallucination rate (1 minus faithfulness). It's the same number, but it's the one people ask about.
• Answer relevance: whether the answer addresses the question asked. One common approach is to have a model write the question the answer seems to be responding to, then compare it with the real question.
• Correctness: how much of the answer matches the ground truth. It's the most obvious score and the most work, because it needs a written ground truth for every question.
• Context recall: how much of the ground truth was present in the retrieved chunks at all. This connects the two halves. If context recall is high but correctness is low, the model isn't using what it was given. If context recall is low, it's a retrieval problem. For a domain where wrong answers have consequences, faithfulness is the score that matters. People will forgive an answer that's incomplete. They won't forgive one that confidently makes things up.

Using a model as the judge

Scoring 35 questions on every rubric by hand isn't something you can do after every change, so a model does the scoring. Two rules make that work.

First, the judge should be a stronger model than the one being tested, because a model is bad at catching its own mistakes. Second, check the judge. I regularly score a sample by hand and compare. If the judge disagrees with me more than about one time in ten, I fix the rubric before trusting the numbers.

The rubric prompt matters more than which judge model you pick. "Rate faithfulness from 1 to 5" gives you noise. "List every factual claim in the answer, find the supporting text in the retrieved chunks for each one or mark it unsupported, then score" gives you something you can use.

Judges also have known biases. They tend to prefer longer answers and confident wording, and they sometimes invent supporting text that isn't there. Read up on this before you let a judge block deploys.

What the first run showed

Here are the results from the first full run, averaged per category on the 1-5 scale:

Overall, retrieval averaged 3.06 and generation 3.77 across all 35 questions, and about half had poor retrieval. The adversarial row is the odd one out: retrieval is low because there's nothing to find, and generation is high because declining is the right answer.

Three things stood out.

Retrieval was the bottleneck. Generation scored higher than retrieval in every category. The model was doing reasonably well with the chunks it got. It often just wasn't getting the right ones. So the next work had to go into retrieval, not prompts.

Table and image questions were weak because of the parser. Both depend on visual structure in the source documents, and the default parser was throwing that away. That made a structure-aware parser the next change to make.

Synthesis and multi-hop questions needed more chunks. They depend on several passages, and with a top-k of 3 the system often couldn't see enough of them. That's the case for choosing k per question type.

Before this table, "the parser is bad at tables" was a hunch I'd have struggled to defend in a planning meeting. With the per-category scores it took one slide.

Making it block bad changes

An evaluation set you run once and keep in a spreadsheet is only half useful. The other half is running it automatically.

The plan is for every change to chunk size, parser, top-k, prompt or model to trigger a run of the full set. The judge scores it, the results are compared with the last baseline, and if average retrieval drops by more than about 0.2 points, the change doesn't ship without someone signing off.

Today I run it on demand against staging. Wiring it in so it blocks deploys is next on the list. That's the point where it stops being a dashboard and becomes a test.

What it has already changed

The parser change happened because of a number. Table-based retrieval at 3.42 and image questions at 2.67 settled an argument that intuition alone couldn't.

Top-k is now chosen per question type. The scores showed synthesis questions needed more chunks while simple ones were fine with fewer. Without the category breakdown, that argument wouldn't have come up.

Some "improvements" never shipped. A longer system prompt and a fancier retrieval filter both looked good on the few questions they were tried on, and made the overall scores worse. Without the set, they'd probably have gone out.

Where to start

If you're building one of these, build the question set before the second feature. A few dozen questions spread across the hard categories is a few days of work. Score retrieval and generation separately, break the results down by category, keep an eye on faithfulness, and check your judge. Once you have real users, refresh the questions from what they actually ask.

It's the least glamorous part of the system, and it's the reason I can now say what's working and what isn't. For the architecture side and the production gotchas, see Part 1.