RAGAS metrics and the "I don't know" rate

Meera called a meeting. Not a long one. She had three questions written on an index card, and she read them out.

“One: when the bot answers, is the answer true? Two: when it doesn’t know, does it admit it? Three: when you change something, like last week’s search upgrade, how do you know you made it better and not worse?”

Arjun was ready, or thought he was. He opened his laptop and typed five questions into the bot, live. The chiller temperature. The closing checklist. SOP-14. The late-delivery refund. Who the fire warden is. Five good answers. He looked up, pleased.

Priya gently closed the lid. “Those are five questions you happen to remember, and that you’ve already fixed,” she said. “That’s a vibe check. Meera’s asking for a measurement.”

“Then I’ll measure,” Arjun said. That evening he wrote a script that sent each of 20 bot answers to Claude with the prompt “Rate this answer from 1 to 10.” Every answer came back 7, 8 or 9. Average: 8.1. He showed Priya the next morning.

She scrolled to answer 14. A Hadapsar staff member had asked what to do when a customer returns a cake that smells off. The bot had said: “Apologise, offer a replacement or refund, and throw the cake away immediately to avoid contamination.” The judge had given it an 8.

“The handbook says to keep the product sealed and send it to the central kitchen for a quality check,” Priya said. “If there’s a bad batch, that cake is the evidence. Your bot told them to destroy it, and your judge gave it an 8.” She put the laptop down. “A single score from 1 to 10 can’t tell you what is wrong, so it can’t tell you what to fix. Let’s break ‘good’ into pieces we can actually check.”

Why “rate it out of 10” failed

“What did your judge actually check?” Priya asked.

“Whether the answer was… good?”

“Good how? It sounded helpful, polite and confident. Did the judge have the handbook page?”

“No. Just the question and the answer.”

“So it graded the answer against its own idea of what a bakery should do. And ‘throw away spoiled food’ sounds very sensible in general. It’s only wrong for Crumb.”

Two key ideas hide in that exchange. First, a RAG answer can fail in two places: retrieval (the right chunks never arrived) or generation (they arrived, but the model ignored them, added things or dodged). One blended score cannot tell you which half to fix. Second, a judge needs something to judge against: the retrieved chunks, and a reference answer written by someone who knows.

An eval (short for evaluation) is a repeatable test of an AI system: the same questions, scored the same way, every time you change something. You will spend all of Module 23 on evals. This lesson is the RAG-specific version.

The golden dataset

Every RAG eval starts with a golden dataset: a list of test questions, each with a reference answer written or checked by a human who knows the truth.

Priya asked two people to write Crumb’s: Farida, the Koregaon Park store manager, and Kavita from HR. They wrote 40 questions in an afternoon, in a shared spreadsheet, deliberately mixing kinds:

Kind	Example	How many
Simple lookups	“What temperature should the walk-in chiller be?”	10
Multi-part answers	“List every step of the closing checklist.”	6
Codes and names	“What is SOP-14?”	6
Messy real wording	“cake came back smelling weird what do i do”	10
Unanswerable	“What’s the WiFi password at Baner?”, “What is the CEO’s salary?”	8

The last row matters most for Meera’s second question. Those eight have the reference answer “Not in the documents”. Without them, you will never see how often the bot makes something up.

Forty is enough to start. It lives as a JSON file in the repo, and it grows: every real failure from production becomes a new test question.

Following one question through four checks

Priya took the bad cake answer, question 14, and scored it by hand on the whiteboard, one check at a time. This is exactly what the RAGAS library does automatically, so it is worth seeing once with your own eyes.

The question: “What should a store do if a customer returns a cake that smells off?”

The reference answer (from Farida): “Apologise, offer a replacement or refund, keep the product sealed, and send it to the central kitchen for a quality check.”

The four chunks retrieved, in order:

1. Customer returns of spoiled or off-smelling products: apologise, offer a replacement or refund, keep the product sealed and send it to the central kitchen for quality checks.
2. Pest control visits happen every second Tuesday…
3. Refunds above 2,000 rupees need approval from the store manager…
4. Greeting customers: smile, make eye contact…

The bot’s answer: “Apologise, offer a replacement or refund, and throw the cake away immediately to avoid contamination [1].”

Think of a restaurant inspector checking one dish. Two checks look at the ingredients that reached the kitchen (retrieval). Two look at the plate that reached the table (generation).

Check 1: context recall. Did the ingredients include everything needed?

Break the reference answer into separate claims and ask, for each one, “is this somewhere in the retrieved chunks?”

Reference claim	In the chunks?
Apologise	Yes, chunk 1
Offer a replacement or refund	Yes, chunk 1
Keep the product sealed	Yes, chunk 1
Send it to the central kitchen for a quality check	Yes, chunk 1

4 out of 4: context recall = 1.0. Retrieval delivered every fact the answer needed. Low recall would mean the facts never reached the model, and no prompt could fix that.

Check 2: context precision. Were the useful ingredients on top?

For each retrieved position, ask “is this chunk relevant?” Here: yes, no, no, no. The relevant chunk is in position 1. Context precision rewards relevant chunks that appear early and penalises relevant chunks buried under junk. With the only relevant chunk at the top, context precision = 1.0.

Had the same chunk come back third, behind pest control and refunds, precision would drop to about 0.33: same facts, but buried under distractions.

Check 3: faithfulness. Is everything on the plate supported by the ingredients?

Now break the bot’s answer into claims and check each against the retrieved chunks.

Answer claim	Supported by the chunks?
Apologise	Yes
Offer a replacement or refund	Yes
Throw the cake away immediately	No. Chunk 1 says the opposite: keep it sealed

2 out of 3: faithfulness = 0.67. Faithfulness is the RAG version of hallucination. Here the model fell back on general knowledge (“spoiled food goes in the bin”) instead of Crumb’s rule.

Check 4: answer relevancy. Did the plate answer what was ordered?

An answer can be faithful and still useless: “The handbook covers customer returns in Chapter 11” is true but tells the staff member nothing. RAGAS checks relevancy cleverly: a judge LLM reads only the answer and guesses, a few times, what question it was replying to. Those guesses are compared with the real question using embeddings (lists of numbers that capture meaning). For our answer, the judge guessed things like “What should staff do when a customer returns a spoiled cake?”, very close to the real question. Answer relevancy ≈ 0.9.

What the four numbers say together

Metric	Score	Verdict
Context recall	1.0	Retrieval found everything
Context precision	1.0	And put it on top
Faithfulness	0.67	The model added something unsupported
Answer relevancy	≈ 0.9	And it did answer the question

“So the search upgrade worked,” Arjun said slowly, “and the problem is the prompt.”

“Exactly. Your 1-to-10 judge gave it 8 and told you nothing. These four numbers tell you which file to open.”

That pattern is so useful that Priya made it a table for the team wiki:

Low score	What it tells you	Where to look
Context recall	The facts never reached the model	Loading, chunking, search, k, query rewriting
Context precision	Facts arrived, buried under noise	Reranking, chunk size, fewer chunks
Faithfulness	The model added or changed things	Prompt rules, abstention, citations, model choice
Answer relevancy	The model dodged or rambled	Prompt, answer format

Opening the hood: how RAGAS scores automatically

RAGAS is an open-source Python library for scoring RAG systems with these metrics. It does what Priya did on the whiteboard, using an LLM as a judge: a second model that makes small, specific yes-or-no decisions (“is this claim supported by this text?”, “is this chunk relevant to this question?”). RAGAS then turns those decisions into numbers between 0 and 1.

The trick that makes this work, where “rate it out of 10” failed, is decomposition. “Is this answer good?” invites a generous, vague verdict. “Does this exact sentence follow from this exact paragraph?” is a narrow reading task that LLMs are good at.

Two metrics need the reference answer (context recall, and the version of context precision used here). Faithfulness and answer relevancy only need the question, the chunks and the bot’s answer, which means you can also run them on real production traffic where no reference exists.

What it costs: each metric makes one or a few judge calls per question, so 40 questions and four metrics is a few hundred calls per run. With a small judge model that is typically well under a dollar. Cheap enough for every pipeline change, too expensive for every keystroke.

Running RAGAS on Crumb’s golden set

You run your RAG pipeline over every golden question, keep four things per question, and hand them to RAGAS.

# uv add ragas langchain-openai python-dotenv
import json
import os

from dotenv import load_dotenv
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from ragas import EvaluationDataset, evaluate
from ragas.embeddings import LangchainEmbeddingsWrapper
from ragas.llms import LangchainLLMWrapper
from ragas.metrics import (Faithfulness, LLMContextPrecisionWithReference,
                           LLMContextRecall, ResponseRelevancy)

load_dotenv()
MODEL = os.getenv("OPENAI_MODEL", "gpt-5-mini")   # model names change; see the provider's models page

golden = json.load(open("golden.json"))           # [{"question": ..., "reference": ...}, ...]

rows = []
for item in golden:
    chunks = retrieve(item["question"])            # your retriever: list of chunk texts
    rows.append({
        "user_input": item["question"],
        "retrieved_contexts": chunks,
        "response": generate(item["question"], chunks),   # your answer function
        "reference": item["reference"],
    })

Each row has exactly the four things from the whiteboard: the question, what was retrieved, what the bot said, and the human’s reference answer. Now score them:

result = evaluate(
    dataset=EvaluationDataset.from_list(rows),
    metrics=[LLMContextPrecisionWithReference(), LLMContextRecall(),
             Faithfulness(), ResponseRelevancy()],
    llm=LangchainLLMWrapper(ChatOpenAI(model=MODEL)),                       # the judge
    embeddings=LangchainEmbeddingsWrapper(OpenAIEmbeddings(model="text-embedding-3-small")),
)
print(result)                                      # the average of each metric
result.to_pandas().to_csv("ragas_run.csv")         # one row per question: read the worst ones

You should see four averages, something like {'llm_context_precision_with_reference': 0.86, 'context_recall': 0.78, 'faithfulness': 0.84, 'answer_relevancy': 0.91}, and a CSV with a score per question.

Notice that the judge is from a different model family (OpenAI) than the model writing answers (Claude). A judge from the same family can share the same blind spots and tends to like its own style. Using a different family is a cheap way to reduce that.

Library APIs change quickly. If a name has moved, check the docs linked below. RAGAS in particular has renamed things between versions: older tutorials use the column names question, contexts, answer and the metric name answer_relevancy, which newer versions call response relevancy.

The “I don’t know” rate

RAGAS answers Meera’s first and third questions. Her second, “when it doesn’t know, does it admit it?”, needs one more measurement.

Abstention means the system deliberately declines to answer (“I don’t know based on Crumb’s documents”) instead of guessing. There are two ways to get it wrong, and they pull in opposite directions:

	The answer is in the documents	The answer is not in the documents
Bot answers	Good (then check faithfulness)	Hallucination: the dangerous one
Bot abstains	Over-cautious: annoying, erodes trust	Good: correct abstention

Arjun’s first run on the 40 golden questions filled in the table like this: of the 32 answerable questions, 29 answered and 3 abstained. Of the 8 unanswerable, only 3 abstained. The other 5 got confident answers: a made-up WiFi password policy (“WiFi details are available from the store manager”), a guess at the leave carry-over rule for a policy that does not exist, and so on.

“Five out of eight,” Meera said. “So when it doesn’t know, it admits it less than half the time.”

Track two numbers from now on: the correct abstention rate on unanswerable questions (aim above 90%), and the false abstention rate on answerable ones (keep it low; every unnecessary “I don’t know” sends a staff member to call their manager). Pushing one usually hurts the other, so you watch both together. A bot that always says “I don’t know” has a perfect hallucination rate and zero usefulness.

Four ways to teach a bot to say “I don’t know”

From cheapest to strongest.

1. Say it in the prompt, with an exact phrase. “If the context does not contain the answer, reply exactly: I don’t know based on Crumb’s documents.” The exact phrase does two jobs: it gives the model a clear, acceptable way out (models often guess because nothing told them not answering was allowed), and it makes abstentions countable in code with a string comparison.

2. Gate on retrieval scores. If even the best chunk is a poor match, do not call the LLM at all. Remember the cross-encoder reranker from the previous lesson? Its top score is a good signal. On Crumb’s sample, answerable questions had top scores between −8.95 and +7.47, and unanswerable ones (“What is the CEO’s salary?”, “Can staff bring pets to work?”, the WiFi question) all came in between −11.24 and −10.69.

3. Cite or abstain. Require the model to quote the exact sentence it relied on, then check in code that the quote really appears in a retrieved chunk. No valid quote, no answer. This catches the throw-it-away kind of failure, because there is no sentence in the chunks that says “throw it away”.

4. Verify after generating. Run a faithfulness check (a quick judge call: “is every claim in this answer supported by these chunks?”) on the answer before sending it, and abstain if it fails. The most reliable of the four and the slowest, since it adds a second LLM call.

Here is the cheapest combination, techniques 1 to 3, in one function. It builds on hybrid_search, rerank and text_by_id from the previous lesson.

import anthropic

client = anthropic.Anthropic()
ABSTAIN = "I don't know based on Crumb's documents."
MIN_TOP_SCORE = -10.0      # chosen from the golden set; re-check if the reranker changes

SYSTEM = ("Answer only from the numbered context. After your answer, add a line "
          'starting with Quote: and copy, word for word, the sentence you relied on. '
          f"If the context does not contain the answer, reply exactly: {ABSTAIN}")

The system prompt sets up techniques 1 and 3: an exact abstention phrase, and a mandatory quote line.

def answer_or_abstain(question: str) -> str:
    ranked = rerank(question, hybrid_search(question))      # [(chunk_id, score), ...]
    if not ranked or ranked[0][1] < MIN_TOP_SCORE:
        return ABSTAIN                                       # technique 2: the gate

    chunks = [text_by_id[cid] for cid, _ in ranked]
    context = "\n".join(f"[{i + 1}] {c}" for i, c in enumerate(chunks))
    response = client.messages.create(
        model="claude-opus-5-5", max_tokens=500, system=SYSTEM,
        messages=[{"role": "user", "content": f"Context:\n{context}\n\nQuestion: {question}"}],
    )
    reply = next(b.text for b in response.content if b.type == "text")
    if reply.strip() == ABSTAIN or "Quote:" not in reply:
        return ABSTAIN

    answer_text, quote = reply.split("Quote:", 1)
    quote = quote.strip().strip('"')
    if not any(quote in c for c in chunks):                  # technique 3: check the quote
        return ABSTAIN
    return answer_text.strip()

Line by line: the gate returns early, before any LLM cost, when the best reranker score is too low. If the model abstained or forgot the quote, the function abstains. Then quote in c checks the quote word for word against every chunk; an invented or paraphrased quote fails the answer. Only the answer part reaches the user.

After adding this, Arjun re-ran the 40 questions. Correct abstentions: 8 out of 8. False abstentions: 4 out of 32, up from 3. Faithfulness: from 0.84 to 0.93. He looked at the four false abstentions; three were messy-wording questions where the model had paraphrased the quote. He tightened the instruction (“copy the sentence exactly, including punctuation”) and two of them recovered.

Where it breaks

• Judges drift. A provider updates its model and last month’s 0.91 becomes 0.87 with no change on your side. Pin the judge version where you can.
• Averages hide disasters. A 0.93 average can hide one allergen answer at 0.0. Sort by score; read the bottom.
• Small datasets wobble. With 40 questions, one flip moves a metric by 2.5 points. Look at which questions changed before celebrating.
• The golden set goes stale. When the handbook changes, update the reference answers too, or the eval punishes the bot for being right.
• Exact-quote checks are brittle. Curly quotes or a dropped full stop fail the check. Normalise whitespace and punctuation before comparing.

Try it yourself

Write 10 golden questions for any document you have: 7 answerable, 3 not. Run your pipeline and fill in the abstention table by hand. Then add technique 1 (the exact phrase) and fill it in again. Finally, run RAGAS on the 7 answerable questions and find the question with the lowest faithfulness.

Show what usually happens

Without an abstention instruction, most pipelines confidently answer at least one unanswerable question from general knowledge. With the exact phrase, correct abstentions usually reach 3 out of 3, and one answerable question may be wrongly abstained, typically because its chunk was not retrieved (check its context recall to confirm). The lowest-faithfulness answer almost always contains a sentence that “sounds right in general” but is not in the chunks, exactly like the throw-it-away answer.

Common mistakes

• Using one overall “quality” score. It cannot tell you whether to fix retrieval or generation.
• Only testing answerable questions. You will never see your hallucination rate.
• Changing the golden set and the pipeline at the same time. Then you cannot tell which change moved the numbers.
• Setting thresholds by intuition. Choose them from your golden set’s actual scores, and re-check them when a model changes.

Recap

Two days later Arjun put one slide in front of Meera. He explained it without a single metric name.

• We wrote 40 test questions with the right answers, including 8 that the handbook cannot answer. We run every change against all 40.
• We check four things separately: did the search find all the facts, did it put them on top, did the bot stick to them, and did it actually answer the question.
• Each failing check points to a different fix. The cake answer failed only “stuck to the facts”, so we fixed the instructions, not the search.
• When the bot can’t find a good match, or can’t quote the sentence it used, it now says “I don’t know” instead of guessing. It does that for 8 of the 8 impossible questions, and wrongly on only a few real ones.
• Every new mistake from the stores becomes test question 41, 42, 43, so it can’t sneak back.

Meera turned the index card over and wrote three numbers on the back: faithfulness 0.93, correct “I don’t know” 8 of 8, false “I don’t know” 4 of 32. “Now I can trust it a bit,” she said. “And I can ask you next month whether these went up or down.”

On the way out she stopped. “Priya, someone from your old company keeps telling Arjun he should be using something called LlamaIndex. Should he?”

Go deeper

• RAGAS documentation: https://docs.ragas.io
• Module 23 goes much further with evals, LLM judges and RAGAS in practice; tools like Langfuse store these scores over time: https://langfuse.com/docs