RAGAS metrics in practice

The store handbook assistant had been live for a month. Staff at all six Crumb stores could ask it things like “What’s the closing checklist for Baner?” and it would search the 200-page operations handbook and answer in a few seconds. Store managers loved it. Mostly.

Then on Tuesday the Kothrud manager asked: “What temperature should the cream cake display fridge be at?” The assistant said 8°C. The handbook says 4°C. A fridge full of fresh cream cakes at 8°C for a day is a food safety problem, and Meera found out because the Kothrud manager, sensibly, phoned her to double-check.

“How did it get a number that wrong?” Meera asked. “And how many other answers are wrong that nobody phoned about?”

Arjun’s first fix: give it more pages

RAG (retrieval-augmented generation) means fetching relevant passages from your documents and giving them to the model along with the question, so it answers from your facts instead of from memory. You built this handbook assistant in Module 09. For every question, it embedded the question, searched the handbook’s chunks for the 3 most similar, and handed those to the model.

Arjun’s reasoning was quick. “If the answer was wrong, the right page probably wasn’t in the top 3. I’ll fetch the top 10. More context, better answers.”

He changed n_results=3 to n_results=10 and ran the 40-question test set he had put together with the store managers. To check answers, he asked a model whether each answer matched the reference answer.

Setting	Correct answers (of 40)	Prompt size	Cost per question
Top 3 chunks	34	about 1,250 tokens	₹0.30
Top 10 chunks	31	about 3,900 tokens	₹0.85

Fewer correct answers, triple the cost. The fridge question was now right, but three questions that used to be right were now wrong. Arjun stared at the table. “That makes no sense.”

“It makes sense,” Priya said, “once you stop grading the whole system with one number.”

Two workers in a library

“Imagine Crumb has a reference library,” Priya said, “with two staff. A librarian, who goes to the shelves and brings back a few pages that look relevant. And a writer, who reads only those pages and writes the answer. When an answer is wrong, whose fault is it?”

“Depends,” Arjun said. “Either the librarian brought the wrong pages, or the writer misread the right ones.”

“And your test says only ‘wrong’. So when you got 6 wrong, what did you change?”

“The librarian. I told her to bring ten pages instead of three.”

“Without knowing whether the librarian was the problem. And what happens to a writer who gets ten pages, seven of them irrelevant?”

Arjun thought about the three questions that broke. “She gets distracted. Picks up a number from the wrong page.”

“So you need two report cards. One for the librarian: did she bring the right pages, and put the best one on top? One for the writer: did she stick to the pages, and actually answer the question?”

The two halves, then:

• Retrieval failures: the right passage was never fetched, or was buried among noise. Fixes: better chunking (how documents are split into pieces), better search, a reranker, metadata filters.
• Generation failures: the right passage was there, and the model ignored it, misread it, or added things that were not in it. Fixes: a clearer prompt, less noisy context, a stronger model.

RAGAS is an open-source Python library that writes those report cards. Most of its metrics use an LLM as a judge behind the scenes, which matters a lot, and we will come back to it.

Where the analogy breaks: a real librarian understands the question. A retriever only measures how similar the words and meanings look, which is why it can confidently bring back the wrong fridge.

Following the fridge question through the pipeline

Arjun pulled up the trace for the fridge question and walked through it with Priya, step by step.

1. The question. “What temperature should the cream cake display fridge be at?”

2. Retrieval. The question was turned into an embedding (a list of numbers capturing its meaning) and compared with every handbook chunk. The top 3 by similarity:

Rank	Chunk (start)	Section	Similarity
1	“Front-of-house fridges and coolers: keep at 8°C or below. Wipe seals weekly.”	Drinks and snacks	0.71
2	“Display fridge cleaning: empty, wipe with food-safe spray, restock.”	Cleaning	0.66
3	“Cake display: rotate stock so oldest items are at the front.”	Merchandising	0.63

The chunk with the right answer, “Cold chain SOP: cream, custard and fresh-fruit products must be held at 0 to 4°C”, ranked ninth, at 0.48. It never uses the words “fridge” or “display”. The handbook’s food safety team wrote “cold chain” and “held”, while store staff say “fridge”.

3. Generation. The three chunks plus the question went to the model (1,250 tokens). The model read chunk 1, saw “fridges … 8°C or below”, and answered: “The display fridge should be kept at 8°C or below.”

Now grade each worker by hand, the way RAGAS does.

The writer: faithfulness. Break the answer into claims and check each against the retrieved chunks. One claim: “the display fridge should be at 8°C or below”. Is it supported by what the writer was given? Chunk 1 says fridges at 8°C or below. Supported. 1 of 1 claims: faithfulness = 1.0. The writer did her job perfectly with the pages she got.

The writer: answer relevancy. Does the answer address the question asked? Yes, it gives a temperature for the display fridge. High, about 0.9.

The librarian: context recall. Take the reference answer, the correct answer a person wrote for the test set: “Cream cake display fridges must be kept at 4°C or below.” Can its claims be found in the retrieved chunks? No. 0 of 1: context recall = 0.

The librarian: context precision. Of the 3 retrieved chunks, how many are actually relevant to answering the question, and were the relevant ones ranked first? None of them are relevant. Context precision = 0.

Metric	Worker	Fridge question
Faithfulness	Writer	1.0
Answer relevancy	Writer	about 0.9
Context recall	Librarian	0.0
Context precision	Librarian	0.0

“So the writer was blameless,” Arjun said. “The librarian never brought the cold chain page.”

“And your top-10 change?”

“Pulled it in at rank 9, so the fridge question got fixed. But it also filled every other question’s context with seven extra pages of noise.”

Now compare a different failure from the test set. Question: “What’s on the Baner closing checklist?” The right chunk was retrieved at rank 1 (context recall 1.0). The answer listed five steps; four were in the chunk, and the fifth, “lock the back door twice”, appeared nowhere in the handbook. Faithfulness: 4 of 5 claims supported, 0.8. Here the librarian was perfect and the writer made something up. A completely different fix.

Opening the hood: the four metrics

Each score runs from 0 to 1, where higher is better. Here is what each measures and, roughly, how RAGAS computes it.

Faithfulness (writer). The share of claims in the answer that the retrieved context supports. RAGAS asks an LLM to split the answer into separate claims, then asks it to check each claim against the context. Low faithfulness means hallucination, the model stating things the sources do not say.

Answer relevancy (writer), called response relevancy in newer versions. Does the answer address the question? RAGAS asks an LLM to write a few questions the answer would be a good answer to, embeds them, and measures how similar they are to the real question. An answer about fridge cleaning, given to a question about fridge temperature, scores low even if every word is true.

Context precision (librarian). Are the relevant chunks ranked near the top? An LLM judges each retrieved chunk as relevant or not, and the score rewards relevant chunks at rank 1 more than at rank 5. Low precision means the writer is wading through noise.

Context recall (librarian). Did retrieval find everything needed? An LLM splits the reference answer into claims and checks whether each can be found in the retrieved chunks. This one needs a reference answer for every test question, and it is often the most useful number of the four.

Read together, they point at the broken half:

Context recall	Faithfulness	What it means	Fix first
Low	High	Writer faithfully used the wrong pages	Retrieval
High	Low	Right page was there; writer strayed	Prompt or model
Low	Low	Both struggling	Retrieval first, then re-check
High	High	Check answer relevancy, then the test question itself	Usually fine

Building a test set worth measuring

Metrics need questions. You want 30 to 50, each with:

• the question (user_input in RAGAS),
• a reference answer written or checked by someone who knows the handbook (reference).

Then you run each question through your RAG system and record what it retrieved (retrieved_contexts) and what it answered (response).

Arjun’s first 40 questions came from three places: 20 sampled from real staff questions in the traces, 10 from the failures found while reading those traces, and 10 the store managers said they wished the assistant handled better. He did not write them himself. His vocabulary matched the handbook’s; the staff’s did not. “Fridge” versus “cold chain” was exactly the gap that real questions reveal and invented ones hide.

The real code

Install RAGAS alongside an OpenAI client, because RAGAS needs a model to do its judging:

uv add ragas openai pandas python-dotenv

Step 1: one metric, one row. Score the fridge question for faithfulness:

import asyncio
import os
from dotenv import load_dotenv
from openai import AsyncOpenAI
from ragas.llms import llm_factory
from ragas.metrics.collections import Faithfulness

load_dotenv()
# Model names change; the current list is on the provider's models page.
MODEL = os.getenv("OPENAI_MODEL", "gpt-5-mini")
llm = llm_factory(MODEL, client=AsyncOpenAI())

row = {
    "user_input": "What temperature should the cream cake display fridge be at?",
    "reference": "Cream cake display fridges must be kept at 4°C or below.",
    "retrieved_contexts": [
        "Front-of-house fridges and coolers: keep at 8°C or below. Wipe seals weekly.",
        "Display fridge cleaning: empty, wipe with food-safe spray, restock.",
        "Cake display: rotate stock so oldest items are at the front.",
    ],
    "response": "The display fridge should be kept at 8°C or below.",
}

async def main():
    result = await Faithfulness(llm=llm).ascore(
        user_input=row["user_input"],
        response=row["response"],
        retrieved_contexts=row["retrieved_contexts"],
    )
    print(f"faithfulness = {result.value:.2f}")

asyncio.run(main())

Expected output: faithfulness = 1.00. The answer is faithful to what it was given.

Step 2: add the librarian’s report card. Context recall takes the reference instead of the response:

from ragas.metrics.collections import ContextRecall

async def score_row(row: dict) -> dict:
    f = await Faithfulness(llm=llm).ascore(
        user_input=row["user_input"], response=row["response"],
        retrieved_contexts=row["retrieved_contexts"])
    r = await ContextRecall(llm=llm).ascore(
        user_input=row["user_input"], reference=row["reference"],
        retrieved_contexts=row["retrieved_contexts"])
    return {"question": row["user_input"], "faithfulness": f.value, "recall": r.value}

print(asyncio.run(score_row(row)))

Expected output: {'question': 'What temperature ...', 'faithfulness': 1.0, 'recall': 0.0}. Diagnosis: retrieval.

Step 3: score the whole test set, and read the worst rows. rag_answer is your existing RAG function, returning the answer and the chunks it used:

import json
import pandas as pd

with open("handbook_tests.jsonl") as f:
    tests = [json.loads(line) for line in f]       # each has user_input and reference

async def score_all() -> pd.DataFrame:
    rows = []
    for t in tests:
        answer, chunks = rag_answer(t["user_input"])
        rows.append(await score_row({**t, "response": answer, "retrieved_contexts": chunks}))
    return pd.DataFrame(rows)

df = asyncio.run(score_all())
print(df[["faithfulness", "recall"]].mean().round(2))
print(df.sort_values("recall").head(5)[["question", "recall", "faithfulness"]])

Expected output, for Crumb’s first run:

faithfulness    0.94
recall          0.71

followed by the five questions with the lowest recall. Four of the five used staff vocabulary that the handbook did not (“fridge”, “till”, “float”, “rota”).

Library APIs change quickly. If a name has moved, check the docs linked below. RAGAS has changed its interface several times; older tutorials use an evaluate() function with an EvaluationDataset, which you may still meet.

Step 4: a free retrieval check, with no LLM at all. If you record which chunk should answer each test question, you can measure hit rate: the share of questions where that chunk appears in the top k results. It costs nothing, runs in milliseconds and gives the same answer every time:

def hit_rate(tests: list[dict], k: int = 3) -> float:
    hits = 0
    for t in tests:
        ids = search_ids(t["user_input"], n_results=k)   # your retriever, returning chunk ids
        hits += t["answer_chunk_id"] in ids
    return hits / len(tests)

print(f"hit@3 = {hit_rate(tests):.0%}")   # hit@3 = 70%

Run this one on every change. Run RAGAS when you want the fuller picture.

What the numbers told Crumb

Averages across the 40 questions:

Metric	Top 3 (original)	Top 10 (Arjun’s fix)	Top 3 + keyword search
Context recall	0.71	0.88	0.92
Context precision	0.62	0.31	0.78
Faithfulness	0.94	0.86	0.95
Correct answers (of 40)	34	31	38

The story is now readable. The original system had a retrieval problem: recall 0.71, with faithfulness fine. Top 10 raised recall but crashed precision, and faithfulness fell with it: the writer was distracted by noise. The real fix was on the retrieval side. Arjun added hybrid search, which combines meaning-based search with old-fashioned keyword matching, and added staff vocabulary (“fridge”, “chiller”) as keywords on the cold chain chunks. Recall rose to 0.92 with only three chunks, so precision stayed high and the writer stayed focused.

Where RAGAS breaks

It is an LLM judge, so check it. Faithfulness and recall are graded by a model and inherit everything from the LLM-as-judge lesson: noise, bias and occasional nonsense. Meera labelled 30 rows by hand (“is every claim in this answer supported by these chunks?”). RAGAS faithfulness, thresholded at 0.9, agreed with her on 26 of 30. Good enough to diagnose with; not good enough to treat as truth.

Scores are continuous; decisions are not. A faithfulness of 0.83 roughly means 83% of claims were supported. To act, set a threshold (“below 0.9 is a fail”) and track the pass rate, as in the binary evals lesson.

Generic metrics miss your specific failures. RAGAS cannot know that quoting the wrong store’s closing time is Crumb’s most common handbook error. Use it to answer “which half is broken?”, alongside the specific evals your error analysis produced.

It costs money and time. Each metric makes several model calls per row. Forty rows times four metrics at about three calls each is roughly 480 calls, about ₹60 per run at Crumb’s prices and several minutes. Run it when retrieval or prompts change; run hit rate on every commit.

Reference answers go stale. When the handbook changes, references written last quarter become wrong, and context recall starts punishing correct retrieval. Re-check references whenever a document is updated.

Try it yourself

Take the RAG system you built in Module 09. Write 10 test questions with reference answers, at least 5 of them phrased the way a real user would, not the way the document is written. Run them through your system and score faithfulness and context recall for each.

Expected result: a table of 10 rows. Find the row with the lowest recall and open its retrieved chunks. Can you name why the right passage was missed: different vocabulary, a chunk split in the wrong place, or the wrong document entirely? Then try one retrieval fix and re-score.

Common mistakes

• Fixing without diagnosing. Raising top k, swapping models or rewriting prompts before knowing which half failed often makes the other half worse.
• Optimising the average. A 0.91 average can hide a 0.2 on the one food safety question. Always read the lowest-scoring rows.
• Skipping reference answers. Without them you cannot measure context recall, often the most useful number.
• Testing with invented questions. Your vocabulary matches your documents; your users’ does not.
• Treating RAGAS as ground truth. It is a judge. Check it against your own labels before acting on it.

Recap

• A RAG system has two workers: retrieval (the librarian) and generation (the writer).
• Context recall and context precision grade the librarian; faithfulness and answer relevancy grade the writer.
• Read the scores together to find which half to fix, then fix only that half.
• Build a 30 to 50 question test set from real questions, with reference answers checked by an expert.
• RAGAS metrics are LLM-judged: validate them against your labels, threshold them, and read the worst rows.
• A free hit-rate check on retrieval complements RAGAS and can run on every change.

Arjun explained it to Meera on the shop floor at Kothrud, with the manager listening in. “The assistant works in two steps: find the right pages, then write an answer from them. For the fridge question, the writing step was perfect. It faithfully repeated the 8°C rule for drinks coolers, because the finding step brought back the wrong page. The cold chain page uses words staff never use. Now we score each step separately on 40 real questions, and the finding step went from 71% to 92%.”

The manager asked the fridge question on her phone. “0 to 4°C,” she read out. Meera asked her to try it twice more, just to be sure.

That evening, feeling good, Arjun changed the WhatsApp order parser’s prompt to handle “half kg” quantities, ran it on three examples, and pushed it straight to production. He would regret that by Thursday.

Go deeper

• RAGAS documentation
• RAGAS paper: Automated Evaluation of Retrieval Augmented Generation
• Retrieval-Augmented Generation paper (Lewis et al.)