Why RAG exists

It was the second week of October, and Crumb’s Diwali hampers had gone on sale that morning. By 11 a.m. the support inbox had 140 unread emails, and a third of them asked the same kind of question: is this hamper eggless?, does the dry-fruit box have nuts?, can it reach Nashik by Friday?

Arjun had a plan. Over the weekend he had wired the inbox to a large language model with a friendly system prompt. On Monday he showed it to Meera. He typed a real customer email into the test window: “Is the Diwali Delight hamper eggless? My mother is strictly vegetarian.”

The reply came back in under two seconds. “Yes! All of Crumb’s Diwali hampers are 100% eggless, so your mother can enjoy them with complete peace of mind.”

Meera read it twice and went quiet. “The Diwali Delight has the saffron plum loaf in it,” she said. “That loaf has eggs. We changed the recipe in August. If we had sent that, a strictly vegetarian family would have eaten egg because we told them it was safe.”

Arjun checked the logs. Nothing had gone wrong, technically. No error, no timeout. The model had answered, smoothly and wrongly. Priya, who had been listening from the next desk, came over with a whiteboard marker.

“You gave it a closed-book exam,” she said, “on a subject it never studied. Let’s give it the book.”

Why the bot was so sure, and so wrong

“Let’s start with what happened,” Priya said. “Where did ‘all our hampers are eggless’ come from?”

“From the model?” Arjun said. “It must have learned it somewhere.”

“Learned it from where? Has Crumb’s August recipe change ever been on the internet?”

He thought about it. “No. It’s in Meera’s recipe archive. Nowhere public.”

“So the model has never seen it. Then what is it doing when it answers?”

To answer her, you need the picture from Module 01. A large language model (LLM, a program trained to continue text by predicting the next word, one piece at a time) learns during training (a slow, expensive process where the model’s internal numbers are adjusted, again and again, to predict a huge pile of public text better). When training ends, those numbers are frozen. Every time you send a question afterwards, the model is doing inference: using its frozen numbers to produce the most likely continuation of your text.

That has three consequences, and each one showed up in Arjun’s demo.

1. It has a knowledge cutoff. The knowledge cutoff is the date its training data stops. Crumb’s August recipe change happened after, or at least was never in, that data.
2. It has never seen your private documents. The 200-page store operations handbook, the HR policies, the recipe archive and the order database live inside Crumb. The model knows what a typical Indian bakery’s Diwali hamper looks like. It knows nothing about Crumb’s.
3. It does not know what it does not know. When asked something it cannot know, a model rarely says “I have no idea”. It produces the answer that sounds most likely, fluently and politely. A confident answer that is not grounded in fact is called a hallucination.

“So it wasn’t lying,” Arjun said slowly. “It was guessing. And festive hampers from Indian bakeries are often eggless, so ‘yes’ was a good guess.”

“A good guess in general. A dangerous guess for us. The model did exactly what it was built to do. The mistake was asking it a question only our documents can answer.”

The open-book exam

Priya drew two desks on the whiteboard.

At the first desk, a student sits a closed-book exam on a subject they only half studied. They write what they remember. Where they half-remember, they guess, and a confident student guesses confidently.

At the second desk, the same student has the textbook open. Even better, a librarian has already put sticky notes on the three pages that matter for question 4. The student reads those pages and writes the answer. They can even write “see page 87”.

“Which desk was your bot at?” she asked.

“The first one. Closed book.”

“Right. RAG moves it to the second desk. Before the model answers, your code does three things.” She wrote them down the side of the board:

1. Retrieve. Search your documents for the passages most related to the question.
2. Augment. Paste those passages into the prompt, next to the question.
3. Generate. Let the model write the answer, with instructions to use only those passages and to say so when they do not contain the answer.

That is the whole name: Retrieval-Augmented Generation. The idea was named and published in a 2020 research paper, but you will see it everywhere now, under the hood of almost every “chat with your documents” product.

“Where does the analogy break?” Arjun asked. He had learned that Priya always had an answer to that.

“The librarian. A real librarian understands the question. In RAG, the librarian is a search system that you build. It can put sticky notes on the wrong pages. And if it does, the model will answer confidently from the wrong pages, with a citation that makes the wrong answer look trustworthy.” She tapped the board. “Most of this module is about building a better librarian.”

Following one question through a tiny RAG

“Show me the smallest possible version,” Arjun said. “No libraries.”

Priya made him type three snippets from Crumb’s product sheets into a Python dictionary. That was the “library”. Then they followed one question all the way through.

The question: “Is the Diwali Delight hamper eggless?”

Step 1: Retrieve. The simplest search there is: split the question and each document into lowercase words, and count how many words they share. The question’s words are is, the, diwali, delight, hamper, eggless. Counting shared words against each document gives:

Document	Shared words	Score
hamper-diwali-delight	diwali, delight, hamper, eggless	4
hamper-festive-classic	hamper, eggless	2
delivery-policy	none	0

Take the top two (k = 2). The Diwali Delight sheet wins clearly. The Festive Classic sheet comes along as a runner-up, which is harmless.

Step 2: Augment. Glue those two documents into the prompt, each labelled with its ID, under a heading that says “Context”. Together with the instructions and the question, the whole prompt is about 150 tokens (the word-pieces models read and charge by; about 1.3 tokens per English word).

Step 3: Generate. The model receives the instructions (“answer only from the context, cite the ID, say you don’t know if it’s not there”), the two documents and the question. It reads that the Diwali Delight contains a “saffron plum loaf (contains egg)” and answers:

“No. The Diwali Delight hamper includes a saffron plum loaf that contains egg. The rose shortbread in it is eggless, but the hamper as a whole is not. If you need a fully eggless option, the Festive Classic hamper is fully eggless. [hamper-diwali-delight] [hamper-festive-classic]”

Arjun stared at it. “It even suggested the other hamper.”

“Because the librarian handed it the Festive Classic sheet as a runner-up. Sometimes a slightly generous search helps.”

Here is the code they wrote.

# uv add anthropic python-dotenv   (or: pip install anthropic python-dotenv)
import re

import anthropic
from dotenv import load_dotenv

load_dotenv()                  # reads ANTHROPIC_API_KEY from a .env file
client = anthropic.Anthropic()

# Our whole "library": three snippets from Crumb's product sheets.
docs = {
    "hamper-diwali-delight": "Diwali Delight hamper (2026): kaju katli, saffron plum "
        "loaf (contains egg), rose shortbread (eggless).",
    "hamper-festive-classic": "Festive Classic hamper: fully eggless. Besan ladoo, "
        "nankhatai, pista biscotti.",
    "delivery-policy": "Same-day delivery in Pune for orders placed before 2 pm.",
}

def words(text: str) -> set[str]:
    """Lowercase, keep only letters and digits, return the set of words."""
    return set(re.findall(r"[a-z0-9]+", text.lower()))

def retrieve(question: str, k: int = 2) -> list[str]:
    """Rank docs by how many words they share with the question. Crude but honest."""
    q = words(question)
    ranked = sorted(docs.items(), key=lambda item: len(q & words(item[1])), reverse=True)
    return [f"[{doc_id}] {text}" for doc_id, text in ranked[:k]]

def answer(question: str) -> str:
    context = "\n".join(retrieve(question))           # step 1 + step 2
    response = client.messages.create(                 # step 3
        model="claude-opus-5-5",
        max_tokens=500,
        system=("You are Crumb's support assistant. Answer ONLY from the context. "
                "Cite the [id] of every document you use. If the context does not "
                "contain the answer, reply: I don't know, let me check with the team."),
        messages=[{"role": "user",
                   "content": f"Context:\n{context}\n\nQuestion: {question}"}],
    )
    return next(b.text for b in response.content if b.type == "text")

print(answer("Is the Diwali Delight hamper eggless?"))

Line by line, the new parts:

• words() lowercases text and pulls out runs of letters and digits, so “eggless?” and “(eggless).” both become eggless. Without this, punctuation would stop words from matching.
• retrieve() scores each document by the size of the overlap between two sets of words (q & words(...)), sorts from highest to lowest, and returns the top k, each tagged with its ID so the model can cite it.
• answer() is the “augment and generate” half. The system prompt does three jobs: restricts the model to the context, asks for citations, and gives it an exact sentence to use when the context is silent.

Three things changed from Arjun’s first bot, and all three matter: the model saw the real product sheet, it was told to stay inside that sheet, and it was told what to say when the sheet does not answer the question.

Three ways to teach a model your facts

Arjun was not fully convinced. “Why go to the trouble of searching? Why not paste all our documents into every prompt? Or train the model on them properly?”

“Both are real options,” Priya said. “Let’s do the maths on each.”

Option 1: stuff everything in. Paste the entire handbook into every prompt. The limit is the context window (the maximum amount of text a model can read in one request, counted in tokens). Watch what happens as a window fills up:

Crumb’s handbook is 200 pages, about 100,000 words, which is roughly 130,000 tokens. Several modern models can hold that much. So it fits. The question is what it costs.

Suppose input tokens cost $3 per million (a typical mid-range price; prices change often, so check your provider’s pricing page) and a dollar is about ₹88. Then:

	Tokens per question	Cost per question	2,000 questions a month
Whole handbook in every prompt	~130,000	$0.39 (about ₹34)	$780 (about ₹68,600)
RAG: four short passages	~1,200	$0.0036 (about ₹0.32)	$7.20 (about ₹630)

Meera looked at the table for a long time. “Sixty-eight thousand rupees a month to answer ‘what time does the Kothrud store open?’”

There is a speed cost too. The model has to read all 130,000 tokens before it writes the first word of the answer, so long prompts take noticeably longer to start. And models tend to read the middle of a very long prompt less carefully than the beginning and end, so a rule on page 87 can be overlooked even though it was technically “in the prompt”.

To be fair to option 1: if your documents are small (a ten-page FAQ) and change rarely, stuffing them in can be exactly right. Prompt caching (which you will meet in Module 21) lets the provider reuse a long, repeated prompt cheaply, which changes the maths a lot for medium-sized documents. Do not build a search system for a document that fits on a napkin.

Option 2: fine-tune. Fine-tuning means continuing a model’s training on your own examples so its frozen numbers shift towards your data. It sounds like the “proper” way to teach facts. It is not, for three reasons. Fine-tuning is good at changing how a model writes (tone, format, a narrow skill), and unreliable at storing what is true; the model may still mix your facts with general knowledge. Every recipe change would need another training run. And a fine-tuned model cannot tell you which page a fact came from. You will learn fine-tuning properly in Module 33, where it is the right tool for the right jobs.

Option 3: RAG. Search, then paste only what is relevant. Cheap per question, easy to update (change a document, re-index it, done), and every answer can point to its source.

	Stuff everything	Fine-tune	RAG
Good for	Small, stable documents	Style, tone, format	Large or changing knowledge
Cost per question	High, grows with documents	Low after training	Low
Updating a fact	Edit the document	Retrain	Re-index one document
Shows its sources	Not easily	No	Yes
Main risk	Cost, slow start, missed middle	Mixes up facts	Search finds the wrong passage

“For Crumb,” Priya said, “the handbook is big, the recipes change every season, and Meera wants every allergy answer to point to its source. That’s RAG.”

Where the tiny RAG breaks

Priya asked Arjun to try three more questions before celebrating. Here is what happened.

“Which hamper is fully eggless?” The Festive Classic sheet scored 3 (it shares hamper, fully and eggless), the Diwali Delight scored 2. Both were retrieved. The model answered “the Festive Classic” and cited it. Good.

“What is the price of the Festive Classic?” The Festive Classic sheet scored 2 and was retrieved, but it has no price on it. The model replied, “I don’t know, let me check with the team.” Good. That is the instruction working: the right document was found, it did not contain the answer, and the model admitted it instead of inventing ₹1,499.

“Do you deliver to Mumbai?” This one went wrong, in an instructive way. The question’s words are do, you, deliver, to, mumbai. The delivery policy says “delivery”, not “deliver”. Different strings, so the overlap is zero. Every document scored zero, the sort kept the first two in the dictionary, and the model received two hamper sheets for a delivery question. It said it did not know, which was honest, but the right document was sitting there unread.

“Deliver and delivery,” Arjun groaned. “Same idea, different letters.”

“That is the central weakness of matching words,” Priya said. “Customers don’t use the words your documents use. ‘Money back’ versus ‘refund’. ‘Cake smells off’ versus ‘product spoilage’. We need a librarian who matches meaning, not spelling. You already met the tool for that in Module 07: embeddings.”

And there is a second, quieter risk in that Mumbai answer. Suppose the model had said “No, we don’t deliver to Mumbai.” That sounds reasonable. But the delivery document never said that. It said Crumb delivers same-day in Pune. The model would have stretched “Pune” into “not Mumbai”, which might be wrong (perhaps Crumb ships hampers by courier). An answer that goes beyond its sources, even slightly, is still a hallucination, only a smaller one. In the RAGAS lesson you will learn to measure this precisely.

What RAG does not fix

Arjun was now an enthusiast. Priya slowed him down with a list of things RAG will not do on its own.

• It does not fix bad documents. If the handbook says two different things on pages 40 and 140, RAG will retrieve whichever it finds and repeat it confidently.
• It does not guarantee the right passage is found. Search quality is now the most common source of wrong answers. You will spend most of this module improving it.
• It does not stop the model adding things. Instructions reduce it; measurement catches it; it never quite reaches zero.
• It does not replace a database. “How many orders did the Baner store take yesterday?” is not a document search. That is a query against the order database, a job for the function calling you learned in Module 06.

Try it yourself

Run the tiny RAG above with the four questions from this lesson, and print the retrieved documents before the answer for each one. Then fix the Mumbai problem in the most naive way you can think of, and notice what it costs you.

Show the answer

One naive fix is to cut words down to a common stem: treat any word starting with “deliver” as the same word, for example by keeping only the first six letters (deliver and delivery both become delive). That makes the Mumbai question score 1 against the delivery policy, so it is retrieved, and the model should reply that the policy covers same-day delivery in Pune and it does not know about Mumbai. But the six-letter trick also merges unrelated words (“presen” for present and presentation) and still cannot connect “money back” with “refund”. Every rule you add fixes one case and breaks another. That frustration is exactly why the next lesson replaces word matching with meaning matching.

Common mistakes

• Expecting a prompt instruction to supply missing facts. “Don’t make things up” cannot give the model knowledge it never had. Put the facts in the prompt.
• Forgetting to say what to do when the answer is missing. Without an explicit “I don’t know” sentence, the model fills the gap from general knowledge, which is the hallucination you were trying to remove.
• Building RAG for tiny documents. If everything fits comfortably in one prompt and rarely changes, paste it in. Add prompt caching if it is used often.
• Fine-tuning to add facts. Fine-tune for behaviour and style; retrieve for knowledge.
• Trusting a cited answer more than an uncited one, automatically. A citation only proves the model saw a passage, not that it read it correctly or that the right passage was found. Check citations in your evaluations.

Recap

That evening Meera asked Arjun to explain it again, “without the jargon, the way you’d explain it to the store managers”. He did.

• The model’s knowledge was frozen when it was trained, and it has never seen Crumb’s private documents.
• When it cannot know something, it guesses fluently. That guess is called a hallucination, and the eggless answer was one.
• RAG means: search our documents for the relevant passages, paste them into the question, and tell the model to answer only from them, or say it doesn’t know.
• Pasting the whole handbook in every time also works, but costs about a hundred times more per question and is slower.
• Fine-tuning changes how a model writes, not reliably what it knows.
• The weak spot moves to the search: if it finds the wrong passage, the answer will be confidently wrong.

Meera nodded. “So the important part is the librarian.”

“The librarian,” Arjun agreed. “And right now my librarian thinks ‘deliver’ and ‘delivery’ are unrelated words.”

Priya was already wiping the board. “Then tomorrow we build a proper one. An index for the whole handbook, and a search that understands meaning.”

Go deeper

• The original RAG paper (Lewis et al., 2020): https://arxiv.org/abs/2005.11401
• Anthropic documentation: https://docs.anthropic.com