The RAG pipeline: indexing and retrieval

On Wednesday morning the Koregaon Park store manager, Farida, messaged the support channel with a photo of a chocolate truffle cake sitting under the display lights. “New staff member asking how long cream cakes can stay out here. Bot says ‘clean the counter every 2 hours’. That’s not what she asked.”

Arjun had pointed his tiny keyword RAG at the whole 200-page store operations handbook overnight. He had split it into pages, kept them in a Python dictionary, and let the word-overlap search pick the top two. For Farida’s question, “How long can a cream cake stay on the display counter?”, the winning page was the one about counter cleaning schedules. It contained the word “counter” four times and “display” twice. The page with the actual rule, page 87, said: “Fresh-cream products: maximum 4 hours outside refrigeration.” It shared almost no words with the question. Not “cream cake”, not “display”, not “counter”. “Cream” was hiding inside “fresh-cream”, and his word splitter had broken it into “fresh” and “cream”, so even that matched only weakly.

Meera saw the thread and walked over. “Is this a small bug or a big one?”

“Big,” Priya said. “The search is matching spelling, not meaning. And it’s searching whole pages, so even when it finds the right page, the model gets a wall of text about six different topics.” She split the whiteboard with a vertical line. On the left she wrote Once, ahead of time. On the right, Every question.

“Every RAG system in the world has these two halves,” she said. “Get the left half right, and the right half becomes easy.”

Why searching pages by their words failed

“Two separate problems,” Priya said. “Name them.”

Arjun looked at the board. “The search matched ‘counter’ instead of the meaning. And… the pages are too big?”

“Why does page size matter?”

“Because page 87 talks about cream products, chocolate tempering, display lighting and the cake-cutting procedure. Even if the search picked it, the model would have to dig through all that.”

“And the search itself would have a harder time,” Priya added. “A page about six things is a little bit about everything. We want each searchable piece to be about one thing.” She drew a library card catalogue. “Imagine a big public library. Before any visitor arrives, the librarians do the slow work. They receive books, break them into sections, write an index card for each section that describes what it is about, and file the cards in a cabinet where cards about similar topics sit next to each other. That’s the left side of the board.”

“And the right side?”

“A visitor asks a question. The librarian walks to the part of the cabinet where cards like that question live, pulls the closest few, fetches those sections, and hands them over. The visitor reads them and writes the answer.”

“So the visitor is the LLM.”

“The visitor is the LLM. The librarian is your code. And the cabinet,” she drew a box, “is a vector database.”

Where the analogy breaks: real index cards are written by people who understand the book. Ours are written by an embedding model, which is fast, tireless and consistent but has no idea what matters to Crumb. It will happily file “4 hours outside refrigeration” right next to “chiller temperature log”, because both are about keeping things cold, even though only one answers Farida’s question.

The left half: indexing, once

flowchart LR
  A["Handbook PDF"] --> B["Load: extract text"]
  B --> C["Clean: drop headers and footers"]
  C --> D["Chunk: split into passages"]
  D --> E["Embed: text to vectors"]
  E --> F[("Vector database")]

Each box is a step you will see in every RAG system, under different names.

Load. Get raw text out of your files: PDFs, Word documents, web pages, spreadsheets, a database. This is messier than it sounds, and the next lesson is about it.

Clean. Remove junk that repeats on every page, like “Crumb & Co. Confidential, v7, Page 88 of 200”. Text that appears everywhere makes every passage look alike to search.

Chunk. Chunking means splitting long text into short passages, usually a few hundred words each, called chunks. You search over chunks, not documents, so that each searchable piece is about one thing and you can hand the model the paragraph about cream cakes, not the whole chapter.

Embed. An embedding is a list of numbers that captures what a piece of text means. You met them in Module 07: an embedding model reads text and outputs a fixed-length list of numbers (a vector), placing texts with similar meanings at nearby points on a map with hundreds of dimensions. “Cream cake on the display counter” and “fresh-cream products outside refrigeration” land close together even though they share almost no words.

Store. A vector database (Module 08) is a database built to answer one question quickly: “which stored vectors are closest to this one?”. For each chunk you save three things: the vector, the original text, and metadata (labels like the source file, page number and chapter). The text is what you will paste into the prompt; the metadata is what lets you cite, filter and debug.

You run indexing when your documents change, not when users ask questions. For Crumb, that means whenever operations publishes a new handbook version, plus a nightly job to catch smaller edits.

How expensive is it? The handbook becomes roughly 640 chunks of about 200 tokens each, about 128,000 tokens in total. Embedding that with a small free model on a laptop takes well under a minute. With a paid embeddings API like OpenAI’s text-embedding-3-small, priced at the time of writing around $0.02 per million tokens, it costs about a quarter of a US cent, well under one rupee, for the whole handbook. Indexing is cheap. It is slow only compared with a single search.

The right half: retrieval and generation, every question

flowchart LR
  Q["User question"] --> E["Embed the question"]
  E --> S["Search: nearest k chunks"]
  DB[("Vector database")] --> S
  S --> P["Build prompt: rules, chunks, question"]
  P --> L["LLM generates"]
  L --> A["Answer with citations"]

Embed the question with the same embedding model you used for the chunks.

Search the vector database for the chunks whose vectors are closest to the question’s vector. Closeness is usually measured with cosine similarity, a score of how much two vectors point in the same direction: 1 means the same direction, 0 means unrelated. You ask for the top-k results, where k is how many chunks you want back, often between 3 and 8.

Build the prompt. Combine your instructions, the retrieved chunks (numbered, with their sources) and the question. This is the “augmented” in Retrieval-Augmented Generation.

Generate. The LLM reads the chunks and writes an answer that cites them.

The part of your code that takes a question and returns relevant chunks has a name you will hear constantly: a retriever. Everything after the retriever is ordinary prompting, which you already know from Module 03.

Following Farida’s question all the way through

Priya made Arjun build a four-chunk version of the handbook and trace Farida’s question through every step, printing the real numbers. Here is the trace.

The four chunks in the index:

ID	Text (shortened)	Page
ops-087-1	Fresh-cream products: maximum 4 hours outside refrigeration. After 4 hours, move to the staff discount tray or discard.	87
ops-087-2	Display counters are wiped with food-safe sanitiser every 2 hours and at closing.	87
ops-112-1	Walk-in chiller must stay between 1 and 4 degrees Celsius. Log the temperature at opening and closing.	112
ops-140-3	Allergy complaints go to the store manager within 30 minutes and are logged in the incident register.	140

Step 1: embed the question. “How long can a cream cake stay on the display counter?” goes into all-MiniLM-L6-v2 and comes out as 384 numbers. The first five are 0.043, 0.025, -0.015, 0.029, -0.024. None of them means anything on its own; meaning is spread across all 384.

Step 2: compare with every chunk. Cosine similarity between the question vector and each chunk vector:

Chunk	Cosine similarity	Rank
ops-087-1 (cream, 4 hours)	0.553	1
ops-087-2 (counter cleaning)	0.456	2
ops-140-3 (allergy complaints)	0.199	3
ops-112-1 (chiller temperature)	0.162	4

Look at what happened. The right chunk won, even though it shares almost no words with the question. The model “knows” that cream cakes are fresh-cream products and that sitting on a display counter means being outside refrigeration. But notice the runner-up: the counter-cleaning chunk scored 0.456, not far behind, because it is genuinely about display counters. Meaning search is better than word search, not magic.

Step 3: take the top k. With k = 2, the retriever returns ops-087-1 and ops-087-2.

Step 4: build the prompt. The code numbers the chunks and adds their page numbers:

Context:
[1] (page 87) Fresh-cream products: maximum 4 hours outside refrigeration. After 4 hours, move to the staff discount tray or discard.

[2] (page 87) Display counters are wiped with food-safe sanitiser every 2 hours and at closing.

Question: How long can a cream cake stay on the display counter?

With the system prompt, that is about 120 tokens of input.

Step 5: generate. Claude reads both chunks, recognises that [1] answers the question and [2] is related but off-topic, and replies:

“A cream cake can stay on the display counter for a maximum of 4 hours outside refrigeration. After that, move it to the staff discount tray or discard it [1].”

Step 6: what it cost. Embedding the question with a local model: free, about 10 milliseconds. The search over four chunks: under a millisecond (over 640 chunks, still a few milliseconds). The LLM call: about 120 input tokens and 40 output tokens, a fraction of a rupee, and one to three seconds of waiting. Almost all of the time and money in a RAG answer is the final LLM call. Remember that; it matters a lot in Module 10, when hundreds of people ask at once.

“Six steps,” Arjun said. “And only the last one is the AI part everyone talks about.”

“The last one is the easy part,” Priya said. “Steps 1 to 4 decide whether it has any chance of being right.”

Building it, one runnable step at a time

The whole pipeline fits in one short file, with no framework. This uses Chroma, a vector database that runs on your laptop and ships with a small default embedding model (a version of the same all-MiniLM-L6-v2), so you do not need an embeddings API key.

Step 1: index the chunks.

# uv add chromadb anthropic python-dotenv
import chromadb

chunks = [
    ("ops-087-1", "Fresh-cream products: maximum 4 hours outside refrigeration. "
                  "After 4 hours, move to the staff discount tray or discard.", 87),
    ("ops-087-2", "Display counters are wiped with food-safe sanitiser every "
                  "2 hours and at closing.", 87),
    ("ops-112-1", "Walk-in chiller must stay between 1 and 4 degrees Celsius. "
                  "Log the temperature at opening and closing.", 112),
    ("ops-140-3", "Allergy complaints go to the store manager within 30 minutes "
                  "and are logged in the incident register.", 140),
]

db = chromadb.PersistentClient(path="./crumb_db")        # saves to a folder on disk
collection = db.get_or_create_collection("handbook")
collection.add(
    ids=[c[0] for c in chunks],                           # unique ID per chunk
    documents=[c[1] for c in chunks],                     # Chroma embeds these for you
    metadatas=[{"source": "ops-handbook", "page": c[2]} for c in chunks],
)
print(collection.count())

You should see 4. The first run also downloads the small embedding model, which takes a few seconds. PersistentClient writes everything to the crumb_db folder, so the index survives after the script ends. That is the left half of the board, done.

Step 2: retrieve, and look at what came back.

question = "How long can a cream cake stay on the display counter?"
hits = collection.query(query_texts=[question], n_results=2)

for doc, meta, dist in zip(hits["documents"][0], hits["metadatas"][0], hits["distances"][0]):
    print(f"page {meta['page']}  distance {dist:.3f}  {doc[:60]}")

You should see something like:

page 87  distance 0.894  Fresh-cream products: maximum 4 hours outside refrigerati
page 87  distance 1.088  Display counters are wiped with food-safe sanitiser every

Two things to notice. Chroma returns results as lists inside lists ([0]) because you can send several questions at once and it returns one list per question. And it reports distance, not similarity: lower is closer. By default it uses squared straight-line distance, which for these normalised vectors works out as 2 minus 2 times the cosine similarity. That is why 0.553 similarity shows up as about 0.894 distance. Mixing up “higher is better” and “lower is better” is a classic bug when you later add a relevance threshold.

Step 3: build the prompt.

def build_context(hits) -> str:
    lines = []
    for i, (doc, meta) in enumerate(zip(hits["documents"][0], hits["metadatas"][0])):
        lines.append(f"[{i + 1}] (page {meta['page']}) {doc}")
    return "\n\n".join(lines)

print(build_context(hits))

This prints exactly the context block from the trace above. Numbering the chunks lets the model cite [1] instead of quoting whole passages, and keeping the page lets a human check the source.

Step 4: generate.

import anthropic
from dotenv import load_dotenv

load_dotenv()                     # ANTHROPIC_API_KEY
client = anthropic.Anthropic()

SYSTEM = ("You answer questions for Crumb & Co. store staff. Use only the numbered "
          "context. Cite sources like [1]. If the context does not contain the "
          "answer, say: I don't know based on the handbook.")

def answer(question: str, k: int = 2) -> str:
    hits = collection.query(query_texts=[question], n_results=k)
    response = client.messages.create(
        model="claude-opus-5-5",
        max_tokens=500,
        system=SYSTEM,
        messages=[{"role": "user",
                   "content": f"Context:\n{build_context(hits)}\n\nQuestion: {question}"}],
    )
    return next(b.text for b in response.content if b.type == "text")

print(answer("How long can a cream cake stay on the display counter?"))

Expected output, roughly: “Up to 4 hours outside refrigeration; after that, move it to the staff discount tray or discard it [1].”

In a real project, steps 1 and 4 live in separate scripts: an index.py you run when documents change, and an app.py that answers questions. If you run step 1 twice, Chroma will warn that those IDs already exist and skip them. That is the left half reminding you it only needs to run once.

Where it breaks: the day the rule changed

Two weeks later, the food safety consultant tightened the rule. Page 87 now said: “Fresh-cream products: maximum 3 hours outside refrigeration.” Operations updated the PDF. Arjun re-ran his indexing script with a new chunk ID, ops-087-1-v8.

The next morning, Farida’s new staff member asked the same question and got: “The handbook gives two different limits: 4 hours [1] and 3 hours [2]. Please check with your manager.”

“At least it noticed,” Arjun said.

“It noticed because both versions were retrieved,” Priya said. “Next time the old one might rank higher on its own, and it’ll say 4 hours with a citation. Your index is a copy of the documents. When a document changes, the copy doesn’t update itself.”

Stale chunks are one failure. Priya made Arjun write the full list on a sticky note and put it on his monitor. When an answer is wrong, walk the pipeline from left to right and find the first step that failed. Do not change anything until you know which one it was.

Step	Typical failure	Symptom you see
Load	Tables become word soup; scanned pages have no text	A fact you know exists is never retrieved
Clean	Repeated headers and footers left in	Many unrelated chunks look alike to search
Chunk	A rule split across two chunks; chunks too big or too small	Half-answers, or the right page with the wrong detail
Embed	Model weak on your jargon (“SOP-14”, “mawa”)	Codes and product names not found
Store	Old versions never deleted	Two conflicting answers, or an outdated one
Search	k too small; similar-but-wrong chunk wins	Confident answer about a neighbouring topic
Generate	Model ignores the context or goes beyond it	Answer not supported by its own citation

Experienced engineers have one habit above all others here: print the retrieved chunks before you read the answer. Most “the model is wrong” bugs turn out to be “retrieval handed it the wrong text” bugs. The model was faithfully summarising the wrong page.

Choosing k, honestly

Arjun’s last question was about the number 2. “Why k = 2? Why not 10, to be safe?”

“What happens if k is too small?”

“We might miss the chunk with the answer.”

“And too big?”

He thought. “More tokens, so more cost. And… the model has to find the right chunk among more wrong ones?”

“Exactly. Ten chunks of 200 tokens is 2,000 extra tokens per question, which is still cheap. The bigger cost is distraction: the more near-miss chunks you hand over, the more likely the model blends one into the answer. Start around 4, then let your evaluation decide.” In the hybrid search lesson you will meet a better answer: fetch many candidates cheaply, then use a sharper model to pick the best few.

Try it yourself

Add these two chunks to the collection with new IDs and page numbers of your choice:

• “Diwali hampers must be packed no earlier than 48 hours before dispatch.”
• “Staff may buy discount-tray items at 50% off at the end of their shift.”

Then ask, printing the retrieved chunks and distances before each answer: “Can I pack hampers three days early?” and “What happens to cream cakes after four hours?”

Show the answer

The first question should retrieve the 48-hour packing chunk as the clear winner, and the answer should be “No, at most 48 hours before dispatch”, with a citation. The second should retrieve the 4-hour cream rule first and, quite possibly, the staff discount chunk second, because both mention the discount tray. That is fine: retrieval returning a related extra chunk is normal, and the model should use only what answers the question. If the model instead says “staff can buy them at 50% off”, it has blended a near-miss chunk into the answer, which is exactly the distraction problem from the section on choosing k.

Common mistakes

• Embedding questions and chunks with different models. The vectors are not comparable, and search fails silently.
• Re-running indexing on every question. Indexing is the slow, once-per-change half. Keep it in its own script.
• Adding new versions without deleting old ones. Use stable chunk IDs and overwrite, or delete by document before re-adding.
• Treating distance as similarity. Chroma’s distance is lower-is-better. Check which one your database returns before writing any threshold.
• Throwing away metadata. Without page and source you cannot cite, filter or debug.
• Only reading the final answer. Print the retrieved chunks first, every time.

Recap

That afternoon, Meera asked for the two-minute version for the store managers’ WhatsApp group. Arjun wrote it on the whiteboard:

• Once, ahead of time: we split the handbook into short passages, turn each into a “meaning fingerprint” (an embedding), and file them in a special database.
• Every question: we fingerprint the question the same way, pull out the closest few passages, and give them to the AI with the instruction “answer only from these, and cite them”.
• The search works on meaning, so “cream cake on the counter” finds “fresh-cream products outside refrigeration”.
• When the handbook changes, we re-file the changed pages and delete the old ones. Nothing needs retraining.
• When an answer is wrong, we check which step failed, starting with what the search found.

Meera read it and added one line of her own underneath: “Wrong passage in, wrong answer out.” Priya said it was the best summary of RAG she had ever seen.

Then Arjun opened the real handbook PDF. Two hundred pages, tables of oven temperatures, a scanned fire-safety certificate on page 3, and a footer on every page. “I’m not typing 640 chunks into a Python list,” he said.

“Good,” said Priya. “Let’s get a loader and a splitter to do it for you.”

Go deeper

• Chroma documentation: https://docs.trychroma.com
• The original RAG paper: https://arxiv.org/abs/2005.11401