Agentic RAG

The email arrived at 9:40 on a Saturday in December:

“Hi, my son has a tree-nut allergy. Is your Christmas plum cake safe for him? I want to order two from the Aundh store for his school party on Monday.”

Crumb’s allergy assistant answered in four seconds. It was the RAG system Arjun had built in Module 09: retrieval-augmented generation, where the system first searches Crumb’s documents for relevant passages, then gives those passages to the model to answer from. It searched the handbook and recipe archive, pulled back the four closest passages, and replied:

“Good news! Our classic Christmas plum cake is walnut-free and almond-free. It’s made with raisins, candied peel and spices. Enjoy the party!”

Sunita, who handled support on weekends, read the draft before sending, as she did with every allergy reply. Something nagged at her. She opened the 200-page store operations handbook and searched for “nuts”. Page 143: “All Christmas cakes at Aundh are baked on the same line as the almond fruitcake. Label as MAY CONTAIN NUTS.” She deleted the draft, wrote the reply herself, and sent Arjun a message he read on Monday morning with a cold feeling in his stomach.

Why the assistant missed page 143

Arjun pulled up the logs. The retrieval step had searched once, with the customer’s words, and returned these four passages, ranked by distance (how far apart two pieces of text are in meaning; lower means closer):

Rank	Passage ID	Distance	What it said
1	recipe-plum-2	0.31	Plum cake ingredients: raisins, candied peel, spices…
2	menu-xmas-4	0.38	“Our classic walnut-free plum cake, a family recipe…”
3	recipe-plum-1	0.44	Soak the fruit overnight in orange juice…
4	faq-allergy-1	0.52	“We take allergies seriously. Ask staff for details.”

Page 143, ops-143-2, was ranked eleventh. It never reached the model.

“Why?” Arjun asked Priya. “It’s literally about nuts and Christmas cakes.”

“Read it again. What words does it use?”

He read it aloud. “Same line… almond fruitcake… label as may contain nuts.”

“And the customer wrote?”

“Tree-nut allergy… safe… plum cake.” He saw it. “Different words. The question is about safety; the passage is about production lines. The search matched the recipe because the recipe mentions plum cake most strongly.”

“Right. Your pipeline searched once, took the top four, and answered. It’s like asking a librarian for a book, and she brings you four, and you’re not allowed to go back to the desk. However good the first search is, some questions need a second look.”

The librarian and the researcher

Priya drew two figures on the whiteboard. “Classic RAG is the librarian on a very strict schedule. You hand her a question. She makes one trip to the shelves, brings back four books, and leaves. You must answer from those four, whatever they say.”

“And the other one?”

“A careful researcher. Suppose Sunita were answering that email herself. Walk me through what she’d do.”

Arjun thought about it. “Check the recipe for nuts. It doesn’t have any. Then… she’d think, but is it made near nuts? So she’d look up how the Aundh kitchen works. And maybe check what the label says.”

“Three lookups. Each one decided by what the previous one found. And when would she stop?”

“When she’s confident. Or when she knows she can’t be sure, and says so.”

“That’s agentic RAG. Retrieval becomes a tool, the model becomes the researcher, and the agent loop you built in Your first agent lets it search, read, notice a gap, and search again with different words.”

Where the analogy breaks: a human researcher remembers which books she has already read. An agent calling a plain search tool does not get that for free: if it asks a similar question twice, it gets the same four passages twice and pays to read them twice. Fixing that is what makes the search tool stateful, which you will build in a moment.

Show the answer

Several would. Retrieve more passages (top 12 instead of top 4) and let a reranker (a model that re-scores passages against the question) pick the best. Or rewrite the query before searching (“plum cake nuts allergy cross-contamination shared equipment”), which you met as query rewriting in Module 09. Or, for allergy questions specifically, always retrieve the allergen policy section, hard-coded. These are cheaper and more predictable than an agent. Agentic RAG earns its cost when questions are varied enough that you can’t predict which extra lookups they will need. For allergies, as you’ll see, the best design uses both.

Following the email through an agentic RAG loop

Here is the same email, answered by an agent whose only tool is search_handbook(query, store). Every intermediate value is shown.

Step 1. First search. The model reads the email and searches with its own words:

search_handbook(query="Christmas plum cake ingredients nuts", store="Aundh")

The tool returns four new passages: recipe-plum-2 (ingredients, no nuts listed), menu-xmas-4 (“walnut-free”), recipe-plum-1 (method), and ops-140-1 (Aundh Christmas production schedule: “Plum cakes, almond fruitcake and stollen baked Mon–Wed on Line 2”).

Step 2. The model notices a gap. Its reasoning (abridged): “The recipe has no nuts, but passage ops-140-1 says plum cake and almond fruitcake share Line 2. I need the cross-contamination policy before answering.” It searches again with different words:

search_handbook(query="shared production line cross-contamination nut label", store="Aundh")

The tool skips the passages already shown and returns three new ones, including:

[ops-143-2] (distance 0.29)
All Christmas cakes at Aundh are baked on the same line as the almond fruitcake.
Label as MAY CONTAIN NUTS. Staff must not describe these products as nut-free.

Step 3. One more check. The model wonders whether another store bakes plum cake separately, since the customer might be flexible:

search_handbook(query="nut-free Christmas cake production dedicated line")

Result: ops-151-1: “No Crumb store currently has a dedicated nut-free line. For severe allergies, recommend our sealed, certified nut-free brownie box (supplier-made).”

Step 4. Answer. No more tool calls:

“Thank you for checking with us. I’m sorry: our Christmas plum cake doesn’t contain nuts as an ingredient, but at Aundh it’s baked on the same line as our almond fruitcake, so it’s labelled ‘may contain nuts’ [ops-143-2]. None of our stores currently has a dedicated nut-free line [ops-151-1]. For a tree-nut allergy, our sealed, certified nut-free brownie box is the safe option [ops-151-1]. Would you like two of those for Monday?”

The cost: four model calls, three searches, about 11,000 input tokens and 900 output tokens. At $4and$20 per million, roughly $0.06, or ₹5, and about fifteen seconds. The classic pipeline cost about ₹1 and took four seconds. It was also wrong in a way that could have put a child in hospital.

Opening the hood: from pipeline to loop

Classic RAG is a fixed workflow (rung 1 on the ladder from What is an AI agent?): retrieve, then generate, always once.

flowchart LR
  Q["Question"] --> R["Retrieve top 4"] --> G["Generate answer"] --> A["Answer"]

Agentic RAG puts retrieval inside the agent loop:

flowchart TD
  Q["Question"] --> T["Model thinks: what do I still need?"]
  T -->|"search(query, filters)"| S["Search tool"]
  S -->|"new passages with IDs"| T
  T -->|"enough evidence"| A["Answer with citations"]
  T -->|"budget used or nothing found"| D["Say what is unknown"]

The model gains four abilities a fixed pipeline lacks:

1. Query rewriting, on the fly. It chooses its own search words, and changes them when results disappoint. Step 2’s “shared production line cross-contamination” was a query no customer would write.
2. Multi-hop retrieval. It follows a trail: the schedule mentioned Line 2, so it searched for Line 2’s policy. Each hop depends on the last.
3. Filtering. It can pass store="Aundh" when the question is store-specific, and drop it when it isn’t.
4. A sufficiency judgement. It decides when it has enough evidence, or when to say “I don’t know”. This is the most powerful ability and, as you’ll see, the least reliable.

Why the search tool should remember things

A plain search function is stateless: same query in, same passages out. Agents rephrase themselves a lot, and similar queries return largely the same passages. Without memory, step 2 would have returned recipe-plum-2 again, and the model would have paid to read it again, and might have concluded there was nothing new to find.

A stateful search tool is an object that keeps track of its own history during one conversation:

• Seen passages. It skips anything already shown and returns only new passages, so every search adds information.
• A query log. It records each query, which is gold for debugging (“why didn’t it search for the label policy?”) and for evals later.
• A budget. It counts searches and refuses politely after, say, six, telling the model to answer with what it has.

You create a fresh tool object for each customer conversation, so state never leaks from one customer to another.

Building it

Step 1: index the handbook in Chroma

You built a vector store in Module 08. Here is a tiny one: a handful of passages with IDs and a store label, embedded automatically by Chroma’s default embedding model. (An embedding is a list of numbers that captures what a piece of text means; similar meanings get similar numbers.)

uv add chromadb "smolagents[litellm]" python-dotenv

# index_handbook.py
import chromadb

PASSAGES = [
    ("recipe-plum-1", "all",   "Plum cake method: soak the fruit overnight in orange juice, then cream butter and sugar."),
    ("recipe-plum-2", "all",   "Plum cake ingredients: raisins, candied peel, mixed spice, flour, butter, sugar, eggs."),
    ("menu-xmas-4",   "all",   "Our classic walnut-free plum cake, a family recipe since 1998."),
    ("ops-140-1",     "Aundh", "Aundh Christmas production: plum cakes, almond fruitcake and stollen baked Mon-Wed on Line 2."),
    ("ops-143-2",     "Aundh", "All Christmas cakes at Aundh are baked on the same line as the almond fruitcake. "
                               "Label as MAY CONTAIN NUTS. Staff must not describe these products as nut-free."),
    ("ops-151-1",     "all",   "No Crumb store currently has a dedicated nut-free line. For severe allergies, "
                               "recommend the sealed, certified nut-free brownie box (supplier-made)."),
    ("faq-allergy-1", "all",   "We take allergies seriously. Ask staff for ingredient details."),
    ("ops-022-3",     "all",   "Delivery partners must keep cakes upright and below 25 degrees C."),
]

client = chromadb.PersistentClient(path="./db")
handbook = client.get_or_create_collection("handbook")
handbook.add(
    ids=[p[0] for p in PASSAGES],
    metadatas=[{"store": p[1]} for p in PASSAGES],
    documents=[p[2] for p in PASSAGES],
)
print(handbook.count(), "passages indexed")

Run it once. You should see 8 passages indexed. In real life this is your full chunked handbook, thousands of passages, built with the chunking pipeline from Module 09.

Step 2: the stateful search tool

# handbook_tool.py
from smolagents import Tool

class HandbookSearch(Tool):
    name = "search_handbook"
    description = (
        "Search Crumb's operations handbook and recipe archive. Returns up to 4 NEW passages, "
        "each with an ID to cite. Passages already shown are skipped, so search again with "
        "different words to find different passages."
    )
    inputs = {
        "query": {"type": "string", "description": "What to look for, in plain words."},
        "store": {"type": "string", "nullable": True,
                  "description": "Optional store name, e.g. 'Aundh', to include that store's rules."},
    }
    output_type = "string"

    def __init__(self, collection, max_searches: int = 6):
        super().__init__()
        self.collection = collection
        self.max_searches = max_searches
        self.seen: set[str] = set()        # state 1: passages already shown
        self.queries: list[str] = []       # state 2: what was searched, in order

    def forward(self, query: str, store: str | None = None) -> str:
        if len(self.queries) >= self.max_searches:                 # state 3: a budget
            return "Search budget used up. Answer from what you have, or say you don't know."
        self.queries.append(query)

        where = {"store": {"$in": [store, "all"]}} if store else None
        res = self.collection.query(query_texts=[query], n_results=10, where=where)

        fresh, skipped = [], 0
        for pid, text, dist in zip(res["ids"][0], res["documents"][0], res["distances"][0]):
            if pid in self.seen:
                skipped += 1
                continue
            self.seen.add(pid)
            fresh.append(f"[{pid}] (distance {dist:.2f})\n{text}")
            if len(fresh) == 4:
                break

        if not fresh:
            return "No new passages for that query. Try different words, or conclude."
        note = f"\n({skipped} already-seen passages skipped)" if skipped else ""
        return "\n\n".join(fresh) + note

What each part adds:

• self.seen makes every search return only new information.
• self.queries gives you a readable trail after every run.
• max_searches turns an open-ended loop into a bounded one.
• where with $in filters by store while still including passages that apply to all stores.
• "nullable": True tells smolagents that store is optional, matching the None default in forward.
• IDs in brackets let the model cite exactly which passage supports each claim, so Sunita can check them in seconds.

Step 3: the agent

# allergy_agent.py
import chromadb
from dotenv import load_dotenv
from smolagents import ToolCallingAgent, LiteLLMModel
from handbook_tool import HandbookSearch

load_dotenv()
handbook = chromadb.PersistentClient(path="./db").get_collection("handbook")
model = LiteLLMModel(model_id="anthropic/claude-opus-5-5")

INSTRUCTIONS = (
    "You answer allergy questions for Crumb & Co. Search the handbook as many times as you need. "
    "Before saying any product is safe for an allergy, you MUST find and cite the cross-contamination "
    "policy for that store. Cite passage IDs in square brackets. If the evidence is unclear, say so "
    "and suggest calling the store."
)

def answer(email: str) -> str:
    tool = HandbookSearch(handbook)               # fresh state for every conversation
    agent = ToolCallingAgent(tools=[tool], model=model, max_steps=8)
    reply = agent.run(f"{INSTRUCTIONS}\n\nCustomer email:\n{email}")
    print("Queries used:", tool.queries)          # the trail, for logs and evals
    return reply

print(answer("My son has a tree-nut allergy. Is your Christmas plum cake from Aundh safe for him?"))

Run it. You should see two or three searches in the log, a Queries used: line that reads like a researcher’s notes, and a reply that cites [ops-143-2] and says the cake may contain nuts. The exact wording will vary from run to run.

Library APIs change quickly. If a name has moved, check the docs linked below.

Show the answer

The state would leak between customers. After the first allergy email, ops-143-2 would be in seen, so the next customer asking about nuts would never be shown it: the tool would skip it as “already seen”. The search budget would also run out after a few customers. Stateful tools need a clear lifetime, usually one conversation. This is a classic bug: state that is helpful inside one task is harmful across tasks.

Where it breaks

Agentic RAG fixed the plum cake. Two weeks of testing on 150 past allergy emails taught Priya and Arjun where it still breaks.

It sometimes stops too early. On 9 of 150 emails, the model found an ingredients list with no nuts and answered at once, without looking for the cross-contamination policy, despite the instruction. Models are often confident after a single good-looking result. The sufficiency judgement is the weakest link.

It loops on rephrasings. Without the seen-set, a test run made nine near-identical searches. With it, the tool’s “no new passages” message ended the loop after two.

It costs more. On average 3.2 searches and 4.4 model calls per email: about ₹5 instead of ₹1, and fifteen seconds instead of four. For allergy questions, an easy trade. For “what time does Baner open?”, a waste: route those to classic RAG.

Evaluation gets harder. Two runs on the same email can take different search paths. Score the outcomes: did the reply cite the cross-contamination passage? Did it say “may contain nuts”? How often did it say “I don’t know” when it should have? You met the RAGAS metrics and the “I don’t know” rate in Module 09, and you’ll build proper evals in Module 23. The query log from the stateful tool makes this much easier.

Try it yourself

1. Run allergy_agent.py and copy the Queries used: line.
2. Remove the cross-contamination sentence from INSTRUCTIONS and run five times. How many replies cite ops-143-2?
3. Add a second tool, read_passage(passage_id: str) -> str, that returns the passages immediately before and after a given ID (for example ops-143-1 and ops-143-3). When would the agent use it?

Show the expected result

1. Typically two or three queries, the second mentioning lines, equipment or cross-contamination.
2. Expect some runs to stop after the first search and call the cake nut-free. This is exactly the early-stopping failure, and why the mandatory policy lookup belongs in code.
3. A neighbouring-passage tool helps when a passage is cut mid-thought by chunking (“…see the exceptions below”). The agent reads the neighbours instead of hoping a search finds them. It’s the same idea as the “expand context” step from Module 09, made available as a tool the model calls when it needs it.

Common mistakes

• Using agentic RAG for every question. Simple lookups (“opening hours”) don’t need a researcher. Route them to the cheap pipeline.
• Search tools without memory. Agents rephrase constantly. Skip passages already shown, or you pay repeatedly for the same text.
• Sharing tool state across conversations. Create the stateful tool per conversation.
• Trusting the model to look for the critical thing. Hard-code mandatory retrievals for safety and compliance.
• No IDs, no citations. Without passage IDs, a human reviewer cannot check the answer quickly, and you cannot score retrieval in evals.

Recap

• Classic RAG retrieves once and answers. Agentic RAG gives the model a search tool and lets it search, read, notice gaps and search again.
• The agent gains on-the-fly query rewriting, multi-hop lookups, filters, and a sufficiency judgement (its most useful and least reliable ability).
• A stateful search tool remembers seen passages, logs queries and enforces a budget. Create one per conversation.
• It costs more and runs slower, so use it where questions vary and mistakes matter. Keep cheap questions on the cheap path.
• For safety-critical facts, combine both: mandatory retrieval in code, open-ended searching by the agent, and a human reviewing the result.

Arjun explained the fix to Meera, with Sunita listening. “The old system searched once and answered from whatever came back. Page 143 used different words, so it never showed up. The new one searches like Sunita does: recipe first, then how it’s made, then the label policy. And for allergies, the label policy is always fetched, whether the AI thinks to ask for it or not.”

“So Sunita still checks?”

“Every allergy reply,” said Priya. “But now she’s checking an answer with sources attached, not catching a dangerous one from scratch.”

Meera nodded. Then she brought up the thing everyone had been avoiding. “Christmas hampers. Twelve new ones. Menu, stock in six stores, pricing, and the descriptions for the website. Last year that took four people two weeks. Can a pile of these agents do it?”

Priya smiled. “Before you build a pile of agents, you need to know the five shapes they can be arranged in. Most of the time, you won’t need a pile at all.”

Go deeper

• smolagents documentation, including its agentic RAG example.
• Chroma documentation, for metadata filters like $in.
• Building effective agents (Anthropic).
• The original RAG paper, for where retrieve-then-generate began.