Reading traces by hand and bucketing failures

On Monday morning Meera forwarded a WhatsApp screenshot to the engineering channel with no comment. A customer had asked CrumbBot, Crumb’s support assistant, whether the walnut brownie was safe for her nut-allergic son. CrumbBot replied that “all our brownies are baked fresh every morning” and wished her a lovely evening. It never answered the question.

CrumbBot handles about 600 conversations a day across the website chat and the WhatsApp line. Arjun had built it in Module 13, wired it to the order database, and tuned its prompt eleven times in the last month. Each tweak fixed the thing someone had complained about. Each week, someone complained about something else.

Meera’s follow-up message came ten minutes later: “Is the bot safe to leave running? Yes or no. And how would we even know?”

Arjun’s first instinct: patch the prompt

Arjun already had the system prompt open. He added a line in capitals:

ALWAYS answer allergy questions carefully and accurately.

Then he tested it with five questions he typed himself: “Do brownies have nuts?”, “Is the plum cake eggless?”, and three more like them. All five answers looked fine. He was about to deploy when he had a second idea. Why not get an AI to find the problems for him? He pasted twenty conversations into a chat with Claude and asked: “What are the common problems with this support bot?”

The answer came back in seconds: “1. Occasional hallucinations. 2. Inconsistent tone. 3. Could be more concise. 4. May not always handle edge cases.”

Arjun read it twice. Every point was true of every chatbot ever built. None of them told him what to change.

Priya had been watching over his shoulder. “Two questions. How many real conversations have you read this week, start to finish, every tool call?”

“Maybe ten? I mostly test with my own questions.”

“And the five you just tested with. Who wrote them?”

“Me.”

“So you’re testing the bot against the customers you imagine, and asking a model to list problems it guessed. Neither of you has looked.”

The critic who never ate at the restaurant

Priya pulled a chair over. “Say Meera noticed that sales at the Kothrud store dropped 20% last month. What would she do first? Build a sales dashboard?”

“No. She’d go and stand in the store for a day.”

“Why?”

Arjun thought. “Because she doesn’t know what’s wrong yet. Maybe the queue is long. Maybe the display fridge is broken. A dashboard only shows what you decided to put on it.”

“Exactly. You can’t measure a problem until you know what the problems are. Standing in the store is how you find out. For an AI app, standing in the store means reading traces.”

A trace is the full record of one request through your AI app: what the user typed, the system prompt, any documents retrieved, every tool the model called and what came back, and the final reply. You set up tracing in Module 22. If a log line is a till receipt, a trace is the CCTV footage of the whole transaction.

An eval (short for evaluation: a repeatable check of how well your AI app does its job) only measures what you tell it to measure. Pick the wrong thing and you get a beautiful chart about a problem nobody has. So the first step is not measuring at all. It has a name: error analysis, a structured way of reading your app’s outputs to discover how it actually fails.

Error analysis is the first stage of a loop you will run for the life of the product:

flowchart LR
  A["Analyse: read traces, find failure modes"] --> M["Measure: build evals for the big ones"]
  M --> I["Improve: change prompt, retrieval, tools"]
  I --> A

You analyse to find out what goes wrong, measure how often each problem happens, and improve the system. Then you analyse again, because every fix changes the behaviour and can create new failures. The rest of this module walks around that loop.

“Fine,” Arjun said. “How many do I read?”

“A hundred. Random ones. And Meera reads some too.”

Following one conversation through its trace

Priya opened Langfuse and picked the walnut brownie conversation, trace 014. This is what it contained, step by step:

Step	What happened	Detail
1. Input	Customer message	“hi is the walnut brownie ok for my son, he has a nut allergy”
2. Prompt	System prompt loaded	2,140 tokens: tone rules, order tool instructions, refund policy, nothing on allergies
3. Retrieval	Menu search for “walnut brownie nut allergy”	Top 3 chunks: “Brownies, baked fresh every morning” (similarity 0.82), “Our story: family recipes since 1998” (0.61), “Gift boxes and hampers” (0.58)
4. Tools	None called
5. Output	Reply	“Hi! All our brownies are baked fresh every morning using our family recipe. Have a lovely evening!”
6. Cost	One model call	2,610 tokens in, 31 out, about ₹0.35 and 1.8 seconds

“So what went wrong?” Priya asked.

“The reply ignored the allergy,” Arjun said.

“That’s the symptom. Go further up. What did the model have to work with?”

Arjun looked at step 3. None of the three retrieved chunks said anything about ingredients. He searched the menu index for “allergen”. Zero results. The allergen sheet, a PDF the kitchen updated monthly, had never been loaded into the search index.

“It couldn’t have answered even if it wanted to,” he said slowly. “And the prompt doesn’t tell it what to do when it doesn’t know. So it filled the gap with something friendly.”

“Now write the note.”

Arjun typed: Allergy question. Retrieval returned only marketing copy; allergen sheet not in index. No instruction for ‘I don’t know’. Reply dodged.

That note took him 90 seconds and pointed at two concrete fixes: index the allergen sheet, and tell the bot what to say when it lacks ingredient data. His capital-letters prompt line would have fixed neither. With no ingredient data in context, “answer carefully and accurately” leaves the model with nothing accurate to say.

Open coding: writing notes, not scores

What Arjun just did has a name borrowed from qualitative research, the field that studies interviews and field notes. Open coding means reading each trace and writing a short, free-form note about what went wrong, without any categories decided in advance. It is the restaurant critic’s first visit: no checklist, only scribbles like “soup lukewarm”, “waiter forgot the bread”. The checklist comes later, from the scribbles.

A few rules turn the scribbles into something useful:

• Note the first failure, the one furthest upstream. Errors cascade. If the bot looked up the wrong order and then gave a wrong delivery time, the wrong lookup is the root. Fix it and the downstream error often disappears.
• Be specific and observable. “Bad answer” teaches nothing. “Promised full refund; policy says store credit for damaged items” points at a fix.
• Write “fine” when it is fine. You need the denominator, the good traces as well as the bad.
• Let the person who knows the business judge. Arjun could not tell whether a refund promise matched Crumb’s policy. Meera could. Practitioners call this person the domain expert, and the analysis is only as good as their judgement. One trusted expert making the calls beats a committee.

Here is a slice of the spreadsheet after the first hour:

Trace	What the customer wanted	Note
014	Is walnut brownie safe for nut allergy?	Allergen sheet not indexed; reply dodged
022	Where is order 3381?	Called order tool with phone number instead of order ID
031	Refund for squashed cake	Promised full refund; policy says store credit for damage
037	Diwali hamper prices	Fine
045	Eggless options at Kothrud store	Listed items only sold at Baner
052	Order 3402 late	Said “out for delivery”; tool returned “cancelled”

How many traces is enough?

Reading 100 traces at around 90 seconds each takes about two and a half hours. That feels like a lot until you compare it with a month of blind prompt tweaks.

Start with 50 to 100, chosen like this:

• Mostly random. About 80 sampled at random from the last week, so you see normal traffic, not only the dramatic cases.
• Some on purpose. About 20 picked from places failures hide: thumbs-down ratings, conversations longer than ten turns, conversations where a human agent took over.

Then keep going until about twenty traces in a row teach you nothing new. Researchers call this point saturation: new data stops producing new kinds of note. For CrumbBot it came at trace 85. A bot with more varied traffic, like one handling both wholesale and retail orders, might need 200.

From sticky notes to buckets: axial coding

After 100 notes, you have a pile of sticky notes. Axial coding is the second pass: you group the notes into a small set of named failure modes (specific, recurring ways your app goes wrong). Think of sorting post into pigeonholes. You read each note and drop it into the hole it belongs in, making a new hole when nothing fits. Then you merge holes that turn out to be the same thing.

Arjun and Meera sorted on a whiteboard with real sticky notes. It took 40 minutes. The result:

Failure mode	Count (of 100)
Wrong or missing order lookup	14
Recommended items not stocked at that store	11
Promised refund outside policy	9
Ignored or dodged allergy question	6
Reply contradicts tool result	5
Fine	55

Forty-five failures in a hundred conversations. When Priya had asked Arjun that morning to guess, he had said “maybe five percent”.

Good bucket names are specific enough that two people would put the same note in the same bucket, and specific enough to suggest a fix. Compare:

• Too broad: “Hallucination”. It hid three different problems with three different fixes.
• Useful: “Reply contradicts tool result”. You can check it, and you know where to look.

You can ask an LLM to suggest groupings for your notes, and it often gives a decent first draft. The difference from Arjun’s morning attempt is the input. This time the model groups your specific, observed notes rather than guessing problems from raw chat logs. You still review every bucket yourself, because the point is that you understand the failures.

Counting and choosing what to fix first

Now the numbers make decisions for you. Rank by how often each failure happens and how much it hurts when it does. Meera scored severity from 1 (annoying) to 5 (dangerous or costly):

Failure mode	Count	Severity	Count × severity
Ignored allergy question	6	5	30
Wrong or missing order lookup	14	2	28
Promised refund outside policy	9	3	27
Reply contradicts tool result	5	3	15
Item not stocked at store	11	1	11

The allergy failures are rare but could put a child in hospital, so they go first. Stock recommendations are common but cost a mildly annoyed customer, so they wait.

Some failures need no measurement at all. When Priya opened the refund bucket, she found the system prompt never mentioned the damage policy. That is not a mysterious model weakness. It is a missing sentence. Add it, read ten fresh refund traces, move on. You build evals for failures that survive a clear prompt; you will see why in the three gulfs lesson.

The code: building your annotation sheet

You do not need special software for error analysis. A spreadsheet works. Here is the workflow in four small steps.

Step 1: export traces to a file. Your tracing tool can export traces; the exact API differs between Langfuse, LangSmith and others, so here we start from a JSONL file (one JSON object per line) of the kind they all produce:

# uv add pandas
import json
import random
import pandas as pd

with open("traces.jsonl") as f:
    traces = [json.loads(line) for line in f]

random.seed(7)                        # same sample every run
sample = random.sample(traces, 80)    # 80 random traces
flagged = [t for t in traces if t.get("feedback") == "thumbs_down"][:20]

rows = [{"trace_id": t["id"], "input": t["input"], "output": t["output"],
         "note": "", "bucket": ""} for t in sample + flagged]
pd.DataFrame(rows).to_csv("to_annotate.csv", index=False)
print(f"Wrote {len(rows)} traces to annotate")

Expected output: Wrote 100 traces to annotate. Open the CSV in any spreadsheet. If your traces include tool calls and retrieval, add columns for them or keep the tracing tool open alongside, so you can see the upstream steps.

Step 2: annotate. Fill in note for every row. Leave bucket empty for now. Resist the urge to invent buckets while you read; they bias what you notice.

Step 3: bucket and count. After grouping, fill in bucket, then count:

df = pd.read_csv("annotated.csv")

counts = df["bucket"].value_counts()
print(counts)

failure_rate = (df["bucket"] != "fine").mean()
print(f"\nFailure rate: {failure_rate:.0%} of {len(df)} traces")

Expected output:

bucket
fine                         55
wrong order lookup           14
item not stocked at store    11
refund outside policy         9
dodged allergy question       6
contradicts tool result       5
Name: count, dtype: int64

Failure rate: 45% of 100 traces

Step 4 (optional): ask a model for a first-draft grouping. Give it only your notes, and ask it to propose buckets you then edit:

import anthropic
from dotenv import load_dotenv

load_dotenv()
client = anthropic.Anthropic()

notes = "\n".join(f"- {n}" for n in df["note"] if n != "fine")
response = client.messages.create(
    model="claude-opus-5-5",
    max_tokens=2000,
    messages=[{"role": "user", "content":
        "These are notes from reviewing a bakery support bot's conversations. "
        "Group them into 4 to 8 specific failure categories. Name each category "
        "so it suggests a fix, and list which notes belong to it.\n\n" + notes}],
)
print(next(b.text for b in response.content if b.type == "text"))

Treat the output as a draft. Merge, split and rename until the buckets match what you saw.

Where this goes wrong

Arjun’s first attempt at error analysis had its own problems, and they are the common ones.

His first thirty notes were vague. “Wrong answer”, “weird”, “not great”. When he came to bucket them, he had to reopen half the traces. Priya’s rule: write the note so that someone who has not seen the trace could guess the fix.

He got tired. After 60 traces his notes got shorter and more of them said “fine”. Reading is real work. Two sessions of 50 beat one session of 100.

The traces held private data. Customer phone numbers, addresses and one person’s medical condition were sitting in a CSV that was about to be shared with Meera and pasted into a model. Redact names, phone numbers and addresses before traces leave your tracing tool, and check what your company allows you to send to an outside model provider. Module 24 returns to this.

He wanted to do it once. Error analysis is not a launch task. A model upgrade, a new menu, or Diwali traffic will change the failure pattern. Priya put a recurring hour in Arjun’s calendar: read 30 fresh traces every Friday.

Try it yourself

Use your own app, or the tracing lab from Module 22. Export 50 traces into the CSV format above. Read each one and fill in note. Then group the notes into buckets and count them.

Expected result: a list of three to eight buckets sorted by count, “fine” probably the largest single bucket, and a failure rate higher than you would have guessed.

Warm-up exercise: put these six notes into buckets.

1. “Said order 3390 was delivered; tool said ‘preparing’”
2. “Gave Christmas plum cake price from last year”
3. “Answered in English when customer wrote in Marathi”
4. “Told customer order was ‘on the way’; tool returned ‘cancelled’”
5. “Quoted old Diwali hamper price”
6. “Replied in English to a Hindi question”

Show the answer

Three buckets, two notes each. Reply contradicts tool result: 1 and 4. Stale prices: 2 and 5 (probably the menu document is out of date, which is a data problem, not a model problem). Wrong reply language: 3 and 6. Each bucket suggests a different fix, which is how you know the grouping is useful.

Common mistakes

• Starting with generic metrics. “Helpfulness” and “coherence” scores sound scientific but measure problems you have not shown exist. Read first, measure second.
• Handing the whole job to an LLM. It will produce tidy categories that miss what only a baker would notice. Let it draft groupings of your notes, never replace your reading.
• Testing only with questions you wrote. Your questions reflect your vocabulary and your assumptions. Real customers write “brownie ok for nut allergy kid??” at 11 pm.
• Noting symptoms, not roots. “Dodged question” leads to prompt tweaks. “Allergen sheet not indexed” leads to the real fix.
• Reading only reported failures. Most unhappy customers never click thumbs-down. Random samples catch the quiet failures.

Recap

• A trace is the full record of one request: input, prompt, retrieval, tool calls, output and cost.
• Error analysis means reading 50 to 100 traces before building any metric, until new traces stop surprising you.
• Open coding: for each trace, write a specific note about the first thing that went wrong.
• Axial coding: group the notes into named failure modes, then count them.
• Prioritise by frequency times severity, and fix obvious prompt gaps directly instead of measuring them.
• It is a loop: analyse, measure, improve, and analyse again.

By four o’clock, Arjun had read all hundred traces and Meera had read forty. He walked her through the whiteboard.

“Out of a hundred conversations, forty-five went wrong somehow. Six were allergy questions it dodged, because we never gave it the allergen sheet and never told it to say ‘I don’t know’. Those are the dangerous ones, so they’re first. Nine promised refunds our policy doesn’t allow, because I never wrote the policy into its instructions. That’s a ten-minute fix. The rest are lookup and stock problems we can now count.” Meera nodded. “So that’s the answer to ‘how would we know’. We look.”

“We look first,” Priya said. “Next we turn each of these buckets into a number you can track every week. But it has to be the right kind of number.”

Go deeper

• Your AI Product Needs Evals (Hamel Husain)
• AI Evals: Everything You Need to Know (Hamel Husain and Shreya Shankar)
• Langfuse documentation