Binary pass/fail evaluations

After the error analysis, Arjun wanted a number. Something he could put on a screen, watch every week, and point at when things got better. On Tuesday he built it: every night a script sent fifty CrumbBot conversations to a model, asked it to rate each reply from 1 to 5 for “helpfulness”, and averaged the results. The average went on a big green tile at the top of a dashboard: 3.7.

He showed it to Meera on Wednesday, quietly proud.

She looked at it for a long moment. “Is 3.7 good?”

“It was 3.6 last week. So it’s better.”

“Better at what? If I want it to be 4.2 by Diwali, what do you change?”

Arjun opened his mouth, and closed it again.

Arjun’s helpfulness dashboard

Here is the heart of what Arjun built. It looks reasonable, and it is the most common first eval people write:

RATING_PROMPT = """Rate how helpful this support reply is, from 1 (useless) to 5 (perfect).
Reply with only the number."""

def rate(customer_message: str, reply: str) -> int:
    response = client.messages.create(
        model="claude-opus-5-5", max_tokens=10, system=RATING_PROMPT,
        messages=[{"role": "user", "content": f"Customer: {customer_message}\nReply: {reply}"}],
    )
    return int(next(b.text for b in response.content if b.type == "text").strip())

scores = [rate(t["input"], t["output"]) for t in traces]
print(f"Helpfulness: {sum(scores) / len(scores):.1f}")   # Helpfulness: 3.7

It runs. It produces a number. The number goes up and down. And nobody, including Arjun, could say what it meant.

Two people, one reply, two scores

Priya did not argue. She picked ten replies from the dashboard and printed them. “Arjun, Meera, rate these yourselves, 1 to 5. Don’t talk to each other.”

Their scores:

They gave the same score on 4 of 10 replies. They then argued for five minutes about reply 6. Arjun thought it was a 4, “it’s polite and it answers”. Meera gave it a 2: “It says the plum cake has no eggs. It does. I make it.” Neither of them had been rating the same thing.

Priya then asked a different question about the same ten replies. “Forget helpfulness. Does each reply state the correct order status, the one the order tool returned? Yes or no.” This time they agreed on all ten, in under two minutes.

“That’s the whole lesson,” Priya said. “Ask a question two people answer the same way.”

She used an analogy Meera knew well. “When the food safety inspector visits the Baner store, does she give the kitchen 3.7 out of 5 for cleanliness?”

“No. She has a checklist. Fridge below 5°C, yes or no. Handwash station stocked, yes or no. Raw and cooked kept apart, yes or no.”

“And when she leaves, what do you know?”

“Exactly which boxes we failed, and what to fix tomorrow morning.”

Why the average lied

The disagreement is one problem. The bigger one is what averaging does. Here are the actual ratings, on a 1 to 5 scale, for ten replies in two different weeks:

Week 2 looks like a clear win: up half a point. Now look at the two 1s. Both were allergy questions where the bot said an item was nut-free when it contained almonds. Week 1 had no dangerous replies at all; it was consistently decent. The average went up while the bot got more dangerous.

An average squashes everything into one number. A reply that could hurt a child and a reply that was a bit wordy both pull the average down by a similar amount, so the average cannot tell them apart.

Opening the hood: binary evals and pass rates

A 1-to-5 rating is called a Likert scale, after the psychologist who popularised it in surveys. It feels richer than yes or no. For evaluating AI output it causes four problems you have now seen:

1. Nobody agrees on the middle. The difference between a 3 and a 4 lives in each rater’s head, and in a model’s head too.
2. Averages hide the dangerous cases. Mediocre-everywhere and perfect-with-two-disasters can share an average.
3. It is not actionable. A score says how good, not what is wrong.
4. Small changes are noise. Because ratings wobble from rater to rater and day to day, a move from 3.6 to 3.7 can easily be chance.

The alternative is a binary eval: one yes-or-no question about one specific failure mode, asked of one output at a time. Run it across many outputs and you get a pass rate, the share of outputs that passed.

You already have the questions. They are the failure modes from your error analysis, turned around:

Each question is about one thing. “Is the reply helpful, accurate and polite?” is three evals pretending to be one. When it fails, you do not know which part failed.

How precise is a pass rate? Honest answer: on small samples, not very. If 40 of 50 replies pass (80%), the real rate could plausibly be anywhere from about 69% to 91%. That margin, about plus or minus 11 points, comes from a standard statistics formula and shrinks as you add examples: with 200 replies it is about plus or minus 6. So a move from 80% to 84% on 50 replies is not news; a move from 80% to 50% is. The difference from Likert averages is that when a pass rate drops, you know exactly which behaviour broke.

Following one reply through a written definition

A question alone is not enough. Meera and Arjun’s first attempt at “does the reply handle the allergy question well?” produced the same arguments as the 1-to-5 ratings. “Well” is the problem word. So they wrote a definition, with an example of each side:

Allergy handling. Applies only when the customer mentions an allergen or asks whether something is safe for a condition. Pass: the reply states whether the item contains or may contain the allergen, and includes the exact warning: “Our kitchens handle nuts, gluten and dairy, so we cannot guarantee any item is allergen-free.” Fail: the reply does not address the allergen, guesses, or leaves out the warning.

Now trace three real replies through it, one at a time.

Reply A. Customer: “is walnut brownie ok for nut allergy”. Bot: “All our brownies are baked fresh every morning!” Does the definition apply? Yes, an allergen is mentioned. Does the reply state whether the item contains nuts? No. Fail.

Reply B. Customer: “Does the Christmas plum cake have dairy?” Bot: “Yes, our plum cake contains butter. Our kitchens handle nuts, gluten and dairy, so we cannot guarantee any item is allergen-free.” Applies? Yes. States the allergen? Yes, butter. Exact warning? Yes. Pass.

Reply C. Customer: “What time does the Aundh store close?” Bot: “We close at 9 pm.” Applies? No allergen mentioned. Does not apply. This reply is left out of the allergy pass rate entirely.

Over a week, 13 of 100 traces mentioned an allergen. Eight passed. The allergy pass rate is 8 out of 13, about 62%, not 8 out of 100. Counting “does not apply” as a pass would have reported 95% and hidden the problem completely.

Code checks first, a model judge second

Some evals can be answered by plain code. These code-based evals, also called assertions, are fast, free and give the same answer every time, so reach for them first:

• Does the order status in the reply match the tool’s output? Compare two strings.
• Is every recommended item on the store’s stock list? Look it up.
• Is the output valid JSON? Try to parse it.
• Does the reply contain the exact allergy warning? Search for it.

Others need judgement. “Does the reply correctly state whether the item contains the allergen?” cannot be answered by string search, because there are a thousand correct phrasings and the code would need to know every recipe. For those you use a second model as a grader, which is the next lesson. Each model-graded check costs money (about ₹0.20 to ₹0.50 per reply for a short judge call with a large model), adds a second or two, and can itself be wrong. Code costs nothing and is never wrong about what it checks.

Building the eval suite, one step at a time

Step 1: one eval, as a function. Each eval takes one trace and returns True (pass), False (fail) or None (does not apply):

WARNING = "we cannot guarantee any item is allergen-free"
ALLERGENS = ["nut", "walnut", "almond", "cashew", "gluten", "dairy", "egg"]

def eval_allergy_warning(trace: dict) -> bool | None:
    """Pass if any allergy question gets the standard warning."""
    if not any(word in trace["input"].lower() for word in ALLERGENS):
        return None                       # no allergen mentioned: does not apply
    return WARNING in trace["output"].lower()

print(eval_allergy_warning({"input": "nut allergy?", "output": "All baked fresh!"}))  # False
print(eval_allergy_warning({"input": "closing time?", "output": "9 pm"}))            # None

Step 2: a second eval that uses the trace’s tool result. Traces store what each tool returned, so you can check the reply against it:

def eval_status_matches_tool(trace: dict) -> bool | None:
    """Pass if the reply repeats the status the order tool returned."""
    status = trace.get("tool_result", {}).get("status")
    if status is None:
        return None                       # no order lookup in this conversation
    return status.lower() in trace["output"].lower()

Step 3: run the suite and report a pass rate per eval.

import json

EVALS = [eval_allergy_warning, eval_status_matches_tool]

with open("traces.jsonl") as f:
    traces = [json.loads(line) for line in f]

for check in EVALS:
    results = [check(t) for t in traces]
    applicable = [r for r in results if r is not None]
    if applicable:
        rate = sum(applicable) / len(applicable)
        print(f"{check.__name__:28} {rate:4.0%} pass  ({len(applicable)} traces apply)")

Expected output:

eval_allergy_warning          62% pass  (13 traces apply)
eval_status_matches_tool      88% pass  (41 traces apply)

Step 4: check the eval against your own judgement. This is the step people skip. Arjun ran the allergy check on 30 traces Meera had already labelled by hand. They disagreed on 2. In both, the bot had written the warning in its own words (“we can’t promise anything is completely nut-free as our kitchen uses nuts”). Meera counted that as a pass; the string match did not.

They had two choices: loosen the check, or tighten the product. They tightened the product. The warning became a fixed sentence that the app code appends whenever an allergen is detected, rather than something the model is asked to remember. The check went from 28 of 30 agreement to 30 of 30, and the bot got more consistent too. This happens surprisingly often: writing a precise eval exposes a vague product decision.

Reading pass rates over time

Instead of one tile saying 3.7, the dashboard now has a row per failure mode:

Meera can read this one. The allergy warning is fixed, because the code now adds it. Refunds jumped after the policy went into the prompt. Stock recommendations did not move, so that is next. And when a number drops after a deploy, you know exactly which behaviour broke.

The targets are business decisions, set by risk. Allergy safety gets 99 or 100%. Store recommendations at 85% is fine for now. Meera sets the bar; engineering hits it.

Where binary evals break

Rare failures have tiny denominators. Only 13 of 100 random traces were allergy questions. A pass rate on 13 examples swings wildly. For rare but important failure modes, build a targeted test set: collect 50 real allergy questions from the past three months and run the bot on all of them.

Evals can be gamed. After the warning was enforced, Arjun briefly tried asking the model to “always include the allergy warning”. It started adding the warning to replies about delivery times. The allergy eval stayed at 100% while the bot became annoying. Every eval measures one thing, so watch for fixes that improve the measured thing by damaging an unmeasured one. Here, the fix was to put the warning in code, triggered only by allergen detection.

String checks are brittle. The status check fails if the tool says “out_for_delivery” and the bot says “on its way”. Normalise both sides (a small mapping of synonyms), or move that check to a model judge.

Binary loses some nuance, on purpose. You give up “slightly better” in exchange for “clearly fixed or not”. When you need nuance, add another binary eval for the specific thing you care about, rather than going back to a scale.

Try it yourself

Pick one failure mode from your own error analysis, or use “reply is too long to read comfortably on WhatsApp”. Write:

1. The yes-or-no question.
2. A one-paragraph pass/fail definition, including when it does not apply.
3. One pass example and one fail example.
4. Whether it can be a code check or needs a judge.
5. If it is a code check, write the function and run it on ten traces.

def eval_whatsapp_length(trace: dict) -> bool | None:
    if trace.get("channel") != "whatsapp":
        return None
    words = len(trace["output"].split())
    lists = trace["output"].count("\n- ") + trace["output"].count("\n1.")
    return words < 80 and lists <= 1

Common mistakes

• Bundling criteria. “Helpful, accurate and polite” in one eval means a failure tells you nothing. One failure mode per eval.
• Skipping the written definition. Without it, pass rates drift with your mood. Write it down, with examples, and share it.
• Counting not-applicable as pass. This silently inflates results. Return “does not apply” and leave those traces out.
• Using a model when code would do. A judge costs money, adds delay and makes mistakes. String checks and lookups do not.
• Trusting an eval you never checked. Run every new eval on traces you labelled by hand and look at the disagreements.
• Chasing 100% everywhere. Set thresholds by how much each failure hurts.

Recap

• 1-to-5 scales are inconsistent between raters, hide dangerous cases inside averages and do not say what to fix.
• A binary eval asks one yes-or-no question about one failure mode; across many traces it gives a pass rate.
• Every eval needs a written definition with pass and fail examples, and a “does not apply” case.
• Use code checks wherever possible; save model graders for real judgement calls.
• Check each eval against your own labels; disagreements often expose vague product decisions.
• Track one pass rate per failure mode, with a target set by risk.

Arjun deleted the green 3.7 tile. At Friday’s review he explained the new table to Meera. “Each row is one thing that went wrong in the conversations we read. Each number is the share of conversations where it went right this week. If a row drops, I know exactly what broke, and I can show you the conversations.”

“And the allergy row says ‘not built yet’,” Meera said. “That’s the one I care about most.”

“Because I can’t write code that knows whether the bot got the ingredients right,” Arjun said. “For that one we need a judge.” Priya nodded. “And then we need to check whether the judge can be trusted.”

Go deeper

• AI Evals: Everything You Need to Know (Hamel Husain and Shreya Shankar)
• Your AI Product Needs Evals (Hamel Husain)
• Using LLM-as-a-Judge for Evaluation (Hamel Husain)