LLM-as-judge and judge alignment

The allergy row on the dashboard still said “not built yet”. It was the row Meera cared about most, and it was the one Arjun could not write in code. “Does the reply correctly say whether the item contains the allergen?” has a thousand correct phrasings, and checking it means knowing every recipe.

So on Thursday Arjun did what felt natural. If a model wrote the replies, a model could grade them. He wrote a second prompt, ran it over 200 allergy conversations from the last three months, and brought the results to Priya.

“Almost everything scores 8 or 9 out of 10,” he said. “So we’re mostly fine?”

Priya scrolled down to trace 014, the walnut brownie conversation that had started all this. It had scored a 7.

Meera, passing by with a tray of samples for a tasting, stopped and read over their shoulders. “Wait. You’re using the AI to check the AI? How is that not the fox guarding the henhouse?”

“That,” said Priya, “is exactly the right question. Pull up a chair.”

Arjun’s first judge

Here is the prompt Arjun used:

You are an expert evaluator of customer support conversations.
Rate the allergy safety of this reply from 1 to 10.
Reply with only the number.

And what it produced across 200 conversations:

Score	1-5	6	7	8	9	10
Conversations	3	9	21	88	71	8

Three things are wrong here, and you have met two of them already.

First, it is a scale again. You learned in the last lesson that scores hide what is wrong. What does a 7 mean? Is 7 safe to send to a mother whose son has a nut allergy?

Second, “allergy safety” is undefined. The judge invented its own standard, and its standard was clearly lenient: a reply that never mentioned nuts at all got a 7.

Third, and most important: nobody knows if the judge is any good. Arjun had replaced “I don’t know if the bot is right” with “I don’t know if the judge is right”. That is not progress.

Hiring a quality inspector

Priya turned to Meera. “Suppose you hire a new quality inspector for the central kitchen. Day one. How do you make sure she grades cakes the way you would?”

Meera did not hesitate. “I show her our standard. The written one: weight within 20 grams, no cracks in the icing, writing centred. Then I walk her through some cakes that passed and some that failed, especially the borderline ones.”

“And then you leave her to it?”

“No. For the first week I grade the same cakes she does. If we disagree, we talk about why. Once we agree almost every time, I stop checking every cake.”

“What if she passes everything?”

“Then she’s useless, however friendly she is. The point of an inspector is catching the bad ones.”

Arjun saw where this was going. “So the judge needs a written standard, examples, and someone checking it against their own grades.”

“And you check it separately on the good cakes and the bad cakes,” Priya said. “Because an inspector who passes everything looks great on a day when most cakes are good.”

An LLM-as-judge is a model call whose job is to grade another model’s output against a written standard. The analogy holds well, with one important difference: a human inspector gets better with experience. A judge prompt does not learn on its own. It only improves when you change the prompt.

What goes into a judge prompt

A judge that works has five parts:

1. One failure mode. Allergy content, not “overall quality”. A judge checking allergies, tone and refunds at once does all three badly.
2. The written pass/fail definition, word for word, from the previous lesson.
3. Reference information it needs. This one surprised Arjun. To know whether “the plum cake contains no eggs” is correct, the judge needs the plum cake’s ingredient list. A judge cannot check facts it does not have.
4. A few labelled examples, ideally borderline ones, showing how the standard applies.
5. A critique before the verdict, in a fixed shape. Asking for one or two sentences of reasoning first makes the verdict follow from the reasoning, and it gives you something to read when the judge disagrees with you. A fixed output shape means your code can count verdicts without parsing prose.

Here is Arjun’s rewritten judge prompt:

You check one thing: whether a bakery support bot correctly tells customers
if an item contains an allergen they asked about.

You will receive the customer's message, the bot's reply, and the item's
ingredient list from the kitchen.

PASS if the reply correctly states whether the item contains, or may contain,
the allergen asked about, consistent with the ingredient list.
FAIL if the reply ignores the allergen, states it wrongly, or guesses
without the ingredient list supporting it.
Treat "may contain traces" on the ingredient list as CONTAINS.

Example (FAIL). Customer: "Is the walnut brownie ok for my nut-allergic son?"
Ingredients: flour, sugar, butter, walnuts, cocoa, eggs.
Reply: "All our brownies are baked fresh every morning!"
Critique: The reply does not mention nuts at all. Verdict: fail

Example (PASS). Customer: "Does the plum cake have dairy?"
Ingredients: flour, butter, eggs, dried fruit, spices, rum.
Reply: "Yes, our plum cake contains butter."
Critique: Butter is dairy and the reply says so. Verdict: pass

Write a one or two sentence critique, then the verdict.

Following one reply through the judge

Take one real conversation and follow it all the way through.

The input. Trace 187. Customer: “Hi, does the almond rusk have gluten? My daughter is coeliac.” Bot reply: “Our almond rusk is made with wholewheat flour, so it contains gluten and isn’t suitable for a coeliac diet.” Ingredient list from the recipe archive: wholewheat flour, almonds, sugar, butter, cardamom.

What gets sent. The judge prompt above (about 380 tokens) as the system prompt, plus a user message holding the customer message, reply and ingredients (about 90 tokens).

What comes back.

{"critique": "Wholewheat flour contains gluten and the reply says so clearly, matching the ingredient list.", "verdict": "pass"}

What it cost. About 470 tokens in and 30 out: roughly ₹0.15 at the prices Crumb was paying, and 2.1 seconds. For 600 conversations a day, if every one were judged, that is about ₹90 a day, or ₹2,700 a month. Worth knowing before you judge everything.

Now a harder one. Trace 203. Customer: “Are the coconut cookies nut-free?” Reply: “Yes! Coconut cookies have no nuts in them.” Ingredient list: flour, desiccated coconut, sugar, butter; may contain traces of cashew (shared line).

The judge said pass, with the critique “The cookies do not list nuts as an ingredient.” Meera read the same conversation and said fail immediately: the shared-line warning means they may contain cashew, and the bot told a customer they were nut-free.

That single disagreement is why the line “Treat ‘may contain traces’ as CONTAINS” ended up in the prompt you saw above. But Arjun only found it because someone checked the judge against a human. That checking has a method.

Who judges the judge: labels and three piles

Judge alignment means measuring, with numbers, how well the judge agrees with a careful human, and improving it until it agrees well enough.

Step 1: label a set by hand. Meera and Arjun sat down with 120 allergy conversations and the recipe archive. For each one, Meera decided pass or fail using the written definition; Arjun pulled up ingredients. It took about two and a half hours over two afternoons. Final count: 79 passes, 41 fails. They had deliberately added fails found during error analysis, because random traffic alone would have had too few to learn from.

Step 2: split the labels into three piles. Think of preparing a student for an exam:

Pile	Count	Used for
Training examples	18	The worked examples you paste into the judge prompt
Development set	48	Measuring the judge while you improve its prompt
Hold-out set	54	Measured once, at the very end, for an honest score

The training examples are the worked solutions you show the student. The development set is the practice papers you mark together. The hold-out set is the real exam paper, sealed in an envelope.

Two numbers, not one

Arjun ran the first version of the judge on the 48-conversation development set. Of those 48, Meera had labelled 32 as pass and 16 as fail. Here is how the judge’s verdicts lined up with hers:

	Judge said pass	Judge said fail
Meera said pass (32)	31	1
Meera said fail (16)	7	9

“40 out of 48 right,” Arjun said. “83%. Not bad.”

“Look at the bottom row,” Priya said.

Of the 16 replies Meera failed, the judge only caught 9. It waved through 7 bad replies, and in this eval a bad reply means telling a parent something unsafe.

This is why you measure two numbers instead of one accuracy figure:

• True positive rate (TPR): of the replies the human marked pass, what share did the judge also pass? Here 31 of 32, about 97%.
• True negative rate (TNR): of the replies the human marked fail, what share did the judge also fail? Here 9 of 16, about 56%.

Accuracy blends the two, and because passes outnumber fails, the blend is dominated by the easy part. To see how badly that can mislead, imagine a lazy judge that answers “pass” to everything. On this development set it gets 32 of 48 right: 67% accuracy, which sounds mediocre rather than disastrous. Its TNR is 0%. It catches nothing. On a week where 95% of replies pass, the lazy judge would score 95% accuracy.

You will see the names TPR and TNR used the other way round in some write-ups, which treat “fail” as the positive case. The names matter less than the habit: always measure agreement on real passes and on real fails separately.

Reading the disagreements

The numbers say how bad the judge is. Reading the disagreements says why. Arjun opened all eight.

• Four were “may contain traces” products like the coconut cookies. The definition did not say how to treat them. Fix: one sentence in the prompt.
• Two were cases where the bot said “this is nut-free” and the judge believed it, because Arjun’s first version did not give the judge the ingredient list. It was checking whether the reply sounded correct. Fix: include ingredients.
• One was a reply that answered about the wrong product (the customer asked about the almond rusk; the bot answered about the almond croissant). The judge had not checked that the item matched. Fix: a line in the definition and a new example.
• One was Meera’s mistake. On re-reading, she changed her label. Humans make errors too, and disagreements are where you find them.

Version 2, with ingredients and the traces rule: TPR 97%, TNR 88%. Version 3, with the wrong-item example: TPR 97%, TNR 94%. Priya and Meera had agreed in advance on a target of at least 90% on both, so Arjun stopped iterating and opened the sealed envelope.

On the 54-conversation hold-out set, measured once: TPR 95%, TNR 93%. That is the number that goes in the report, slightly lower than on the development set, as it nearly always is.

The real code

Step 1: the judge function, with structured output. You met messages.parse and Pydantic models for typed output in Module 06:

from typing import Literal
import anthropic
from dotenv import load_dotenv
from pydantic import BaseModel

load_dotenv()
client = anthropic.Anthropic()

class Verdict(BaseModel):
    critique: str                        # reasoning, written first
    verdict: Literal["pass", "fail"]     # the only two allowed answers

JUDGE_PROMPT = open("allergy_judge_prompt.txt").read()   # the prompt shown above

def judge(customer: str, reply: str, ingredients: str) -> Verdict:
    response = client.messages.parse(
        model="claude-opus-5-5",
        max_tokens=1000,
        system=JUDGE_PROMPT,
        messages=[{"role": "user", "content":
            f"Customer: {customer}\n\nReply: {reply}\n\nIngredients: {ingredients}"}],
        output_format=Verdict,
    )
    return response.parsed_output

v = judge("Are the coconut cookies nut-free?",
          "Yes! Coconut cookies have no nuts in them.",
          "flour, desiccated coconut, sugar, butter; may contain traces of cashew")
print(v.verdict, "|", v.critique)

Expected output, with wording that will vary: fail | The ingredient list says may contain traces of cashew, but the reply says nut-free.

Step 2: run the judge on your labelled development set. Store labels in a CSV with columns customer, reply, ingredients, human:

import pandas as pd

dev = pd.read_csv("allergy_dev.csv")
dev["judge"] = [judge(r.customer, r.reply, r.ingredients).verdict
                for r in dev.itertuples()]

Step 3: compute TPR and TNR, and list every disagreement.

def agreement(df: pd.DataFrame) -> tuple[float, float]:
    """Return (TPR, TNR), treating 'pass' as positive."""
    real_pass = df[df.human == "pass"]
    real_fail = df[df.human == "fail"]
    tpr = (real_pass.judge == "pass").mean()
    tnr = (real_fail.judge == "fail").mean()
    return tpr, tnr

tpr, tnr = agreement(dev)
print(f"TPR {tpr:.0%}   TNR {tnr:.0%}")

for r in dev[dev.human != dev.judge].itertuples():
    print(f"\nHuman: {r.human}  Judge: {r.judge}\n  {r.customer}\n  {r.reply}")

Expected output for version 1: TPR 97% TNR 56%, followed by the eight disagreements to read.

Step 4: correct the judge’s pass rate on new data. Once you know TPR and TNR, you can adjust what the judge reports on unlabelled production traffic. If the judge says 80% of this week’s replies pass, with TPR 0.95 and TNR 0.93:

def corrected_pass_rate(observed: float, tpr: float, tnr: float) -> float:
    return (observed + tnr - 1) / (tpr + tnr - 1)

print(f"{corrected_pass_rate(0.80, 0.95, 0.93):.0%}")   # 83%

The estimate is slightly higher than the raw 80%, because this judge fails a few good replies (TPR below 100%). You do not need this on day one. It is a reminder that a judge’s numbers are estimates, not facts.

Where judges break

The judge model changes. When your provider updates the model behind the judge, its behaviour shifts. Re-run the hold-out set after any model change. Pin the model version where your provider allows it.

Your standard changes. Two months later, Crumb started labelling “egg-free” separately from “eggless” (one has no egg; the other may use egg substitutes processed on shared lines). The definition changed, so the old labels and the old alignment score went out of date. Relabel a fresh sample when the standard moves.

Judges have biases. Research on LLM judges has found that they tend to favour longer answers, answers in the first position when comparing two, and sometimes answers written by their own model family. Binary verdicts on a single output, with a tight definition, avoid most of this. Comparing two answers side by side invites it.

Judges cost money and time. At ₹0.15 per verdict, judging every conversation is ₹2,700 a month for one eval, and five evals is five times that. Most teams judge a daily random sample of 100 to 200 conversations, plus every conversation flagged by a cheap code check.

The judge only sees what you give it. Arjun’s version 1 could not catch wrong ingredients because it had no ingredients. If a human would need to look something up to grade a reply, the judge needs it too.

Try it yourself

1. Take 40 traces from your own app for one failure mode and label each pass or fail by hand, using a written definition.
2. Split them: 5 as examples, 20 as a development set, 15 held out.
3. Write a judge prompt with the five parts above, using your 5 examples.
4. Run it on the development set and compute TPR and TNR with the code above.
5. Read every disagreement and write one sentence about why it happened. Change the prompt, re-measure.
6. When you are happy, run the hold-out set once.

Expected result: your first judge will score noticeably lower on one of the two numbers than you hoped, and at least one disagreement will reveal a place where your own definition was fuzzy.

Common mistakes

• Asking for scores instead of verdicts. “Rate 1 to 10” brings back every problem from the last lesson, with a model’s quirks added.
• Trusting a judge you never checked. A judge that has never been compared with human labels is an opinion, not a measurement.
• Reporting accuracy only. When failures are rare, accuracy flatters a judge that misses them. Report TPR and TNR.
• Tuning on the hold-out set. Every peek turns the exam into homework. Keep it sealed.
• Starving the judge of facts. If grading needs a recipe, a policy or a tool result, put it in the judge’s input.
• One judge for everything. One judge per failure mode, each with its own alignment numbers.

Recap

• An LLM-as-judge is a model prompt that gives a pass/fail verdict on one failure mode against a written standard.
• A good judge prompt has one criterion, a definition, the reference facts it needs, examples, and a critique before a structured verdict.
• Align the judge with human labels split into training examples, a development set and a sealed hold-out set.
• Measure TPR (agreement on real passes) and TNR (agreement on real fails), not accuracy alone.
• Read every disagreement: they reveal fuzzy definitions, missing context and occasionally human mistakes.
• Re-check alignment when the model, the product or the standard changes.

At the Friday review, Arjun explained it to Meera. “The judge is a second AI with one job: checking whether the bot got the allergen right, using the actual recipe. You and I graded 120 conversations by hand. We used some of them to teach the judge, some to improve it, and kept 54 hidden. On those 54, it agreed with you on 95% of the good replies and caught 93% of the bad ones.”

“So it misses about one bad reply in fourteen,” Meera said.

“Yes. Which is why every reply it fails goes to a human, and why we re-check it every time something changes.” Meera thought about it. “It’s not the fox guarding the henhouse, then. It’s an inspector we tested against a hundred cakes we’d already graded ourselves.”

Then she asked the question that had been bothering her for weeks. “But why does something new break every time you fix something?” Priya was already uncapping a whiteboard marker.

Go deeper

• Using LLM-as-a-Judge for Evaluation (Hamel Husain)
• Who Validates the Validators? (Shankar et al.)
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Zheng et al.)