JSON outputs and structured response handling

Arjun’s ticket router had been live for three days. Every message in Crumb’s support inbox went to Claude with a prompt that ended “Reply in JSON”, and the answer flowed into a small dashboard: category, urgency, order number. Meera checked it every morning with her chai. On a busy day it sorted 400 messages, and the food-safety team said allergy questions now reached them in minutes instead of hours.

On the fourth morning the dashboard was blank.

Arjun opened the logs. At 2:14 a.m. a customer had written a long, angry message about a Christmas plum cake. The model had answered with a friendly sentence, “Sure! Here is the JSON you asked for:”, followed by the JSON wrapped in a Markdown code fence. His code had called json.loads() on the whole reply, crashed on the word “Sure”, and the crash stopped the nightly job. The 212 messages behind it in the queue were never sorted. Two of them were allergy questions.

“It worked a hundred times,” he told Priya. “This is random.”

“It’s not random,” Priya said. “It’s text. You asked a writer for a spreadsheet and got a letter with a spreadsheet inside it. A model’s output is untrusted input, exactly like a form a stranger filled in. Let’s treat it that way.”

Arjun’s first attempt, and the three ways it fails

Here is the heart of what Arjun had shipped. It is the code almost everyone writes first.

import json
import anthropic
from dotenv import load_dotenv

load_dotenv()
client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY

PROMPT = """Classify this Crumb & Co. support message.
Reply in JSON with keys: category, urgency, order_id, summary.

Message:
"""

def triage(message: str) -> dict:
    response = client.messages.create(
        model="claude-opus-5-5",
        max_tokens=2000,
        messages=[{"role": "user", "content": PROMPT + message}],
    )
    text = next(b.text for b in response.content if b.type == "text")
    return json.loads(text)   # <- trusts the model completely

JSON (JavaScript Object Notation) is a plain-text format for structured data: curly braces, keys in double quotes, and values that are strings, numbers, true/false, lists or null. Programs love it because its rules are exact. Language models write it well, but remember from Module 01 how they work: they predict text one token (a chunk of text, roughly three-quarters of a word) at a time. Nothing in that process forces the result to obey JSON’s rules. Most of the time it does. “Most of the time” is the problem when you process 400 messages a day.

When Priya and Arjun went through a week of logs, the failures fell into three families:

The third family is the dangerous one. The first two crash loudly, so you notice. A wrong shape slips through and corrupts a report three systems away. The dashboard had been filtering on urgency == "high" for three days, and every “VERY HIGH!!” ticket had been quietly invisible.

Priya’s order slips

Priya grabbed a marker. “Think about the order slips at the Baner store counter. What’s on them?”

“Boxes,” Arjun said. “Item, quantity, pickup time, phone number.”

“Why boxes? Why not a blank sheet where customers write whatever they want?”

“Because then the kitchen gets ‘two of the chocolate ones, the nice ones, by evening-ish’.”

“Right. Now, even with boxes, a customer can write ‘lots’ in the quantity box. So what does the person at the counter do?”

“Checks the slip before it goes to the kitchen. Hands it back if it’s wrong.”

“That’s the whole lesson,” Priya said. “The boxes are a schema: a written description of the exact shape data must have, which keys, which types, which allowed values. The person checking slips is a validator. You had neither. You gave the model a blank sheet and sent whatever came back straight to the oven.”

She drew a ladder with four rungs.

flowchart LR
    A["1. Ask clearly in the prompt"] --> B["2. Parse defensively"]
    B --> C["3. Validate against a schema"]
    C --> D["4. Let the API enforce the schema"]

“Each rung catches what the one below misses. Prompting reduces failures. Defensive parsing survives the wrapped ones. Validation catches the wrong shapes. And the top rung, where the API itself refuses to produce anything outside the schema, makes most of the problems disappear. We’ll climb it with one real message.”

Where the analogy breaks: a customer filling in a slip knows they are filling in a form. The model does not “know” anything; it predicts likely text. That is why rung 4, which changes how the text is generated, is so much stronger than asking politely.

Following one message up the ladder

Here is the message they used, a real one from the log, lightly edited:

“Hi!! My Diwali hamper CR-2210 still hasn’t come and guests arrive at 7. Also does the kaju katli have nuts, my nephew is allergic”

Rung 1: ask clearly

Arjun’s prompt said “keys: category, urgency, order_id, summary” but never said what values were allowed. “Urgency” could be anything from “high” to “OMG”. So rung 1 is a better prompt: list every allowed value, say what to do when something is missing, and show one example.

PROMPT = """Classify this Crumb & Co. support message.
Reply with ONLY a JSON object, no other text, with exactly these keys:
- category: one of "late_delivery", "allergy_question", "refund_request", "other"
- urgency: one of "low", "medium", "high"
- order_id: the order id like "CR-1042" as a string, or null if none is given
- summary: one short sentence in English

Example: {"category": "refund_request", "urgency": "medium", "order_id": "CR-1042", "summary": "Customer wants a refund for a squashed cake."}

Message:
"""

This prompt is about 150 tokens; with the message, about 190. On Arjun’s run, Claude replied:

Here's the classification:

```json
{
  "category": "late_delivery",
  "urgency": "VERY HIGH",
  "order_id": "CR-2210",
  "summary": "Diwali hamper not delivered; guests arriving at 7. Also asks about nuts.",
}
```

Better, but look closely: it is wrapped in prose, it says "VERY HIGH", and there is a trailing comma after the last value, which strict JSON forbids. One reply, all three failure families. A clearer prompt lowers the failure rate, often a lot, but it never brings it to zero. That is the first misconception to drop: “If I say ONLY JSON firmly enough, it will always comply.” It won’t, especially on long or emotional inputs that pull the model toward being conversational.

Rung 2: parse defensively

A defensive parser pulls the JSON object out of whatever surrounds it:

import json
import re

def extract_json(text: str) -> dict:
    """Pull the first JSON object out of a model reply, even if wrapped in prose or a code fence."""
    fenced = re.search(r"```(?:json)?\s*(\{.*?\})\s*```", text, re.DOTALL)
    if fenced:
        candidate = fenced.group(1)
    else:
        # No fence: take everything from the first "{" to the last "}"
        candidate = text[text.find("{") : text.rfind("}") + 1]
    return json.loads(candidate)  # still raises json.JSONDecodeError if the JSON itself is broken

Line by line: re.search looks for three backticks, an optional word json, then a {...} block, then closing backticks. re.DOTALL lets . match newlines, because the JSON spans several lines. If there is no fence, the fallback slices from the first { to the last }.

On the reply above, extract_json finds the fenced block and calls json.loads on it, which raises:

json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 6 column 1 (char 178)

That is the trailing comma. The wrapper is solved; the broken syntax is not. Rung 2 fixes only one family, and you should not try to “repair” broken JSON with more regular expressions. That road ends in a 200-line function nobody understands. Instead, the next rung gives the model a precise error and one more chance.

One more broken case deserves a mention: the reply that simply stops. Every response carries a stop reason (the API’s note on why the model stopped writing). If response.stop_reason == "max_tokens", the model hit the output limit you set, mid-sentence, and the JSON is incomplete by definition. Check it before parsing, and raise max_tokens or ask for less.

Rung 3: validate with Pydantic

Pydantic is a Python library that turns a class with type hints into a validator. You met it in the Python refresher; here is where it earns its keep. You describe the slip once:

from typing import Literal
from pydantic import BaseModel, Field, ValidationError

class TicketTriage(BaseModel):
    category: Literal["late_delivery", "allergy_question", "refund_request", "other"]
    urgency: Literal["low", "medium", "high"]
    order_id: str | None = Field(description="Order id like CR-1042, or null if the customer gave none")
    summary: str = Field(description="One short sentence in English")

Literal[...] means “only these exact strings”. str | None means “a string, or null”. Field(description=...) attaches a note that will matter on rung 4.

Suppose the trailing comma had not been there and extract_json returned this dictionary:

data = {"category": "late_delivery", "urgency": "VERY HIGH", "order_id": "CR-2210",
        "summary": "Diwali hamper not delivered; guests arriving at 7."}
TicketTriage.model_validate(data)

Pydantic refuses it, precisely:

1 validation error for TicketTriage
urgency
  Input should be 'low', 'medium' or 'high' [type=literal_error, input_value='VERY HIGH', input_type=str]

That message is good enough to send straight back to the model. So rung 3 is a small repair loop: try, and if parsing or validation fails, show the model its own reply plus the exact error, and ask once more.

def triage(message: str, attempts: int = 2) -> TicketTriage:
    messages = [{"role": "user", "content": PROMPT + message}]
    for _ in range(attempts):
        response = client.messages.create(model="claude-opus-5-5", max_tokens=2000, messages=messages)
        if response.stop_reason == "max_tokens":
            raise ValueError("Reply was cut off; raise max_tokens")
        text = next(b.text for b in response.content if b.type == "text")
        try:
            return TicketTriage.model_validate(extract_json(text))
        except (json.JSONDecodeError, ValidationError) as err:
            # Show the model its own reply and the exact error, then ask again
            messages += [
                {"role": "assistant", "content": text},
                {"role": "user", "content": f"That reply was not valid.\nError: {err}\nReply with only the corrected JSON."},
            ]
    raise ValueError("Model did not return valid triage JSON")

On Arjun’s message, attempt one failed on the trailing comma, and attempt two came back clean: urgency was "high", the comma was gone. It worked. But count the cost. The first call was about 190 input tokens and roughly 150 output tokens. The repair call re-sends everything, so about 420 input tokens and another 120 output. At Claude Opus 5.5’s prices ($4permillioninputtokens,$20 per million output tokens), the first call costs about $0.0038andtherepairabout$0.0041. The retry more than doubled the cost of that ticket, from about ₹0.33 to ₹0.70 at ₹88 to the dollar. Small per ticket; across a 4% failure rate on 400 tickets a day, it adds up, and every retry adds a few seconds for the customer.

Rung 4: let the API enforce the schema

Now the top rung. Modern APIs offer structured outputs: you hand the API your schema and it guarantees the reply matches it. With the Anthropic SDK you pass the Pydantic class directly and get a validated object back. Here is the version with Priya’s fix from the pause above:

class TicketTriage(BaseModel):
    categories: list[Literal["late_delivery", "allergy_question", "refund_request", "other"]]
    urgency: Literal["low", "medium", "high"]
    order_id: str | None = Field(description="Order id like CR-1042, or null if the customer gave none")
    summary: str = Field(description="One short sentence in English")

response = client.messages.parse(
    model="claude-opus-5-5",
    max_tokens=2000,
    system="You triage Crumb & Co. support messages. Allergy questions are always high urgency.",
    messages=[{"role": "user", "content": "Hi!! My Diwali hamper CR-2210 still hasn't come and guests "
               "arrive at 7. Also does the kaju katli have nuts, my nephew is allergic"}],
    output_format=TicketTriage,
)
ticket = response.parsed_output   # a TicketTriage instance, already validated
print(ticket)

Output from Arjun’s run:

categories=['late_delivery', 'allergy_question'] urgency='high' order_id='CR-2210'
summary='Diwali hamper CR-2210 not delivered before a 7 pm party; customer also asks if kaju katli contains nuts due to an allergy.'

No prompt listing the allowed values. No regex. No repair loop. One call, about ₹0.35. Notice also that the prompt got shorter: the schema now carries the rules, so the instructions only say what the job is.

Under the hood: how an API can promise a shape

How can the API promise valid output from a model that predicts one token at a time? By changing which tokens it is allowed to pick.

Remember the picture from Module 01: at every step the model produces a probability for every possible next token, and one is chosen. With structured outputs, the SDK first converts your Pydantic class into a JSON Schema, a standard JSON document describing allowed keys, types and values. You can see it yourself:

import json
print(json.dumps(TicketTriage.model_json_schema(), indent=2))

{
  "properties": {
    "categories": {"items": {"enum": ["late_delivery", "allergy_question", "refund_request", "other"], "type": "string"}, "type": "array", ...},
    "urgency": {"enum": ["low", "medium", "high"], "type": "string", ...},
    "order_id": {"anyOf": [{"type": "string"}, {"type": "null"}], "description": "Order id like CR-1042, or null if ...", ...},
    "summary": {"description": "One short sentence in English", "type": "string", ...}
  },
  "required": ["categories", "urgency", "order_id", "summary"],
  "type": "object", ...
}

The API turns that schema into a grammar, a set of rules for which characters may come next. Then, at each step of generation, it masks out every token that would break the grammar. After "urgency": ", the only tokens allowed are ones that can spell low, medium or high. “VERY” is not discouraged; it is impossible. This technique is called constrained decoding. It is why a trailing comma or a chatty preamble cannot happen: those tokens never get a chance.

Two practical consequences follow. First, your description texts matter: they go to the model inside the schema and act like a mini prompt for each field. Second, the first request with a brand-new schema can be a little slower while the grammar is prepared; providers cache it after that.

OpenAI offers the same idea under the same name; you will compare the two SDKs side by side in Module 06.

Where it still breaks

Structured outputs remove the syntax and shape problems. They do not remove the need to think. Three things still bit Arjun in the following weeks.

Valid is not true. One morning a ticket came back with order_id: "CR-22100". Perfectly valid string. The customer had typed “CR-2210 0” with a stray zero. The schema cannot know your order table. This is the second big misconception: “If it passes validation, it’s correct.” Shape and truth are different checks. Before you refund, email or update anything, look the value up in your real data.

Forced boxes invite invention. Early on, Arjun made order_id: str (not str | None) because “every ticket should have one”. For messages with no order id, the model, constrained to produce some string, filled in plausible-looking ids. If a value might be missing, make “missing” a legal answer with | None. Otherwise you are asking the model to hallucinate (produce confident, fluent, false content) to satisfy your form.

Refusals and cut-offs. If the model declines a request (stop_reason == "refusal") or runs out of max_tokens, there may be no valid object to return. In the Anthropic SDK, check that response.parsed_output is not None before using it, and decide what your app does instead. The next lesson is entirely about that kind of “success that isn’t”.

And when you cannot use rung 4 at all, for example with a small local model in Module 20 or a provider without structured outputs, rungs 1 to 3 are still your safety net. That is why you learned them first.

What changes in the rest of the app

Once triage() returns a TicketTriage object instead of a loose dictionary, the code after it becomes boring in the best way:

if "allergy_question" in ticket.categories:
    route_to("food-safety-team", ticket)
if ticket.urgency == "high":
    route_to("priority-queue", ticket)

Arjun’s editor autocompletes ticket.urgency. A typo like ticket.urgancy is an error he sees immediately, not a silent None. Nobody downstream ever sees "VERY HIGH!!". Validation happens once, at the edge, and everything inside the app can trust what it receives. Engineers call this parse, don’t validate everywhere: convert untrusted text into a trusted type at the boundary, then pass the type around.

Try it yourself

1. Create triage.py with the multi-category TicketTriage model and the messages.parse call above.
2. Run it on these three messages:
   • “Does the chocolate truffle cake contain peanuts? My son is allergic.”
   • “Order CR-3301 arrived squashed. I want my money back.”
   • “Love the new sourdough!”
3. Print each result with ticket.model_dump_json(indent=2), and print response.usage to see the tokens.

Expected: ["allergy_question"] with high urgency and order_id null; ["refund_request"] with order_id "CR-3301"; ["other"] with low urgency. Each call should use a few hundred input tokens.

for msg in [
    "Does the chocolate truffle cake contain peanuts? My son is allergic.",
    "Order CR-3301 arrived squashed. I want my money back.",
    "Love the new sourdough!",
]:
    response = client.messages.parse(
        model="claude-opus-5-5", max_tokens=2000,
        system="You triage Crumb & Co. support messages. Allergy questions are always high urgency.",
        messages=[{"role": "user", "content": msg}],
        output_format=TicketTriage,
    )
    print(response.parsed_output.model_dump_json(indent=2))
    print(response.usage.input_tokens, "in /", response.usage.output_tokens, "out")

Common mistakes

• Calling json.loads on the raw reply. Works in testing, fails at 2 a.m. Use structured outputs, or at least defensive parsing plus validation.
• Using str where you mean a fixed set of values. urgency: str accepts “VERY HIGH!!”. Use Literal[...] so bad values fail at the edge.
• Leaving “missing” off the menu. If a field might be absent, make it X | None. Otherwise the model invents a value.
• Ignoring stop_reason. A reply cut off by max_tokens is broken JSON, and a refusal has no object at all. Check before parsing.
• Treating valid as true. Schema-valid data can still be wrong. Check ids and amounts against your own records.
• Fixing the model when the schema is wrong. If real inputs do not fit your boxes, change the boxes.

Explaining it to Meera

Meera stopped by that afternoon. “So what was wrong with my dashboard?”

“The AI writes text, not data,” Arjun said. “I was pouring its text straight into the dashboard. Usually the text happened to be neat. Once it wasn’t, and everything stopped. And for three days some tickets said ‘VERY HIGH’ instead of ‘high’, so the urgent filter never showed them.”

“And now?”

“Now it’s like our order slips. I give the AI a form with fixed boxes and fixed choices, and the system won’t let it write outside the boxes. Then I still check the order number against our real orders, because a neat slip can still have the wrong number on it.”

“What does that cost?”

“About thirty-five paise a ticket. Slightly less than before, because I stopped paying for retries.”

Recap

• Model output is untrusted text until you have parsed and validated it.
• Prompt-only JSON fails in three ways: wrapped, broken, or wrong shape, and the last one fails silently.
• Pydantic models describe the shape once and give precise errors you can feed back to the model in a small repair loop.
• Structured outputs (messages.parse(..., output_format=Model)) use constrained decoding so the reply cannot leave the schema.
• Design the schema for real inputs: allow None for missing values and lists where several answers apply.
• Valid shape does not mean a true answer: check important values against real data.

The dashboard came back, and it has not gone blank since. But Priya was already looking past it. “JSON was the polite failure. It happens after the API has answered. Wait until the API doesn’t answer at all, on the busiest day of the year.”

Go deeper

• Pydantic documentation: https://docs.pydantic.dev
• Anthropic docs (structured outputs): https://docs.anthropic.com
• OpenAI docs (structured outputs): https://platform.openai.com/docs
• Python tutorial (the json module and exceptions): https://docs.python.org/3/tutorial/