Python for AI workflows

Meera has a new request, and it’s a big one. “The bot sorts the weekend emails beautifully now. Can it also draft the replies? The team would check each one before sending, but starting from a draft instead of a blank page would save hours.”

Priya adds a condition. “Before we commit to one AI provider, let’s try the same task on Claude, OpenAI and Gemini and compare quality and cost. Models and prices change every few months. Our code shouldn’t have to.”

Arjun builds the first version in an afternoon, against OpenAI’s API, because that’s the tutorial he found first. It works. Then he swaps in Claude to compare, and eleven lines across four files break at once. Then, with OpenAI again, a reply that starts “Here is the triage:” crashes his JSON parser. By evening, he has three half-working scripts and no comparison.

“This is the lesson where Python turns into AI engineering,” Priya says. “Everything you’ve learned this week comes together: dictionaries, classes, errors, files, Pydantic, HTTP. The model is one function call. The engineering is everything around it.”

Arjun’s first attempt: welded to one provider

Here’s the core of Arjun’s OpenAI version, simplified. To run without a key, it uses a reply exactly as OpenAI’s API returned it on one of his test emails:

import json

# What OpenAI's API sent back for one email (trimmed)
resp = {
    "choices": [{
        "message": {"role": "assistant",
                    "content": 'Here is the triage:\n{"category": "refund", "urgency": 4}'},
        "finish_reason": "stop",
    }],
}

text = resp["choices"][0]["message"]["content"]   # this line appears in 11 places
data = json.loads(text)
print(data["category"])

json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Two problems are tangled here.

The first is the shape. resp["choices"][0]["message"]["content"] is how you reach the text in an OpenAI reply. Claude and Gemini put it somewhere else entirely. Arjun copied that path into eleven places, so his whole app is welded to one provider.

The second is the contents. He asked for JSON, and the model mostly obliged, but it added a friendly sentence first. json.loads reads the very first character, H, finds it isn’t valid JSON, and gives up: “Expecting value: line 1 column 1 (char 0)”. Models are helpful by nature, and helpful text isn’t always parseable text.

Three couriers, three boxes

Priya sketches three delivery vans on the whiteboard.

“Crumb gets flour from three suppliers. Same flour, but one sends sacks, one sends boxes with an inner bag, one sends tins. What does the kitchen do?”

“Unpack them at the back door and pour everything into our own bins.”

“So the bakers never care who the supplier was. Where should the ‘unpacking’ code live in your app?”

“In one place, at the door,” Arjun says. “Not in eleven.”

Here are the three boxes. These are the shapes of the raw JSON each provider’s API sends back for the same email, trimmed to the fields that matter. Save this as sample_responses.py; we’ll use these real shapes to build and test everything without spending a rupee.

# sample_responses.py
"""The same answer, in three providers' response shapes (trimmed)."""

FENCE = "`" * 3      # three backticks, the Markdown "code fence" models love to add

ANSWER = (
    '{"category": "refund", "order_id": "CR-2210", "urgency": 4, '
    '"draft_reply": "Hi Ananya, I am so sorry your Diwali hamper arrived damaged. '
    'I have passed order CR-2210 to our team for a refund, and you will hear from us within 24 hours."}'
)

CLAUDE = {
    "id": "msg_01AbCdEf",
    "type": "message",
    "role": "assistant",
    "model": "claude-opus-5-5",
    "content": [{"type": "text", "text": ANSWER}],
    "stop_reason": "end_turn",
    "usage": {"input_tokens": 231, "output_tokens": 71},
}

OPENAI = {
    "id": "chatcmpl-AbCdEf",
    "object": "chat.completion",
    "model": "gpt-5-mini",
    "choices": [{
        "index": 0,
        "message": {"role": "assistant",
                    "content": f"Here is the triage:\n{FENCE}json\n{ANSWER}\n{FENCE}"},
        "finish_reason": "stop",
    }],
    "usage": {"prompt_tokens": 224, "completion_tokens": 78, "total_tokens": 302},
}

GEMINI = {
    "candidates": [{
        "content": {"role": "model", "parts": [{"text": ANSWER}]},
        "finishReason": "STOP",
    }],
    "usageMetadata": {"promptTokenCount": 219, "candidatesTokenCount": 70, "totalTokenCount": 289},
    "modelVersion": "gemini-2.5-flash",
}

Compare where each one keeps the same four things:

A token is the unit models read and write: a chunk of text, roughly ¾ of an English word on average. Providers bill per token, separately for input (everything you send) and output (everything the model writes), which is why every response reports both.

Why is Claude’s content a list? Because a reply can contain several kinds of blocks: text, and later in the course, tool calls (the model asking your code to run a function). OpenAI’s and Gemini’s shapes have room for multiple answers (choices, candidates) and multiple parts. In practice you’ll read the first, but your code should know the structure is a list.

One more thing: the SDKs give you Python objects, not dictionaries. With the Anthropic SDK you write response.content[0].text instead of response["content"][0]["text"], but the shape is identical, because the SDK objects mirror the JSON. The Anthropic and OpenAI SDK objects even have .model_dump() to turn them back into dictionaries, since they’re Pydantic models underneath.

The unpacking station: one shape for every reply

Now the back door. We’ll build a module, llm_utils.py, in parts. Create the file with this first part:

# llm_utils.py
"""Reusable helpers for calling LLMs and handling their replies."""
from dataclasses import dataclass


@dataclass
class LLMReply:
    """One model reply, in OUR shape, whoever the provider was."""
    provider: str
    text: str
    input_tokens: int
    output_tokens: int
    truncated: bool          # True if the model ran out of room mid-answer


def from_claude(raw: dict) -> LLMReply:
    return LLMReply(
        provider="claude",
        text="".join(b["text"] for b in raw["content"] if b["type"] == "text"),
        input_tokens=raw["usage"]["input_tokens"],
        output_tokens=raw["usage"]["output_tokens"],
        truncated=raw["stop_reason"] == "max_tokens",
    )


def from_openai(raw: dict) -> LLMReply:
    choice = raw["choices"][0]
    return LLMReply(
        provider="openai",
        text=choice["message"]["content"] or "",
        input_tokens=raw["usage"]["prompt_tokens"],
        output_tokens=raw["usage"]["completion_tokens"],
        truncated=choice["finish_reason"] == "length",
    )


def from_gemini(raw: dict) -> LLMReply:
    candidate = raw["candidates"][0]
    return LLMReply(
        provider="gemini",
        text="".join(p.get("text", "") for p in candidate["content"]["parts"]),
        input_tokens=raw["usageMetadata"]["promptTokenCount"],
        output_tokens=raw["usageMetadata"]["candidatesTokenCount"],
        truncated=candidate["finishReason"] == "MAX_TOKENS",
    )

Each from_... function is an adapter: it knows one provider’s box and pours the contents into our bin, the LLMReply dataclass. Nothing else in the app will ever touch a provider’s raw shape. Try them:

from llm_utils import from_claude, from_gemini, from_openai
from sample_responses import CLAUDE, GEMINI, OPENAI

for reply in [from_claude(CLAUDE), from_openai(OPENAI), from_gemini(GEMINI)]:
    print(f"{reply.provider:7} in={reply.input_tokens} out={reply.output_tokens} "
          f"truncated={reply.truncated} text starts: {reply.text[:22]!r}")

claude  in=231 out=71 truncated=False text starts: '{"category": "refund",'
openai  in=224 out=78 truncated=False text starts: 'Here is the triage:\n``'
gemini  in=219 out=70 truncated=False text starts: '{"category": "refund",'

Three boxes in, one shape out. Notice the token counts differ slightly for the same email: each provider splits text into tokens its own way, which matters when comparing prices. And OpenAI’s text still has its chatty preamble. The adapter’s job is to unpack the box, not to clean the contents; that comes next.

Prompts are code: templates you can read and test

Arjun’s original prompt was built with + in the middle of a function, a little differently in each script. Priya’s rule: prompts are code. Keep them in one place, as named templates, with clear slots for the variable parts.

Add this to llm_utils.py:

# llm_utils.py (continued)

SYSTEM_PROMPT = """You are the support assistant for Crumb & Co., a family bakery chain in Pune.
Read the customer's email and reply with JSON only, no other text, with these keys:
- "category": one of "late_delivery", "refund", "allergy", "other"
- "order_id": the order number like "CR-1234", or null if there isn't one
- "urgency": 1 to 5, where 5 means a health risk or a very upset customer
- "draft_reply": a warm, short reply for a staff member to review before sending
Never promise a specific refund amount. Never give medical advice."""

USER_TEMPLATE = """Customer email:
<email>
{email}
</email>

Order details from our database:
{order_context}"""


def build_messages(email: str, order_context: str = "No matching order found.") -> list[dict]:
    """Fill the template and wrap it in the messages list every chat API expects."""
    content = USER_TEMPLATE.format(email=email.strip(), order_context=order_context)
    return [{"role": "user", "content": content}]

The parts of a prompt:

• The system prompt holds the standing instructions: who the assistant is, the output format, the rules. It’s the same for every email.
• The user message holds what changes each time: this email, this order’s details.
• The messages list is the list of dictionaries you’ve seen since the basics lesson. Each has a role ("user" or "assistant") and content. For a multi-turn chat, it grows with each exchange.

Wrapping the email in <email> tags makes clear to the model where the customer’s words start and stop, which helps it keep instructions and data apart. And because build_messages is an ordinary function, you can test it, print it and review it like any other code:

from llm_utils import build_messages

email = "Hi, my Diwali hamper (order CR-2210) arrived with the box crushed and two kaju katli broken. I'd like a refund. - Ananya"
messages = build_messages(email, order_context="CR-2210: 1 x Diwali hamper, ₹1,499, delivered 3 Nov, Baner store")
print(messages[0]["role"])
print(messages[0]["content"])

user
Customer email:
<email>
Hi, my Diwali hamper (order CR-2210) arrived with the box crushed and two kaju katli broken. I'd like a refund. - Ananya
</email>

Order details from our database:
CR-2210: 1 x Diwali hamper, ₹1,499, delivered 3 Nov, Baner store

When a reply is bad, the first question is always “what exactly did we send?” Now you can answer it in one line.

Following Ananya’s email from inbox to draft

Before writing the rest, Priya and Arjun trace one email through the whole workflow on the whiteboard, with the real numbers from a Claude test run.

Steps 3, 6 and 10 are written. Steps 7 to 9 are next.

Getting JSON out of a chatty reply

Models wrap JSON in code fences, add a sentence before it, or occasionally return something that isn’t JSON at all. Add a forgiving extractor and a strict validator to llm_utils.py:

# llm_utils.py (continued)
import json
import re
from typing import Literal

from pydantic import BaseModel, Field, ValidationError

FENCE = "`" * 3


class LLMResponseError(Exception):
    """The model replied, but not in a shape we can use."""


class Triage(BaseModel):
    category: Literal["late_delivery", "refund", "allergy", "other"]
    order_id: str | None = None
    urgency: int = Field(ge=1, le=5)
    draft_reply: str


def extract_json(text: str) -> dict:
    """Find the JSON object inside a model's reply, tolerating fences and chatter."""
    fenced = re.search(FENCE + r"(?:json)?\s*(.*?)" + FENCE, text, re.DOTALL)
    candidate = fenced.group(1) if fenced else text
    start, end = candidate.find("{"), candidate.rfind("}")
    if start == -1 or end == -1:
        raise LLMResponseError(f"No JSON object in reply: {text[:60]!r}")
    try:
        return json.loads(candidate[start:end + 1])
    except json.JSONDecodeError as error:
        raise LLMResponseError(f"Reply had broken JSON: {error}") from error


def parse_triage(reply: LLMReply) -> Triage:
    """Turn a normalised reply into a validated Triage, or raise LLMResponseError."""
    if reply.truncated:
        raise LLMResponseError("Reply was cut off (raise max_tokens or shorten the task)")
    try:
        return Triage.model_validate(extract_json(reply.text))
    except ValidationError as error:
        problems = "; ".join(f"{e['loc'][0]}: {e['msg']}" for e in error.errors())
        raise LLMResponseError(f"Reply failed validation: {problems}") from error

What each piece does:

• extract_json first looks for a fenced block. re.search with a regular expression (a pattern language for matching text) finds three backticks, an optional json, then captures everything up to the closing fence; re.DOTALL lets the match span several lines. If there’s no fence, it uses the whole text. Then it takes everything from the first { to the last }, which skips chatter before and after.
• parse_triage checks for truncation first. A reply that ran out of tokens mid-answer usually has broken JSON, and “it was cut off” is a much more useful error than “Expecting ‘,’ delimiter”.
• Triage is the Pydantic model from the type hints lesson, doing its job at the door.
• Every failure becomes one exception type, LLMResponseError, with a clear message. Callers only need to catch one thing.

Now all three providers’ replies, including OpenAI’s chatty one, produce the same validated object:

from llm_utils import LLMReply, LLMResponseError, from_claude, from_gemini, from_openai, parse_triage
from sample_responses import CLAUDE, GEMINI, OPENAI

for reply in [from_claude(CLAUDE), from_openai(OPENAI), from_gemini(GEMINI)]:
    triage = parse_triage(reply)
    print(reply.provider, "->", triage.category, triage.order_id, triage.urgency)

bad_replies = [
    LLMReply("claude", "Sorry, I can't help with that.", 300, 9, False),
    LLMReply("openai", '{"category": "refund_request", "urgency": 9, "draft_reply": "Hi"}', 300, 20, False),
    LLMReply("gemini", '{"category": "refund", "urgency": 4, "draft_re', 300, 1000, True),
]
for reply in bad_replies:
    try:
        parse_triage(reply)
    except LLMResponseError as error:
        print(reply.provider, "->", error)

claude -> refund CR-2210 4
openai -> refund CR-2210 4
gemini -> refund CR-2210 4
claude -> No JSON object in reply: "Sorry, I can't help with that."
openai -> Reply failed validation: category: Input should be 'late_delivery', 'refund', 'allergy' or 'other'; urgency: Input should be less than or equal to 5
gemini -> Reply was cut off (raise max_tokens or shorten the task)

Three good replies, three different failure modes, each named precisely. That’s the difference between a demo and something you can run on 2,300 emails.

What did that cost?

Meera will ask. Add a small helper to llm_utils.py:

# llm_utils.py (continued)


def estimate_cost(reply: LLMReply, input_per_million: float, output_per_million: float) -> float:
    """Dollar cost of one reply, given prices per million tokens."""
    return (reply.input_tokens * input_per_million + reply.output_tokens * output_per_million) / 1_000_000


def estimate_tokens(text: str) -> int:
    """Very rough: about 4 characters per token for English text."""
    return max(1, round(len(text) / 4))

Prices vary by provider and model, and change often, so always look them up on the provider’s pricing page. To see how the arithmetic works, suppose a model charges $3permillioninputtokensand$15 per million output tokens:

from llm_utils import estimate_cost, from_claude
from sample_responses import CLAUDE

reply = from_claude(CLAUDE)
cost = estimate_cost(reply, input_per_million=3.00, output_per_million=15.00)   # illustrative prices
print(f"One email:  ${cost:.4f}  (about ₹{cost * 88:.2f})")
print(f"400 emails: ${cost * 400:.2f}  (about ₹{cost * 400 * 88:.0f})")

One email:  $0.0018  (about ₹0.15)
400 emails: $0.70  (about ₹62)

(The rupee figures assume about ₹88 to the dollar.) Notice that output tokens cost five times more per token at these prices, so the 71 output tokens cost more than the 231 input tokens. Asking for a shorter draft is the cheapest optimisation there is. You’ll go much further with caching and model choice in Module 21.

Calling the real models

Now the one function that talks to a real provider. Each SDK is imported inside its branch, so you only need to install the ones you use (uv add anthropic, uv add openai, uv add google-genai). Add the last part to llm_utils.py:

# llm_utils.py (continued)
import os


def complete(provider: str, system: str, messages: list[dict], max_tokens: int = 1000) -> LLMReply:
    """Send one request to the chosen provider and return a normalised reply."""
    if provider == "claude":
        import anthropic
        client = anthropic.Anthropic()                       # reads ANTHROPIC_API_KEY
        response = client.messages.create(
            model="claude-opus-5-5", max_tokens=max_tokens, system=system, messages=messages,
        )
        return LLMReply(
            provider="claude",
            text="".join(b.text for b in response.content if b.type == "text"),
            input_tokens=response.usage.input_tokens,
            output_tokens=response.usage.output_tokens,
            truncated=response.stop_reason == "max_tokens",
        )
    if provider == "openai":
        from openai import OpenAI
        client = OpenAI()                                    # reads OPENAI_API_KEY
        resp = client.chat.completions.create(
            model=os.getenv("OPENAI_MODEL", "gpt-5-mini"),
            messages=[{"role": "system", "content": system}, *messages],
        )
        choice = resp.choices[0]
        return LLMReply(
            provider="openai",
            text=choice.message.content or "",
            input_tokens=resp.usage.prompt_tokens,
            output_tokens=resp.usage.completion_tokens,
            truncated=choice.finish_reason == "length",
        )
    if provider == "gemini":
        from google import genai
        client = genai.Client()                              # reads GEMINI_API_KEY
        user_text = "\n\n".join(m["content"] for m in messages)
        response = client.models.generate_content(
            model=os.getenv("GEMINI_MODEL", "gemini-2.5-flash"),
            contents=f"{system}\n\n{user_text}",
        )
        usage = response.usage_metadata
        return LLMReply(
            provider="gemini",
            text=response.text or "",
            input_tokens=usage.prompt_token_count,
            output_tokens=usage.candidates_token_count,
            truncated=False,                                 # kept simple here
        )
    raise ValueError(f"Unknown provider {provider!r}")

The SDK objects mirror the JSON shapes from earlier, with dots instead of square brackets. A few honest notes:

• Model names change. The OpenAI and Gemini model names are read from environment variables with a default, so you can switch models without editing code. Check each provider’s models page for the current list.
• For Gemini, we put the system prompt at the start of the content to keep the example minimal; Gemini also offers a dedicated system-instruction setting, and its SDK can report a finish reason, which we skip here.
• Library APIs change quickly. If a name has moved, check the docs linked below.

Everything downstream, parse_triage, estimate_cost, the CLI below, works the same whichever branch ran.

Letting the model fix its own mistake

When validation fails, the most effective fix is often to tell the model exactly what was wrong and ask once more. Because complete is passed in as a parameter (the “swappable” idea from the classes lesson), you can test this logic with a fake model that fails on purpose. Add one more function to llm_utils.py:

# llm_utils.py (continued)


def triage_email(email: str, complete_fn, order_context: str = "No matching order found.") -> Triage:
    """Triage one email, giving the model one chance to correct an unusable reply."""
    messages = build_messages(email, order_context)
    reply = complete_fn(SYSTEM_PROMPT, messages)
    try:
        return parse_triage(reply)
    except LLMResponseError as error:
        messages = messages + [
            {"role": "assistant", "content": reply.text},
            {"role": "user", "content": f"Your reply could not be used: {error}. Reply again with JSON only."},
        ]
        return parse_triage(complete_fn(SYSTEM_PROMPT, messages))   # a second failure raises

Then test it with a fake:

from llm_utils import LLMReply, triage_email
from sample_responses import ANSWER


class FlakyFakeModel:
    """Fails the first time, behaves the second. Lets us test without an API key."""
    def __init__(self):
        self.calls = 0

    def __call__(self, system, messages):
        self.calls += 1
        if self.calls == 1:
            return LLMReply("fake", '{"category": "refund_request", "urgency": 4, "draft_reply": "Hi"}', 300, 20, False)
        print("Second attempt was told:", messages[-1]["content"][:72], "...")
        return LLMReply("fake", ANSWER, 380, 71, False)


model = FlakyFakeModel()
triage = triage_email("My hamper arrived crushed. CR-2210", model)
print(triage.category, triage.order_id, "after", model.calls, "calls")

Second attempt was told: Your reply could not be used: Reply failed validation: category: Input s ...
refund CR-2210 after 2 calls

The fake is a class with a __call__ method, which lets its objects be called like functions: model(system, messages). To the workflow, it’s indistinguishable from complete. That’s how you test AI code without paying for, or waiting on, a real model.

The cleaner path: structured outputs

Everything above works with any model. But providers increasingly offer structured outputs: you hand over a schema, and the model’s reply is constrained to match it. With the Anthropic SDK, you can pass the Pydantic model directly:

import anthropic
from dotenv import load_dotenv

from llm_utils import SYSTEM_PROMPT, Triage, build_messages

load_dotenv()
client = anthropic.Anthropic()

response = client.messages.parse(
    model="claude-opus-5-5",
    max_tokens=2000,
    system=SYSTEM_PROMPT,
    messages=build_messages("My hamper arrived crushed. Order CR-2210. Refund please!"),
    output_format=Triage,
)
triage = response.parsed_output          # already a validated Triage object
print(triage.category, triage.urgency)

No extracting, no fences, no chatty preambles. The Triage class from the type hints lesson does double duty: its JSON Schema tells the model what to produce, and Pydantic checks the result. Module 05 covers structured outputs across providers in depth.

A command-line tool the team can use

The support team doesn’t run Python scripts with edits in the middle. They need a tool. A CLI (command-line interface: a program you run from the terminal with options) is the quickest way to share a workflow. Python’s built-in argparse module turns options into variables, and writes the --help text for you. Save this as triage.py, next to llm_utils.py:

# triage.py
"""Triage a customer email and draft a reply. Run with --help for options."""
import argparse
import sys
from pathlib import Path

from dotenv import load_dotenv

from llm_utils import SYSTEM_PROMPT, LLMResponseError, build_messages, complete, estimate_tokens, triage_email


def main() -> int:
    parser = argparse.ArgumentParser(description="Triage a customer email with an LLM.")
    parser.add_argument("email_file", type=Path, help="text file containing the email")
    parser.add_argument("--provider", choices=["claude", "openai", "gemini"], default="claude")
    parser.add_argument("--dry-run", action="store_true",
                        help="print the prompt and a token estimate without calling a model")
    args = parser.parse_args()

    email = args.email_file.read_text(encoding="utf-8")

    if args.dry_run:
        user_message = build_messages(email)[0]["content"]
        print(user_message)
        print(f"\n~{estimate_tokens(SYSTEM_PROMPT + user_message)} input tokens (rough estimate)")
        return 0

    load_dotenv()
    try:
        triage = triage_email(email, lambda system, messages: complete(args.provider, system, messages))
    except LLMResponseError as error:
        print(f"Could not triage this email: {error}", file=sys.stderr)
        return 1

    print(f"Category: {triage.category} | Urgency: {triage.urgency} | Order: {triage.order_id or '-'}")
    print(f"\nDraft reply:\n{triage.draft_reply}")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Try the free path first. Put Ananya’s email in email.txt and run:

uv run triage.py email.txt --dry-run

Customer email:
<email>
Hi, my Diwali hamper (order CR-2210) arrived with the box crushed and two kaju katli broken. I'd like a refund. - Ananya
</email>

Order details from our database:
No matching order found.

~184 input tokens (rough estimate)

The real count the provider reports will differ a little from this estimate (our test run reported 231 for a slightly longer prompt with order details), but it’s close enough to catch a prompt that has accidentally grown to 20,000 tokens.

Then for real, once your .env has a key: uv run triage.py email.txt --provider claude. And uv run triage.py --help prints a usage guide built from your help= strings.

What’s new in this file:

• argparse.ArgumentParser collects the options. A plain name ("email_file") is a required positional argument; a name starting with -- is optional. type=Path converts the text to a Path; choices= rejects unknown providers with a helpful message; action="store_true" makes --dry-run a yes/no switch.
• if __name__ == "__main__": runs main() only when the file is run directly, not when another module imports it. __name__ is "__main__" for the file you ran.
• raise SystemExit(main()) ends the program with main’s return value as its exit code: 0 for success, anything else for failure. Other programs and scripts check exit codes to know whether a command worked.
• Errors go to sys.stderr, the error stream, so they stay separate from normal output if someone saves the output to a file.
• The --dry-run path calls no model and needs no key. Every AI tool should have one: it makes the prompt reviewable and the tool safe to try.

The final layout is small, and every piece is reusable. Next month, when Arjun builds a web endpoint for the same workflow, it will import triage_email from llm_utils.py too.

crumb-inbox/
├── .env               # API keys (never committed)
├── llm_utils.py       # adapters, prompts, parsing, cost, complete(), triage_email()
├── sample_responses.py
└── triage.py          # the CLI: argparse + calls into llm_utils

Where it breaks

The other failure modes Arjun meets in the first month:

• Braces in templates. str.format treats { and } as slots. A template containing a JSON example, like {"category": ...}, raises KeyError when formatted. Double literal braces ({{ and }}), or keep JSON examples in the system prompt, which isn’t formatted.
• Truncation that looks like bad JSON. If max_tokens is too low for long drafts, replies get cut off. Always check the stop reason before parsing, as parse_triage does.
• Silent provider drift. A model update changes the style of replies: more preamble, different category wording. Your validation catches the format changes; a fixed set of test emails, run after every change, catches the quality changes. That’s the evaluation work of Module 23.
• Costs growing in the background. Retries double the cost of failed emails, and longer prompts cost on every call. Log input_tokens and output_tokens for every call, from day one.
• Personal data in prompts. Emails contain names, phone numbers and addresses. Send only what the task needs, and check your provider’s data-retention terms before sending customer data.

Arjun explains it to Meera

A week later, Arjun has run the same 200 test emails through all three providers using one command per provider. Same prompt, same parser, same validation. He shows Meera a small table: categories correct, drafts the support team rated “send as is”, and cost per 1,000 emails.

“How did you compare them so fast?” she asks.

“Each company’s AI sends its answer back in a differently shaped box,” Arjun says. “Before, my code reached into one particular box in eleven different places, so switching meant rewriting everything. Now there’s one unpacking station at the door that turns any box into our own standard tray. After that, the rest of the app doesn’t know or care whose AI it was. The instructions live in one template, so all three got exactly the same question. And every answer goes through the same checker, so a reply in the wrong format is caught and retried once, instead of reaching the team.”

“And what does it cost?”

“Every reply reports how many tokens it used, so we know to the paisa. The current numbers are in the table. When prices change, I change two numbers.”

Try it yourself

Add a fourth provider, a local model (a model running on your own computer, which you’ll set up in Module 20), without changing parse_triage or triage_email. Its raw response looks like this:

LOCAL = {
    "model": "llama3.2",
    "message": {"role": "assistant", "content": '{"category": "allergy", "order_id": null, "urgency": 5, "draft_reply": "Thank you for checking. Our plum cake contains walnuts, so please avoid it."}'},
    "done_reason": "stop",
    "prompt_eval_count": 290,
    "eval_count": 41,
}

1. Write from_local(raw) -> LLMReply. The text is in message.content, input tokens are prompt_eval_count, output tokens are eval_count, and a done_reason of "length" means truncated.
2. Pass the result to parse_triage and print the category, urgency and whether order_id is None.

Expected:

local allergy 5 True

from llm_utils import LLMReply, parse_triage

LOCAL = {
    "model": "llama3.2",
    "message": {"role": "assistant", "content": '{"category": "allergy", "order_id": null, "urgency": 5, "draft_reply": "Thank you for checking. Our plum cake contains walnuts, so please avoid it."}'},
    "done_reason": "stop",
    "prompt_eval_count": 290,
    "eval_count": 41,
}


def from_local(raw: dict) -> LLMReply:
    return LLMReply(
        provider="local",
        text=raw["message"]["content"],
        input_tokens=raw["prompt_eval_count"],
        output_tokens=raw["eval_count"],
        truncated=raw["done_reason"] == "length",
    )


reply = from_local(LOCAL)
triage = parse_triage(reply)
print(reply.provider, triage.category, triage.urgency, triage.order_id is None)

local allergy 5 True

Common mistakes

• Reaching into raw provider responses all over the app. Normalise once, in an adapter, and pass your own type around.
• json.loads on raw model text. Models add fences and chatter. Extract, then validate.
• Ignoring the stop reason. A truncated reply is a different problem from a badly formatted one, with a different fix.
• Prompts scattered through the code. Keep them as named templates in one module, so they can be reviewed, versioned and tested.
• No dry-run or fake model. If every test costs money and needs a network, you’ll test less. Make the model swappable.

Recap

• Claude, OpenAI and Gemini return the same kind of answer in different shapes; write one adapter per provider that converts replies into your own dataclass, and use only that everywhere else.
• Prompts are code: a fixed system prompt plus a user template with named slots, filled by a function you can print and test.
• Model output is untrusted text: check the stop reason, extract the JSON, validate it with Pydantic, and give the model one clear chance to correct itself.
• Every reply reports input and output tokens; multiply by the provider’s current prices to track cost per call.
• Package the workflow as small reusable functions, then wrap them in a CLI with argparse, including a --dry-run that calls no model.

Meera loves the comparison table. Then she asks a different kind of question. “We have six months of orders. Which store sells the most plum cake per day? Do rainy days hurt sales? Could the bot predict tomorrow’s croissant demand?” Arjun opens the export: 52,000 rows. His trusty for loops crawl. “Time to meet the tools data scientists use,” Priya says. “And the idea underneath every neural network: arrays of numbers.”

Go deeper

• Anthropic documentation: the Messages API, response format and structured outputs.
• OpenAI documentation: Chat Completions and the newer Responses API.
• Gemini API documentation: generate_content and response structure.
• argparse tutorial in the Python docs.