Choosing a model: open vs closed, big vs small

Crumb’s AI pilot is going well enough that Meera wants a plan for real. She lists three jobs on the whiteboard:

1. Sort the inbox. Label every incoming email (late delivery, allergy, refund, other) so the right person sees it first. About 9,000 emails a month.
2. Draft replies. Write a first draft for each email, grounded in the order record, for the team to review.
3. WhatsApp ordering. Chat with customers in real time, in English, Hindi and Marathi. Customers on phones expect an answer within two or three seconds.

“Which model do we use?” she asks. “There seem to be hundreds. And Arjun, before you answer, remember I have to pay for it.”

Arjun has already decided. He found a public leaderboard that ranks models by test scores, and picked the one at the very top. He wired it into all three jobs. “Best model, best results. Done.”

Then the first week’s numbers come in. The inbox sorter alone, a job that outputs a single word per email, is on track to cost more than Arjun had budgeted for the whole pilot. The WhatsApp bot takes six to nine seconds to reply, because the top model thinks carefully before every message, including “Hi”. And Kavita from support flags something else: a draft had to be thrown away because the leaderboard champion, given a Marathi email, replied in formal Hindi.

Priya draws a triangle on the whiteboard and writes quality, speed and price at its corners.

Arjun’s leaderboard choice, and what it cost

Priya asks Arjun to price the inbox sorter with the champion model. Labelling one email takes about 600 input tokens (instructions plus the email) and about 150 output tokens (a one-word label, plus the model’s hidden thinking, which you pay for at output prices). At the champion’s price of $10permillioninputtokensand$50 per million output tokens:

	Per email	Per month (9,000 emails)
Input: 600 tokens × $10 per million	$0.0060	$54.00
Output: 150 tokens × $50 per million	$0.0075	$67.50
Total	$0.0135	$121.50 (about ₹10,700)

“For writing one word,” says Priya. “How hard is the job? Could a human do it on their first day?”

“Easily. Most emails say what they are about in the first line.”

“So why hire the head pastry chef to sort the post?”

Hiring for the job, not for the title

Crumb has a head pastry chef who builds three-tier wedding cakes, an experienced counter team, and a teenager on weekends who packs boxes. You would not pay the pastry chef to pack boxes, and you would not ask the weekend helper to design a wedding cake.

“Model choice is a hiring decision,” says Priya. “For each job, ask three questions. How good does it need to be? How fast? How much can each one cost? Then you hire the cheapest one that clears the bar.”

“Why not just go for the best on all three?”

“Because the triangle is real. A model that is smarter is usually bigger and does more thinking, which makes it slower and more expensive. A model that is very fast and cheap is usually smaller and makes more mistakes on hard tasks. You can push one corner, but it pulls on the others.”

Where the analogy breaks: hiring a person is slow and switching is painful. Switching models is one line of code. That makes it cheap to try several and measure, which is exactly what you should do.

What actually differs between models

Here are the dimensions that matter, starting with the one that most affects the triangle.

Size. A model’s size is the number of parameters (the internal numbers that store what it learned). More parameters generally mean more knowledge and better reasoning, but each token takes more computation, so bigger models are slower and pricier. Providers usually sell families in tiers. Anthropic has Haiku (small, fast), Sonnet (middle) and Opus (large); OpenAI has full, “mini” and “nano” sizes; Google has Gemini Pro, Flash and Flash-Lite. At Anthropic’s prices at the time of writing, the spread is enormous:

Claude model	Input, per million tokens	Output, per million tokens
Claude Haiku 5.5	$0.10	$0.50
Claude Sonnet 5.5	$2.00	$10.00
Claude Opus 5.5	$4.00	$20.00

The largest tier costs forty times as much as the smallest, per token. Prices change often; always check the provider’s pricing page.

Reasoning (thinking) models. Some models write hidden reasoning before answering. That makes them much better at multi-step problems (a tricky refund case, debugging code, planning) and slower and costlier on everything, because thinking tokens are billed as output. Many let you control how much they think: Claude has an effort setting, OpenAI’s reasoning models have reasoning_effort, Gemini has a thinking budget. For “label this email”, low effort is plenty. For “work out what went wrong across this customer’s three orders”, more thinking pays off.

Context window. The maximum tokens per request. Most current large models handle hundreds of thousands to a million tokens, so this rarely decides things for Crumb, but small and local models can be much more limited.

Inputs and outputs it supports. Can it read images (a photo of a squashed cake)? PDFs? Audio? Can it call tools? Check before you choose.

Languages. Models differ in how well they handle Indian languages. The only way to know for Marathi is to test Marathi.

Open weights versus closed

There is one more split, and it is about control.

A closed model (sometimes called closed-weights or proprietary) is one whose parameters stay private: you can only use it through the provider’s API. GPT, Gemini and Claude are closed.

An open-weights model is one whose parameters are published for anyone to download and run on their own hardware. Meta’s Llama, Alibaba’s Qwen, Mistral’s models, Google’s Gemma and DeepSeek’s models are well-known examples. (The licences vary; “open weights” is not always “open source” in the full sense.)

	Closed (via API)	Open weights (self-hosted)
Top-end quality	Usually the frontier	Close behind, and improving fast
Setup	An API key	GPUs, serving software, maintenance
Cost model	Per token	Per hour of hardware, whether busy or idle
Your data	Sent to the provider	Stays on your machines
Customisation	Limited	Full: you can fine-tune it
Size you can run	Any	What your hardware fits

Size bites hard when you run models yourself. A 405-billion-parameter open model stored at 2 bytes per parameter needs about 810 GB of GPU memory before it answers a single question, which means a rack of top-end GPUs. A 3-billion-parameter model, compressed, fits in a few gigabytes and runs on a laptop. You will run open models locally with Ollama in Module 20, and on Hugging Face in Module 11.

Benchmarks, and why Arjun’s leaderboard misled him

A benchmark is a fixed set of test questions with known answers, used to compare models: school-style exams across many subjects, competition maths, real-world coding tasks, and so on. Public leaderboards rank models by these scores, and some rank them by people voting between two anonymous answers.

They are genuinely useful for a first shortlist. But they misled Arjun in three ways.

1. They test somebody else’s job. A top score on competition maths says little about labelling Crumb’s emails or writing in Marathi.
2. They saturate and leak. Once a benchmark is famous, its questions can end up in training text, and models get tuned towards it. Scores rise faster than real-world ability.
3. They ignore your other two corners. A leaderboard rarely shows price per task or seconds per reply for your inputs.

The answer is your own eval (short for evaluation): a small set of real examples from your own work, each with the answer you consider correct, that you run every candidate model against. Twenty to fifty examples is enough to start. Module 23 is entirely about building good evals; here you will build the simplest useful one.

Following one decision: the inbox sorter bake-off

Kavita picks 20 real emails from last month (with names and phone numbers removed) and writes the correct label for each. Arjun runs all 20 through three Claude tiers at low effort, recording accuracy, time per email and tokens. His results:

Model	Correct labels	Average time per email	Avg tokens in / out	Cost per 1,000 emails
Claude Haiku 5.5	19 / 20	0.8 s	600 / 150	$0.14
Claude Sonnet 5.5	20 / 20	1.9 s	600 / 150	$2.70
Claude Opus 5.5	20 / 20	3.1 s	600 / 150	$5.40

These are Arjun’s numbers on his twenty emails; yours will differ, which is the whole point of running your own. Here is how the cost column is calculated, for Haiku: 600 × $0.10+150\times$0.50 = $60+$75 = $135permillionemail{s}^{\prime }worthoftokens,whichis$0.000135 per email, or $0.14 per thousand.

Now look at the one Haiku got wrong. The email said: “Cake was fine but the delivery guy was rude and also can I get the eggless version next time?” Kavita had labelled it other. Haiku said late_delivery. Opus said other.

“Is that one mistake worth paying forty times more?” asks Priya.

Arjun thinks about it. “A wrong label means an email lands in the wrong person’s queue for a few hours. Nobody gets hurt. At 9,000 emails a month, Haiku costs about $1.22against$48.60 for Opus.”

“And the drafts?”

“Different job. A bad draft could promise the wrong thing to a customer. And there are fewer of them, because only some emails get drafts. I’d run the same bake-off on drafts, and I expect the bigger model to earn its price there.”

Where your data goes

Meera has one more question, and it is the one that can overrule all the others. “These emails have customers’ names, phone numbers and addresses. Where does that information go?”

When you call a hosted model, your request travels to the provider’s servers. Before choosing a provider for real customer data, check four things in their current terms:

• Training. Will they use your data to improve their models? Paid API tiers from the major providers generally say no by default. Free tiers may say yes, which is why the setup lab warned you to use made-up data there.
• Retention. How long do they keep requests, and why? Many keep data for a limited period to detect abuse. Some offer zero-data-retention agreements for business customers.
• Location. Where are the servers? Some industries and countries require data to stay in a particular region.
• Your own obligations. India’s data protection law and your own promises to customers apply whatever the provider says. Send only what the model needs: the order record, not the customer’s full history.

If the data must never leave your building, open-weights models on your own hardware are the answer, with the costs described above.

Opening the hood: the decision, as a flowchart

Priya turns the discussion into a process Arjun can repeat for every new job:

flowchart TD
  A["New job for an LLM"] --> B{"Can this data go to a hosted API?"}
  B -->|No| C["Shortlist open-weights models you can host"]
  B -->|Yes| D["Shortlist two or three models across sizes"]
  C --> E["Build an eval: 20 to 50 real examples with correct answers"]
  D --> E
  E --> F["Run every candidate: accuracy, time, cost"]
  F --> G["Pick the cheapest model that clears your quality and speed bar"]
  G --> H["Re-run the eval when models or prices change"]

The last box matters. Models are released and retired every few months, and prices often drop. A choice that was right in March can be wrong by September. Because your model names live in configuration (remember OPENAI_MODEL and GEMINI_MODEL in .env), switching is cheap, and your eval tells you whether the switch is safe.

The real code: your first bake-off

Here is a small eval harness. It runs labelled examples through several models and prints accuracy, average time and cost. Use your own examples; these four are placeholders to get you started.

Step 1: the examples and the candidates.

# bakeoff.py
import time
import anthropic
from dotenv import load_dotenv

load_dotenv()
client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY

EXAMPLES = [
    ("Where is my order CR-20931? It was due yesterday.", "late_delivery"),
    ("Does the walnut brownie contain peanuts? My daughter is allergic.", "allergy"),
    ("The cake arrived squashed, I want my money back.", "refund"),
    ("Loved the plum cake! Do you ship to Bengaluru?", "other"),
]

# (model, input $ per million, output $ per million) at the time of writing
CANDIDATES = [
    ("claude-haiku-5-5", 0.10, 0.50),
    ("claude-sonnet-5-5", 2.00, 10.00),
    ("claude-opus-5-5", 4.00, 20.00),
]

SYSTEM = ("Classify the customer email. Reply with exactly one label from: "
          "late_delivery, allergy, refund, other. No other words.")

Step 2: run one model over all examples.

def run(model: str, price_in: float, price_out: float) -> None:
    correct, seconds, dollars = 0, 0.0, 0.0
    for email, expected in EXAMPLES:
        start = time.perf_counter()
        r = client.messages.create(
            model=model,
            max_tokens=2000,
            output_config={"effort": "low"},
            system=SYSTEM,
            messages=[{"role": "user", "content": email}],
        )
        seconds += time.perf_counter() - start
        label = next((b.text for b in r.content if b.type == "text"), "").strip().lower()
        correct += label == expected
        dollars += (r.usage.input_tokens * price_in + r.usage.output_tokens * price_out) / 1_000_000
    n = len(EXAMPLES)
    print(f"{model:20} {correct}/{n} correct  {seconds / n:.1f}s avg  "
          f"${dollars / n * 1000:.2f} per 1,000 emails")

for model, p_in, p_out in CANDIDATES:
    run(model, p_in, p_out)

You should see one line per model, something like:

claude-haiku-5-5     4/4 correct  0.7s avg  $0.11 per 1,000 emails
claude-sonnet-5-5    4/4 correct  1.6s avg  $2.31 per 1,000 emails
claude-opus-5-5      4/4 correct  2.6s avg  $4.95 per 1,000 emails

With four easy examples, every model scores perfectly, which tells you nothing about quality. That is the most important output of this script. Replace the examples with 20 or more real, messy ones from your own work, including the ambiguous cases and other languages, and the differences appear.

Step 3 (optional): add other providers. Add a branch that calls OpenAI or Gemini with the same system prompt, using the adapter from the anatomy lesson, and compare across providers too.

Library APIs change quickly. If a name has moved, check the docs linked below.

Where it breaks in production

• Choosing by leaderboard. It tests someone else’s job and ignores price and speed. Use it to shortlist, then run your own eval.
• One model for everything. Different jobs have different triangles. Most real systems use two or three models.
• Testing on easy examples. If every model scores 100%, your eval is too easy to tell them apart. Include the hard, ambiguous, multilingual cases.
• Ignoring thinking costs. A reasoning model at high effort can spend more on thinking than on the answer. Measure output tokens, not only answer length.
• Hard-coding model names. Models get retired. Keep names in configuration, and re-run your eval when you switch.
• Forgetting the data question. Quality, speed and price are irrelevant if the data is not allowed to go there.

Try it yourself

1. Run bakeoff.py with ten examples of your own, including two that are genuinely ambiguous and one in Hindi or Marathi.
2. Run it again with "effort": "medium" instead of "low". What changes in time, cost and accuracy?
3. Work out Meera’s monthly bill for the inbox sorter on each model, at 9,000 emails a month, using your per-1,000 figures.

Show what you should expect

1. The ambiguous examples are where models disagree, and where you may find that even you are unsure of the “right” label. That is useful: it tells you the instructions need a clearer rule for those cases.
2. Medium effort usually means more output tokens (more thinking), so higher cost and slower replies. Accuracy may improve on the hard examples and stay the same on easy ones. If accuracy does not change, low effort is the better deal.
3. Multiply each per-1,000 cost by 9. With Arjun’s figures: Haiku about $1.22,Sonnetabout$24.30, Opus about $48.60 a month.

Common mistakes

• “The biggest model is always best.” It is often the slowest and most expensive, and for simple jobs no more accurate.
• “Benchmarks tell me which model to use.” They give you a shortlist. Your eval gives you the answer.
• “Open weights are free.” The weights are free; the GPUs, power and engineering time are not.
• Testing two or three examples. Small evals hide real differences. Twenty to fifty real cases is a good start.
• Sending customer data to a free tier. Check training, retention and location terms before any real data goes anywhere.

Recap

• No model is best at everything; pick per job by trading off quality, speed and price.
• Bigger and thinking models are smarter but slower and costlier; small models are fast, cheap and often good enough.
• Closed models are used through an API; open-weights models can be self-hosted, with your data staying in-house.
• Benchmarks shortlist; your own eval, on your real examples, decides.
• Data privacy rules can overrule every other factor.

Arjun brings Meera a one-page plan. “The inbox sorter uses the smallest Claude model: nineteen out of twenty in our test, about a dollar a month. Drafts use Opus, because mistakes there reach customers, and only about a third of emails need one. WhatsApp uses a fast model at low effort, because people are waiting. Everything runs on a paid tier that does not train on our data, and we re-test every quarter.”

Meera reads it twice. “This I can sign.” Then she taps the WhatsApp line. “So when does the WhatsApp bot actually exist? I have customers asking.”

“That,” says Priya, “is the next module. Arjun is about to build his first real chatbot.”

Go deeper

• Anthropic models overview and pricing: https://docs.anthropic.com
• OpenAI models and pricing: https://platform.openai.com/docs
• Gemini models and pricing: https://ai.google.dev/gemini-api/docs
• Hugging Face, for exploring open-weights models: https://huggingface.co/docs