Staying current without drowning

The meeting room above the Koregaon Park store smelled of cardamom buns. Meera sat at the head of the table, with all six store managers, Raju from the central kitchen, Priya, and Tanvi on her third day. Arjun plugged in his laptop. His hands were steady, which surprised him.

He started with Kavya’s four browser tabs, then the numbers. Forty-one staff used Crumb Copilot. Last month they asked it 3,200 questions, at about ₹2.4 each: under ₹8,000 in model costs. The median first reply to a support email had gone from nine hours to two. In the two pilot stores, 81% of Google reviews now got a reply within 48 hours, up from 22%. It passed 18 of 20 questions on its eval set, and he showed the two it failed, and why.

The Wakad manager asked the question Arjun had been waiting for. “What if it tells my staff something wrong?”

He typed her question live. The answer came back with handbook p. 112 under it. “Then you can see where it came from, and check. And if the handbook doesn’t say, it tells you that instead of guessing. That was the hardest part to build.”

Meera let the room go quiet, then asked the last question. “Everyone tells me AI changes every week. Will all this be out of date by Diwali?”

“Some of it will,” Arjun said. “The important parts won’t. And we’ll know which is which.”

He believed it when he said it. The next morning tested that belief.

The morning after the demo

At 7 a.m., Arjun’s phone was already full. A lab he had barely heard of had released a new model, claiming the best scores ever on three reasoning benchmarks. A new agent framework had gained 30,000 GitHub stars in nine days. His feed was split between “RAG is dead” and “agents are dead”, sometimes from the same people. Someone had posted a chart showing his current model in fourth place.

He felt, all at once, behind.

So he did what felt responsible. He spent Saturday migrating Crumb Copilot to the new framework and switching to the new model. The code got shorter. The demo looked slick. On Monday morning, he ran the eval out of habit.

Fifteen of twenty. Down from eighteen. Time to first word: 3.4 seconds, up from 2.7. Worse, he could not easily see what prompt the framework was sending to the model, so he could not tell why three questions had broken. And on Sunday night, the framework had published a new version that renamed two of the functions he had just learned.

That evening he counted his subscriptions: 23 newsletters, four podcasts, three Discord servers, and 140 “read later” links he would never read.

The trade fair and the test loaf

He told Priya the whole story. She did not laugh, which he appreciated.

“Every year, Raju goes to the bakery trade fair in Mumbai,” she said. “Hundreds of new ovens, mixers, proofers. Each stall says theirs is the best in the world. How many does he buy?”

“Usually none. Once, a new deck oven.”

“How did he decide on that one?”

“He took a tray of our sourdough dough with him. Asked them to bake it, right there. Compared it with ours at home: crust, crumb, how long, how much gas.” Arjun smiled. “Then he asked two other bakers who’d owned one for a year.”

“So the stall’s brochure…”

“…was the least important thing. The test loaf was the most important.”

“Your eval set is your test loaf,” Priya said. “Twenty real questions, from real staff, with known good answers. Every new model, every new framework, bakes your loaf before it gets near your kitchen. You did it backwards: bought the oven, then baked the loaf.”

“And the twenty-three newsletters?”

She drew a river and a small bucket. “The field is the river. Nobody drinks a river. You fill a bucket at the same time each week, from the same clean spot, and you drink that.”

Two misconceptions came apart in that conversation. The first: being current means knowing about every release. Nobody does that, including the people who seem to; the engineers you admire know what lasts deeply and check the rest on a schedule. The second: the newest model is the best one for you. Benchmarks measure someone else’s loaf. Only your eval measures yours.

What changes fast, and what barely moves

Priya’s first suggestion was to sort what Arjun knew into two piles.

Changes every few months	Has barely changed in years
Model names, rankings and prices	Tokens, context windows and why models forget
Framework APIs and import paths	Embeddings, similarity and retrieval
Which vendor has which feature	Chunking, hybrid search and reranking trade-offs
Benchmark leaderboards	How to build and trust an eval set
Context window sizes	The agent loop: think, act, observe
Today’s hot pattern names	Prompt injection and least-privilege tools
Exact cost per token	Thinking in cost per task, latency and value

The right-hand column is most of this course. What you learned in Context windows and why LLMs forget, What are embeddings? and Why RAG exists was true before most of today’s models existed and will hold for the next ones. The left-hand column is a lookup problem: check the docs when you need it.

This is also what the “X is dead” posts miss. When context windows grew, people declared RAG dead. But retrieval was never only about fitting text in a window; it was about cost, speed and finding the right page, as Crumb Copilot’s ₹37-to-₹2.4 story showed. The left column changed. The right did not.

Show the answer

(b) and (d). The import path (a) may change in months, and you will look it up when you need it. The price © changes too, usually downwards; what lasts is the habit of computing cost per task, not the number. (b) and (d) are ideas about how these systems work and how to measure them, and they transfer to every model and framework you will ever use.

Following one new model through Arjun’s test kitchen

The following week, Priya and Arjun set up a routine for exactly this moment. Here is how they evaluated the new model, step by step, with the real numbers.

Step 1: what problem would it solve for us? Before anything else, Arjun wrote one line: “Copilot is good enough on quality; we’d switch for lower cost or faster answers, but not for worse quality on policy questions.” That sentence decides what to measure, and what “better” means.

Step 2: read the model card. A model card is the document a model’s maker publishes describing what it is, how it was tested, its limits, price and data policies. Ten minutes told him the price (a little over half the current model’s per question), a data-retention policy to check with Meera, and that the headline benchmarks were maths and coding, not document question-answering.

Step 3: bake the test loaf. He ran Copilot’s golden dataset (the fixed set of real questions with known good answers that every change is tested against) against both models: same prompts, same retrieval, only the model changed.

# compare_models.py: the same golden set, two models, side by side
import json, statistics, time
import anthropic

client = anthropic.Anthropic()
SYSTEM = open("system_prompt.txt").read()

def call_model(model: str, question: str) -> tuple[str, int, int]:
    """Swap this for your multi-provider gateway to compare models from different companies."""
    r = client.messages.create(model=model, max_tokens=1000, system=SYSTEM,
                               messages=[{"role": "user", "content": question}])
    text = next((b.text for b in r.content if b.type == "text"), "")
    return text, r.usage.input_tokens, r.usage.output_tokens

def run(model: str, cases: list[dict]) -> dict:
    passed, seconds, tokens = 0, [], []
    for case in cases:
        start = time.perf_counter()
        text, tin, tout = call_model(model, case["q"])
        seconds.append(time.perf_counter() - start)
        tokens.append((tin, tout))
        ok = case["must_include"].lower() in text.lower()
        passed += ok
        if not ok:
            print(f"  FAIL [{model}] {case['q'][:60]}")
    return {"model": model, "passed": f"{passed}/{len(cases)}",
            "median_seconds": round(statistics.median(seconds), 1), "tokens": tokens}

cases = [json.loads(line) for line in open("golden_set.jsonl")]
for model in ["claude-opus-5-5", "the-new-model-name"]:
    print(run(model, cases))

The call_model function is the only place a model name appears. If the new model comes from a different provider, you swap in the gateway you built in A multi-provider LLM gateway with fallback, and nothing else changes.

First results:

	Current model	New model
Golden set	18 / 20	17 / 20
Cost per question	about ₹2.4	about ₹1.4
Median time to answer	4.1 s	2.6 s

Step 4: read every failure. This is the step most people skip. The numbers said “slightly worse”. The failures said something more useful. Two of the new model’s three failures were formatting: it wrote “p112” instead of “[handbook p. 112]”, so the check missed them. One extra line in the prompt fixed both. The third was real and serious: it told a staff member the late-delivery refund was 25%, when the handbook says 20%.

Step 5: rerun, because twenty questions is a small sample. A difference of one question in twenty is within the noise; run the same set three times and the scores wobble. Arjun ran each model three times with the fixed prompt. Current model: 18, 18, 17. New model: 19, 18, 18. Effectively a tie on quality, at a little over half the cost and noticeably faster. But the refund error appeared in two of the three runs.

Step 6: decide, with a rule written in advance. They did not switch everything. They added three more refund-policy questions to the golden set (every real failure should become a permanent test), switched the review-reply drafts to the new model straight away (low-risk, green-square work from AI product thinking), and kept the current model for policy questions. They agreed to recheck in a month.

The cost of all this: 20 questions × 2 models × 4 runs is 160 calls, about ₹300 and 40 minutes including reading the failures. The Saturday migration had cost a whole day and a broken eval.

Opening the hood: benchmarks, model cards and your golden set

Three words carry this lesson, so it is worth being precise about them.

A benchmark is a public, standardised test for models: a fixed set of tasks (maths problems, coding challenges, exam questions) with a scoring method, so models can be compared. Useful for a rough sense of strengths, with three well-known limits:

• Task mismatch. A benchmark of competition maths says little about answering questions from a bakery handbook in Hinglish.
• Contamination. Benchmark questions are public. If they leak into a model’s training data, the model can score well partly by memory.
• Saturation. When top models all score 95% or more, small differences between them are mostly noise.

A model card is the maker’s own description of a model. Read it for facts (price, limits, data policies), and read its benchmark claims as the brochure at the trade fair stall.

Your golden dataset is the only one of the three that measures your problem: real questions with checked answers, growing every time something breaks in production. It is the eval idea from Binary pass/fail evaluations, used for a new purpose: deciding whether to change anything at all.

Show the answer

Not without testing on Crumb’s own data. The leaderboard’s tasks are not Crumb’s handbook, with its product names like “kaju katli box” and its mix of English and Marathi. And switching embedding models is not a small change: vectors from different models cannot be compared with each other, so all 4,000 chunks must be re-embedded, and every query too. The right move: take 50 real staff questions with known correct chunks, measure how often the right chunk lands in the top 5 with each model, and switch only if the gain is clear. The leaderboard tells you which models are worth testing, not which to use.

Judging a new framework or tool

New frameworks are like new ovens with exciting features and no track record. Before adopting one, Priya and Arjun now ask six questions:

1. What problem of ours does it solve? If you cannot name one, stop. “Everyone is using it” is not a problem.
2. Can I see what it sends to the model? Hidden prompts cannot be debugged or traced (Tracing model calls, tools and retrieval).
3. How stable is it? Frequent breaking releases mean rewriting your code on its schedule, as Arjun did on Sunday night.
4. Who maintains it? Read a few open issues. Are they answered?
5. What would it cost to leave? Plain Python calling a small part of it is easy to leave; an app shaped around it is a rewrite.
6. Does it beat my test loaf? Try it on a branch, on your golden set, with a two-hour limit. Not clearly better in two hours? Not worth a migration.

The framework Arjun tried that Saturday failed questions 2 and 3. That did not make it bad; it made it wrong for Crumb, for now.

Spotting hype without becoming cynical

Some announcements matter a great deal. The goal is not to dismiss everything, which is as lazy as believing everything, but to look for evidence.

Be careful with: “X is dead” headlines; benchmark charts with no method; demos that show only successes; star counts offered as proof of quality; predictions with dates; and anyone selling the thing they are excited about.

Pay attention to: evaluations you could rerun; honest limitations; clear pricing; independent practitioners reporting results on their own tasks weeks later; and anything that touches the right-hand column of the table above.

A useful rule: when something seems important, wait two weeks before acting. By then the hype has settled, independent people have tried it, and the first bugs are fixed. You will rarely regret waiting.

A habit you can keep for years

Here is the routine Arjun replaced his 23 newsletters with. It takes about 30 minutes a week and two hours a month.

Weekly, 30 minutes, same time every week (his is Friday, 4 p.m.):

• One or two curated sources you trust. A practitioner’s blog or digest that explains why things matter, not just that they happened.
• The changelogs and release notes for the providers and libraries you actually use. These matter more than any newsletter, because they affect your code directly.
• Add anything interesting to a “try later” list. Do not try it now.

Monthly, two hours, hands-on:

• Pick one item from the “try later” list and actually build something small with it. Reading about a tool and using it are different kinds of knowing.
• Run your golden sets against any new model that seems relevant to your problem.

Quarterly, one hour: delete half the “try later” list (most of it no longer matters, which is the point), and note what changed in the right-hand column, if anything.

Always: keep a learning log, a plain text file with a dated sentence or two about anything you learned. Over a year it becomes a record of how far you have come, material for posts and interview stories, and a cure for feeling behind.

Why this works: depth compounds and breadth evaporates. A week understanding why retrieval fails improves every future RAG system; a week skimming 40 announcements leaves almost nothing.

Try it yourself

Set up your own system this week. It takes about two hours, and then 30 minutes a week.

1. Choose your sources. At most three, plus the changelogs of the two or three providers and libraries you use most. Unsubscribe from the rest.
2. Build a golden set for one of your projects. Twenty questions from real use (or realistic ones if you have no users yet), each with the fact the answer must contain, in golden_set.jsonl. Include two questions it should decline.
3. Write compare_models.py as above, with call_model as the only place a model name appears. Run it against two models you can access. Record pass rate, median time and cost per question.
4. Read every failure and write one sentence about each: format problem or real error?
5. Block 30 minutes a week in your calendar, and create learning-log.md with today’s first entry.

Expected result: a small table comparing two models on your task, a note on what each failure was, and a calendar entry that will still be there next month.

Show an example learning-log entry

12 Nov. Ran the Copilot golden set (20) against the new smaller model. 17/20 first run; two failures were citation format, fixed with one prompt line; then 18/20, 19/20, 18/20. Half the cost. One real error on refund percentage in 2 of 3 runs. Lesson: always rerun at least 3 times; one-question differences are noise. Added 3 refund questions to the golden set.

Common mistakes

• Migrating first, measuring second. Bake the test loaf before buying the oven.
• Trusting benchmarks over your own eval. Benchmarks tell you what to test, not what to choose.
• Reading one run of a small eval as truth. Rerun it and read the failures before deciding.
• Following everything. Fewer, better sources, on a schedule.
• Swinging to cynicism. “It’s all hype” is as lazy as “it changes everything”. Look for evidence, then test.

Recap

• Sort knowledge into what changes fast (names, prices, APIs) and what lasts (tokens, retrieval, evals, agents, security, product thinking). Learn the second deeply; look up the first.
• Judge every new model or tool on your own golden dataset, not on benchmarks or excitement. Read every failure, and rerun before deciding.
• A model card gives you facts (price, limits, policies); a benchmark gives you a hint; your golden set gives you an answer.
• Before adopting a framework, ask what problem it solves, whether you can see its prompts, how stable it is, and what leaving would cost. Timebox the trial.
• Keep a small habit: 30 minutes a week, two hands-on hours a month, a learning log, and a two-week pause before acting on hype.

The end of this story, and the start of yours

On Friday, Arjun went back to Meera with the answer to her question from the meeting.

“You asked if Copilot would be out of date by Diwali. Here’s how we’ll know. We have a set of real questions from your staff, with the right answers. Every time a new model or tool comes out that looks useful, we test it on those questions, the way Raju test-bakes our sourdough at the trade fair. We switched the review replies to a new model this week, because it was just as good and half the price. We didn’t switch the policy answers, because it got a refund rule wrong. And we’ll check again next month.”

Meera nodded. “So you don’t have to chase every new thing.”

“No. Just test the ones that might matter, on our loaf.”

After Meera left, Priya stayed by the whiteboard. It still had the faint ghost of a drawing from months ago: a single box labelled model, with an arrow going in and an arrow coming out. Arjun had drawn it on his first week, when he had asked her, a little embarrassed, what a token actually was.

“Do you remember what you said when you pasted the whole handbook into the system prompt?” she asked.

“‘It’s only one line of code.’”

“And today you told the founder which things the system can’t do, how you know, and what you’ll check next month.” She capped her marker. “That’s the job, Arjun. Not knowing everything. Knowing how to find out, and being honest about the rest. You don’t need me looking over your shoulder any more.”

He looked at the old drawing for a while. Then he wiped it off and started a new one. Tanvi had a question about chunking.

---

And you. You started this course perhaps not knowing what a token was, either. Since then, you have watched a model predict one word at a time, turned text into vectors, built retrieval that cites its pages and admits what it does not know, given models tools and watched them loop, built agents with memory and voices, measured all of it with evals, defended it against prompt injection, looked inside the transformer, and fine-tuned a model of your own. Then you pulled it all together into projects that would make any engineering team look twice.

The field will keep moving. New models will arrive every month, and some weeks it will feel impossible to keep up. When that happens, come back to the right-hand column. The ideas there are yours now, and they will carry you through whatever comes next.

Build things for real people. Measure whether they work. Be honest about where they don’t. Share what you learn. And keep a little time each week to be curious.

Thank you for learning with Arjun, Priya and Meera. Go build something good.

Go deeper

• Simon Willison’s blog, one of the most careful, hands-on practitioners writing about new models and tools as they arrive.
• Anthropic documentation, OpenAI documentation and Gemini documentation: the release notes and model pages here matter more than any headline.
• Hugging Face documentation, where open-weights models publish their model cards.
• arXiv, for when a new idea really matters and you want the original paper.
• Building effective agents, a reminder that the simplest design that works usually wins.