Minimising model risk in production

Diwali eve, 7:40 p.m. The WhatsApp bot was handling about 600 messages an hour, four times a normal evening, mostly people asking where their hampers were. Then the model provider had a partial outage. Nearly a third of requests failed. The ones that worked took 40 seconds instead of two.

Customers did what people do when an app goes quiet: they sent the message again. And again. “Hello?” “Is anyone there??” “ORDER 9120 WHERE IS IT”.

Arjun’s phone lit up with alerts. He was at his family’s Diwali puja. He opened his laptop on the sofa, surrounded by relatives and sweets.

Arjun makes it worse

Arjun’s reasoning was quick and confident: requests are failing, so retry them more. The code retried each failed model call twice. He changed it to five, committed, and started a deploy. The deploy took eleven minutes.

When the new version went live at 7:51, things got worse, not better.

	7:30 (normal Diwali eve)	7:55 (after Arjun’s change)
Customer messages per 10 min	100	230 (people re-sending)
Model calls per 10 min	210	1,690
Provider “too many requests” errors	0	410
Reply time, slowest 5%	3 s	95 s
Spend in the last 30 minutes	₹290	₹2,600

Each failing message now tried up to six times, and each attempt waited up to 40 seconds. Customers re-sent their messages while waiting, which created more conversations, which created more retries. The provider, already struggling, started rejecting Crumb’s requests outright for sending too many. Arjun had built a retry storm: retries that multiply load exactly when the system is weakest.

At 7:58 Priya, from her own home, opened Crumb’s admin page and flipped one switch, labelled ai_support_enabled. Within seconds every new message got this reply:

“Our assistant is taking a short break. A team member will reply within 15 minutes. You can track any order at crumb.co.in/track.”

Anita’s support team, already on standby for Diwali, took over the queue. No code was deployed. Spend dropped to zero.

The fuse box

The next morning, over leftover sweets, Priya drew a fuse box.

“Why does your house have a fuse box?”

“So if the kettle short-circuits, it trips one switch instead of setting the house on fire.”

“Can you promise the kettle will never short-circuit?”

“No.”

“So you don’t try. You make sure when it does, only the kitchen goes dark, for a minute, and you can switch it back on. Last night you tried to prevent the failure by retrying harder. I contained it by switching one circuit off.”

How much a failure can damage, how many users, how much money, how much data, for how long, is called its blast radius. Production risk work is mostly about shrinking it, ahead of time.

Priya asked a question to make it concrete. “If the refund tool went completely wrong last night, what’s the most money we could have lost?”

Arjun thought it through. “The tool can only request refunds up to ₹500, and staff approve every one. So… zero rupees automatically. The worst case is a lot of annoying requests in Anita’s queue.”

“That’s a small blast radius. You designed that weeks ago, and it’s why the refund tool was never last night’s problem. Now let’s do the same for everything else.”

Where the analogy breaks: a fuse trips automatically on a clear physical signal, too much current. AI failures are often fuzzy, a bot that is subtly wrong rather than crashed, so some of your switches need a human to decide, and good monitoring to tell them when.

Following one message through a better design

Priya and Arjun redesigned the system, then replayed the outage in staging with one message traced end to end: “Where is my hamper? Order 9120”, sent at 7:52 p.m., with the primary provider failing a third of requests and taking 40 seconds for the rest.

Rung 1: the primary model, with a short leash. The call gets an 8-second timeout and one retry at most. It times out. Elapsed: 8.0 s.

Rung 2: the backup model, from a different provider. It has its own 8-second timeout. It answers in 2.3 seconds: “Your hamper (order 9120) left the Baner kitchen at 6:10 p.m. and should arrive by 8:30 p.m.” Elapsed: 10.3 s. The customer gets a correct answer in ten seconds instead of nothing in ninety-five.

Had the backup also failed, rung 3: a stored answer. For order-status questions, the bot can skip the model entirely: look up the order in the database and fill a template. “Order 9120: out for delivery, expected by 8:30 p.m.” Elapsed: about 0.1 s more.

Rung 4: the human handover. If all else fails, the friendly break message, and the conversation goes into Anita’s queue.

Meanwhile, the rate limiter caps each customer at 20 messages per 10 minutes, so the frantic re-senders cannot multiply load without limit. And a spend breaker watches total cost: if the last hour passes ₹3,000, it flips ai_support_enabled off automatically and pages Priya.

Replaying the whole outage with this design: 98% of order-status customers got a correct answer within 12 seconds, model calls stayed under 260 per 10 minutes, and nobody needed to deploy anything.

Opening the hood: five ways to shrink the blast radius

1. Fallbacks. A fallback is what the system does when the main path fails. Arrange them as a ladder from best to most basic: retry briefly, then a backup model, then a stored or template answer, then a human. Two rules: every rung has a short timeout, and every rung that talks to customers must pass your evals. A backup model that gives wrong allergy advice is worse than a polite handover. You built a multi-provider gateway in the streaming and fallbacks module; this is where it pays off.

2. Feature flags and kill switches. A feature flag is a setting your app reads while running that turns a feature on or off, or on for some users, without deploying code. A kill switch is a flag whose job is emergency shutdown. Crumb now has three levels:

Switch	What it turns off	Flip it when
ai_support_enabled	The whole bot; everyone gets the handover message	Provider outage, a bad answer spreading, cost spike
tool_refund_enabled	Only the refund-request tool	Someone has found a way to abuse refunds
model_route	Switches primary to backup model	Primary is slow or behaving oddly

The crucial property: the app checks flags on every request, so a change applies in seconds. Arjun’s eleven-minute deploy is exactly what flags exist to avoid.

3. Rate limits. Rate limiting caps how many requests a user, an address or the whole system can make in a period. Set it at three levels: per customer (20 messages per 10 minutes), per conversation (agent loops capped at six steps, replies capped with max_tokens), and on total spend (alert at one level, automatic shutdown at a higher one).

4. Staged rollouts. A staged rollout releases a change to a growing share of users in steps, checking health at each step. It applies to new prompts, new models, new tools, and turning the bot back on after an incident.

flowchart LR
  S["Shadow: log only"] --> I["Staff only"]
  I --> C10["10 percent"]
  C10 --> C50["50 percent"]
  C50 --> A["Everyone"]
  C10 -->|"worse"| R["Roll back"]
  C50 -->|"worse"| R

5. Monitoring with owners. Every switch above is useless if you learn about the problem from a customer’s tweet. Alert on signals with thresholds agreed in advance, and send alerts to a named person with a short runbook.

Signal	Crumb’s alert
Error rate	Over 5% of model calls failing for 5 minutes
Latency	Slowest 5% of replies over 10 seconds
Cost	Spend in the last hour over 3 times the usual for that hour
Guardrail hits	Output validation failures doubling
Human handovers	Over 20% of conversations
Feedback	Thumbs-down over 10% in the last hour

Building the switches in code

Step 1: a fallback ladder with short timeouts. Both SDKs accept a per-request timeout; setting max_retries on the client stops it from retrying quietly in the background.

# fallback.py: try each rung within a fixed time budget
import os
import anthropic
from openai import OpenAI

primary = anthropic.Anthropic(max_retries=1)          # at most one quiet retry
backup = OpenAI(max_retries=0)
BACKUP_MODEL = os.getenv("OPENAI_MODEL", "gpt-5-mini")  # names change: check the models page

HANDOVER = ("Our assistant is taking a short break. A team member will reply within "
            "15 minutes. You can track any order at crumb.co.in/track.")

def call_primary(message: str) -> str:
    r = primary.messages.create(model="claude-opus-5-5", max_tokens=500, timeout=8.0,
                                system=SYSTEM, messages=[{"role": "user", "content": message}])
    return next(b.text for b in r.content if b.type == "text")

def call_backup(message: str) -> str:
    r = backup.with_options(timeout=8.0).chat.completions.create(model=BACKUP_MODEL,
        messages=[{"role": "system", "content": SYSTEM}, {"role": "user", "content": message}])
    return r.choices[0].message.content

def answer(message: str) -> str:
    for rung in (call_primary, call_backup, template_answer):  # template_answer: database + template
        try:
            reply = rung(message)
            if reply:              # template_answer returns None if it can't help
                return reply
        except Exception as error:  # timeouts, connection errors, 5xx, 429
            log_failure(rung.__name__, error)
    queue_for_human(message)
    return HANDOVER

SYSTEM, template_answer, log_failure and queue_for_human are your app’s own functions. Catching every exception is deliberate here: in a fallback ladder, any failure on one rung should move to the next.

Step 2: flags read on every request. Flags must live somewhere every server can see and change instantly. Redis (a fast shared key-value store you met in the scalable RAG module) is a common choice.

# flags.py: shared flags, read on every request
import os
import redis

store = redis.Redis.from_url(os.environ["REDIS_URL"], decode_responses=True)

def flag(name: str, default: bool = True) -> bool:
    try:
        value = store.get(f"flag:{name}")
    except redis.RedisError:
        return default            # if the flag store is down, use a safe default
    return default if value is None else value == "on"

def handle(message: str) -> str:
    if not flag("ai_support_enabled"):
        return HANDOVER
    return answer(message)

Flipping the switch is then one command, which a support lead can run from an admin page:

redis-cli SET flag:ai_support_enabled off

Notice the default. For ai_support_enabled, defaulting to True keeps the bot running if Redis hiccups. For a risky tool, you might default to False, so it is off unless explicitly on. Choose each default by asking which failure is worse.

Step 3: a rate limit that works across servers. An in-memory counter forgets everything on restart and cannot see other servers. A Redis counter per customer per time window can.

# rate_limit.py: at most 20 messages per customer per 10-minute window
import time

WINDOW_SECONDS, MAX_MESSAGES = 600, 20

def allowed(customer_id: str) -> bool:
    window = int(time.time() // WINDOW_SECONDS)
    key = f"rate:{customer_id}:{window}"
    count = store.incr(key)             # add one, atomically, return the new total
    if count == 1:
        store.expire(key, WINDOW_SECONDS)   # clean up after the window ends
    return count <= MAX_MESSAGES

This “fixed window” approach is slightly generous at window edges, which is fine for a chat app. Your hosting platform or an API gateway may offer rate limiting built in.

Step 4: a staged rollout that is consistent per customer. A customer should not switch between versions mid-conversation. Hashing their ID puts them in the same bucket every time.

import hashlib

def in_rollout(customer_id: str, percent: int) -> bool:
    bucket = int(hashlib.sha256(customer_id.encode()).hexdigest(), 16) % 100
    return bucket < percent

print([in_rollout(f"cust_{i}", 10) for i in range(8)])

[False, True, False, False, False, False, False, False]

Store the current percentage as a flag too, so widening from 10% to 50% is another one-line change.

Step 5: an automatic spend breaker. A person might be at a puja. A breaker is not.

# breaker.py: run every minute
HOURLY_LIMIT_INR = 3000

def check_spend(spend_last_hour_inr: float) -> None:
    if spend_last_hour_inr > HOURLY_LIMIT_INR and flag("ai_support_enabled"):
        store.set("flag:ai_support_enabled", "off")
        page_on_call(f"Spend ₹{spend_last_hour_inr:.0f} in last hour; AI support switched off")

The breaker switches off automatically but never back on. Turning things back on is a human decision, made with a staged rollout.

Where it breaks

Untested fallbacks. A backup provider with an expired API key is not a backup. Once a month, force each rung to fail in staging and check the next one takes over.

Backups that behave differently. The backup model may be weaker at Hinglish or allergy questions. Run your eval gate against it too, and consider limiting it to simple routes such as order status.

Flags read only at startup. Then flipping the switch still needs a restart or deploy, which is slow exactly when you need speed.

Alert fatigue. Alerts that fire daily for harmless reasons get muted, and then the real one is missed. Every alert should be rare, actionable and owned by a named person.

Retry storms. Retries without limits multiply load on an already-struggling provider. Cap retries, add backoff (as you learned in the retries module), and set timeouts per rung.

Try it yourself

Write a one-page incident runbook for any AI app you have built in this course. Include:

1. Your fallback ladder, with a timeout for each rung.
2. Your kill switches, their safe defaults, and who may flip them.
3. Three alerts with thresholds and owners.
4. Your staged plan for switching the feature back on.

Show the answer

A good runbook is short enough to read during a crisis. Example for a document Q&A bot: rungs are the primary model (10 s), a smaller backup (10 s), a “search results only” mode with no generation, then a contact form. Switches: qa_enabled (default on; any on-call engineer or the product owner), upload_enabled (default off if the flag store is down). Alerts: error rate over 5% for 5 minutes (on-call engineer); hourly cost over 3 times the 7-day average (product owner); validation failures over 1% (AI engineer). Re-enable: staff only for 15 minutes, then 10%, 50%, 100% at 15-minute steps, rolling back if any alert fires.

Common mistakes

• Trying to prevent every failure instead of containing it. Design for “when”, not “if”.
• Retrying harder during an outage. More retries mean more load on a struggling provider. Cap them and fall back instead.
• Rolling back by deploying. If undoing a change needs a deploy, it is too slow. Make prompts, models and features switchable by flag.
• In-memory limits on many servers. Each server counts separately, so the real limit is many times higher than you think. Use a shared store.
• Alerts with no owner. An alert that pings an unwatched channel at 8 p.m. on Diwali is decoration.

Recap

• Blast radius is how much a failure can damage; production risk work shrinks it in advance.
• A fallback ladder (retry, backup model, stored answer, human) works only with short timeouts and capped retries.
• Feature flags and kill switches change behaviour in seconds without a deploy; read them on every request.
• Rate limits per customer, per conversation and on total spend stop abuse and runaway costs; automatic breakers switch off, humans switch back on.
• Staged rollouts and owned alerts mean problems reach few users and are seen within minutes.

At 8:41 p.m. on Diwali eve, the provider recovered. Priya set the rollout to 10%, watched for fifteen minutes, then 50%, then everyone. Total time in human-only mode: 43 minutes. The redesign shipped the next week. Arjun explained it to Meera: “Last night I tried to fix an outage by retrying harder, and it made things eight times worse. Now, if the AI provider fails, the bot quietly switches to a backup, and if that fails it looks up the order itself or hands over to Anita’s team. Anyone on the support team can switch the bot off in one click, and if spending goes crazy, it switches itself off and phones Priya.”

Meera raised an eyebrow. “And if the office loses power? The bot runs on that computer under your desk.” Arjun went pale. It had happened twice last monsoon. That is the next lesson.

Go deeper

• Anthropic docs, including errors, timeouts and rate limits: https://docs.anthropic.com
• OpenAI docs, including error handling: https://platform.openai.com/docs
• OpenTelemetry, for metrics and alerts: https://opentelemetry.io/docs/
• Building effective agents, on keeping agents simple and bounded: https://www.anthropic.com/research/building-effective-agents