Exponential backoff and retry strategies

On the first big Saturday of the Diwali season, Crumb ran a “pre-order your hamper” campaign on Instagram. A normal Saturday brings about 40 support chats a minute. By 11 a.m. it was 210 a minute, and the streaming widget from the last lesson was answering every one of them in real time.

At 11:07 it started failing. Arjun opened the logs and saw the same line hundreds of times:

anthropic.RateLimitError: Error code: 429 - rate_limit_error

In twenty minutes, 312 customers got “Sorry, something went wrong.” Some of them were in the middle of spending ₹2,400 on a hamper.

Arjun’s fix took two minutes to write. He was about to deploy it when Priya walked over with her coffee. She read it, put the coffee down, and said: “Before you ship that, let’s work out what it does at 11:07 on a Saturday.”

Arjun’s two-minute fix

Here is what he had written:

def ask(question: str):
    while True:
        try:
            return client.messages.create(
                model="claude-opus-5-5",
                max_tokens=1000,
                messages=[{"role": "user", "content": question}],
            )
        except Exception:
            continue  # it failed, so try again

“It can’t fail any more,” he said. “It keeps trying until it works.”

Priya pulled up the staging test he had run with a pretend load of 200 customers. In the first minute, his app had sent 4,800 requests. Almost all of them got 429s. The few that did get through were delayed by everyone else’s retries. And one request never stopped at all: a customer had pasted a message so long that the API returned 400 Bad Request, “prompt is too long”. The loop had retried that same impossible request 1,100 times in a minute and was still going.

“Two problems,” Priya said. “You’re retrying things that can never work. And you’re retrying the things that can work in the worst possible way.”

The busy phone line

Priya drew a phone on the whiteboard. “Diwali morning. The bakery has two phone lines. Two hundred customers call at once. One hundred and ninety-eight hear a busy tone. What happens if they all hit redial instantly?”

“The line’s still busy,” Arjun said. “The two lucky ones are still talking.”

“And if they keep redialling instantly?”

“It stays busy. Forever, until people give up.”

“So what should a polite caller do?”

“Wait a bit before calling back.”

“How long?”

Arjun thought. “A minute?”

“All 198 of them wait exactly one minute,” Priya said. “What happens at minute one?”

He laughed. “They all call back at the same second. Busy again.”

“That’s called a thundering herd. Waiting is half the answer. The other half is waiting differently from everybody else.” She drew a scatter of dots spread across the minute instead of a single spike. “If each caller waits a random amount, the calls trickle in. The lines can handle a trickle.”

Then she asked the second question. “A customer calls to order a cake for pincode 999999, which doesn’t exist. The bakery says no. Does redialling help?”

“No. They need to fix the pincode.”

“That’s your 400 error. Some failures mean ‘try later’. Others mean ‘your request is wrong’. Only the first kind deserves a retry.”

Where the analogy breaks: a phone line is either busy or free. An API’s limit is more like a water tank that refills at a steady rate. You’ll see that in a moment.

What the API was actually telling you

A rate limit is a cap on how much you can ask of an API in a period of time. An API (application programming interface) is a service your code talks to over the internet. LLM providers usually limit three things per minute: requests, input tokens and output tokens. A token is a small chunk of text, roughly three-quarters of an English word.

Many providers, Anthropic included, implement these limits like a bucket that refills continuously rather than a counter that resets on the minute. If your limit is 50 requests per minute, you get roughly one new request “credit” every 1.2 seconds. Burst through the bucket and you get 429 Too Many Requests until it refills a little.

Every failed response carries an HTTP status code, a three-digit number that says what kind of failure it was. Sort them into two buckets:

Retry: it might work later	Do not retry: the request itself is wrong
429 rate limited	400 bad request (malformed, or prompt too long)
500, 502, 503, 504 server errors	401 invalid API key
529 overloaded (Anthropic’s “too busy right now”)	403 not allowed
408 request timeout	404 model or endpoint not found
409 conflict (a temporary clash on the server)	413 request too large
Network timeouts and dropped connections (no status at all)	422 validation failed

The rule of thumb: 4xx means “you did something wrong”, 5xx means “we did”, and 408, 409 and 429 are the 4xx codes that mean “try later”. If the same request will fail in the same way next time, retrying wastes money and time.

Show the answer

(a) Retry: the server is temporarily too busy. (b) Do not retry: the API key is wrong or revoked, and it will still be wrong in two seconds. Alert a human. © Retry, carefully: you got no answer at all, so it might be a passing network problem. (In the next lesson you’ll see why “no answer” is trickier than it looks when the request does something.) (d) Do not retry: your code sent an invalid value. Fix the code.

Following one request through the retry loop

Priya rewrote Arjun’s loop on the whiteboard with three rules: retry only the left column, double the waiting ceiling each time, and pick a random wait under that ceiling. Then they traced a single customer’s question, “Is the dry fruit hamper eggless?”, through it during the 11:07 rush. These are the real values from a test run:

Attempt	Sent at	Result	Wait rule	Ceiling	Actual wait
0	11:07:00.00	429, no retry-after header	random(0, ceiling)	1 s	0.62 s
1	11:07:00.95	529 overloaded	random(0, ceiling)	2 s	1.37 s
2	11:07:02.66	429, header retry-after: 4	use the header	(ignored)	4.00 s
3	11:07:06.98	200 OK, 212 output tokens

The customer’s first words appeared about 7.8 seconds after they pressed Enter: slower than usual, but an answer instead of an error. The three failed attempts cost nothing in tokens, because a rejected request is not processed by the model.

Two things are worth noticing. The ceiling doubles: 1, 2, 4, 8 seconds. That is exponential backoff: each failure makes you back off twice as far. And at attempt 2 the server said how long to wait. A retry-after header is the server’s own advice, usually a number of seconds. The provider knows when your bucket will refill; your formula is only guessing. When the header is there, use it.

Now zoom out from one customer to all 200. Priya simulated the same rush twice: once with waits of exactly 1, 2 and 4 seconds, once with a random wait under those ceilings. Then she counted when the retries arrived:

	Exact waits (no randomness)	Random waits under the ceiling
First retry wave	All 200 arrive in the same instant, at 1.000 s	200 arrive spread across 0 to 1 s, about one every 5 ms
Second retry wave	The 190 still failing arrive together at 3.000 s	Spread across the following 0 to 2 s
Most arrivals in any 100 ms	200	about 20

Without randomness, the herd arrives in spikes and keeps hitting the limit together. With it, the same retries smear across the window, so each one has a fair chance of landing in a moment when the bucket has a free slot. The randomness is called jitter.

Opening the hood: backoff, jitter and limits

The formula is short. For attempt number n, starting at 0:

ceiling = min(cap, base × 2^n)
wait    = random number between 0 and ceiling      ("full jitter")

With base = 1 second and cap = 30 seconds, the ceilings go 1, 2, 4, 8, 16, 30, 30. The cap stops waits growing absurdly long. A separate maximum number of attempts (often 3 to 6) stops a request from trying forever.

There are a few flavours of jitter. “Full jitter”, above, is the most common and spreads load the most. “Equal jitter” waits half the ceiling plus a random amount up to the other half, so you never retry instantly. Either is fine for an LLM app; what matters is having some randomness.

Providers also send headers describing your remaining allowance, such as how many requests or tokens you have left in the current window and when it refills. You don’t need them to retry correctly, but they make useful dashboard numbers and let you slow down before you hit the limit.

Building the retry loop, one layer at a time

The SDKs already retry for you, as you’ll see at the end. Writing it yourself once makes the idea stick and prepares you for the gateway at the end of this module, where retries span several providers.

Layer 1: decide whether an error is worth retrying

import anthropic

RETRYABLE = {408, 409, 429, 500, 502, 503, 504, 529}


def is_retryable(err: Exception) -> bool:
    if isinstance(err, anthropic.APIConnectionError):  # network trouble, including timeouts
        return True
    if isinstance(err, anthropic.APIStatusError):      # the server replied with an error code
        return err.status_code in RETRYABLE
    return False                                        # a bug in our own code: never retry

The SDK raises APIConnectionError (and its child APITimeoutError) when no reply arrives, and APIStatusError when one arrives with an error code. Anything else is probably your own bug, and retrying would hide it.

Layer 2: decide how long to wait

import random


def wait_seconds(err: Exception, attempt: int, base: float = 1.0, cap: float = 30.0) -> float:
    if isinstance(err, anthropic.APIStatusError):
        header = err.response.headers.get("retry-after")
        if header and header.isdigit():
            return float(header)                          # the server told us
    return random.uniform(0, min(cap, base * 2**attempt))  # full jitter

retry-after is occasionally a date rather than a number of seconds. This version falls back to backoff in that case, which is safe.

Layer 3: the loop

import time

from dotenv import load_dotenv

load_dotenv()
client = anthropic.Anthropic(max_retries=0)  # turn off built-in retries for this exercise


def ask_with_retries(question: str, max_attempts: int = 5):
    for attempt in range(max_attempts):
        try:
            return client.messages.create(
                model="claude-opus-5-5",
                max_tokens=1000,
                messages=[{"role": "user", "content": question}],
            )
        except Exception as err:
            if not is_retryable(err) or attempt == max_attempts - 1:
                raise  # wrong request, or out of attempts: let the caller show an error
            wait = wait_seconds(err, attempt)
            print(f"Attempt {attempt + 1} failed ({type(err).__name__}), waiting {wait:.2f}s")
            time.sleep(wait)

Read it top to bottom: try; if it fails, decide whether it is worth another go; if so, pick a wait and sleep; give up after five attempts. During the rush, the output looked like this:

Attempt 1 failed (RateLimitError), waiting 0.62s
Attempt 2 failed (InternalServerError), waiting 1.37s
Attempt 3 failed (RateLimitError), waiting 4.00s

followed by the answer. Compare it with Arjun’s first version: it retries the 400 zero times instead of 1,100, and spreads the 429 retries out instead of firing 4,800 in a minute.

Layer 4: let the SDK do it

In real code, start with what the SDK gives you. Both the Anthropic and OpenAI Python SDKs automatically retry connection errors, timeouts, 408, 409, 429 and 5xx errors with exponential backoff, and both default to 2 retries. You can tune both numbers:

import anthropic
from openai import OpenAI

claude = anthropic.Anthropic(max_retries=4, timeout=60.0)       # timeout in seconds
openai_client = OpenAI(max_retries=4, timeout=60.0)

# Or for one call only:
claude.with_options(max_retries=6).messages.create(...)

If you write your own loop on top, set the SDK’s max_retries=0. Otherwise you stack two retry systems, which leads straight to the next problem.

Where it breaks

Arjun shipped Layer 4 with max_retries=4, and the Saturday rush flattened into slightly slower answers instead of errors. Then Priya asked him to check three things that bite experienced engineers too.

Stacked retries. Sneha had added her own retry loop of 5 attempts around a helper that already used the SDK’s default 2 retries. Every one of her attempts was really up to 3 SDK attempts.

Show the answer

5 × 3 = 15 requests. If each one times out at 60 seconds, that is 15 minutes of timeouts, plus all the backoff waits in between. Nobody waits 15 minutes in a chat widget. Decide your total patience first (say 30 seconds for a chat reply) and choose the timeout and retry count to fit inside it. And keep only one retry layer.

Retry storms. Retries add load. If the provider is down for everyone, your app’s retries multiply your traffic while helping nobody. Three attempts each means three times the requests against a server that is already struggling. The circuit breaker at the end of this module stops calling a provider that is clearly down.

Streams. If a stream fails before the first token, retry freely: the customer has seen nothing. If it fails halfway, the customer has seen half an answer. Do not silently start a new answer and glue it on; the second reply will be worded differently. Show the error event from the last lesson with a “try again” button, or clear the bubble and restart visibly.

Retries that do things. A model call that only produces text is safe to repeat. A request that issues a refund or sends a WhatsApp message might do it twice. Hold that thought.

There are also libraries for retry policies, such as tenacity, which wraps a function with configurable backoff and jitter in a few lines. Library APIs change quickly. If a name has moved, check the docs linked below.

Try it yourself

Simulate a flaky provider without spending any money. Replace the API call with a function that fails at random:

import random


def flaky_call():
    if random.random() < 0.6:  # fails 60% of the time
        raise TimeoutError("pretend timeout")
    return "Here is your answer"

Write a loop that retries flaky_call() up to 5 times with full jitter (base 1 second, cap 30), printing each wait. Run it ten times. How often does it give up, and what do the waits look like?

Show the answer

Each attempt fails with probability 0.6, so all five fail with probability 0.6 to the power 5, about 0.078. Expect roughly one give-up in every thirteen runs. The printed waits should look random but grow on average: under 1 s, under 2 s, under 4 s, under 8 s. If every run prints the same waits, the jitter is missing.

Common mistakes

• Retrying immediately in a tight loop. It deepens the overload and may get your key throttled harder. Always wait.
• Retrying every error. A 400 or 401 fails the same way every time. Retry only temporary errors.
• Backoff without jitter. Your clients retry in lockstep and collide again. Add randomness.
• Ignoring retry-after. The server told you when to come back. Use it.
• Stacking retry layers. Your loop around the SDK’s loop multiplies attempts and waiting time. Keep one.
• No overall deadline. Without a cap, a maximum attempt count and a sensible timeout, a bad hour turns into requests that never return.

Recap

• A 429 means “slow down and try later”, not “your code is broken”.
• Retry 408, 409, 429, 5xx (including 529), timeouts and dropped connections. Do not retry other 4xx errors.
• Exponential backoff doubles the waiting ceiling each attempt; jitter picks a random wait under it so retries do not arrive in herds.
• When the response includes retry-after, wait exactly that long.
• The Anthropic and OpenAI SDKs retry automatically (2 retries by default); tune max_retries and timeout before writing your own loop, and never stack two loops.

That evening Meera asked what had gone wrong in the morning. Arjun tried the phone-line version. “The AI company only lets us make so many calls a minute. When we went over, our app hung up and redialled instantly, all two hundred customers at once, so the line stayed jammed. Now each one waits a little, a different little, and waits longer if it’s still busy. And if the request itself is wrong, it stops instead of redialling forever.” Meera liked that. “So it’s polite now.”

Then a message arrived in the support inbox: “Why did I get two refund confirmations for one cake?” Arjun’s retries had worked a little too well.

Go deeper

• Anthropic errors and rate limits: https://docs.anthropic.com
• OpenAI error codes and rate limits: https://platform.openai.com/docs
• HTTPX, the HTTP library under both SDKs, with timeouts explained: https://www.python-httpx.org