Rate-limit retries and token-limit management

Arjun had two problems on his whiteboard, and at first they looked unrelated.

The first was the review tagger from the last lesson. At 8 requests at a time it was fine. At 50 at a time, a third of the calls came back with 429 Too Many Requests, and the run took longer than at 8.

The second came from the WhatsApp ordering line. Mrs. Kulkarni, a wedding caterer, had been planning a three-tier cake and 400 dessert boxes with Crumb for three weeks, all in one chat thread, now 140 messages long. The assistant drafted every reply for the WhatsApp team. Arjun pulled up the usage numbers for that one conversation: ₹1,450 so far, and the latest reply had cost ₹20 on its own. The first reply, three weeks ago, had cost 60 paise.

Worse, yesterday a reply had stopped mid-sentence: “For the middle tier we recommend the Belgian chocolate with a layer of”. The WhatsApp team had sent it to Mrs. Kulkarni before noticing.

Priya drew a single line connecting the two problems. “Both are about limits. One limits how fast you can ask. The other limits how much you can say and hear in one go. Real apps budget for both, instead of discovering them in production.”

Arjun’s first attempts

For the 429s, Arjun’s instinct was to retry harder:

while True:
    try:
        return client.messages.create(**kwargs)
    except anthropic.RateLimitError:
        continue   # try again immediately!

With 50 workers each doing this, the API received hundreds of requests a second, rejected nearly all of them, and his account stayed over the limit the entire time. The more he retried, the fewer succeeded.

For the cut-off reply, his instinct was to look at max_tokens. He had set it to 300 a month earlier “to keep costs down”. He raised it to 2,000, which fixed the cut-off. But the bill for the next reply was still ₹20. The output was never the expensive part.

“Two instincts, both reasonable, both wrong,” Priya said. “Let’s start with the queue.”

The Sunday queue at the Koregaon Park store

“What happens at the Koregaon Park counter on a Sunday morning?” Priya asked.

“Long queue. They use the token machine. You take a number, you wait.”

“Imagine a customer who, every two seconds, walks to the counter and asks ‘Is it my turn?’ What does that do?”

“Annoys everyone. Slows the staff down.”

“Now imagine fifty of them.”

“The counter stops working.”

“So what’s the polite version?”

“Wait. If it’s still busy when you check, wait longer before checking again.”

“And if fifty people all decide to wait exactly one minute?”

Arjun laughed. “They all come back together at once and it’s the same crush.”

“So?”

“Everyone waits a slightly different amount.”

“That’s the algorithm,” Priya said. “You just invented exponential backoff with jitter.”

Opening the hood on rate limits

A rate limit is a cap a provider puts on how much you can use per minute. Providers typically measure three things:

• Requests per minute: how many calls.
• Input tokens per minute: how much text you send. A token is a chunk of text, about three-quarters of an English word on average.
• Output tokens per minute: how much the model writes back.

Exceed any one and you get HTTP 429, usually with a retry-after header saying how many seconds to wait. Your limits depend on your account tier, which rises as your usage history and spend grow; the current numbers are on your provider’s console.

Under the hood, Anthropic describes its limits with a token bucket: picture a jar that holds, say, 50 request tokens and is refilled steadily, almost a token every second. Each request takes one token from the jar. If the jar is empty, you get a 429. This has a practical consequence: short bursts are fine as long as the jar has tokens, but a sustained rate above the refill speed always ends in 429s, no matter how cleverly you retry. Retrying only helps you get through bursts. If your steady load is too high, you need a lower concurrency, a higher tier, the Batch API, or fewer tokens per request.

Exponential backoff means: after each failure, wait twice as long as last time (1 second, 2, 4, 8…), up to a ceiling. Jitter means adding a small random amount to each wait. Without jitter, fifty workers that failed together at second 0 all retry at second 1, fail together, all retry at second 3, fail together again. Engineers call this a thundering herd. With jitter, they spread out across each window, and most of them slip through.

flowchart LR
    A["Call API"] --> B{"429, 5xx or network error?"}
    B -->|"No"| C["Use the reply"]
    B -->|"Yes"| D{"Attempts left?"}
    D -->|"No"| E["Give up: fallback"]
    D -->|"Yes"| F["Wait retry-after, or 2^n seconds, plus jitter"]
    F --> A

Following one request through a rate-limited minute

Here is one review-tagging request traced through Arjun’s final retry function (built below), from the log of a run at 30 concurrent workers:

12:04:10.0  attempt 1 -> 429 rate_limit_error, retry-after: 3
12:04:10.0  waiting 3.0s (from retry-after) + 0.4s jitter
12:04:13.4  attempt 2 -> 429 rate_limit_error (no retry-after header)
12:04:13.4  waiting 2.0s (2^1) + 0.3s jitter
12:04:15.7  attempt 3 -> 529 overloaded_error
12:04:15.7  waiting 4.0s (2^2) + 0.9s jitter
12:04:20.6  attempt 4 -> 200 OK  "negative"

Four attempts, 10.6 seconds, one success, and the API was never hit more than once every couple of seconds by this request. Multiply that politeness across 30 workers and the account stays just under its limit instead of far over it. In the full run, 30 workers with backoff finished the 3,014 reviews in 11 minutes with zero lost reviews. The earlier 50-worker run with instant retries had taken 19 minutes and lost 412.

The real code: backoff you control

Good news first: both the Anthropic and OpenAI SDKs already do a basic version of this. They retry 408, 409, 429, 5xx and connection errors twice by default, with exponential backoff, and respect retry-after. You can turn the dial:

import anthropic
client = anthropic.Anthropic(max_retries=5, timeout=60.0)

# Or for one call only:
client.with_options(max_retries=8).messages.create(...)

For an overnight job that can afford to wait, more retries is usually right. For a live chat, fewer retries plus a fast fallback is kinder to the customer.

Sometimes you want your own loop: to log each wait like the trace above, to stop at a hard deadline, or to share one pause across all workers. Here is a complete, readable version.

Step 1: decide what is retryable.

import random
import time
import anthropic
from dotenv import load_dotenv

load_dotenv()
client = anthropic.Anthropic(max_retries=0)   # turn off SDK retries: we handle them here

def is_retryable(err: Exception) -> bool:
    if isinstance(err, anthropic.APIConnectionError):     # includes timeouts
        return True
    if isinstance(err, anthropic.APIStatusError):
        return err.status_code in (408, 409, 429) or err.status_code >= 500
    return False

Step 2: wait the right amount, then try again.

def create_with_backoff(max_attempts: int = 6, base: float = 1.0, cap: float = 60.0, **kwargs):
    for attempt in range(1, max_attempts + 1):
        try:
            return client.messages.create(**kwargs)
        except Exception as err:
            if not is_retryable(err) or attempt == max_attempts:
                raise                                       # 400s, 401s, or out of attempts
            retry_after = None
            if isinstance(err, anthropic.APIStatusError):
                retry_after = err.response.headers.get("retry-after")
            # Prefer the server's advice; otherwise double the wait each time, up to the cap
            delay = float(retry_after) if retry_after else min(cap, base * 2 ** (attempt - 1))
            delay += random.uniform(0, delay * 0.25)        # jitter: spread retries out
            print(f"Attempt {attempt} failed ({type(err).__name__}). Waiting {delay:.1f}s")
            time.sleep(delay)

Line by line, the important parts: max_retries=0 on the client prevents double retrying (the SDK’s retries inside yours would multiply the attempts). base * 2 ** (attempt - 1) gives 1, 2, 4, 8, 16 seconds. min(cap, ...) stops the wait growing forever. random.uniform(0, delay * 0.25) adds up to 25% jitter. In async code, use await asyncio.sleep(delay) instead of time.sleep, which would freeze the event loop.

Combine this with the semaphore from the batching lesson and you have both halves of polite API use: don’t send too much at once, and back off when told to.

Now the quieter limit: tokens

Back to Mrs. Kulkarni. To see where ₹1,450 came from, Priya made Arjun trace the thread turn by turn.

The key fact is from Module 01: an LLM API is stateless. It remembers nothing between calls. To make a chat feel continuous, your app re-sends the whole conversation every time. Each exchange in this thread averaged about 150 tokens from the customer and 250 from the assistant, so the history grows by about 400 tokens per turn. Add a 300-token system prompt.

(Prices: Claude Opus 5.5 at $4permillioninputtokensand$20 per million output tokens, ₹88 to the dollar.)

The output cost never changed. The input grew every turn, and the total grows with the square of the conversation’s length: twice as many turns costs about four times as much. Adding up all 140 replies gives about 3.9 million input tokens, roughly $16, which is the ₹1,450 on Arjun’s screen. And each reply got slower too, because the model has to read 56,000 tokens before writing a word.

Three ceilings that beginners mix up

Priya wrote three lines on the whiteboard.

1. The context window: the most tokens the model can consider in one request, input and output together. On current Claude models it is about a million tokens, so Mrs. Kulkarni’s 56,000 was nowhere near it. Being under the window does not make a request cheap or fast.
2. max_tokens: the most tokens you allow one reply to use, thinking included. Hit it and stop_reason is "max_tokens", and the reply ends mid-sentence.
3. Your budget: the limit you choose for cost and speed. For a support chat it might be 8,000 input tokens. Nobody enforces it but you.

The clip shows what happens at the first ceiling: the oldest messages fall out. The practical lesson is that you should hit your own budget, ceiling 3, long before the context window, and decide deliberately what falls out.

Measuring before you send

You cannot manage what you do not measure. The Anthropic SDK can count the tokens in a request without running the model:

count = client.messages.count_tokens(
    model="claude-opus-5-5",
    system="You are Crumb's catering assistant.",
    messages=history,
)
print(count.input_tokens)   # e.g. 56050

For quick estimates, “about four characters per English token” is a fine rule of thumb, but Hindi, Marathi, code and emoji use more tokens per character, so count properly when it matters. After every call, response.usage.input_tokens and response.usage.output_tokens tell you exactly what you were billed for. Log them; they are the numbers that would have flagged Mrs. Kulkarni’s thread on day four instead of week three.

Trimming history to a budget

The simplest fix: keep the system prompt and the most recent turns, and drop the oldest until the request fits.

def trim_history(history: list[dict], system: str, budget: int = 8000) -> list[dict]:
    """Drop the oldest turns until the request fits the token budget."""
    trimmed = list(history)
    while len(trimmed) > 1:
        tokens = client.messages.count_tokens(
            model="claude-opus-5-5", system=system, messages=trimmed
        ).input_tokens
        if tokens <= budget:
            break
        trimmed = trimmed[2:]   # drop the oldest user + assistant pair; the list still starts with "user"
    return trimmed

With an 8,000-token budget, every reply in the thread would cost at most about ₹3.30, and the whole thread about ₹400 instead of ₹1,450.

But read the comment again: it drops the oldest turns. For Mrs. Kulkarni that would have forgotten the agreed flavours from week one. Trimming trades memory for money. A better pattern keeps a short running summary of what was dropped:

def summarise(old_turns: list[dict]) -> str:
    """Compress dropped turns into a few lines of facts."""
    transcript = "\n".join(f"{m['role']}: {m['content']}" for m in old_turns)
    response = client.messages.create(
        model="claude-opus-5-5",
        max_tokens=2000,
        output_config={"effort": "low"},
        messages=[{"role": "user", "content": "List the agreed facts and open questions from this "
                   "catering conversation as short bullet points:\n\n" + transcript}],
    )
    return next(b.text for b in response.content if b.type == "text")

# When the history is over budget:
#   summary = summarise(history[:-20])
#   system = BASE_SYSTEM + "\n\nWhat has been agreed so far:\n" + summary
#   history = history[-20:]

The summary for Mrs. Kulkarni was eleven bullet points, about 180 tokens: “Wedding 14 Feb, Sangli. Three tiers, eggless. Bottom: vanilla bean. Middle: Belgian chocolate. Top: mango (seasonal; confirm availability). 400 dessert boxes, mixed mithai…” All the important facts, at a fraction of the size. You met this idea in Module 03’s context-engineering lesson; Module 15 builds proper long-term memory for agents.

Then, on every reply, check the other end:

response = create_with_backoff(model="claude-opus-5-5", max_tokens=4000,
                               system=system, messages=trim_history(history, system))
if response.stop_reason == "max_tokens":
    # The reply was cut off. Don't send it: raise the limit, ask for a shorter answer, or stream.
    ...

Where it still breaks

• Summaries lose detail. A summary is the model’s judgement of what mattered. Keep anything legally or financially important (prices, dates, allergens) in your database, not only in a summary.
• Counting costs calls. count_tokens is free of token charges but has its own rate limit. Estimate first, count when you are close to the budget.
• Retries and budgets interact. A retried 56,000-token request costs ₹20 each time. Trim before you retry, not after.
• Shared limits. Your review tagger and your live chat share one account’s limits. A big overnight job can starve the morning’s chat if it overruns. Run bulk jobs through the Batch API or give them a separate workspace with its own limits.

Try it yourself

1. Build a fake history of 40 messages: alternate {"role": "user", "content": "Tell me about cake option N in detail."} and an assistant reply of a few sentences.
2. Print count_tokens for the full history.
3. Run trim_history with budget=1500 and print how many messages remain, the new count, and the role of the first message.

Expected: the full count is a few thousand tokens; after trimming, the count is at or below 1,500 and the first message is still "user".

history = []
for i in range(20):
    history.append({"role": "user", "content": f"Tell me about cake option {i} in detail."})
    history.append({"role": "assistant", "content": "It is a two-layer sponge with seasonal fruit, "
                    "whipped cream and a light sugar glaze. " * 3})
print("before:", client.messages.count_tokens(model="claude-opus-5-5", messages=history).input_tokens)
small = trim_history(history, system="You are Crumb's catering assistant.", budget=1500)
print(len(small), "messages left; first role:", small[0]["role"])

Common mistakes

• Retrying instantly in a tight loop. It keeps your account over the limit. Back off and add jitter.
• Ignoring retry-after. When the server tells you how long to wait, believe it.
• Double retries. Your own loop around an SDK that also retries multiplies attempts. Set max_retries=0 when you write your own.
• Using max_tokens as a cost control. You pay only for tokens generated; a tight cap mostly produces broken replies. Control cost through what you send.
• Letting chat history grow forever. Cost grows with the square of the length. Budget, trim and summarise.

Explaining it to Meera

Meera wanted the short version. “Why did one customer’s chat cost ₹1,450?”

“Because the AI forgets everything between messages, so every time we ask it to draft a reply, we send it the entire three-week conversation again. Message one cost sixty paise. Message 140 cost twenty rupees, for the same kind of short answer.”

“And now?”

“Now we keep the last twenty messages plus a short summary of everything agreed before. The summary is like the order card pinned on the board in the kitchen: wedding date, tiers, flavours, eggless. Each reply costs about three rupees, and it still remembers the mango top tier.”

“And the half-finished message she got?”

“I had capped the reply length to save money. It didn’t save money and it cut the message off. The cap is higher now, and if a reply is ever cut off, the system won’t let the team send it.”

Recap

• Rate limits cap requests and tokens per minute; exceeding them returns 429.
• Retry only temporary failures, with exponential backoff plus jitter, respecting retry-after, with a cap on attempts.
• Retries smooth over bursts; a sustained load above your limit needs less concurrency, a higher tier, or the Batch API.
• Chat apps re-send history every turn, so input cost grows with the square of the conversation length.
• Measure with count_tokens and usage, keep requests within your own budget by trimming and summarising, and always check for max_tokens cut-offs.

With backoff and budgets in place, the tagger ran at 30 workers without losing a review, and Mrs. Kulkarni’s next reply cost ₹3. “Right,” said Priya. “Your code is sturdy now. But it’s still a script on your laptop, and the WhatsApp team writes Node, not Python. Your AI needs a front door other programs can knock on.”

Go deeper

• Anthropic docs (rate limits, token counting): https://docs.anthropic.com
• OpenAI docs (rate limits): https://platform.openai.com/docs
• Python asyncio (for async retries): https://docs.python.org/3/library/asyncio.html