Token budgets

Meera’s printout had a yellow highlighter line across one row. A single customer, in a single afternoon, had cost Crumb ₹700 (about $8) in AI fees. The Baner store’s support chat, all 350 conversations of it, had cost ₹525 for the entire week.

Arjun opened the conversation. The customer was planning a 300-guest wedding and had pasted a 60-page catering contract into the chat. Then she asked, forty times with small changes, “Is this clause okay for my order?” Each reply was long and helpful. Nothing in Arjun’s code had ever said “that’s enough”.

“It’s not her fault,” Meera said. “She’s a good customer. But if ten people do this on a Saturday, I’m paying for a second delivery van in chatbot fees. You set a flour budget for every store. Where’s the budget for this?”

“That’s the right question,” Priya said. “And the answer is going to be measured in tokens.”

Arjun’s quick fix

Arjun knew one setting that controlled length: max_tokens. He dropped it from 2,000 to 150. “Shorter answers, smaller bill,” he said, and replayed the wedding conversation against staging to prove it.

The replies came back like this:

Clause 7 sets the deposit at 30 percent, payable within seven days of signing. Clause 9
covers cancellation: if you cancel within 14 days of the event, the caterer keeps the deposit
and may also charge

Cut off mid-sentence. In the replay, the simulated customer did what real customers do: asked “and may also charge what?” That added more turns, each carrying the whole contract again. The replayed conversation cost more than the original, not less.

Priya pulled up the cost breakdown from the original afternoon. “Before you change anything, where did the ₹700 go? To what the model wrote, or to what it read?”

Arjun looked. Output: about ₹49. Input: about ₹650. He had been squeezing the 7 percent.

The flour budget

Priya drew three nested boxes on the whiteboard and labelled them like Crumb’s own budgets.

“How does flour spending work at Crumb?”

“Each tray has a recipe, so a set amount per tray,” Arjun said. “Each store has a daily allowance. And Meera watches the monthly total for the whole chain.”

“If one baker uses ten kilos on one tray, which limit catches it?”

“The recipe. The per-tray limit.”

“And if a store bakes a thousand reasonable trays in one day?”

“The daily store allowance.”

“Then we need the same three for tokens,” Priya said. “A limit per reply, a limit per request and a limit per customer per day. Your max_tokens was a per-tray limit set so low the trays came out half-baked. And it doesn’t touch the real cost, which was the ingredients going in.”

Where the analogy breaks: flour used is gone. Tokens are re-used in a strange way, as the next section shows: the same contract was paid for forty times.

Why turn forty costs more than turn one

A token is a small chunk of text the model reads or writes, roughly three-quarters of an English word (Hindi and Marathi usually take more tokens per word). Providers charge per token, with separate prices for:

• Input tokens: everything you send: the system prompt, any documents, the whole chat history and the new message.
• Output tokens: everything the model writes back. These cost several times more per token; Module 21 explains why.

Remember from Module 02 that chat APIs are stateless: the model remembers nothing between calls, so your app re-sends the whole conversation every time. Turn 1 sends the contract. Turn 2 sends the contract, turn 1 and the new question. Turn 40 sends the contract plus 39 earlier turns.

Show the answer

Turn 1 sends about 31,000 (contract) + 300 (system prompt) + 60 (question) = 31,360 tokens. Turn 40 sends the same plus 39 earlier exchanges of 60 + 700 = 760 tokens each, which is 29,640 more: about 61,000 tokens, nearly double. Every turn pays for the contract again, and for all the history before it.

Following the wedding conversation token by token

Priya and Arjun rebuilt the afternoon in a spreadsheet, using Claude Opus 5.5’s prices at the time of writing: $4permillioninputtokensand$20 per million output tokens, with about ₹88 to the dollar.

Turn	Input tokens	Output tokens	Cost of this turn
1	31,360	700	$0.125+$0.014 = $0.139 (₹12.2)
2	32,120	700	$0.128+$0.014 = $0.142 (₹12.5)
20	45,800	700	$0.183+$0.014 = $0.197 (₹17.3)
40	61,000	700	$0.244+$0.014 = $0.258 (₹22.7)
All 40	about 1,847,000	28,000	$7.39+$0.56 = $7.95 (about ₹700)

Three things jumped out at Arjun:

1. Input was 93 percent of the bill. Output tokens are pricier per token, but there were 66 times more input tokens.
2. The contract alone was 1,240,000 of those input tokens: 31,000 tokens, paid for 40 times.
3. A per-request check would have caught it at turn 1. No support question needs a 31,000-token prompt.

Then they replayed the afternoon with the budgets Priya proposed: a per-request limit of 12,000 input tokens, a per-reply limit of 800 output tokens and a per-customer limit of 200,000 tokens a day.

Step	Value	Decision
Customer pastes contract and asks	Counted: 31,360 input tokens	Over the 12,000 per-request limit
Bot replies without calling the model	“That document is too long for chat. Our catering team can review it: catering@crumb”	Cost of the counting call: $0
Customer emails the catering team		A human closes a ₹40,000 wedding order

The afternoon would have cost almost nothing, and a person with real authority would have read the contract, which is what the customer needed anyway.

Opening the hood: three limits and how each works

Per reply: max_tokens

Every Anthropic request must include max_tokens: the most output tokens the model may produce. (OpenAI has the same idea; its parameter name differs between its APIs, so check its docs.) When the model reaches the cap, it stops immediately, even mid-word, and the response’s stop_reason is "max_tokens" instead of "end_turn".

Here is the misconception Arjun held: he thought the model aims for max_tokens, writing shorter answers when the cap is lower. It does not. The model never sees the number. It writes as it normally would and gets cut off. To get shorter answers, ask for them in the prompt (“Answer in at most three sentences”) and set max_tokens comfortably above that as a safety ceiling.

One more subtlety: some models, including Claude Opus 5.5, think before they answer, and on those models the thinking counts as output tokens and counts toward max_tokens too. A very low cap can run out during the thinking, before a single visible word is written. Another reason not to squeeze it.

Per request: count before you send

You can ask how many input tokens a request will use without running it. Anthropic offers a token-counting endpoint, free to call. For OpenAI models, the tiktoken library counts tokens locally on your machine. Each provider uses a different tokenizer (the rules that split text into tokens, from Module 01), so the same contract might be 31,000 tokens for one model and 29,000 for another. Always count with the tokenizer of the model you are about to call.

Once you know the number, you choose before paying:

• Reject with a friendly message and a better route (email, a human, an upload form).
• Trim the oldest turns of history, or replace them with a short summary.
• Retrieve instead of pasting: store the document and send only the relevant sections, the RAG approach from Module 09.

Per customer: a usage ledger

The third limit is yours to build. Every response reports exactly what it used, in response.usage.input_tokens and response.usage.output_tokens. Record those against the customer and check the running total before the next call.

A budget is not the same thing as a rate limit, the per-minute cap you met two lessons ago. A rate limit protects the provider’s servers from bursts and is enforced by the provider. A budget protects your wallet over a day or a month and is enforced by you. You need both.

Building the budgets, one layer at a time

Layer 1: a sensible ceiling, and noticing when it is hit

import anthropic
from dotenv import load_dotenv

load_dotenv()
client = anthropic.Anthropic()
SYSTEM = "You are Crumb's friendly support assistant. Answer in at most four sentences."

response = client.messages.create(
    model="claude-opus-5-5",
    max_tokens=800,  # ceiling, not target: the prompt asks for brevity
    system=SYSTEM,
    messages=[{"role": "user", "content": "Summarise your refund policy."}],
)
print(response.stop_reason, response.usage.output_tokens)
if response.stop_reason == "max_tokens":
    print("Warning: the answer was cut off")

end_turn 96

Ninety-six tokens, a complete answer, well under the ceiling. The prompt controls the length; max_tokens only catches runaways.

Layer 2: check the size of a request before sending it

MAX_INPUT_TOKENS = 12_000


def input_tokens(messages: list[dict]) -> int:
    count = client.messages.count_tokens(model="claude-opus-5-5", system=SYSTEM, messages=messages)
    return count.input_tokens


messages = [{"role": "user", "content": open("contract.txt").read() + "\n\nIs clause 9 okay?"}]
n = input_tokens(messages)
print(n, "tokens")
if n > MAX_INPUT_TOKENS:
    print("Too long for chat: route to the catering team instead")

31346 tokens
Too long for chat: route to the catering team instead

Layer 3: trim history when a long conversation grows too big

def trim_history(messages: list[dict], limit: int = MAX_INPUT_TOKENS) -> list[dict]:
    """Drop the oldest exchanges until the request fits. Keeps the latest message."""
    trimmed = list(messages)
    while len(trimmed) > 1 and input_tokens(trimmed) > limit:
        trimmed = trimmed[2:]  # drop the oldest user + assistant pair
    return trimmed

This keeps messages in user, assistant order because it removes them in pairs. It is deliberately simple. Module 15 shows better approaches, such as summarising old turns or storing key facts separately.

Layer 4: a per-customer daily ledger

from datetime import date

DAILY_LIMIT = 200_000  # tokens per customer per day
ledger: dict[tuple[str, date], int] = {}  # use Redis or a database table in production


def answer(customer_id: str, messages: list[dict]) -> str:
    messages = trim_history(messages)
    estimate = input_tokens(messages) + 800  # input plus the largest possible reply
    used = ledger.get((customer_id, date.today()), 0)
    if used + estimate > DAILY_LIMIT:
        return "You've reached today's chat limit. Our team can help on WhatsApp."

    response = client.messages.create(model="claude-opus-5-5", max_tokens=800,
                                      system=SYSTEM, messages=messages)
    spent = response.usage.input_tokens + response.usage.output_tokens
    ledger[(customer_id, date.today())] = used + spent  # record what really happened
    return next(b.text for b in response.content if b.type == "text")

The check uses an estimate (input plus the maximum reply), because you must decide before calling. The ledger records the actual usage afterwards, because that is what you pay.

Layer 5: a step limit for agents

Agents (programs where the model chooses tools in a loop until a task is done, from Module 13) can make ten model calls for one customer message. Give every agent run two extra limits: a maximum number of loop steps, and a token total across all steps.

MAX_STEPS = 8
for step in range(MAX_STEPS):
    response = client.messages.create(...)  # one agent step
    run_total += response.usage.input_tokens + response.usage.output_tokens
    if response.stop_reason != "tool_use" or run_total > 60_000:
        break
else:
    hand_over_to_human()  # the loop used all its steps without finishing

A stuck agent calling the same tool over and over is one of the most common causes of surprise bills.

Where it breaks

The budgets went live on Thursday. Within a day, three edge cases showed up.

Two requests at once. A customer with the chat open in two tabs sent two messages in the same second. Both read used from the ledger before either wrote back, and both passed. For a daily limit, an overshoot of one request is tolerable. When it is not, reserve the estimate in the ledger before the call, then correct it to the actual figure afterwards, using an atomic increment in Redis or a database.

Trimming forgets things. A customer gave her order number in the first message. Thirty turns later, trimming had removed it, and the bot asked for it again. She was not pleased. Trimming is a blunt tool. Pin important facts (order number, allergy notes) in the system prompt or a summary, and drop only the chit-chat.

Show the answer

For English text, characters divided by four is usually within about 10 to 20 percent, which is fine for a rough budget check far from the limit. It is risky near the limit, and badly wrong for Hindi or Marathi, emoji or code, which use more tokens per character. A practical pattern: estimate first, and call the exact counter only when the estimate is within, say, 20 percent of a limit.

Different models, different counts. When the gateway at the end of this module sends a request to a backup provider, the same text may count as a different number of tokens there. Budget with some headroom rather than at the exact edge.

Try it yourself

Take any long text you have, such as a long article. Count its tokens with client.messages.count_tokens. Then work out, at $4 per million input tokens:

1. What one message carrying it costs, in dollars and rupees.
2. What the first ten turns cost in input alone, if every turn re-sends the text plus 760 tokens of history per earlier turn.

Show the answer

Say the text is 20,000 tokens. One message costs 20,000 × $4/1,000,000=$0.08, about ₹7.0. Over ten turns the input is 10 × 20,000 + 760 × (0 + 1 + … + 9) = 200,000 + 34,200 = 234,200 tokens, about $0.94 or ₹82, and 200,000 of those tokens are the same text paid for ten times. That repetition is why trimming, retrieval and the prompt caching in Module 21 matter so much.

Common mistakes

• Setting max_tokens very low to save money. Answers get cut off, customers ask again, and the bill goes up. Ask for brevity in the prompt and keep the cap as a ceiling.
• Ignoring stop_reason. A truncated answer looks complete unless you check.
• Budgeting only output. On chat apps with history and documents, input is usually most of the bill.
• Counting with the wrong tokenizer. Counts differ between providers and model families.
• No step limit on agents. One confused loop can cost more than a day of normal traffic.

Recap

• You pay per token, for input and output separately; output costs more per token, but input is often most of the bill.
• Chat history is re-sent on every turn, so long conversations snowball.
• max_tokens is a ceiling the model never sees; ask for short answers in the prompt and check stop_reason.
• Count tokens before sending and reject, trim or retrieve when a request is too big.
• Keep a per-customer ledger from response.usage, give agents step limits, and set a provider-side spending cap.

On Friday afternoon Arjun walked Meera through it. “Every message now has three checks, like flour. Each reply has a maximum size. Each request has a maximum size, and if someone pastes a whole contract we send them to a person instead. And each customer has a daily allowance, so nobody can cost more than a few hundred rupees a day, and almost everyone costs a few rupees.” Meera did the maths on her phone. “And the wedding customer?” “Catering closed her order by phone. Forty thousand rupees.”

That evening, at 7:40, the busiest hour of the week, the provider itself went down for forty minutes. No budget in the world helped. “We have one supplier,” Priya said, watching the error graph climb. “No bakery runs on one flour supplier.”

Go deeper

• Anthropic token counting and pricing: https://docs.anthropic.com
• OpenAI tokens and pricing: https://platform.openai.com/docs
• To see where token counts come from: