Cost management and prompt compression

Meera put the invoice on the table: ₹1,80,000 for one month of the WhatsApp bot.

“That bot answered about 36,000 messages,” she said. “So each answer cost us five rupees. Our support team handles a message for less than that when they’re busy. Is the bot worth it or not?”

Arjun had never thought about cost per message. He knew the monthly total because it arrived as an email. He did not know which kinds of questions were expensive, why, or what would happen at Diwali, when traffic usually triples. Triple ₹1.8 lakh was a number nobody wanted to say out loud.

“Give me two days,” he said.

Arjun’s two-day fix

Arjun did the obvious thing. He found the cheapest small model on the provider’s pricing page, switched the bot to it, and, for good measure, set max_tokens=80 so replies could not run long. He ran a quick calculation: about 80% cheaper. He was preparing a triumphant message to Meera when Priya asked if he had run the evals.

He had not. He ran the 212-case eval set from the last lesson.

	Before	After Arjun’s fix
Overall pass rate	94.3%	79.2%
Allergy cases	12 / 12	10 / 12
Replies cut off mid-sentence	0	23

One of the cut-off replies read: “Your order 4471 is out for delivery and should reach you by”. The response’s stop_reason field said max_tokens: the model had been stopped mid-thought because it hit the cap.

“You made it cheaper,” said Priya, “and worse, and you don’t yet know where the money was going. That’s like cutting the bakery’s costs by buying cheaper butter before checking whether the real problem is the electricity bill.”

The suitcase in every parcel

Priya opened a trace of one ordinary conversation. Customer: “Where is my order 4471?” Bot: “It’s out for delivery and should reach you by 5:40 p.m.”

“How many tokens do you think that took?” she asked. A token is the chunk of text a model reads and writes, roughly three quarters of an English word.

Arjun guessed. “The question is ten words, the answer fifteen. Maybe 40 tokens?”

“Nearly 19,000.”

Priya drew a courier van. “Imagine a courier who charges by weight, both directions. You pay to send a parcel, and more per kilo for the reply. Now imagine that every time a customer sends a postcard, we pack it inside a suitcase with our entire rulebook, the tool manual, six chapters of the handbook and the whole conversation so far, and post the lot. Twice.”

“Why twice?”

“Because the bot is an agent. First call: the model reads everything and decides to call the order-lookup tool. Second call: we send everything again plus the tool’s result, and the model writes the reply. Each call carries the full suitcase.”

That is the shape of LLM pricing. Providers charge per million tokens, with separate prices for input (what you send) and output (what the model writes). Output usually costs several times more per token. And with agents and chat history, input is usually the bigger bill, because it is resent on every call.

Weighing one conversation, token by token

Priya and Arjun unpacked the suitcase for that one order-status conversation. For this lesson we will use illustrative prices: $3permillioninputtokens,$15 per million output tokens, and about ₹88 to the dollar. Real prices differ by model and change often, so always check the provider’s pricing page.

What was inside each call:

Part of the request	Tokens	Needed for “Where is my order?”
System prompt (rules, tone, 14 worked examples)	3,100	Partly: ten examples cover rare cases
Tool definitions (8 tools)	1,400	Only 2 tools are relevant
Retrieved handbook chunks (6)	3,600	None: order status needs no handbook text
Conversation history	1,260	Only the last couple of turns
Customer’s message	40	Yes
Input per call	9,400

The two calls:

	Input tokens	Output tokens
Call 1: decide to look up the order	9,400	45 (the tool request)
Call 2: write the reply, with the 120-token tool result added	9,520	60
Total	18,920	105

The cost:

• Input: 18,920 × $3/1,000,000=$0.0568
• Output: 105 × $15/1,000,000=$0.0016
• Total: $0.0583, or about ₹5.13

Output, the part everyone worries about, was under 3% of the cost. Ninety-seven per cent was the suitcase.

Step 1: make cost visible

You cannot cut what you cannot see, so the first job is to record the cost of every call and tag it with what the call was for.

# cost.py: the cost of one call, from the usage the API returns
import anthropic

client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY

# Illustrative US dollar prices per million tokens. Check the pricing page.
PRICES = {"claude-opus-5-5": {"input": 3.00, "output": 15.00, "cache_read": 0.30}}
INR_PER_USD = 88

def cost_usd(response) -> float:
    price = PRICES[response.model]
    u = response.usage
    cached = u.cache_read_input_tokens or 0      # tokens served from the prompt cache
    return (u.input_tokens * price["input"]
            + cached * price["cache_read"]
            + u.output_tokens * price["output"]) / 1_000_000

response = client.messages.create(
    model="claude-opus-5-5", max_tokens=400,
    messages=[{"role": "user", "content": "Where is order 4471?"}],
)
usd = cost_usd(response)
print(f"in={response.usage.input_tokens} out={response.usage.output_tokens} "
      f"cost=${usd:.5f} (₹{usd * INR_PER_USD:.2f})")

in=14 out=38 cost=$0.00061 (₹0.05)

This bare test call is cheap because it carries no suitcase. In the real bot, log this cost on every trace together with a feature tag (“order_status”, “menu_question”, “hamper_change”, “complaint”) and a conversation ID. Then a few lines add it up:

# report.py: cost per feature from a day of trace records
from collections import defaultdict

traces = [  # in practice, loaded from your tracing tool or log files
    {"conversation": "c1", "feature": "order_status", "usd": 0.0584},
    {"conversation": "c2", "feature": "complaint", "usd": 0.1310},
    {"conversation": "c3", "feature": "order_status", "usd": 0.0571},
]

totals, conversations = defaultdict(float), defaultdict(set)
for t in traces:
    totals[t["feature"]] += t["usd"]
    conversations[t["feature"]].add(t["conversation"])

for feature, usd in sorted(totals.items(), key=lambda kv: -kv[1]):
    per_conv = usd / len(conversations[feature]) * INR_PER_USD
    print(f"{feature:13} ₹{usd * INR_PER_USD:7.2f} total, ₹{per_conv:.2f} per conversation")

order_status  ₹  10.16 total, ₹5.08 per conversation
complaint     ₹  11.53 total, ₹11.53 per conversation

A week of this answered Meera’s real question. Order-status questions were 58% of conversations and 47% of the bill, and a person could not answer them faster than the bot. The number that decides whether a feature is worth it is the cost per resolved conversation compared with the alternative, sometimes called the feature’s unit economics.

Step 2: send fewer tokens

Prompt compression means shrinking what you send without losing anything the model needs. Arjun worked down the suitcase table, checking the eval set after each change.

1. Trim the system prompt. Remove repetition and polite filler, and drop examples for rare cases. Fourteen examples became four. 3,100 tokens became 900, and the eval pass rate did not move.
2. Send only relevant tools. A cheap first step labels the message (“order status”), then the main call offers only the tools that route needs. 1,400 tokens became 350.
3. Retrieve only when needed. Order-status questions skip the handbook search entirely. For menu questions, a reranker (a model that re-sorts search results by relevance) lets you keep the best two chunks instead of six.
4. Summarise old history. Keep the last two turns word for word and replace older turns with a short running summary. 1,260 tokens became about 400. The context window (the maximum text a model can read at once) is a ceiling, not a target.
5. Prefer structure to prose. order: 4471 | status: out_for_delivery | eta: 17:40 carries the same facts as a paragraph in a quarter of the tokens.
6. Automatic compression, carefully. Research tools such as Microsoft’s LLMLingua use a small model to delete low-information tokens from long prompts. They can cut a lot, and occasionally they cut a crucial word like “not”. Keep them away from safety rules.

You can measure each change before sending anything, because the API counts tokens for free:

# measure.py: how big is the system prompt?
from pathlib import Path

def input_tokens(system: str) -> int:
    return client.messages.count_tokens(
        model="claude-opus-5-5", system=system,
        messages=[{"role": "user", "content": "Where is order 4471?"}],
    ).input_tokens

before = input_tokens(Path("prompts/support_v1.md").read_text(encoding="utf-8"))
after = input_tokens(Path("prompts/support_v2.md").read_text(encoding="utf-8"))
print(f"{before} -> {after} tokens ({100 * (before - after) / before:.0f}% smaller)")

3121 -> 921 tokens (70% smaller)

Step 3: pay less for what you still send

Prompt caching. Many requests start with the same long prefix: system prompt and tool definitions. Prompt caching lets the provider keep its work on that prefix for a few minutes, and cached input tokens are billed at a steep discount (on Anthropic, reads cost around a tenth of the normal input price; writing to the cache costs a little extra the first time). You mark where the stable prefix ends:

response = client.messages.create(
    model="claude-opus-5-5", max_tokens=400,
    system=[{"type": "text", "text": system_prompt,
             "cache_control": {"type": "ephemeral"}}],   # cache everything up to here
    messages=[{"role": "user", "content": "Where is order 4471?"}],
)
print(response.usage.cache_read_input_tokens)   # 0 the first time, then the prefix size

The rule that makes caching work: stable things first, changing things last. A timestamp or customer name near the top of the system prompt changes every request and breaks the cache for everything after it. You studied the mechanics in the caching module.

Model routing. Model routing sends each request to the cheapest setup that handles it well. Order status is easy; an angry complaint about a ruined anniversary cake is not.

# router.py: cheap rules first; a small classifier can replace them later
def route(message: str) -> str:
    text = message.lower()
    if any(word in text for word in ("allerg", "refund", "complaint", "sick")):
        return "careful"        # largest model, full context, higher effort
    if "order" in text and any(ch.isdigit() for ch in text):
        return "order_status"   # two tools, no retrieval, low effort
    return "general"

print(route("Where is order 4471?"), route("I want a refund, the cake was crushed"))

order_status careful

Each route maps to its own model, tools and settings. On Claude you can also lower the effort for easy routes with output_config={"effort": "low"}. Your evals must pass on every route, not only overall.

Batching. Work that does not need an instant answer, like tagging yesterday’s 1,200 chats by topic, can go through a provider’s batch API, which typically costs about half price in exchange for results within hours.

Cache whole answers. “What time does the Aundh store open?” has one right answer. Store it and skip the model entirely.

flowchart LR
  Q["Customer message"] --> C{"Answer cached?"}
  C -->|yes| A["Send stored answer"]
  C -->|no| R{"Route"}
  R -->|order_status| S["Small context, low effort"]
  R -->|careful| L["Full context, high effort"]

The same conversation, after

Here is the order-status conversation again, after compression and caching. The cached prefix is the system prompt plus the two tools: 1,250 tokens.

	Cached input	Uncached input	Output	Cost
Call 1	1,250	440	45	$0.00237
Call 2	1,250	560	60	$0.00296
Total				$0.00533, about ₹0.47

From ₹5.13 to ₹0.47, roughly eleven times cheaper, with the eval pass rate at 94.8%, slightly higher than before, because the model was no longer distracted by six irrelevant handbook chunks.

Step 4: cap the worst day

Averages are for planning. Budgets are for disasters.

• A token budget is a hard cap on tokens per request, per conversation or per user per day. It stops one stuck agent loop or one abusive user from burning money.
• Step limits on agent loops: six tool calls, then hand over to a human.
• Spend alerts on the provider dashboard, daily, not monthly.
• A hard monthly limit at the provider as a last safety net.

# budget.py: refuse to continue a conversation that has blown its budget
MAX_TOKENS_PER_CONVERSATION = 40_000

def within_budget(conversation_tokens: int, next_call_estimate: int) -> bool:
    return conversation_tokens + next_call_estimate <= MAX_TOKENS_PER_CONVERSATION

Where it breaks

Compressing the safety rules. The allergy policy is the last thing to trim. Keep a short list of prompt sections that no compression may touch.

Routes that misfire. A keyword router sends “my order 4471 made my son sick” to order_status if “sick” is not on the list. Route to the careful path when in doubt; a wrong cheap answer costs far more than an expensive right one.

Caches that never hit. If cache_read_input_tokens stays at 0, something at the top of your prompt changes every request. Check before celebrating.

Agent loops. Ten tool calls each resending a growing history can cost fifty times a single answer. Measure per conversation, not per call.

Exchange rates and price changes. Your rupee costs move with the dollar and with the provider’s price list. Re-check your unit economics every quarter.

Try it yourself

Take any system prompt you have written in this course. Count its tokens with count_tokens. Rewrite it to be at most half as long: remove repetition, cut rare examples, use lists instead of paragraphs. Count again, then run your eval cases against both versions and record the pass rates.

Show the answer

First-draft prompts of around 1,500 tokens usually shrink to 600 to 800 tokens with no measurable quality loss. If an eval case starts failing, you cut something load-bearing: put back the smallest piece that fixes it. A good write-up states three numbers: tokens saved per request, rupees saved per 1,000 requests at your model’s price, and pass rate before and after.

Common mistakes

• Cutting cost before measuring it. Switching models or capping output blindly cuts quality along with cost. Find the biggest line item first.
• Looking only at the monthly total. By the time the invoice arrives, the money is spent. Track cost per conversation and per feature daily.
• Putting changing data at the top of the prompt. It breaks prompt caching for everything after it.
• Forgetting that agents multiply calls. Every tool step resends the whole context.
• Compressing without evals. Every cut needs a before-and-after eval run.

Recap

• Cost is input tokens plus output tokens times price; in chat and agent apps, resent input is usually the biggest part.
• Measure cost per call, per conversation and per feature before changing anything.
• Prompt compression trims prompts, tools, retrieved chunks and history; evals confirm nothing important was lost.
• Prompt caching, routing, batching and stored answers make the remaining tokens cheaper.
• Token budgets, step limits and spend alerts protect you from the worst day.

Two days became five. Then Arjun sat down with Meera. “A ‘where is my order’ chat used to cost about five rupees, because we were posting a suitcase of instructions and handbook pages with every message, twice. Now it costs under fifty paise, and the answers are slightly better. Complaints still use the big model on purpose. The whole bill should land around ₹38,000 a month, even with Diwali, and there’s an alarm if any day goes over ₹3,000.”

Meera smiled for the first time that week. Then she frowned. “One more thing. A hotel chain wants to order corporate hampers through the bot, and their legal team has sent a questionnaire about ‘AI governance’. And someone told me we should fine-tune our own model. What do I say?” That is the next lesson.

Go deeper

• Anthropic docs, including prompt caching, token counting and message batches: https://docs.anthropic.com
• OpenAI docs, including the Batch API and pricing: https://platform.openai.com/docs
• Langfuse, for tracking cost per trace and per feature: https://langfuse.com/docs