Prefix caching and cache-friendly prompt design

Arjun had been confident on Monday. The handbook bot’s cache was working, the bill had fallen by four-fifths, and Meera had asked him to do the same for the customer support chat. The support chat was the busiest thing Crumb ran: in the weeks before Diwali it handled about 12,000 customer messages a month about late deliveries, allergy questions and refunds.

So on Monday afternoon he added a cache_control breakpoint to the support bot’s system prompt and deployed it.

On Wednesday morning his dashboard said: cache hit rate 0.0%. Not low. Zero. And the projected bill for the support chat had gone up, from about ₹26,600 a month to about ₹33,200.

“I did exactly what worked for the handbook,” he told Priya. “Breakpoint at the end of the system prompt. Shared part first, question after it.”

“Show me the system prompt,” she said. “Not the code that builds it. The actual text the model sees, for two different customers.”

Arjun’s first attempt

Here is the function that built the support bot’s system prompt. It had been in the codebase since Module 03, and nobody had looked at it in months.

from datetime import datetime

def build_system(customer_name: str) -> list[dict]:
    header = f"Today is {datetime.now()}. Customer: {customer_name}.\n\n"
    return [{
        "type": "text",
        "text": header + SUPPORT_INSTRUCTIONS + "\n\n" + POLICIES_AND_MENU,
        "cache_control": {"type": "ephemeral"},   # added on Monday
    }]

It looked harmless. The model needs to know today’s date to answer “Will my hamper arrive by Friday?”, and greeting customers by name is friendly. The tools list (the four functions from Module 06, such as order_status and start_refund) came first, then this system prompt of about 5,300 tokens, then the conversation.

Arjun printed the usage for two customers who wrote within a few seconds of each other:

Rohan:   cache_creation_input_tokens=6200  cache_read_input_tokens=0  input_tokens=100
Anjali:  cache_creation_input_tokens=6200  cache_read_input_tokens=0  input_tokens=100

Every request wrote 6,200 tokens to the cache. No request read any. That is the signature you met in lesson 1: the breakpoint is on something that is different every time.

“But the instructions and the policies are identical for both customers,” Arjun said. “That’s 99% of the prompt.”

“Where in the prompt?”

He looked again. The 99% that was identical came after the 1% that was not.

The stack of plates

Priya drew a stack of plates on the whiteboard.

“Think of the cache as a stack of plates in Crumb’s kitchen. Each plate is a token. The provider builds the stack from the bottom up, in the order your prompt is rendered. When the next request comes in, it compares plates from the bottom. As long as each plate matches, it can reuse that part of the stack. At the first plate that differs, it has to take that plate and every plate above it off and build them again.”

“And my bottom plate is the time.”

“Your bottom plate is the time to the microsecond. datetime.now() gives you something like 10:31:07.482913. Two requests never share it. So the stack is rebuilt from almost the very bottom, every single time.”

“If I’d put the date at the top of the stack instead…”

“Then only the top plate changes, and the other 6,200 get reused. Change the bottom plate and you rebuild the whole stack. Change the top plate and you rebuild one plate.”

Arjun asked where the analogy broke, because he had learned to.

“Two places. First, the provider does not reuse plates one by one wherever they happen to match. It only reuses a stack it saved at a breakpoint, so matching plates that sit between breakpoints are not automatically cached. Second, you do not choose the order of the main sections. On Anthropic the stack is always built tools first, then system, then messages. A change in the tools list changes a plate under your system prompt too.”

Finding the first different byte

“Let’s stop guessing,” Priya said. “Render both requests as text, exactly as the provider would see them in order, and find the first character where they differ.”

This is the single most useful debugging technique for caching, so here it is in full. It does not call the API at all.

import json
import os

def render(tools: list[dict], system: list[dict], messages: list[dict]) -> str:
    """Approximate what the provider sees, in its order: tools, system, messages."""
    return (json.dumps(tools, ensure_ascii=False)
            + "\n" + "".join(block["text"] for block in system)
            + "\n" + json.dumps(messages, ensure_ascii=False))

def first_difference(a: str, b: str) -> None:
    n = len(os.path.commonprefix([a, b]))     # length of the identical start
    print(f"Request length: {len(a):,} characters")
    print(f"Identical prefix: {n:,} characters (about {n // 4:,} tokens)")
    print(f"A continues: {a[n:n + 30]!r}")
    print(f"B continues: {b[n:n + 30]!r}")

a = render(TOOLS, build_system("Rohan Kulkarni"), [{"role": "user", "content": "Where is my order?"}])
b = render(TOOLS, build_system("Anjali Deshpande"), [{"role": "user", "content": "Is the plum cake eggless?"}])
first_difference(a, b)

os.path.commonprefix is a standard library function that returns the longest common starting string of a list of strings. It works on any strings, not only file paths. The “about 4 characters per token” rule is a rough estimate for English; use the provider’s token counter when you need an exact number.

The output for Arjun’s two customers:

Request length: 25,212 characters
Identical prefix: 3,627 characters (about 906 tokens)
A continues: '7.482913. Customer: Rohan Kulk'
B continues: '9.115274. Customer: Anjali Desh'

Read it carefully. The first 3,600 characters (the tools) match, plus 27 characters of the system prompt: Today is 2026-11-04 10:31:0. Then Rohan’s request has a 7 where Anjali’s has a 9, because they arrived two seconds apart. From character 3,628 onward, nothing can be reused, which means the instructions and policies, about 5,300 tokens, are recomputed for every customer.

The layered layout

Priya wiped the plates off the board and drew five horizontal layers instead, from the slowest-changing at the bottom to the fastest at the top.

flowchart TD
    L1["1. Tools: fixed list, sorted by name"] --> L2["2. System instructions: frozen text"]
    L2 --> L3["3. Reference docs: policies, menu, handbook"]
    L3 --> B1["Breakpoint: shared by every customer"]
    B1 --> L4["4. Conversation history: append-only"]
    L4 --> L5["5. Latest message plus volatile context"]
    L5 --> B2["Automatic breakpoint: this conversation"]

The rule for each layer:

1. Tools never vary per request. Build the list once at start-up, in a fixed order. If some customers should not use a tool, do not remove it from the list; tell the model in the instructions, or reject the call in your code.
2. System instructions are frozen text. No dates, no names, no request IDs, no “you are talking to a premium customer”.
3. Reference documents (the delivery policy, the allergen menu, the store hours) change rarely, perhaps once a season. When they change, the cache is cold for one request and then warm again.
4. Conversation history is append-only. Each turn adds to the end. Earlier turns are never edited, reordered or reformatted.
5. The latest message and volatile context go last: today’s date, the customer’s name, their order number, any documents your RAG search retrieved for this question (Module 09). All of these change per request, so they belong at the very top of the stack.

Here is Arjun’s rewrite:

from datetime import date

TOOLS = sorted(ALL_TOOLS, key=lambda tool: tool["name"])   # fixed list, fixed order

SYSTEM = [
    {"type": "text", "text": SUPPORT_INSTRUCTIONS},          # frozen
    {"type": "text", "text": POLICIES_AND_MENU,              # changes once a season
     "cache_control": {"type": "ephemeral"}},                # breakpoint 1: shared by all
]

def latest_message(customer_name: str, text: str, retrieved: str = "") -> dict:
    """Volatile context rides along with the newest user message, at the end."""
    context = (f"<context>\nToday: {date.today():%A %d %B %Y}\n"
               f"Customer name: {customer_name}\n{retrieved}</context>\n\n")
    return {"role": "user", "content": context + text}

def reply(history: list[dict], customer_name: str, text: str):
    messages = history + [latest_message(customer_name, text)]
    return client.messages.create(
        model="claude-opus-5-5",
        max_tokens=1000,
        tools=TOOLS,
        system=SYSTEM,
        messages=messages,
        cache_control={"type": "ephemeral"},   # automatic breakpoint 2: follows the conversation
    )

What changed, line by line:

• sorted(ALL_TOOLS, ...) fixes the tool order once. The list is identical for every request.
• The instructions and the policies are separate blocks with no f-strings anywhere. The breakpoint sits on the policies block, so tools plus system are cached as one entry that every customer shares.
• latest_message() puts the date and name inside a small <context> section at the start of the newest user message. The date is now a day, not a microsecond, which is all the model needs to reason about deliveries.
• The top-level cache_control turns on automatic caching for the conversation. On turn 4, the history of turns 1 to 3 is read from the cache that turn 3 wrote. Explicit and automatic breakpoints can be combined like this; the automatic one uses one of the four breakpoint slots.

A subtle point: the old messages in history still contain their old context sections with yesterday’s date or a different time. That is fine. They were written once and are never edited, so they stay byte-identical turn after turn.

Running first_difference() on Rohan and Anjali again:

Request length: 25,231 characters
Identical prefix: 24,786 characters (about 6,196 tokens)
A continues: 'Rohan Kulkarni\n</context>\n\nWh'
B continues: 'Anjali Deshpande\n</context>\n\nI'

The first difference is now the customer’s name, inside the context block at the very end. Everything before it matches, including the date, because both customers wrote on the same day. The identical prefix grew from about 906 tokens to about 6,196, and the 6,180 tokens before the breakpoint are now read from the cache.

Measuring the hit rate

A single request’s usage tells you whether that request hit the cache. To know whether your design works, you need a rate across many requests. The cache hit rate is the share of input tokens that were served from the cache:

hit rate = cache_read / (cache_read + cache_creation + input_tokens)

All three numbers come from the usage fields. Arjun added a small tracker:

from collections import Counter

totals = Counter()

def track(response) -> None:
    u = response.usage
    totals["read"] += u.cache_read_input_tokens or 0
    totals["written"] += u.cache_creation_input_tokens or 0
    totals["uncached"] += u.input_tokens

def hit_rate() -> float:
    all_input = totals["read"] + totals["written"] + totals["uncached"]
    return totals["read"] / all_input if all_input else 0.0

Now the worked numbers, before and after, for one ordinary message. Prices are for claude-opus-5-5 at the time of writing: $4permillioninputtokens,$5 per million for cache writes, $0.20 per million for cache reads. Rupees use about ₹88 per dollar (approximate). Output tokens are the same in both versions, so they are left out.

Before (timestamp at the front): written 6,200, read 0, uncached 100.

• Hit rate: 0 / (0 + 6,200 + 100) = 0%.
• Input cost: 6,200 × $5/M+100\times$4/M = $0.0314, about ₹2.76.
• For comparison, with no caching at all: 6,300 × $4/M=$0.0252, about ₹2.22. Arjun’s caching made each message 25% more expensive.

After (layered layout, a customer writing during a busy hour): written 0, read 6,180, uncached 120.

• Hit rate: 6,180 / (6,180 + 0 + 120) = 98.1%.
• Input cost: 6,180 × $0.20/M+120\times$4/M = $0.0017, about ₹0.15.

Over a month of 12,000 messages, Arjun’s logs showed that about 5% of requests still wrote a fresh entry (the first message after a quiet spell overnight). So:

The fix was moving one line of text from the start of the prompt to the end.

The silent invalidators

Arjun’s timestamp was one member of a family. Priya made him write the whole family on a sticky note and put it on his monitor. Every item here breaks the cache without an error, a warning or a wrong answer.

• Timestamps and dates in the prefix. datetime.now(), “Last updated 10:31”, even date.today() if it sits at the front (the cache would then reset at midnight, which is tolerable, but it still belongs at the end).
• IDs. A request ID, a session ID or a UUID in the system prompt for “tracing”. Put IDs in your logs or request metadata, not in the prompt.
• Unsorted JSON. If you put a dictionary into the prompt with json.dumps(menu), its key order follows how the dictionary was built. Build it differently in two places and the bytes differ. Use json.dumps(menu, sort_keys=True).
• A tool list that varies. Adding the start_refund tool only for logged-in customers, or building the list from a Python set (whose order of strings can change between processes), changes the very first section of the prompt and invalidates everything after it.
• Per-user personalisation in the system prompt. “You are talking to Rohan, a gold member” makes every customer’s prefix unique, so no customer can reuse another’s entry. Move it to the latest message.
• Switching models. Caches are kept per model, and per workspace. The Module 19 gateway’s fallback to another provider starts cold, and so does an upgrade to a new model version.
• Editing earlier history. Reformatting old messages, stripping whitespace, or redacting a phone number from turn 2 after the fact all change bytes in the middle of the conversation, so everything after that point is recomputed.
• Invisible differences. Windows line endings (\r\n) on Sneha’s machine versus \n on the server, a trailing space, a different Unicode form of an apostrophe. You cannot see them; first_difference() can.

Where it breaks: trimming the history

Arjun’s next surprise came from his own earlier work. In Module 19 he had added a token budget: when a conversation passed 20,000 tokens, his code replaced the oldest turns with a short summary to keep costs down. Customers who chatted for a long time about a complicated catering order were exactly the ones who triggered it.

Every time the summary was rewritten, the conversation’s history changed in the middle. The cache entry for that conversation became useless and the whole thing was written again at the 1.25 times price.

“So the token budget and the cache are fighting,” Arjun said.

“They pull in different directions,” Priya said. “Trimming saves tokens on every later request. Caching saves money on tokens you keep identical. You cannot have both on the same turn. What you can do is trim rarely and in big steps. Summarise once when the conversation reaches 20,000 tokens, cutting it to 5,000, rather than nibbling one old message off every turn. Each trim costs one cold write. Nibbling costs a cold write every turn.”

Two other honest limits:

• A/B tests cost cache. If you test two versions of the system prompt, each version has its own cache, and each starts cold. With low traffic, the test itself can raise your bill.
• Short prefixes still do not cache. The layered layout cannot help a 300-token prompt: it is below the minimum cacheable size, which on Anthropic models ranges from 512 to 4,096 tokens.

Try it yourself

Write a test that would have caught Arjun’s bug before it shipped. It should build the prompt twice, for two different customers, one second apart, and fail if the stable layers are not byte-identical.

# uv add --dev pytest
import time

def stable_prefix(customer_name: str) -> str:
    """Everything before the conversation, built the way the app builds it."""
    return render(TOOLS, SYSTEM, [])
    # Arjun's old code would be: render(TOOLS, build_system(customer_name), [])

def test_prefix_is_identical_across_customers_and_time():
    first = stable_prefix("Rohan Kulkarni")
    time.sleep(1)
    second = stable_prefix("Anjali Deshpande")
    assert first == second, "Stable prefix changed between requests"

def test_volatile_context_goes_in_the_latest_message():
    msg = latest_message("Rohan Kulkarni", "Where is my order?")
    assert "Rohan" not in stable_prefix("Rohan Kulkarni")
    assert msg["content"].startswith("<context>")

Common mistakes

• Putting “helpful” context at the top. Dates, names and order numbers feel like they belong first. They belong last. The model reads the whole prompt either way.
• Assuming identical bytes are automatically cached. Reads land only where a breakpoint wrote an entry earlier. Put a breakpoint at the end of each layer that is worth caching (up to four).
• Building the tool list dynamically. A tool list that depends on the user, the time or a set changes the first section of the prompt. Keep it fixed and sorted.
• Using json.dumps without sort_keys=True for anything in the prefix. Equal dictionaries can serialise to different strings.
• Trimming or redacting history one turn at a time. Every edit to the middle of the conversation causes a full cold write. Trim rarely, in large steps.
• Judging caching by one request. Measure the hit rate over hours of real traffic, and watch it after every deployment.

Recap

• The cache matches from the first byte and stops at the first difference; everything after a difference is recomputed.
• Order the prompt from slowest-changing to fastest-changing: tools, system instructions, reference documents, append-only history, then the latest message with volatile context.
• Timestamps, IDs, unsorted JSON, varying tool lists, personalisation, model switches and edited history are silent invalidators.
• Find problems by rendering two requests and locating the first different character. Measure success with the hit rate: read / (read + written + uncached).
• Trimming history and caching pull against each other; trim rarely and in big steps.

On Friday Arjun showed Meera the support chat panel: hit rate 93% over the week, input cost down from ₹26,600 a month to about ₹3,400.

“What was wrong?” she asked.

“The very first line of the prompt was the current time, down to the microsecond. The cache can only reuse a prompt from the start up to the first thing that’s different, and the first thing was always different. I moved the time to the end.”

“One line.”

“One line.” He paused. “Sneha asked me something in code review I couldn’t answer, though. When the provider says it ‘cached’ our prompt, what is it actually keeping? It isn’t the text. We send the text every time.”

Priya overheard. “That,” she said, “is the most interesting question in this module. Tomorrow.”

Go deeper

• Anthropic documentation, prompt caching guide (placement, invalidation, breakpoints): https://docs.anthropic.com
• OpenAI documentation, prompt caching guide: https://platform.openai.com/docs
• The Python tutorial, for json, os.path and friends: https://docs.python.org/3/tutorial/