Prompt caching on Anthropic and OpenAI

The first of the month at Crumb & Co. meant one thing for Meera: invoices. She went through them at the big table in the back office with a cup of cutting chai, and this month she stopped at the one from the AI provider.

“Arjun. What is this line? Twenty-two thousand three hundred rupees for ‘store operations assistant’.”

Arjun knew the bot well. He had built it after Module 20, when the team decided the handbook was not private enough to need a local model and a cloud model gave better answers. Store managers asked it questions on a tablet behind the counter: “What is the closing checklist?”, “Who approves a refund over ₹2,000?”, “What temperature should the display fridge be?” About 2,000 questions a month across the six stores.

“Every question sends the store procedures part of the handbook,” he said. “About 30,000 tokens (the word-pieces a model reads and charges by, roughly three-quarters of an English word each). The model needs the handbook to answer.”

“Every question sends the same handbook?”

“Yes.”

“And the managers say it sits there for three or four seconds before it says anything. The Baner manager reads the answer off the laminated sheet faster.” Meera put her pen down. “We are paying to send the same book two thousand times a month, and it is slow. Can it remember the book?”

Priya, walking past with her laptop, stopped. “It can. That is exactly what prompt caching is for.”

Paying to read the same book two thousand times

Priya pulled a chair up to the whiteboard and drew a stick figure behind a counter.

“Imagine you hire a temp worker for the Baner store. Every time a manager asks a question, the temp reads the entire procedures section of the handbook, cover to cover, then answers. Then they forget it. Next question, they read it all again.”

“That’s absurd,” Arjun said.

“It’s what your bot does. A model is stateless, remember from Module 01: it keeps nothing between requests. So every request has to carry the whole handbook, and the model has to read all 30,000 tokens before it writes the first word of the answer. You pay for that reading, and the manager waits for it.”

“So what does caching change?”

“The temp keeps their reading notes on the desk for a few minutes. If the next question comes with the same handbook, they skip the reading and go straight to the question. The provider charges you a small fee for keeping the notes, and a much smaller price when the notes get used.”

She wrote three words on the board: write, read, expire.

• A cache write happens the first time. The provider processes your prompt normally and saves its work. You pay a little extra for that: 1.25 times the normal input price on Anthropic.
• A cache read (also called a cache hit) happens when a later request starts with exactly the same text. The provider loads the saved work instead of redoing it. You pay about 0.1 times the normal input price for those tokens.
• The saved work expires after a few minutes of not being used. Five minutes is the default on Anthropic, and every read resets the clock for free.

“Where does the analogy break?” Arjun asked.

“Two places. The notes are not a summary: the provider saves the model’s internal working for every token of the prefix, which we open up in lesson 3. And a real temp could reuse notes if you changed one page. The model cannot. If one character near the start differs, everything after it is redone. That rule drives the next lesson.”

That rule has a name. Prompt caching is prefix caching: the provider can only reuse work for the prefix (the beginning of the prompt, from the very first character up to some point) that is byte-for-byte identical to a prompt it has seen recently. The point where you say “cache everything up to here” is called a cache breakpoint.

Arjun’s first attempt

Arjun had the docs open before Priya finished her chai. He found the cache_control field, added it to his code, and deployed. Here is what his request looked like:

response = client.messages.create(
    model="claude-opus-5-5",
    max_tokens=1000,
    system="You are Crumb's store operations assistant. Answer from the handbook only.",
    messages=[{
        "role": "user",
        "content": [{
            "type": "text",
            "text": f"Handbook:\n{HANDBOOK}\n\nQuestion: {question}",
            "cache_control": {"type": "ephemeral"},   # "cache this, please"
        }],
    }],
)
print(response.usage)

He sent three questions from his test list and printed the usage each time:

cache_creation_input_tokens=30430  cache_read_input_tokens=0  input_tokens=0
cache_creation_input_tokens=30427  cache_read_input_tokens=0  input_tokens=0
cache_creation_input_tokens=30436  cache_read_input_tokens=0  input_tokens=0

cache_creation_input_tokens is how many tokens were written to the cache in this request. cache_read_input_tokens is how many were served from the cache. Three writes, zero reads. No error, no warning.

“It’s caching,” he said. Then he did the maths. Every request now paid the write premium of 1.25 times on 30,000 tokens, and no request ever read anything back. If he left it running, the monthly bill would go up by a quarter, from about ₹22,300 to about ₹27,600.

Where the bookmark belongs

Priya drew a long horizontal bar on the board and split it into three coloured sections.

“Every request the provider sees is rendered in a fixed order,” she said. “On Anthropic it is tools, then system, then messages. The cache reads that bar from the left. A breakpoint is a bookmark that says: save everything to the left of here.”

“So I want the bookmark right after the handbook and before the question.”

“Exactly. Move the handbook into the system prompt, put the breakpoint on it, and let the question live in the user message where it belongs.”

flowchart LR
    T["Tools (none here)"] --> S1["System: instructions"]
    S1 --> S2["System: handbook, 30,394 tokens"]
    S2 --> BP["Breakpoint: cache up to here"]
    BP --> U["User: the question, 18 tokens"]

Because tools render first and system second, a breakpoint on the last system block caches the tools and the system prompt together. Anything after the breakpoint, here the question, is billed at the normal input price and reported as input_tokens.

Following one question through the cache

Arjun rewrote the request, then he and Priya watched the real traffic one morning. Here are three requests, traced end to end. The shared prefix (instructions plus handbook) came to 30,412 tokens. Prices are for claude-opus-5-5 at the time of writing: $4permillioninputtokens,$5 per million for cache writes (1.25x), $0.20permillionforcachereads(0.05xonthismodel;mostmodelschargeabout0.1x),and$20 per million output tokens. Rupees use about ₹88 to the dollar, which is approximate.

Request 1, 9:02 a.m. The Baner manager asks: “What is the closing checklist for the Baner store?” Nobody has asked anything for twelve hours, so the cache is empty.

cache_creation_input_tokens = 30,412   (written: instructions + handbook)
cache_read_input_tokens     = 0
input_tokens                = 18       (the question, after the breakpoint)
output_tokens               = 240

Cost: 30,412 × $5/M=$0.1521 for the write, 18 × $4/M=$0.0001 for the question, 240 × $20/M=$0.0048 for the answer. Total **$0.157,about₹13.81\ast \ast .Thatis\ast more\ast thanthesamerequestwithoutcaching($0.127, about ₹11.13), because of the write premium.

Request 2, 9:05 a.m. The Kothrud manager asks: “What is the closing checklist for the Kothrud store?” Three minutes later, same handbook, same instructions, different question.

cache_creation_input_tokens = 0
cache_read_input_tokens     = 30,412   (served from the cache)
input_tokens                = 18
output_tokens               = 240

Cost: 30,412 × $0.20/M=$0.0061 for the read, plus $0.0001and$0.0048 as before. Total $0.011, about ₹0.97. Notice that the cache is shared across everyone using the same prefix in your organisation’s workspace. The Kothrud manager benefited from the Baner manager’s question.

The read also reset the five-minute clock, so the entry now lives until 9:10.

Request 3, 9:21 a.m. The Viman Nagar manager asks a question after sixteen quiet minutes. The entry expired at 9:10. The provider writes it again: 30,412 creation tokens, $0.157, exactly like request 1.

Arjun put the three side by side:

Request	Cache result	Cost (USD)	Cost (INR, approx.)	Without caching
1, 9:02	write	$0.157	₹13.81	$0.127 (₹11.13)
2, 9:05	read	$0.011	₹0.97	$0.127 (₹11.13)
3, 9:21	write (expired)	$0.157	₹13.81	$0.127 (₹11.13)

Then the month. Questions at Crumb come in bursts: opening at 7 a.m., the lunch rush, closing at 9 p.m. Arjun’s logs showed that about one question in ten arrived after a quiet gap longer than five minutes. So out of 2,000 questions, about 200 are writes and 1,800 are reads:

• Without caching: 2,000 × $0.127=\ast \ast$253, about ₹22,300**. That was Meera’s invoice.
• With caching: 200 × $0.157+1,800\times$0.011 = $51, about ₹4,500.

The managers noticed something too. The time to first token (how long before the first word of the answer appears, which you met in Module 19) fell from about 3.4 seconds to about 1.3 seconds on cache hits in Arjun’s measurements. Your numbers will differ with model, prompt length and time of day, but the direction is always the same: the model skips reading 30,000 tokens, so it starts sooner.

The real code, step by step

Here is the version Arjun shipped, built up in three steps. Install the SDK first:

uv add anthropic python-dotenv

Step 1: a request with an explicit breakpoint. The system parameter accepts a list of text blocks instead of a single string. That lets you mark one block with cache_control.

from pathlib import Path

import anthropic
from dotenv import load_dotenv

load_dotenv()                      # reads ANTHROPIC_API_KEY from .env
client = anthropic.Anthropic()

INSTRUCTIONS = ("You are Crumb's store operations assistant. Answer store managers' "
                "questions using only the handbook below. Quote the section number.")
HANDBOOK = Path("store_procedures.md").read_text(encoding="utf-8")  # ~30,000 tokens

def ask(question: str):
    response = client.messages.create(
        model="claude-opus-5-5",
        max_tokens=1000,
        system=[
            {"type": "text", "text": INSTRUCTIONS},
            {"type": "text", "text": HANDBOOK,
             "cache_control": {"type": "ephemeral"}},   # breakpoint: cache up to here
        ],
        messages=[{"role": "user", "content": question}],   # unique part, after it
    )
    return response

Line by line, the new parts:

• system=[...] is a list of two blocks. The instructions come first, the handbook second. Order matters: both are part of the cached prefix.
• "cache_control": {"type": "ephemeral"} on the handbook block is the breakpoint. “Ephemeral” means the entry is temporary, with the default five-minute lifetime.
• The question goes in messages, after the breakpoint, so it never becomes part of the cache key.

Step 2: read the usage and turn it into money. Never assume caching works. The response tells you exactly what happened.

# $ per million tokens for claude-opus-5-5, at the time of writing. Check the pricing page.
PRICE = {"input": 4.00, "write": 5.00, "read": 0.20, "output": 20.00}
INR_PER_USD = 88   # approximate

def report(response) -> float:
    u = response.usage
    written = u.cache_creation_input_tokens or 0
    read = u.cache_read_input_tokens or 0
    usd = (u.input_tokens * PRICE["input"] + written * PRICE["write"]
           + read * PRICE["read"] + u.output_tokens * PRICE["output"]) / 1_000_000
    print(f"written={written:,} read={read:,} uncached={u.input_tokens} "
          f"output={u.output_tokens}  ${usd:.4f}  ₹{usd * INR_PER_USD:.2f}")
    return usd

for q in ["What is the closing checklist for the Baner store?",
          "What is the closing checklist for the Kothrud store?"]:
    r = ask(q)
    report(r)

Output on Arjun’s machine:

written=30,412 read=0 uncached=18 output=240  $0.1569  ₹13.81
written=0 read=30,412 uncached=18 output=242  $0.0110  ₹0.97

The or 0 guards against the fields being missing on responses that did not involve caching.

Step 3: automatic caching for conversations. The handbook bot answers one question at a time. Crumb’s support chat from Module 02 is different: it is a conversation, and every turn resends the whole history. There the thing worth caching is the growing history itself. Anthropic offers automatic caching for this: pass cache_control once at the top level of the request, and the provider places the breakpoint on the last cacheable block for you, moving it forward as the conversation grows.

response = client.messages.create(
    model="claude-opus-5-5",
    max_tokens=1000,
    cache_control={"type": "ephemeral"},   # automatic: breakpoint follows the conversation
    system=SUPPORT_INSTRUCTIONS,
    messages=history,                       # all earlier turns plus the newest message
)

On turn 5, the prefix of turns 1 to 4 was written on turn 4, so it is read from the cache, and only the newest exchange is new. Automatic caching suits conversations. For a shared document with a different question each time, like the handbook bot, place the breakpoint yourself at the end of the shared part. You can use up to four explicit breakpoints in one request, which helps when you have layers that change at different speeds (you will use this in the next lesson).

What the provider actually keeps

Arjun asked the question every engineer eventually asks. “When it says ‘cached’, is it storing my handbook text somewhere?”

“Not the text,” Priya said. “You send the text every time anyway. What it stores is the model’s work on that text.”

When a model reads your prompt, it does a large amount of arithmetic for every token: for each layer of the network it computes numbers that describe what that token means in context. That arithmetic is most of the cost of reading a long prompt, and it is the reason for the slow first token. Prompt caching saves those computed numbers, keyed by the exact prefix. In lesson 3 you will meet them by name: the KV cache.

Knowing this explains every rule you have seen so far:

• Exact prefix match. The numbers for token 5,000 depend on every token before it. Change token 10, and token 5,000’s numbers are different too. So only an identical prefix can reuse the work.
• Per model. Those numbers come from one model’s internal weights. Another model cannot use them. Caches are also kept separately per workspace, so another company’s requests never read yours.
• A lifetime. The saved numbers are large and occupy expensive memory on the provider’s machines, so they cannot be kept forever.
• A write premium. Saving and storing them costs the provider something.
• A minimum size. Below a certain length the bookkeeping is not worth it. On Anthropic the minimum is 512 to 4,096 tokens depending on the model. Shorter prefixes are silently not cached: cache_creation_input_tokens stays at 0, with no error.

It also explains what caching does not do: store answers. Request 2 was not a replay of request 1. The model read the Kothrud question and wrote a fresh answer about Kothrud. Caching changes cost and speed, not what the model says.

OpenAI does it for you, mostly

“Is this only Anthropic?” Meera asked. “Our Module 19 fallback goes to OpenAI.”

OpenAI caches automatically. For prompts of 1,024 tokens or more, it reuses the longest prefix it has recently seen, with no change to your code. You read the result from the usage:

import os

from dotenv import load_dotenv
from openai import OpenAI

load_dotenv()
client = OpenAI()                                  # reads OPENAI_API_KEY
MODEL = os.getenv("OPENAI_MODEL", "gpt-5-mini")    # model names change; see the models page

resp = client.chat.completions.create(
    model=MODEL,
    messages=[
        {"role": "system", "content": INSTRUCTIONS + "\n\n" + HANDBOOK},  # stable first
        {"role": "user", "content": "What is the closing checklist for the Baner store?"},
    ],
    prompt_cache_key="store-ops-handbook",   # optional hint: these requests share a prefix
)
u = resp.usage
print(u.prompt_tokens, u.prompt_tokens_details.cached_tokens)

Running it twice within a few minutes printed:

29870 0
29870 29824

The counts differ from Claude’s because each provider uses its own tokenizer (the rule for cutting text into tokens). On the newer Responses API, the number lives at usage.input_tokens_details.cached_tokens. The optional prompt_cache_key helps OpenAI route requests that share a prefix to the same place, which can raise the hit rate. The discount varies by model and entries are evicted after some minutes of inactivity; check OpenAI’s current docs, because these figures change.

Gemini has both flavours: implicit caching that happens automatically, and explicit context caching where you create a named cache and refer to it.

On every provider, caching rewards prompts whose beginning stays identical, and only the usage numbers tell you it is working.

Where it breaks

Arjun’s shipped version worked. Over the next week, Priya made him find each way it could quietly stop working.

The prefix is too short. Arjun’s first test used a two-page sample handbook of about 400 tokens: zero written, zero read, no complaint. Test with a realistic prompt.

Traffic is too sparse. The Viman Nagar store gets few questions after lunch. Almost every afternoon request there is a write. That is fine as long as the busy stores carry the average. Watch the monthly ratio of reads to writes, not one request.

The model changes. The gateway Arjun built in Module 19 falls back to OpenAI when Claude is down. The fallback model has its own cache, which starts cold. And when Crumb upgrades to a newer Claude model, every cache entry is useless on day one. Expect a short spike in cost after any model switch.

Something in the prefix changes when you were not looking. This is the big one, and it gets its own lesson next.

Try it yourself

Take any long document you have (a policy, a long README, a chapter of a book) of at least 5,000 tokens. Put it in the system prompt with a breakpoint, and send three different questions in a row with the report() function above.

1. Predict the written, read and uncached numbers for each request before you run it.
2. Then move the breakpoint into the user message, attached to the question, and run the three questions again. What changes?
3. Wait six minutes and send a fourth question. What do you see?

Show the answer

1. Request 1 shows a write of the full prefix and a read of 0. Requests 2 and 3 show a write of 0 and a read equal to the prefix size. uncached is the size of each question, a few dozen tokens.
2. With the breakpoint on the question, every request shows a large write and a read of 0, because the cache key now includes each unique question. Your cost per request goes up by about 25% on the input side.
3. After six minutes of silence the entry has expired, so the fourth request writes it again. If your fourth request read from the cache instead, something else (another test, a colleague) used the same prefix within the window and refreshed it.

Common mistakes

• Putting the breakpoint after the unique part. The cache key includes everything up to the breakpoint. Put it at the end of what requests share, never after the question.
• Assuming caching is on because the code says so. Read cache_read_input_tokens (Anthropic) or cached_tokens (OpenAI). Zero reads on repeated traffic means something is wrong.
• Thinking the cache stores answers. It stores the model’s work on your prompt. Every answer is generated fresh. If you want to reuse whole answers for identical questions, that is a separate response cache you build yourself, with its own risks of serving stale answers.
• Testing with a tiny prompt. Prefixes below the model’s minimum (512 to 4,096 tokens on Anthropic, 1,024 on OpenAI) are not cached and nothing tells you.
• Picking the one-hour lifetime by default. It costs 2 times the base price to write. Use it when traffic is steady but sparser than once every five minutes.

Recap

• Long, repeated prompts cost money and time on every request because the model is stateless and must re-read them.
• Prompt caching saves the provider’s work on an identical prefix for a few minutes. Writes cost a bit more than normal input; reads cost about a tenth (even less on some newer models).
• On Anthropic you mark a cache breakpoint with cache_control, at the end of the shared part, or use top-level automatic caching for growing conversations. OpenAI caches prompts of 1,024 tokens or more automatically.
• The usage fields tell you the truth: tokens written, tokens read, and uncached tokens after the breakpoint.
• With the default five-minute lifetime, caching pays off from the second request that shares a prefix.

That evening Arjun showed Meera a new dashboard tile: “Handbook bot, this week: 90% of handbook tokens read from cache.”

“Explain it to me like I’m a store manager,” she said.

“The model used to re-read the whole handbook for every question. Now, if someone asked anything in the last five minutes, it keeps its reading notes and goes straight to the question. Reading the notes costs about a twentieth of reading the book. So we pay full price around two hundred times a month instead of two thousand. And it answers faster.”

“Good. Do the same for the customer chat.”

He had already tried. The customer chat’s dashboard tile said something less pleasing: 0.0%. And its bill had gone up.

Go deeper

• Anthropic documentation, including the prompt caching guide and pricing: https://docs.anthropic.com
• OpenAI documentation, including prompt caching and the usage fields: https://platform.openai.com/docs
• Gemini API documentation, including context caching: https://ai.google.dev/gemini-api/docs