Module 21
Prompt Caching and Cost/Latency Optimisation
Crumb's handbook bot and support chat resend the same long prompts thousands of times a month, and the bill and the slow first word show it. You learn how prompt caching on Anthropic and OpenAI works, how to lay out prompts so the cache actually hits, what the provider is really storing (the KV cache), and why a request has a fast-reading prefill phase and a slow-writing decode phase. You finish with a table of levers that cut cost and latency without quietly cutting quality.
By the end of this module you can
- add prompt caching to Anthropic and OpenAI calls and prove it works from the usage fields
- lay out a prompt from stable to volatile and find the first byte that breaks a cache
- measure a cache hit rate and turn it into rupees and dollars saved per month
- explain what a KV cache stores, why it needs an exact prefix, and what it costs in memory
- measure time to first token and decode speed, and explain why output tokens cost more
- choose between caching, shorter outputs, smaller models, streaming and Batch APIs for a given cost or latency problem
Lessons
4 in this module- 01Prompt caching on Anthropic and OpenAIYou can place a cache breakpoint at the end of the shared part of a prompt, read the cache usage fields on Anthropic and OpenAI, and work out what caching saves per request and per month.
- 02Prefix caching and cache-friendly prompt designYou can order a prompt from stable to volatile, hunt down silent cache invalidators by finding the first differing byte between two requests, and measure the cache hit rate.
- 03KV cache fundamentalsYou can explain how queries, keys and values let a model reuse past work while generating, estimate the memory a KV cache needs, and connect it to why prompt caching needs an exact prefix.
- 04Batching, prefill and decodeYou can measure prefill and decode separately, explain why output tokens cost more, and pick the right lever (caching, shorter output, batching, Batch APIs) for a latency or cost problem.