Batching, prefill and decode

Meera had the handbook bot open on her tablet, and she was timing it with the stopwatch on her phone.

“Watch,” she said, and typed: What is the closing checklist for the Baner store?

Nothing for about two seconds. Then the answer began, and it came out steadily, word by word, at roughly the speed she could read it. The whole answer took about nine seconds.

“Two things,” she said. “First: why the pause, and then the slow typing? If it knows the answer, why not show it all at once? Second.” She swiped to the pricing page Arjun had sent her. “Claude Opus 5.5: four dollars per million input tokens, twenty dollars per million output tokens. Why do the words it writes cost five times the words we send? A word is a word.”

Arjun had learned the KV cache yesterday. He had a feeling the answer was nearby, but he could not put it together.

“Both questions have the same answer,” Priya said. “Every request has two phases, and they are limited by completely different things.”

Arjun’s first guess

Arjun had a theory before Priya reached the whiteboard. “The prompt is 30,000 tokens. That’s why it’s slow. If I cut the handbook down to the most-asked half, everything gets twice as fast.”

He tried it that afternoon, on a copy of the bot. He measured the one number he knew how to measure: the total time from sending the question to receiving the full answer.

Full handbook (30,412 tokens):  total 8.9 s
Half handbook (15,206 tokens):  total 8.2 s

Eight per cent faster, not fifty. And the half-handbook bot now answered “I don’t know” to questions about deep-cleaning the ovens, because that section was gone.

“I halved the input and barely moved the time,” he said. “So the prompt isn’t the problem?”

“The prompt is a problem,” Priya said. “You measured two different things added together and called them one number. Let’s separate them.”

The kitchen at seven in the morning

Priya drew Crumb’s central kitchen.

“An order ticket comes in for a big catering job. First, the prep. Twelve cooks read the whole ticket at once and each takes a part: one cracks eggs, one weighs flour, one melts butter. All at the same time. The more cooks you have, the faster prep goes. That’s prefill.”

“Then the plating. Each plate needs the head chef. And here’s the annoying part: for every single plate, the chef has to walk to the walk-in pantry at the back, check the whole recipe binder, and walk back. One plate, one walk. The walk takes the same time whether the chef brings back instructions for one plate or for twelve.”

“That’s decode,” Arjun said. “One token, one walk.”

“Exactly. And what’s in the pantry?”

He thought about Module 01. “The model’s weights?”

“The weights (the billions of learned numbers that make up the model). For every token it writes, the model has to read all of its weights from memory. Prefill reads them once and uses them for thousands of prompt tokens in parallel. Decode reads them once per token.”

Where the analogy breaks: the “walk” is data moving from the GPU’s memory to its arithmetic units, and it takes milliseconds, not minutes.

Prefill: the whole prompt at once

Prefill is the first phase of a request. The model takes every token of your prompt, all 30,000 of them, and runs them through its layers in parallel. Along the way it builds the KV cache from lesson 3: a key and a value for every prompt token at every layer. At the end, it predicts the first token of the answer.

Three things to remember about prefill:

• It is compute-bound. A GPU (graphics processor) has thousands of small arithmetic units, and prefill gives all of them work at once. The limit is how fast the GPU can do arithmetic.
• It sets the time to first token. Time to first token (TTFT, which you met in Module 19) is mostly queueing plus prefill. A longer prompt means more prefill, so a longer wait.
• Prompt caching skips most of it. A cache hit loads the saved keys and values for the prefix instead of computing them. That is why caching cut the handbook bot’s TTFT in lesson 1.

One extra detail for claude-opus-5-5: this model always thinks before answering (it reasons internally, and that reasoning happens before the first visible word). Thinking is generated token by token, so it shows up in TTFT too. You can reduce it with output_config={"effort": "low"} for simple questions.

Decode: one token at a time

Decode is the second phase. The model writes the answer one token at a time. For each token it must:

1. Read all its weights from memory.
2. Read the KV cache for every earlier token.
3. Do a comparatively small amount of arithmetic for one token.
4. Append one new key and value to the cache.

Step 1 dominates. Decode is memory-bandwidth-bound: the limit is how many gigabytes per second the GPU can move from memory, not how fast it can multiply. Most of the arithmetic units sit idle, waiting for data.

You can estimate a ceiling with one division. Take Llama 3.2 3B, the model you ran with Ollama in Module 20. At 4-bit quantisation (storing each weight in 4 bits instead of 16), its weights are about 2 GB. A typical laptop moves roughly 100 GB per second from memory. So:

maximum decode speed ≈ memory bandwidth / bytes read per token ≈ 100 GB/s / 2 GB ≈ 50 tokens/s

Real speed is lower, because of KV cache reads and other overheads. The numbers are illustrative, but the conclusion holds: decode speed is set by model size and memory speed, and barely by prompt length.

flowchart LR
    P["Prompt: 30,412 tokens"] --> PF["Prefill: all tokens in parallel"]
    PF --> KV["KV cache built"]
    PF --> T1["First token: sets TTFT"]
    T1 --> D["Decode step: read weights and KV cache"]
    D --> N["One more token"]
    N --> D
    N --> E["Last token: sets total time"]

Following one answer through both phases

Priya had Arjun measure both phases on his own laptop with Ollama, because Ollama reports the timings directly. Every response from ollama.chat includes:

• prompt_eval_count and prompt_eval_duration: tokens and time for prefill.
• eval_count and eval_duration: tokens and time for decode.
• load_duration: time to load the model into memory, if it was not loaded already.

All durations are in nanoseconds (billionths of a second).

# uv add ollama
import ollama

CLOSING_SECTION = open("closing_checklist.md", encoding="utf-8").read()  # ~1,400 tokens

r = ollama.chat(model="llama3.2", messages=[
    {"role": "system", "content": "Answer from this handbook section only.\n\n" + CLOSING_SECTION},
    {"role": "user", "content": "What is the closing checklist for the Baner store?"},
])

prefill_s = r["prompt_eval_duration"] / 1e9
decode_s = r["eval_duration"] / 1e9
print(f"load:    {r['load_duration'] / 1e9:.2f} s")
print(f"prefill: {r['prompt_eval_count']} tokens in {prefill_s:.2f} s "
      f"= {r['prompt_eval_count'] / prefill_s:.0f} tokens/s")
print(f"decode:  {r['eval_count']} tokens in {decode_s:.2f} s "
      f"= {r['eval_count'] / decode_s:.0f} tokens/s")

Output on Arjun’s laptop (yours will differ):

load:    0.04 s
prefill: 1412 tokens in 2.31 s = 611 tokens/s
decode:  286 tokens in 8.17 s = 35 tokens/s

Trace it end to end:

1. Load, 0.04 s. The model was already in memory; a cold start can take seconds.
2. Prefill, 2.31 s. 1,412 prompt tokens processed at 611 tokens per second. The first token of the answer appears at about 0.04 + 2.31 ≈ 2.35 s. That is Meera’s pause.
3. Decode, 8.17 s. 286 answer tokens at 35 tokens per second. That is Meera’s one-finger typing.

Per token, prefill was about 17 times faster than decode, on the same model, on the same laptop, in the same request. And the decode speed of 35 tokens per second sits below the 50 tokens per second ceiling we estimated, as expected.

One gotcha if you run this twice: Ollama keeps the KV cache of the previous prompt in memory, so a second run with the same system prompt may report a much smaller prompt_eval_count. That is prompt caching again, on your own laptop. Change the system prompt slightly to measure a cold prefill.

Why output tokens cost more

Now Meera’s second question. Look at how the GPU spends its time in each phase.

During prefill, one read of the weights serves thousands of prompt tokens at once, and the arithmetic units are busy. The GPU time per input token is small.

During decode, one read of the weights produces one token per conversation. The GPU spends most of each step moving data, with its arithmetic units waiting. The GPU time per output token is much larger. The conversation also holds its KV cache in scarce memory for the whole time it is decoding.

Providers price that difference. For claude-opus-5-5, at the time of writing:

Token type	Price per million	Relative to a normal input token
Cache read	$0.20	0.05x
Input	$4.00	1x
Cache write (5 minutes)	$5.00	1.25x
Output	$20.00	5x

Thinking tokens are output tokens too: they are generated by decode, one at a time, and billed at the output price even though you do not see them.

Here is one warm handbook answer from lesson 1, broken down, at about ₹88 per dollar (approximate):

• 30,412 cached tokens read: 30,412 × $0.20/M=\ast \ast$0.0061 (₹0.54)**
• 18 question tokens: 18 × $4/M=\ast \ast$0.00007 (₹0.006)**
• 258 output tokens: 258 × $20/M=\ast \ast$0.0052 (₹0.45)**

The 258 output tokens are less than 1% of all the tokens in the request, and almost half its cost. Once the prompt is cached, output is where the money goes.

Measuring the cloud model

Ollama hands you the phase timings. A cloud API does not, but you can measure them from the outside by streaming (Module 19) and noting when the first text arrives.

import time

import anthropic
from dotenv import load_dotenv

load_dotenv()
client = anthropic.Anthropic()

def measure(question: str) -> None:
    start = time.perf_counter()
    first_text_at = None
    with client.messages.stream(
        model="claude-opus-5-5",
        max_tokens=1000,
        output_config={"effort": "low"},     # keep thinking short for a simple lookup
        system=SYSTEM,                       # instructions + cached handbook, from lesson 1
        messages=[{"role": "user", "content": question}],
    ) as stream:
        for text in stream.text_stream:
            if first_text_at is None:
                first_text_at = time.perf_counter()   # the first visible word
        final = stream.get_final_message()
    end = time.perf_counter()

    u = final.usage
    ttft = first_text_at - start
    print(f"TTFT {ttft:.2f} s | total {end - start:.2f} s | "
          f"output tokens {u.output_tokens} | cache read {u.cache_read_input_tokens or 0:,}")

measure("What is the closing checklist for the Baner store?")
measure("What is the closing checklist for the Kothrud store?")

time.perf_counter() is Python’s most precise clock for short intervals. The loop records when the first visible text arrives; everything before that is queueing, prefill and thinking. stream.get_final_message() returns the complete message, including usage.

Output one afternoon:

TTFT 3.41 s | total 8.95 s | output tokens 262 | cache read 0
TTFT 1.32 s | total 6.87 s | output tokens 258 | cache read 30,412

The first request was a cold cache: a long prefill, so a long TTFT. The second was warm: TTFT fell by two seconds. But look at the time after the first word: 5.54 s and 5.55 s. Decode did not care about the cache at all. That is roughly 46 output tokens per second. (It is approximate: output_tokens includes the model’s thinking tokens, which were generated before the first visible word.)

Batching: one walk to the pantry, many plates

“So decode wastes the GPU,” Arjun said. “It sits waiting for weights to arrive.”

“For one user, yes. Now think about the kitchen again. If the chef has to walk to the pantry anyway, what would a smart chef do?”

“Bring back instructions for twelve plates at once.”

That is batching. The serving system runs the decode steps of many conversations together. The weights are read once per step and used for every conversation in the batch. Each conversation reads its own KV cache, so batching is not free, but the expensive part, the weights, is shared.

The illustrative arithmetic for a mid-size model on a data-centre GPU:

Batch size	Speed per user	Total tokens per second (throughput)
1	~150 tokens/s	~150
32	~110 tokens/s	~3,500

Throughput is the total number of tokens a server produces per second for everyone together. Batching raises throughput enormously, which lowers the cost of each token. The price is that each individual user gets a bit slower, because every step now does more work. That trade-off, throughput against per-user latency, is the central dial in serving LLMs.

There are two ways to batch:

• Static batching. Collect a group of requests, run them together until all of them finish, then take the next group. If one customer’s answer is 20 tokens and another’s is 900, the short one’s slot sits empty for 880 steps, and new requests queue outside.
• Continuous batching. After every decode step, finished conversations leave the batch and waiting ones join. No slot sits empty for long, and a new request starts its prefill as soon as there is room. The open-source vLLM serving engine is built around this, together with the paged attention you met in lesson 3. Most production servers now work this way.

On a provider’s API, they batch for you, which is a large part of why their price per token is low. It also explains why the same request is sometimes slower at busy times: bigger batches and longer queues.

When nobody is waiting: Batch APIs

Some work has no customer watching. Every night Crumb wants the day’s online shop reviews sorted into categories (late delivery, quality complaint, allergy concern, praise), so the morning team can act on them. About 1,500 reviews a day.

Nobody needs those answers in two seconds. Anthropic’s Message Batches API and OpenAI’s Batch API both accept a large set of requests at once, process them when the provider has spare capacity, and return results within 24 hours (often much sooner), at about 50% of the normal price.

import anthropic
from dotenv import load_dotenv

load_dotenv()
client = anthropic.Anthropic()

reviews = load_todays_reviews()   # e.g. [{"id": "R-20261104-0001", "text": "Cake arrived squashed..."}]

batch = client.messages.batches.create(requests=[
    {
        "custom_id": review["id"],    # your own ID, to match results later
        "params": {
            "model": "claude-opus-5-5",
            "max_tokens": 1000,
            "output_config": {"effort": "low"},
            "system": CLASSIFY_INSTRUCTIONS,
            "messages": [{"role": "user", "content": review["text"]}],
        },
    }
    for review in reviews
])
print(batch.id, batch.processing_status)   # e.g. msgbatch_... in_progress

The next morning, a second script collects the results:

batch = client.messages.batches.retrieve(BATCH_ID)
if batch.processing_status == "ended":
    for result in client.messages.batches.results(BATCH_ID):
        if result.result.type == "succeeded":
            message = result.result.message
            label = next(b.text for b in message.content if b.type == "text")
            save_label(result.custom_id, label)
        else:
            print(result.custom_id, result.result.type)   # errored, canceled or expired

Results come back in any order, so match them by custom_id, never by position.

The money: each review is about 400 input tokens (instructions plus review) and 20 output tokens. At standard prices that is 400 × $4/M+20\times$20/M = $0.002perreview.For45,000reviewsamonth:\ast \ast$90 (about ₹7,920)** normally, $45 (about ₹3,960) through the Batch API. If the instructions are long, prompt caching can lower the input side further. OpenAI’s Batch API works on the same idea: you upload a file of requests and download a file of results.

The levers, in one table

Arjun and Priya ended the afternoon with a table for the team wiki. Each lever works on a specific phase.

Lever	Improves	Phase	Watch out for
Prompt caching and a stable prefix	TTFT, input cost	Prefill	Write premium; silent invalidators
Shorter prompt (RAG instead of whole documents)	TTFT, input cost	Prefill	Missing information, as Arjun found
Shorter answers (“three bullet points”, sensible max_tokens)	Total time, output cost	Decode	Cutting answers off mid-sentence
Lower effort on claude-opus-5-5	TTFT and output cost (less thinking)	Decode	Weaker answers on hard questions
A smaller, faster model	Both phases, both costs	Both	Quality; measure it (Module 23)
Streaming	How long the wait feels	Neither	Changes nothing about real time
Parallel requests for independent calls	Wall-clock time of multi-call jobs	Neither	Rate limits (Module 19)
Batch APIs	Cost, about 50% off	Neither	Results take up to 24 hours

Where it breaks

Every lever trades something away. Shorter answers can drop the one step a store manager needed. Lower effort or a smaller model can be confidently wrong more often. None of this shows up in a latency graph. Priya’s rule: change one lever at a time, and check quality on a fixed set of real questions before and after. You will build exactly that kind of test set in Module 23.

And measure the right number. Arjun’s “total time” mixed prefill and decode, and sent him off to cut the handbook, which hurt answers and barely helped speed. Log TTFT, total time and output tokens separately for every request.

Try it yourself

Run the Ollama measurement with two prompts: the closing checklist section alone (about 1,400 tokens), and the same section pasted four times (about 5,600 tokens). Keep the question the same.

Predict before you run: which of the four numbers (prefill tokens, prefill tokens per second, decode tokens, decode tokens per second) should change a lot, and which should stay about the same?

Show the answer

prompt_eval_count should roughly quadruple, and so should prompt_eval_duration, so the time to first token grows by several seconds. Prefill tokens per second stays in the same range (it may even rise a little, as the GPU is used more fully, or fall a little as attention over a longer prompt costs more). Decode tokens per second should stay about the same or drop slightly, because each step now reads a longer KV cache. The number of decode tokens depends on the answer, not the prompt, so it should be similar. If the second run shows a tiny prompt_eval_count, Ollama reused its cached prefix: change one word at the start of the prompt and run it again.

Common mistakes

• Measuring only total time. It mixes prefill and decode, which respond to different levers. Log TTFT and output tokens separately.
• Shrinking the prompt to speed up long answers. Prompt length mostly affects the wait for the first word. Long answers are slow because of decode.
• Ignoring output tokens in cost estimates. At five times the input price, and with thinking billed as output, they often dominate once the prompt is cached.
• Using the real-time API for overnight jobs. If nobody is waiting, a Batch API halves the cost.
• Treating streaming as a speed-up. It makes the wait feel shorter (Module 19). The model takes exactly as long.

Recap

• Every request has two phases: prefill processes the whole prompt in parallel and sets the time to first token; decode writes one token per step and sets the typing speed.
• Prefill is limited by arithmetic, decode by memory bandwidth, because every decode step reads all the weights for one token per conversation.
• That is why output tokens cost more (five times input on claude-opus-5-5), and why thinking, which is output, adds to both cost and TTFT.
• Batching shares each read of the weights across many conversations; continuous batching (as in vLLM) keeps the batch full, trading a little per-user speed for much higher throughput.
• Batch APIs give about 50% off for work that can wait up to 24 hours.

Arjun gave Meera the short version the next morning. “The pause is the model reading our handbook and thinking. Caching made that shorter. The slow typing is the model writing, and it can only write one word at a time, because each word depends on the one before. For every word it has to read its whole brain from memory. That’s why writing costs five times reading. So we keep answers short, we cache the handbook, and the overnight review sorting goes through the half-price batch service.”

Meera nodded. “Good. Cheaper and faster.”

Two days later she was back at his desk, and she was not smiling. A customer had written in, furious and frightened. Her son has a severe nut allergy. She had asked the support chat whether the walnut brownie was nut-free, and the bot had said yes. Her mother read the label before the boy took a bite.

Arjun pulled up his logs for that conversation. They said: 200 OK, 3.2s.

“That’s all we have?” Meera asked. “It was fast. That’s what our logs know?”

Go deeper

• vLLM documentation, on continuous batching and paged attention: https://docs.vllm.ai
• Ollama, including the response timing fields: https://ollama.com
• Anthropic documentation, including Message Batches, streaming and effort: https://docs.anthropic.com
• OpenAI documentation, including the Batch API: https://platform.openai.com/docs