Training vs inference: where the cost goes

Meera has been reading the business pages. She drops a newspaper on Arjun’s desk, folded open at a story about an AI lab that spent an eye-watering sum, hundreds of crores of rupees, training its newest model.

“Two questions,” she says. “One: if these models cost that much to make, why are we also paying every single time we use one? We are not building it. Two: should we be building our own? I would rather pay once than pay forever.”

Arjun has an answer ready, and he likes it a lot. “We could train our own Crumb model on our three years of support emails. Then it would know our policies, and we would not pay per token. I priced a cloud GPU: about ₹250 an hour. Even if training takes a week, that is only ₹42,000.”

Priya, at the next desk, puts down her chai. “That is a very reasonable-sounding plan. Let us do the arithmetic before Meera signs anything.”

Arjun’s plan meets the numbers

“How big a model do you want?” asks Priya.

“Something as good as the one we tested. Smart enough to write a polite reply in Marathi.”

“The big commercial labs do not publish their numbers, but Meta published theirs for its open Llama 3.1 models. The largest has 405 billion parameters.” She explains as she writes: parameters are the internal numbers, the dials, that hold everything the model learned. “Meta trained it on about 15 trillion tokens of text, using up to 16,000 top-end GPUs at once. Their published total was about 31 million GPU-hours for that one model.”

Arjun does the multiplication on his phone. “At ₹250 an hour, that is… about ₹775 crore.”

“And that is only the final run. Not the failed experiments, the data collection, the engineers, or the second stage of training that turns a raw text predictor into a helpful assistant. And what about your three years of emails? How many tokens is that?”

Arjun checks. About 300,000 emails, maybe 150 tokens each. “45 million tokens.”

“Against 15 trillion. You would have about 0.0003% of the text a modern model learns from. A model trained only on that would know Crumb’s refund policy and almost nothing else, not even good Marathi grammar.”

Arjun puts the phone down. “So we do not build one.”

“Almost nobody does. A handful of labs build these. Everybody else, including teams at very large companies, uses them. The real questions for us are the other two Meera asked: why we pay per use, and how to keep that cost down.”

Writing the recipe book versus cooking from it

Priya walks Arjun to the kitchen door, where the head baker is pouring batter.

“Our plum cake recipe took twenty years to perfect. Hundreds of test batches, wasted fruit, burnt trays. That was slow, expensive, and it happened once. Now any trained baker follows the finished recipe in two hours on a Tuesday morning.”

• Training is writing the recipe book. The model is shown enormous amounts of text, and its parameters are adjusted, over and over, until it predicts the next token well. Weeks to months, thousands of GPUs, done once.
• Inference is cooking from the book. The parameters are frozen. You send in text, the model runs its fixed calculation, text comes out. Seconds per request, done millions of times a day by everyone who uses the model.

“So when we pay per token,” says Arjun, “we are paying the kitchen to cook. Not to rewrite the book.”

“Exactly. And cooking is not free. Every request runs that enormous calculation on GPUs that the provider has bought, powers and cools. You are renting a slice of a very expensive kitchen, by the minute.”

Where the analogy breaks: a human baker gets better with every batch. A model does not learn anything from your requests during inference. Its dials stay exactly where training left them. If your chatbot gets better next month, it is because you changed the instructions, the data you send, or the model, never because it “learned from customers”.

Why GPUs, and what they are actually doing

A GPU (graphics processing unit) is a chip originally built for video games, which need millions of small calculations done at the same time to draw each frame. A language model’s work is the same shape: huge grids of numbers multiplied together. A GPU does thousands of these multiplications in parallel, where a normal processor does a handful.

During training, each step goes like this: show the model a batch of text with the next tokens hidden, let it guess, measure how wrong each guess was, then work out how to nudge every one of the billions of parameters so the right answers become slightly more likely. That “work out the nudge” step is expensive, roughly twice the cost of the guess itself, and it repeats for trillions of tokens.

During inference, only the guessing happens. No measuring, no nudging. That is why one request takes seconds instead of months. But “only guessing” is still a lot of maths, and it happens once for every token.

Following one request through the kitchen

Let us trace one real Crumb request from the trial: a support reply where the model receives 1,500 input tokens (instructions, the order record and the customer’s email) and writes a 300-token reply. Inference happens in two phases, and they behave very differently.

Phase 1: prefill, reading the order. The model processes all 1,500 input tokens. Because it already has the whole text, it can do this in parallel: all 1,500 tokens go through the model together in one big batch of maths. Picture a baker reading the whole recipe card at a glance. On a modern GPU this takes a fraction of a second. Then the model produces its first output token.

Phase 2: decode, writing the reply. Now it writes the 300 output tokens. This part is sequential: token 2 depends on token 1, token 3 on token 2. The model must do a full pass for each one, one after the other. Picture a baker who must add each ingredient, wait for it to mix, and only then decide on the next.

flowchart LR
  A["Request: 1,500 input tokens"] --> B["Prefill: all 1,500 read in one parallel pass"]
  B --> C["First output token"]
  C --> D["Decode: token 2, then 3, then 4, one pass each"]
  D --> E["Token 300 or end of reply"]

With some rough, illustrative speeds for a fast hosted model, the timeline looks like this:

Moment	Time since request sent
Request travels over the internet to the provider	0.1 s
Prefill finishes; first token appears	0.5 s
Decode writes 300 tokens at about 80 tokens per second	+3.75 s
Reply complete	about 4.3 s

The time until the first token is called time to first token. The total time is the latency of the request (the delay between asking and getting the complete answer). Notice that 1,500 input tokens took under half a second, while 300 output tokens took almost four. Per token, writing is more than forty times slower than reading here (about 12 milliseconds per output token against about 0.3 milliseconds per input token).

And the bill for this one request, at Claude Opus 5.5’s prices at the time of writing ($4permillioninputtokens,$20 per million output):

	Tokens	Cost
Input	1,500	1,500 × $4/1,000,000=$0.006
Output	300	300 × $20/1,000,000=$0.006
Total		$0.012, about ₹1

The 300 output tokens cost as much as the 1,500 input tokens. Providers price output at roughly four to five times input, and the timeline above shows why: output is where the GPU time goes.

Opening the hood: why writing is so much slower than reading

“But why can the model read 1,500 tokens at once and not write them at once?” asks Arjun.

Priya draws a GPU as a small, fast workshop next to a huge warehouse. “The parameters are stored in the GPU’s memory, the warehouse. To do a pass, the workshop has to fetch every parameter from the warehouse. How much is that, for a 70-billion-parameter model?”

Each parameter is commonly stored in 2 bytes, so 70 billion parameters is about 140 gigabytes. Every single pass through the model reads all 140 GB.

• In prefill, one fetch of those 140 GB serves all 1,500 input tokens together. The cost of fetching is shared 1,500 ways.
• In decode, each pass produces only one new token for this conversation. So 140 GB is fetched for a single token, then fetched again for the next one. A top-end GPU can move roughly 3 terabytes per second from its memory. That caps a single conversation at about 20 passes per second if the GPU served only you.

That is the core reason decode is slow and expensive. Providers fight back in two ways. First, they batch: while the parameters are fetched for your token, they also compute the next token for dozens of other customers’ conversations, sharing the fetch. That is why a provider can serve you at 80 tokens per second. Second, they remember earlier work. For every token already processed, the model saves some intermediate results so it does not recompute them on the next pass. That saved store is called the KV cache, and you will open it up properly in Module 21 and again in Module 32.

The hidden tokens: models that think first

There is one more twist. Many current models are reasoning models: before writing the visible answer, they write private reasoning tokens to work through the problem. Claude Opus 5.5 always does this, to a depth you control with an effort setting. OpenAI’s GPT-5 family and Google’s Gemini 2.5 models do it too.

Those thinking tokens are produced by decode, exactly like the visible answer. So they cost output-token prices, and they happen before your first visible token, which makes the time to first token longer. A model that thinks for 400 tokens before a 300-token reply has actually written 700 output tokens. You will see where these show up in your usage numbers in the lesson on the anatomy of an API call, and how to choose when thinking is worth it in the lesson on choosing a model.

The real code: measure prefill and decode yourself

Now see it with your own eyes. This uses Gemini because it has a free tier. You will set up the key in the lab two lessons from now, so bookmark this and come back.

Step 1: time to first token and total time. Streaming (receiving the reply in pieces as it is written, instead of all at the end) lets you see the moment the first token arrives.

# timing.py
import os
import time
from dotenv import load_dotenv
from google import genai

load_dotenv()
client = genai.Client()  # reads GEMINI_API_KEY
MODEL = os.getenv("GEMINI_MODEL", "gemini-2.5-flash")

def timed(prompt: str, label: str) -> None:
    start = time.perf_counter()
    first = None
    usage = None
    for chunk in client.models.generate_content_stream(model=MODEL, contents=prompt):
        if first is None and chunk.text:
            first = time.perf_counter() - start       # first visible token arrived
        if chunk.usage_metadata:
            usage = chunk.usage_metadata               # token counts, filled in as we go
    total = time.perf_counter() - start
    print(f"{label}: first token {first:.2f}s, done {total:.2f}s, "
          f"in={usage.prompt_token_count} out={usage.candidates_token_count}")

time.perf_counter() is Python’s precise stopwatch. The loop receives the reply in chunks; we note the time when the first chunk with text arrives, and the token counts that come with the stream.

Step 2: two experiments that pull in opposite directions.

# A: short input, long output. Decode dominates.
timed("Write a 400-word story about a bakery in Pune.", "Long output")

# B: long input, short output. Prefill dominates.
handbook = "Store rule: always label allergens clearly on every shelf. " * 1500
timed(handbook + "\n\nIn one word, what is this text about?", "Long input")

Run uv run timing.py three times. A typical result looks like this (yours will differ):

Long output: first token 1.10s, done 6.80s, in=13 out=540
Long input: first token 2.40s, done 2.55s, in=15016 out=1

Library APIs change quickly. If a name has moved, check the docs linked below.

Read it like an engineer. Experiment A read 13 tokens and wrote 540: the first token came quickly, but writing took over five seconds. Experiment B read 15,000 tokens and wrote one: the wait for the first token was longer, because prefill had to process all of it, but after that it was done almost instantly. Gemini 2.5 Flash also thinks before answering by default, which adds a little to both “first token” times.

Step 3: tokens per second. Add one line to timed to compute the decode speed: print(f" about {usage.candidates_token_count / (total - first):.0f} tokens/s while writing"). For experiment A you will typically see somewhere between 50 and 250 tokens per second, depending on the model and the time of day.

Where it bites in production

A month later, the WhatsApp ordering team complains. Replies from Arjun’s prototype take nine seconds, and customers are typing “hello??” while they wait. Arjun investigates and finds two causes, both straight out of this lesson.

• He had pasted the entire 200-page operations handbook into every request, about 120,000 tokens, “so it always has the answer”. Prefill on that much text took several seconds before the first word appeared, and cost about $0.48 per message in input tokens alone.
• He had asked for “detailed, thorough replies”, so each answer was 600 tokens long. On a phone, nobody reads 600 tokens.

Sending only the relevant handbook section (about 1,200 tokens) and asking for replies under 80 words brought the first token down to under a second and the full reply to about 1.5 seconds. The cost per message fell by more than 95%. You will learn to find “the relevant section” automatically in Module 09 on retrieval.

Other things experienced engineers watch for:

• Training your own model is almost never the first step. Using an existing model, with good instructions and your data in the request, beats it on cost, speed and quality for nearly every business. Module 33 covers fine-tuning, a much smaller extra round of training on your own examples, and when it is worth it.
• Busy hours are slower. Provider GPUs are shared, so decode speed varies with load.
• Running your own model changes the bill, not the physics. With a self-hosted open model (Module 20), you pay for GPU hours instead of tokens, but long inputs and long outputs still cost you time and hardware.

Try it yourself

1. Run timing.py. Then double the handbook repetition to * 3000 and run experiment B again. Predict first: what happens to the time to first token, and to the time after it?
2. Change experiment A to ask for a 100-word story instead of 400. Predict the effect on total time.
3. Work out the monthly cost of Arjun’s original WhatsApp design: 120,000 input tokens and 600 output tokens per message, 5,000 messages a month, at $4and$20 per million.

Show the answers

1. The time to first token roughly increases, because prefill must read about twice as many tokens. The time after the first token stays tiny, because the answer is still one word.
2. Total time drops sharply, roughly in proportion to the output length, because decode time scales with the number of tokens written. The time to first token barely changes.
3. Input: 120,000 × 5,000 = 600 million tokens × $4permillion=$2,400. Output: 600 × 5,000 = 3 million × $20permillion=$60. Total $2,460 a month (about ₹2.2 lakh). Almost all of it is the handbook being re-read 5,000 times.

Common mistakes

• “We should train our own model.” It costs crores and needs trillions of tokens of text. Use an existing model through an API, and consider fine-tuning later only for a specific reason.
• “The model learns from our users.” During inference the parameters are frozen. Improvements come from you changing the prompt, the data or the model.
• Ignoring output length. Output tokens are the slow, expensive ones. “Answer in under 80 words” is often the single biggest saving.
• Pasting huge inputs into every request. Each one delays the first token and is paid for on every single call.
• Forgetting thinking tokens. Reasoning models write hidden tokens that you pay for at output prices and wait for before the first visible word.

Recap

• Training sets a model’s parameters once, at enormous cost; inference runs the frozen model for every request.
• You pay per token because every token read or written costs GPU work.
• Prefill reads all input tokens in one parallel pass; decode writes output tokens one at a time.
• Output tokens are slower and cost roughly four to five times more than input tokens.
• Long inputs delay the first token; long outputs, and hidden thinking, delay the complete answer.

Arjun explains it to Meera the next morning. “The labs spend crores writing the recipe book, once. We pay the kitchen every time it cooks for us, and the price depends on how much it reads and, even more, how much it writes. So we keep what we send focused and ask for short answers. Training our own would cost more than the whole bakery.”

Meera nods. “Good. Then here is my next problem.” She turns her phone around. It shows a chat with Arjun’s prototype. A customer gave her order number in the first message. Three messages later, the bot asked for it again.

Go deeper

• Claude docs, pricing and token usage: https://docs.anthropic.com
• Gemini API docs, streaming and token counts: https://ai.google.dev/gemini-api/docs
• vLLM docs, for a look at how inference servers batch requests to use GPUs efficiently: https://docs.vllm.ai