What is a model? Training vs inference in plain words

Crumb’s online shop has a problem that has nothing to do with chatbots. Every order confirmation says “Your order will arrive in about 45 minutes.” Every single one. For a customer two streets from the Baner store, that is a pleasant surprise. For a customer 9 km away in Wakad on a Friday evening, it is a lie, and last month 31% of all support emails were some version of “where is my order, you said 45 minutes”.

Meera wants honest delivery estimates. “Not perfect. Just better than the same number for everyone.”

Arjun is also nursing a theory from yesterday. He finds Priya at the coffee machine. “I have figured out models,” he says. “A model is a giant database of answers. Somebody stored every question on the internet, and when you ask something, it finds the closest one.”

Priya hands him a cup. “Then ask Claude for a haiku about our eggless brownie and a delivery van stuck on the Hinjewadi flyover.”

He does. The haiku is good. It is also certainly not in any database: it did not exist until four seconds ago.

“Okay,” he says. “Then what is a model?”

Arjun builds a lookup table

Arjun decides to solve the delivery problem his way first: with a database of answers. He pulls five recent deliveries from the Baner store, chosen to cover a range of distances:

Distance (km)	Actual minutes
1	14
2	18
3	22
5	30
8	42

And writes the obvious code:

past = {1: 14, 2: 18, 3: 22, 5: 30, 8: 42}

def eta(km):
    return past[km]

print(eta(3))   # 22
print(eta(6))   # KeyError: 6

The very first new order is 6 km away, and the program crashes. Nobody has delivered exactly 6 km before. So Arjun “fixes” it: find the closest distance in the table and use that answer.

def eta(km):
    closest = min(past, key=lambda d: abs(d - km))
    return past[closest]

print(eta(6.4))  # 30  (closest is 5 km)
print(eta(6.6))  # 42  (closest is 8 km)

No more crash. But look at those two lines. Two hundred metres of extra distance adds twelve minutes. A customer at 6.6 km is told 42 minutes, while one at 6.4 km is told 30. And if tomorrow’s order is 20 km away, the answer is “42 minutes”, because 8 km is the furthest thing in the table.

Arjun’s lookup table can only repeat what it has seen. It has no idea why the numbers are what they are, so it cannot say anything sensible about a distance it has not seen.

“How long does 6 km take?”

Priya grabs a napkin and copies the table. “Forget code for a second. Look at the numbers. How long would a 6 km delivery take?”

Arjun stares. “From 1 to 2 km it goes up by 4 minutes. From 2 to 3, 4 more. From 3 to 5, 8 more, so 4 per km again. So every kilometre adds 4 minutes.”

“And at zero kilometres?”

“14 minus 4… 10 minutes. Packing the box, finding parking, the stairs.”

“So 6 km is?”

“4 times 6, plus 10. 34 minutes.”

“Congratulations,” Priya says. “You just trained a model in your head. You did not memorise the table. You found a rule with two numbers in it that explains the table, and now it works for any distance, including ones you never saw.”

She writes on the napkin:

minutes = w × distance_km + b

“This is the model. It is a function: put a distance in, get minutes out. It has two knobs. w is how many minutes each kilometre adds. b is the fixed time every delivery takes. You found w = 4 and b = 10. With different knob settings, the same function gives different answers. Same recipe card, different quantities.”

The knobs have a name: parameters. People also call them weights (and the fixed one, b, is often called the bias). When you hear that a language model “has 8 billion parameters”, it means a function with 8 billion knobs. Ours has two.

Show the answer

The formula generalises: it gives sensible answers for inputs it never saw. The table stores answers; the formula stores a pattern. Because the pattern is “4 minutes per km plus 10”, it can answer 6 km, 6.4 km or 6.6 km smoothly. The table can only return something it already holds. That is the heart of the difference between a database and a model, and it is why a model can write a haiku that never existed. (The flip side: a model can also produce a confident answer that is wrong, because nothing checks it against stored facts.)

Training: how a computer finds the knobs

Arjun could see the pattern because there were five neat numbers. Crumb has 40,000 past deliveries, with traffic, weather and time of day all mixed in. Nobody spots the pattern by eye at that scale. And a language model has billions of knobs. So computers find the knob values themselves, by a process called training.

Training needs a dataset: examples where you know the right answer. Then it runs a loop:

flowchart LR
    A["Start: knobs at zero"] --> B["Predict every example"]
    B --> C["Measure how wrong: the loss"]
    C --> D["Nudge each knob to reduce the loss"]
    D --> B
    D --> E["Loss small enough: freeze the knobs"]

The single number that measures “how wrong” is called the loss. A common choice is the average of the squared errors: square each miss (so misses in either direction count, and big misses count a lot), then average them.

The “nudge” step asks, for each knob: if I turned this knob up slightly, would the loss go up or down, and how steeply? Then it turns the knob a little in the downhill direction. Mathematically, that slope is a gradient, and this method is called gradient descent. Picture yourself blindfolded on a hillside, trying to reach the valley floor: you feel which way the ground slopes under your feet and take a small step downhill. Then again. You will learn the calculus in Module 27; for now, the intuition is enough.

One training step, with every number

Here is the first step of training on Crumb’s five deliveries, worked by hand. Both knobs start at zero: the model knows nothing.

Predict. With w = 0 and b = 0, every prediction is 0 minutes.

Measure. The errors (prediction minus actual) are −14, −18, −22, −30 and −42. Squared: 196, 324, 484, 900, 1,764. Average: loss = 733.6. Terrible, as expected.

Nudge. Every error is negative (we guessed too low), so both knobs need to go up. The slope calculation says the loss falls steeply as w rises (slope −240.8) and less steeply as b rises (slope −50.4). Training moves each knob by a small fraction of its slope, here 1% (this fraction is called the learning rate):

• w becomes 0 + 0.01 × 240.8 = 2.408
• b becomes 0 + 0.01 × 50.4 = 0.504

Predict again. Now the predictions are 2.9, 5.3, 7.7, 12.5 and 19.8 minutes. Still far too low, but closer. The loss drops to 257.3.

Step three. w = 3.786, b = 0.815, predictions 4.6, 8.4, 12.2, 19.7, 31.1, loss 100.3.

The loss went 733.6, then 257.3, then 100.3. Each step, the knobs creep towards the values that fit the data. After a few hundred steps they are almost exactly w = 4, b = 10, the values Arjun found by eye. The computer found them without being told the pattern exists.

Show the answer

Because w gets multiplied by the distance, so it affects the predictions more. Changing w by 1 changes the 8 km prediction by 8 minutes; changing b by 1 changes every prediction by only 1 minute. So the loss is much more sensitive to w, its slope is steeper, and gradient descent moves it further. That is also why b takes many more steps to settle. Uneven knobs like this are a real practical issue in training, and you will see the fixes (like scaling your inputs) in Part 2.

Inference: using the frozen model

Once training finishes, the knobs stop moving. They are frozen. Using the trained model to make predictions is called inference. Put in 6 km, get 34 minutes. No learning happens during inference; it is fast and cheap.

The split between the two matters enormously in practice:

	Training	Inference
What happens	Knobs are adjusted	Knobs are frozen; input in, output out
How often	Once, or occasionally	Every single request
Speed	Slow: from seconds (our two knobs) to months (big LLMs)	Fast: microseconds to seconds
Cost for large language models	Enormous: thousands of specialised chips for weeks	A fraction of a cent to a few cents per request
Who does it, for an LLM	The provider (Anthropic, OpenAI, Google, Meta)	You, every time you call the API

Almost everything you do as an AI engineer is inference. When Crumb’s support bot sends a question to Claude, it is running inference on a model Anthropic trained months earlier.

The real code, one step at a time

Now build the trainer yourself in plain Python, no libraries. Run each step after you finish the setup lab (uv run train.py), or with any Python 3 you have.

Step 1: the model, with knobs you set by hand.

data = [(1, 14), (2, 18), (3, 22), (5, 30), (8, 42)]  # (km, actual minutes)

w, b = 5.0, 5.0  # a guess

def predict(km):
    return w * km + b

for km, minutes in data:
    print(km, "km: predicted", predict(km), "actual", minutes)

1 km: predicted 10.0 actual 14
2 km: predicted 15.0 actual 18
3 km: predicted 20.0 actual 22
5 km: predicted 30.0 actual 30
8 km: predicted 45.0 actual 42

Not bad for a guess. But “not bad” is not a number.

Step 2: score it with a loss. Add this below:

def loss():
    return sum((predict(km) - minutes) ** 2 for km, minutes in data) / len(data)

print("loss:", loss())

loss: 7.6

Try w, b = 4.0, 10.0 and the loss becomes 0.0. Now “better” has a number.

Step 3: let the computer turn the knobs. Replace the whole file with the full training loop:

data = [(1, 14), (2, 18), (3, 22), (5, 30), (8, 42)]

w, b = 0.0, 0.0       # the two knobs: they start knowing nothing
learning_rate = 0.01  # how big each nudge is


def predict(km):
    """The model: a function with two knobs."""
    return w * km + b


for step in range(2001):
    # 1. Measure how wrong we are, and which way each knob should move.
    nudge_w = nudge_b = 0.0
    total_error = 0.0
    for km, minutes in data:
        error = predict(km) - minutes          # positive = we guessed too high
        total_error += error ** 2
        nudge_w += 2 * error * km / len(data)  # the slope for w
        nudge_b += 2 * error / len(data)       # the slope for b

    # 2. Turn each knob a little in the downhill direction.
    w -= learning_rate * nudge_w
    b -= learning_rate * nudge_b

    if step % 500 == 0:
        print(f"step {step:4}: w={w:5.2f}  b={b:5.2f}  loss={total_error / len(data):8.2f}")

# Training is over. The knobs are now frozen. Using the model = inference.
print(f"A 6 km delivery should take about {predict(6):.0f} minutes")

The new lines are the two nudge_ sums (the slopes from the worked example) and the two -= lines (the nudge itself). The loss printed on each line is measured before that step’s nudge. You should see:

step    0: w= 2.41  b= 0.50  loss=  733.60
step  500: w= 4.09  b= 9.51  loss=    0.07
step 1000: w= 4.01  b= 9.97  loss=    0.00
step 1500: w= 4.00  b=10.00  loss=    0.00
step 2000: w= 4.00  b=10.00  loss=    0.00
A 6 km delivery should take about 34 minutes

The first line matches the hand-worked step exactly: 733.6, then knobs at 2.41 and 0.50. The program discovered “4 minutes per km plus 10” on its own.

From two knobs to billions

A large language model is the same idea, scaled beyond intuition.

• The input is text, split into small pieces called tokens (whole words or parts of words).
• The function is a neural network: layer after layer of simple multiply-and-add steps, much like w × km + b, stacked dozens of times, with billions of knobs between them.
• The output is a probability for every possible next token. Pick one, add it to the text, repeat. That is how it writes a paragraph, and the haiku.
• Training used trillions of tokens of text and the same predict, measure, nudge loop, run for weeks on thousands of GPUs (chips that do many simple calculations in parallel).

When you download an open model like Meta’s Llama 3.1 8B, the big file you get is mostly its 8 billion knob values. Stored at 2 bytes each, that is about 16 GB of numbers. That is the model. Not a database of sentences: numbers that, combined with the network’s shape, produce good next-token predictions.

Where it breaks

Arjun ships the new estimates on a Monday. Complaints about “45 minutes” fall sharply that week. Then three things happen, and each one teaches something that is true of every model, including the giant ones.

The learning rate was a guess. Arjun experiments with learning_rate = 0.05 to “train faster”. The knobs overshoot the valley, swing back further each time, and by step 2,000 the loss is a number more than 200 digits long. Training is sensitive to settings like this, and big-model training runs are expensive partly because of it.

The model only knows its data. A bulk order comes in from Lonavala, 60 km away. The model happily predicts 4 × 60 + 10 = 250 minutes. The real journey is mostly expressway and takes about 80. All five training examples were short city trips; the model has never seen a highway. Asking a model about situations unlike its training data is called extrapolation, and it is where models are most confidently wrong.

The world changes, the knobs do not. In July, monsoon traffic adds 30% to every trip. The model keeps predicting dry-season times, because its knobs were frozen in March. Nothing about inference updates them. Arjun has to retrain on recent data.

That last one has a twin you will meet constantly with LLMs. A model does not learn from your chat. If you tell a chatbot Crumb’s new phone number, it will not know it tomorrow. If a chatbot seems to remember what you said earlier, the app is re-sending your earlier messages with every request. Its knobs are frozen. You will see exactly how in Module 02.

Try it yourself

Use the full training script from above.

1. Change learning_rate to 0.05 and run it. What happens to the loss?
2. Set it back to 0.01, then add a weird example to data: (4, 60), a delivery stuck behind a wedding procession. How do w and b change, and what is the new 6 km estimate?

Show the answer

1. The loss explodes to astronomically large numbers within a few hundred steps. Each nudge overshoots the valley and lands higher up the opposite slope, so the next nudge is even bigger. Too-large learning rates make training diverge.
2. You should get roughly w = 4.18 and b = 14.96, and a 6 km estimate of about 40 minutes instead of 34. The knobs compromise between the normal deliveries and the outlier, so every prediction gets a little worse. Models learn whatever is in the data, including its accidents. That is why data quality matters so much.

Common mistakes

• “The model looks answers up.” It computes them. That is why it can write things that never existed, and also why it can produce fluent nonsense: there is no lookup that can fail.
• “My chats are training it.” During inference the knobs are frozen. Some providers may use data to train future models depending on your account settings, but your conversation does not change the model you are talking to.
• “More parameters always means better.” Bigger models are often more capable, but slower and pricier. A smaller model is frequently the right engineering choice; Module 01 shows how to choose.
• Mixing up training costs and usage costs. The headline-making millions are training costs, paid by the provider. Your bill is for inference, per request.

Recap

• A model is a function with adjustable knobs called parameters (or weights); it stores patterns, not answers.
• Training finds the knob values from a dataset: predict, measure the loss, nudge each knob downhill, repeat.
• Inference is using the trained model with its knobs frozen; every API call you make is inference.
• Models generalise to new inputs, but are least reliable on inputs unlike their training data, and do not update themselves when the world changes.
• Large language models are the same idea with billions of parameters, text as input and next-token probabilities as output.

That evening, Arjun explains it to Meera with the napkin. “A model is a formula with dials. We showed the computer 40,000 past deliveries and it turned the dials until its guesses matched what really happened. Now the dials are fixed, and we use the formula for every new order. It is why the estimate is better than 45 minutes for everyone. It is also why I have to retrain it when the monsoon comes: it cannot learn on its own.”

Meera studies the napkin. “And the chatbot you keep talking about is the same thing? Just bigger?”

“Billions of dials instead of two,” Arjun says. “Same idea.”

“Then why does everyone in your meetings use thirty different words for it?” she asks. Priya, passing by, laughs. “Tomorrow. We will follow one customer’s question through the whole system and name every word on the way.”

Go deeper

• Hugging Face LLM course: the opening chapters explain models, training and inference with good diagrams.
• PyTorch tutorials: where you will write real training loops in Part 2. Peek now to see how close they are to the one above.