Overfitting, dropout and evaluation

Arjun did not sleep well. His 241-parameter model scored 83.5%, and a rubber stamp that says “on time” to every order scored 80%. Three and a half points of value, measured on the orders the model had practised on.

By morning he had a theory. “The model’s too small. Rush hour, rain, ratings, distance, all interacting. 241 numbers can’t hold all that.” He built a bigger one: two hidden layers of 256 neurons each, about 67,000 parameters. He trained it on one week of orders, about 600, for 300 epochs.

The result was beautiful.

Training accuracy: 99.8%

Five hundred and ninety-nine out of six hundred orders correct. He screenshotted it for Meera before Priya got in.

Priya looked at the screenshot, then at Arjun. “Run it on the next three weeks of orders. The ones it’s never seen.”

He did.

Accuracy on new orders: 78.7%

Worse than the rubber stamp.

The student who memorised the papers

“My cousin did this before his board exams,” Priya said. “He got hold of the last six years of question papers and memorised every answer. Word for word. In mock tests built from those papers he scored 99%.”

“And the real exam?”

“New questions. Same topics, different numbers, different wording. He’d memorised answers instead of learning how to solve problems. He scraped a pass.”

Arjun’s 67,000-parameter model had done exactly that. With only 600 orders, it had enough capacity to memorise each one individually: this 13.4 km, 3-item, rainy, 2pm order was on time; that one was late. Including the orders that were late through pure bad luck, a rider’s puncture or a wrong turn, which no feature could predict. It learned the noise along with the pattern. On new orders, the noise is different, so the memorised answers are worse than useless.

This is overfitting (a model fitting the quirks of its training data so closely that it does worse on new data). Its opposite is underfitting (a model too simple, or trained too little, to capture even the real pattern). The goal sits between them: learn the pattern, ignore the noise.

	Training score	Score on new data	Like a student who…
Underfitting	Poor	Poor	Skimmed the textbook
Good fit	Good	Nearly as good	Understood the methods
Overfitting	Excellent	Poor	Memorised last year’s answers

The tell-tale sign of overfitting is that gap: brilliant on training data, disappointing elsewhere. And you can only see the gap if you measure on data the model did not train on.

Watching it happen, epoch by epoch

Priya had Arjun rerun the big model, this time scoring it after every epoch on 400 orders it never trained on. Here is what the two losses did (cross-entropy, lower is better):

Epoch	Training loss	Loss on unseen orders	Training accuracy	Accuracy on unseen
1	0.363	0.435	85.5%	80.5%
10	0.312	0.410	87.2%	83.2%
20	0.289	0.468	88.5%	83.0%
50	0.209	0.551	91.7%	80.5%
100	0.113	0.812	96.7%	79.3%
300	0.018	1.886	99.8%	78.0%

Read it like a story. For the first ten epochs, both losses fall together: the model is learning real patterns that help everywhere. Around epoch 10, the unseen loss hits its lowest point. After that, the two curves split. Training loss keeps plummeting towards zero while unseen loss climbs and climbs. Every epoch after 10 made the model better at the training orders and worse at everything else.

If you plot the two columns, you get the most important chart in machine learning: two lines that run together and then part like a pair of scissors. Where the scissors open, memorisation has started.

Three piles of data

To catch overfitting, split your data before you train anything. Priya’s exam version:

• Training set (the practice questions): the model learns from these. Typically 60 to 80% of your data.
• Validation set (the mock exams): you check the model on these during development, to pick the number of epochs, the model size, the learning rate. The model never trains on them, but you make decisions based on them.
• Test set (the real board exam): you touch these once, at the very end, to get an honest final score. If you peek at it while making decisions, it stops being honest.

Arjun’s split: the first 600 generated orders for training (one week), the next 400 for validation, and 2,000 for the final test.

import torch
from deliveries import make_deliveries

X, y = make_deliveries(3000, seed=42)
X_train, y_train = X[:600], y[:600]
X_val, y_val = X[600:1000], y[600:1000]
X_test, y_test = X[1000:], y[1000:]

# Standardise using TRAINING statistics only, then apply the same numbers everywhere
mean, std = X_train.mean(dim=0), X_train.std(dim=0)
X_train = (X_train - mean) / std
X_val = (X_val - mean) / std
X_test = (X_test - mean) / std

The last block matters more than it looks. If you compute the mean and standard deviation from all 3,000 orders, a little information about the test orders leaks into training. Here the effect is tiny; with real data it can quietly inflate your scores. Compute on training data, save those numbers, reuse them.

(For real data with dates, you would split by time, train on older orders and test on newer ones, because that is how the model will be used. Module 29’s forecasting lesson makes this a hard rule.)

Three ways to stop memorising

Getting more data is the best cure for overfitting: memorising 60,000 orders is much harder than memorising 600. But you often cannot get more data quickly. These three techniques help with the data you have, and in practice you use them together.

Early stopping: stop when the mock exams stop improving

The simplest fix comes straight from the table. Track validation loss each epoch, save the model whenever it hits a new best, and stop when it has not improved for a while (the patience, say 20 epochs). Then reload the best version. That is early stopping.

It is like a coach who stops drilling past papers the moment mock scores stop rising, before the student starts memorising.

Weight decay: a tax on big weights

Memorising individual orders tends to need large, finely balanced weights, pushing hard in one direction to get one odd order right. Weight decay adds a small penalty for large weights, pulling every weight slightly towards zero on every step. The model can still use a big weight, but only if it pays for itself across many orders. In PyTorch it is one argument: torch.optim.AdamW(model.parameters(), lr=1e-3, weight_decay=1e-2).

Dropout: the kitchen where anyone might be off sick

Priya’s favourite. “Imagine Crumb’s kitchen if, every morning, a random 30% of the staff were told to stay home. Nobody can become the only person who knows the croissant lamination. Everyone has to learn a bit of every station, and the team works no matter who turns up.”

Dropout does that to neurons. During training, each step randomly switches off a fraction $p$ of the neurons in a layer (sets their output to zero). The network cannot rely on any one neuron, or on a fragile conspiracy of neurons that together memorise one order. It has to spread knowledge around, which tends to mean learning patterns that generalise.

from torch import nn

model = nn.Sequential(
    nn.Linear(5, 256), nn.ReLU(), nn.Dropout(p=0.3),    # switch off 30% during training
    nn.Linear(256, 256), nn.ReLU(), nn.Dropout(p=0.3),
    nn.Linear(256, 1),
)

At prediction time you want everyone at work, and you want the same answer every time you ask. So dropout switches itself off when the model is in evaluation mode. (During training, PyTorch scales the surviving outputs up by $1/(1-p)$, so the average signal is the same in both modes.) This is why two lines you might have been ignoring suddenly matter:

model.train()   # dropout ON: use while training
model.eval()    # dropout OFF: use for validation, testing and production

Forget model.eval() and your predictions become slightly random, different each time you ask about the same order. Forget to switch back with model.train() and dropout silently stops working.

Arjun’s second attempt

Arjun combined all three: dropout of 0.3, weight decay of 0.01, early stopping with a patience of 20.

import copy
from torch.utils.data import TensorDataset, DataLoader

loss_fn = nn.BCEWithLogitsLoss()
optimizer = torch.optim.AdamW(model.parameters(), lr=1e-3, weight_decay=1e-2)
loader = DataLoader(TensorDataset(X_train, y_train), batch_size=32, shuffle=True)

best_val, best_state, patience, bad_epochs = float("inf"), None, 20, 0

for epoch in range(1, 301):
    model.train()                                   # dropout on
    for xb, yb in loader:
        loss = loss_fn(model(xb), yb)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

    model.eval()                                    # dropout off
    with torch.no_grad():
        val_loss = loss_fn(model(X_val), y_val).item()

    if val_loss < best_val:                         # new best: remember it
        best_val, bad_epochs = val_loss, 0
        best_state = copy.deepcopy(model.state_dict())
    else:
        bad_epochs += 1
        if bad_epochs >= patience:
            print(f"Stopped at epoch {epoch}; best validation loss {best_val:.3f}")
            break

model.load_state_dict(best_state)                   # roll back to the best epoch

model.state_dict() is a dictionary of every parameter tensor, and copy.deepcopy takes a full snapshot of it, not a reference that keeps changing. That snapshot is a checkpoint, a saved copy of the model at one moment, and you will meet it again in the next module.

Stopped at epoch 24; best validation loss 0.390

It found its best point within a few epochs and stopped long before the memorising began. On the 2,000 test orders it had never seen: 81.7% accuracy. Better than the rubber stamp’s 79.3% on the same orders. Not a triumph. But honest, and a lot better than 78.7%.

Accuracy is the wrong question

Arjun was still unhappy about 81.7%. Priya asked a different question. “Forget accuracy. Of the 414 orders in the test set that actually arrived late, how many did the model warn us about?”

He did not know. Accuracy cannot tell you. For that you need a confusion matrix (a table counting every combination of what the model said and what actually happened).

On the 2,000 test orders, at the default threshold of 0.5 (flag as late if the model says more than 50%):

	Model said “on time”	Model said “late”
Actually on time (1,586)	1,489 (correct)	97 (false alarm)
Actually late (414)	268 (missed!)	146 (caught)

Two numbers summarise it:

• Precision: of the orders the model flagged, how many were really late? $146/(146+97)=60\%$. When it raises the alarm, it is right six times in ten.
• Recall: of the orders that really were late, how many did it flag? $146/(146+268)=35\%$. It misses almost two thirds of late orders.

The F1 score combines the two into one number (their harmonic mean), here 0.44. Useful for comparing models, less useful for explaining to Meera.

The rubber stamp, for comparison, has 0% recall. It catches nothing. Its 79.3% accuracy came entirely from the 1,586 on-time orders. On imbalanced data, where one answer is much more common, accuracy flatters lazy models. Always look at the confusion matrix.

What a mistake costs

Here Meera became the most important person in the room, because the right trade-off is a business decision, not a technical one. Priya asked her what each kind of mistake cost.

• A late order nobody warned about: a refund, about ₹350, plus a likely bad review.
• A late order caught in advance: a call and a ₹100 discount voucher, and the customer usually stays happy. ₹100.
• A false alarm: an unnecessary call and a ₹50 goodwill voucher. ₹50.

“So missing a late order is seven times worse than a false alarm,” Arjun said. “We should flag more aggressively.”

The model outputs a probability, and 0.5 was an arbitrary cut-off. Lower it to 0.3 and more orders get flagged:

Threshold	Caught	Missed	False alarms	Precision	Recall	Cost on 2,000 orders
No model	0	414	0	n/a	0%	₹1,44,900
0.5	146	268	97	60%	35%	₹1,13,250
0.3	256	158	251	50%	62%	₹93,450

At 0.3, accuracy actually drops to 79.5%, yet the business saves ₹51,450 over these 2,000 orders compared with no model, about ₹15,000 a week. The best model for Meera is not the most accurate one.

One more honest number. Priya computed what a perfect model would score, one that knew the hidden rule inside make_deliveries: 84.5% accuracy at the 0.5 threshold. Some orders are late by pure luck and no model can predict them. Arjun’s 81.7% was much closer to the ceiling than it looked. Knowing the ceiling stops you chasing points that do not exist.

Measuring it properly with torchmetrics

Computing these by hand is error-prone. torchmetrics is a library of well-tested metrics for PyTorch. Install it with uv add torchmetrics.

from torchmetrics import MetricCollection
from torchmetrics.classification import (
    BinaryAccuracy, BinaryPrecision, BinaryRecall, BinaryF1Score, BinaryConfusionMatrix,
)

threshold = 0.3
metrics = MetricCollection({
    "accuracy": BinaryAccuracy(threshold=threshold),
    "precision": BinaryPrecision(threshold=threshold),
    "recall": BinaryRecall(threshold=threshold),
    "f1": BinaryF1Score(threshold=threshold),
})
confusion = BinaryConfusionMatrix(threshold=threshold)

model.eval()
with torch.no_grad():
    probs = torch.sigmoid(model(X_test)).squeeze(1)   # shape (2000,)
target = y_test.squeeze(1).int()                      # metrics want integer labels

metrics.update(probs, target)
confusion.update(probs, target)
print({name: round(v.item(), 3) for name, v in metrics.compute().items()})
print(confusion.compute())   # rows: actual (on time, late); columns: predicted

{'accuracy': 0.795, 'f1': 0.556, 'precision': 0.505, 'recall': 0.618}
tensor([[1335,  251],
        [ 158,  256]])

The update then compute pattern exists for big datasets: call update once per batch inside your evaluation loop, then compute once at the end, and torchmetrics combines the batches correctly. (Averaging per-batch precision yourself gives subtly wrong answers.) Call .reset() before evaluating again.

Library APIs change quickly. If a name has moved, check the torchmetrics documentation.

Where evaluation goes wrong

The test set gets used more than once. Every time you look at the test score and change something, it becomes a little less honest. Keep a log of how many times you have touched it. Once is ideal.

The test data does not look like the future. Arjun’s generated test orders follow the same rule as his training orders. Real life does not. In the monsoon, rain is not 30% of days but 80%; a new delivery partner joins with no rating history; Diwali triples order volume. A model that scored 82% in March can quietly drop in July. This is distribution shift, and the only defence is to keep measuring after you ship. Module 25’s MLOps lessons cover monitoring for exactly this.

Forgetting model.eval(). With dropout in the model, evaluation in training mode gives noisy, pessimistic scores, and production predictions that change each time.

Comparing against nothing. Always report the rubber-stamp baseline next to your model. “81.7%” means nothing until you know the baseline is 79.3%.

Chasing accuracy on imbalanced data. Choose the metric that matches the costs. Here that was recall, at a precision Meera could afford.

Try it yourself

1. Train the big 67,000-parameter model without dropout or weight decay, but with early stopping (patience 20). Where does it stop, and what is its test accuracy?
2. Put the regularised model in model.train() mode and predict the same test order five times. What do you see? Now switch to model.eval() and repeat.
3. Compute the cost table for a threshold of 0.2. Is it better or worse for Meera than 0.3?

Show the answer

1. It stops early (within the first 30 or so epochs) and scores around 81%, about the same as the regularised model. With this little data, early stopping does most of the work. Dropout and weight decay matter more for long training runs and bigger models.
2. In training mode, the five probabilities differ, because a different random 30% of neurons is switched off each time. In eval mode, all five are identical.
3. Your exact numbers depend on your model, but typically recall rises further and false alarms rise faster. Work out the rupee total; the threshold with the lowest total cost wins. Very low thresholds eventually flag almost everything, and the ₹50 false alarms add up.

Common mistakes

• Celebrating training accuracy. It measures memory, not skill. Report validation and test scores.
• Tuning on the test set. Use validation data for every decision; touch the test set once.
• Standardising with all the data. Compute statistics on the training split only.
• Forgetting model.eval() and model.train(). Dropout (and BatchNorm, next module) behave differently in each mode.
• Reporting accuracy alone on imbalanced data. Show the confusion matrix, precision and recall, and the baseline.

Recap

• Overfitting is memorising training data, noise included; you spot it as a widening gap between training and validation scores.
• Split data three ways: training to learn, validation to make decisions, test for one honest final score.
• Early stopping, weight decay and dropout all fight memorisation; more data fights it best.
• model.train() switches dropout on; model.eval() switches it off for every prediction.
• A confusion matrix shows the kinds of mistakes; precision and recall summarise them; the threshold should be chosen by what each mistake costs.

That afternoon Arjun explained it to Meera without a single screenshot of accuracy. “The big model memorised last week’s orders, including the unlucky ones, so it was worse than guessing on new orders. The fixed model learns the pattern instead. If we flag every order it gives more than a 30% chance of being late, we catch about six in ten late orders before they leave. We’ll make some calls that weren’t needed, but on these numbers we save about fifteen thousand rupees a week.”

“That’s a number I understand,” Meera said. “Can we start Monday?”

“Monday it runs in shadow mode,” Priya said. “It makes predictions, we check them against what happens, and nobody gets called yet. Meanwhile, Arjun has a side quest: build the same kind of network in TensorFlow, so he learns which ideas belong to deep learning and which belong to PyTorch.”

Go deeper

• torchmetrics documentation for the full list of classification and regression metrics.
• PyTorch tutorials for nn.Dropout, saving and loading state_dict checkpoints, and evaluation loops.
• scikit-learn user guide has the clearest explanations of precision, recall and model evaluation anywhere, even if you never use scikit-learn itself.