CNNs end to end

Crumb’s online shop has about 3,400 product photos, uploaded by staff at six stores, each tagged by hand with one of ten categories: croissants, pain au chocolat, danishes, muffins, éclairs, cinnamon rolls, cookies, cakes, breads and savouries. Meera had someone audit a random 300. Thirty-three were in the wrong category. An éclair filed under breads. A cheesecake under croissants. A customer searching for “croissant” on a Sunday morning got a cheesecake as the third result and emailed to ask if this was a joke.

“Eleven percent wrong,” Meera said. “And they upload forty new photos a week. Can the computer check the category when the photo goes up?”

Arjun had the twenty-layer network from yesterday, now training properly with ReLU and BatchNorm. He pointed it at the shop photos, shrunk to 64×64 pixels.

Arjun flattens the photos

A 64×64 colour photo is $64\times 64\times 3=12,288$ numbers. Arjun’s network flattened each photo into one long list and fed it to dense layers, the same way it had fed five delivery features into the late-order model.

To iterate quickly, Priya had him prototype on CIFAR-10, a standard public dataset of 60,000 tiny 32×32 colour photos in ten classes (aeroplanes, cars, birds, cats, deer, dogs, frogs, horses, ships, lorries). Crumb’s own photos are private and only 3,400, so CIFAR-10 is the practice kitchen: same shape of problem, ten classes, small colour photos, and much more data. Everything built here runs on Crumb’s photos by changing one line, as you will see.

Twenty layers had been overkill, so for the prototype he used a smaller flattened network: two hidden layers of 512 and 256 neurons with BatchNorm, ReLU and dropout. The first layer alone needed $3,072\times 512\approx 1.57$ million weights. In total: 1,708,810 parameters. After 15 epochs:

FlatNet   train accuracy 66.9%   test accuracy 59.0%

Better than guessing (10%), but four photos in ten wrong. Then Priya asked him to take every test photo and shift it 4 pixels down and to the right (the strip that falls off one edge wraps round to the other), the kind of difference you get when someone holds the phone slightly differently.

FlatNet   test accuracy on shifted photos 36.8%

“Your network doesn’t know what a picture is,” Priya said. “To it, pixel 1,000 and pixel 1,001 are as unrelated as ‘distance’ and ‘rain’. When the croissant moves four pixels, every number it learned to look at now holds something else. It has to learn ‘croissant in the top left’ and ‘croissant in the bottom right’ as separate facts, with separate weights. With 3,400 photos, it will never see enough croissants in enough positions.”

The inspector with a magnifying glass

“How does the quality inspector at the central kitchen check a tray?” Priya asked.

Arjun had watched her. “She has a small loupe. She moves it across the tray, spot by spot, and at every spot she checks the same things. Are the layers flaky? Is the top glossy? Any burnt edges?”

“Does she use a different checklist for the top-left corner of the tray?”

“No. Same checklist everywhere.”

“And what does she write down?”

“A sort of map. Burnt edge here and here. Good lamination along this row.”

“Then her supervisor reads those maps, not the tray, and looks for bigger patterns: ‘this whole corner of the oven runs hot’. That’s a convolutional network. The loupe is a filter. The checklist is the filter’s weights. The map is a feature map. And the supervisor reading maps is the next layer.”

A convolutional neural network, or CNN, is built on that idea. Two properties make it right for images:

• Weight sharing. The same small filter is used at every position. A croissant-edge detector learned in one corner works in every corner, and it needs only a handful of weights, not one per pixel.
• Locality. Each filter looks at a small neighbourhood, because in images nearby pixels belong together.

Where the analogy breaks: the inspector chooses her checklist. A CNN learns its filters from data, starting random, by the same gradient descent you have used all along. Nobody tells it to look for flaky edges; it discovers whatever patterns help separate the classes.

One patch, one filter

To see exactly what a filter does, take a tiny 5×5 patch of a greyscale photo: the edge of a croissant against a dark plate. Dark (0.1) on the left, bright pastry (0.9) on the right, with one slightly odd pixel:

$$\left[\begin{array}{ccccc}0.1& 0.1& 0.9& 0.9& 0.9\\ 0.1& 0.1& 0.9& 0.9& 0.9\\ 0.1& 0.1& 0.9& 0.9& 0.9\\ 0.1& 0.2& 0.9& 0.8& 0.9\\ 0.1& 0.1& 0.9& 0.9& 0.9\end{array}\right]$$

And a 3×3 filter that detects vertical edges where it gets brighter from left to right:

$$\left[\begin{array}{ccc}-1& 0& 1\\ -1& 0& 1\\ -1& 0& 1\end{array}\right]$$

A convolution places the filter on the top-left 3×3 window of the patch, multiplies each pair of overlapping numbers, and adds them up. That is a dot product, from Module 27, between the filter and the window. Each row of the window is $(0.1,0.1,0.9)$, which contributes $-0.1+0+0.9=0.8$. Three rows: 2.4.

Slide the filter one step right. Now each row is $(0.1,0.9,0.9)$: $-0.1+0.9=0.8$, three rows, 2.4 again. One more step: $(0.9,0.9,0.9)$ gives $-0.9+0.9=0$. Flat pastry, no edge. Do every position and you get a 3×3 feature map:

$$\left[\begin{array}{ccc}2.4& 2.4& 0.0\\ 2.4& 2.2& 0.0\\ 2.4& 2.2& 0.0\end{array}\right]$$

Large values exactly where the edge is, zero over plain pastry. The odd pixel dents two values slightly. Turn the filter on its side, to detect horizontal edges, and this whole patch scores almost exactly zero, because there is no horizontal edge in it.

That is all a convolution is: one small dot product, slid everywhere. A real convolution layer learns dozens of such filters at once, each producing its own feature map.

Opening the hood: channels, padding and pooling

A colour image has three channels (red, green, blue), so a filter for the first layer is really $3\times 3\times 3$: it looks at a 3×3 neighbourhood in all three colours at once, 27 weights plus one bias. nn.Conv2d(3, 32, kernel_size=3) learns 32 such filters, producing 32 feature maps (32 output channels). Its parameter count:

$$32\times (3\times 3\times 3)+32=896$$

896 parameters, whether the photo is 32×32 or 4,000×3,000. Compare the 1.57 million in the flattened network’s first layer.

Three more settings control the output size:

• Padding adds a border of zeros around the image so the filter can be centred on edge pixels. With a 3×3 filter, padding=1 keeps the output the same height and width as the input.
• Stride is how far the filter moves each step. Stride 1 visits every position; stride 2 skips every other one and halves the output size.
• Pooling shrinks a feature map by summarising small regions. Max pooling with a 2×2 window keeps only the largest value from each 2×2 block, halving the height and width. It keeps “was the pattern here, roughly?” and drops the exact position, which adds a little tolerance to small shifts and cuts computation.

The general size formula, for input size $n$, filter size $k$, padding $p$ and stride $s$:

$$\text{output size}=\frac{n+2p-k}{s}+1$$

For the patch above: $n=5$, $k=3$, $p=0$, $s=1$, so $(5+0-3)/1+1=3$. A 3×3 map, as computed.

Stacking is where the power comes from. A first-layer filter sees a 3×3 neighbourhood. After pooling, a second-layer filter’s 3×3 window covers about 8×8 of the original pixels, a third-layer filter about 18×18. Early layers learn edges and colour blobs; middle layers combine them into textures (lamination, crumbs, glaze); late layers respond to whole parts and objects. Nobody programs that hierarchy. It emerges because it is what helps.

Following one photo through the network

Here is the network Arjun built. Each block is two convolutions with BatchNorm and ReLU (yesterday’s lesson, now in 2-D), followed by max pooling:

import torch
from torch import nn

def conv_block(c_in: int, c_out: int) -> nn.Sequential:
    return nn.Sequential(
        nn.Conv2d(c_in, c_out, kernel_size=3, padding=1), nn.BatchNorm2d(c_out), nn.ReLU(),
        nn.Conv2d(c_out, c_out, kernel_size=3, padding=1), nn.BatchNorm2d(c_out), nn.ReLU(),
        nn.MaxPool2d(2),                                   # halve height and width
    )

class PastryNet(nn.Module):
    def __init__(self, n_classes: int = 10):
        super().__init__()
        self.features = nn.Sequential(
            conv_block(3, 32),      # (3, 32, 32)  -> (32, 16, 16)
            conv_block(32, 64),     #              -> (64, 8, 8)
            conv_block(64, 128),    #              -> (128, 4, 4)
        )
        self.classifier = nn.Sequential(
            nn.Flatten(),                       # (128, 4, 4) -> 2,048 numbers
            nn.Dropout(0.3),
            nn.Linear(128 * 4 * 4, n_classes),  # 2,048 -> 10 class scores
        )

    def forward(self, x):                       # x: (batch, 3, 32, 32)
        return self.classifier(self.features(x))

Follow one photo, shape by shape:

Stage	Shape	What it holds
Input	(3, 32, 32)	Red, green and blue brightness for each pixel
After block 1	(32, 16, 16)	32 maps of simple patterns: edges, colour blobs
After block 2	(64, 8, 8)	64 maps of textures and small parts
After block 3	(128, 4, 4)	128 maps of large parts, each covering a big region
Flatten	2,048	All of that as one list
Classifier	10	One score per class; softmax turns them into probabilities

Total: 308,394 parameters, about a sixth of the flattened network’s 1,708,810. Fewer parameters, but each one is used everywhere in the image.

Teaching it that a shifted croissant is still a croissant

Even a CNN can overfit, especially on 3,400 photos. Data augmentation creates new training examples by randomly changing the originals in ways that do not change the correct answer: a croissant flipped left to right is still a croissant; cropped slightly differently, still a croissant; under warmer lighting, still a croissant. Each epoch, every photo is transformed afresh, so the model never sees exactly the same image twice.

torchvision’s v2 transforms do this:

from torchvision import datasets
from torchvision.transforms import v2

MEAN, STD = (0.491, 0.482, 0.447), (0.247, 0.243, 0.262)   # per-channel stats of the training photos

train_tf = v2.Compose([
    v2.ToImage(),
    v2.RandomCrop(32, padding=4),                      # shift by up to 4 pixels
    v2.RandomHorizontalFlip(),                         # mirror half the time
    v2.ColorJitter(brightness=0.2, contrast=0.2),      # small lighting changes
    v2.ToDtype(torch.float32, scale=True),             # 0-255 -> 0.0-1.0
    v2.Normalize(MEAN, STD),                           # standardise each colour channel
])
test_tf = v2.Compose([                                 # NO augmentation for evaluation
    v2.ToImage(), v2.ToDtype(torch.float32, scale=True), v2.Normalize(MEAN, STD),
])

train_set = datasets.CIFAR10("data", train=True, download=True, transform=train_tf)
test_set = datasets.CIFAR10("data", train=False, download=True, transform=test_tf)

Normalize is the standardising from Module 27, applied per colour channel with statistics from the training photos.

For Crumb’s own photos, arranged in folders by category (photos/train/croissant/*.jpg, photos/train/eclair/*.jpg and so on), the only change is the dataset line:

train_set = datasets.ImageFolder("photos/train", transform=train_tf)   # folder names become labels

(Add v2.Resize((64, 64)) at the start of each transform for larger photos, and recompute the mean and standard deviation from your own training set.)

Training and evaluating it properly

The training loop is the Trainer from this module’s first lesson, with two changes for ten classes: nn.CrossEntropyLoss (softmax and cross-entropy in one) and argmax to pick the predicted class. Arjun used AdamW with a one-cycle learning rate schedule, which ramps the learning rate up and then down over the run and works well for CNNs:

from torch.utils.data import DataLoader

device = "cuda" if torch.cuda.is_available() else "mps" if torch.backends.mps.is_available() else "cpu"
train_dl = DataLoader(train_set, batch_size=128, shuffle=True, num_workers=4)
test_dl = DataLoader(test_set, batch_size=512, num_workers=4)

torch.manual_seed(0)
model = PastryNet().to(device)
epochs = 15
optimizer = torch.optim.AdamW(model.parameters(), lr=1e-3, weight_decay=5e-4)
scheduler = torch.optim.lr_scheduler.OneCycleLR(optimizer, max_lr=3e-3, total_steps=epochs * len(train_dl))
loss_fn = nn.CrossEntropyLoss()

for epoch in range(epochs):
    model.train()
    for x, y in train_dl:
        x, y = x.to(device), y.to(device)
        loss = loss_fn(model(x), y)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        scheduler.step()                    # one-cycle steps every batch, not every epoch

num_workers=4 loads and augments images in four background processes, because with images the data pipeline, not the GPU, is often the bottleneck. (Keep the if __name__ == "__main__": guard from the first lesson, or this will misbehave on macOS and Windows.)

For evaluation, overall accuracy hides too much. Use torchmetrics to get accuracy per class and a confusion matrix:

from torchmetrics.classification import MulticlassAccuracy, MulticlassConfusionMatrix

per_class = MulticlassAccuracy(num_classes=10, average=None).to(device)
confusion = MulticlassConfusionMatrix(num_classes=10).to(device)

model.eval()
with torch.no_grad():
    for x, y in test_dl:
        logits = model(x.to(device))
        per_class.update(logits, y.to(device))
        confusion.update(logits, y.to(device))
print(per_class.compute())        # one accuracy per class
print(confusion.compute())        # rows: true class, columns: predicted class

On Arjun’s M-series MacBook each epoch took about a minute and a half, most of it spent loading and augmenting images rather than in the network itself; a CUDA GPU with more CPU workers is several times faster. Here is what Arjun got after 15 epochs, all scored on the same 10,000 test photos:

Model	Parameters	Train accuracy	Test accuracy	Test accuracy, photos shifted 4 px
Flattened dense network	1,708,810	66.9%	59.0%	36.8%
CNN, no augmentation	308,394	99.5%	86.7%	75.5%
CNN, with augmentation	308,394	90.1%	87.9%	83.7%

Three stories in one table.

Convolutions beat flattening by a mile. With less than a fifth of the parameters, the CNN without augmentation gets 86.7% against the flattened network’s 59.0%. Shifting the photos by four pixels knocks the flattened network down 22 points, to 36.8%; the CNN loses about 11. Weight sharing is doing exactly what the inspector analogy promised.

Without augmentation, the CNN memorises. It reaches 99.5% on training photos but 86.7% on test photos: the overfitting gap from Module 27, in pictures. With augmentation, training accuracy is lower (90.1%), because every epoch shows it new variations, yet test accuracy is higher (87.9%). Lower training accuracy with better test accuracy is what good regularisation looks like.

Augmentation buys robustness more than accuracy. The headline gain is only 1.2 points, but on shifted photos the augmented CNN keeps 83.7% against 75.5%, because random crops showed it shifted photos all through training. For Crumb, where every member of staff holds the phone differently, that column matters more than the headline.

Per class, the augmented CNN is best on cars (94.2%), lorries (93.6%) and ships (92.5%), and worst on cats (74.9%) and birds (81.8%). The confusion matrix shows where the cat errors go: 100 test cats were called dogs and 94 dogs were called cats, by far the largest pair. Those are classes that genuinely look alike at 32×32 pixels. When Arjun later ran the same pipeline on Crumb’s photos, the equivalent pair was croissants and almond croissants.

Where CNNs break

Arjun’s model went into shadow mode on the shop uploads, suggesting a category for each new photo while staff still chose manually. Within a fortnight, Priya had a list.

It learned the tray, not the pastry. Most croissant photos had been taken on the Koregaon Park store’s wooden display tray. A new store photographed a cake on a similar wooden board, and the model said “croissant” with 91% confidence. CNNs learn whatever separates the classes in the training data, including backgrounds. The fixes: photograph products on varied backgrounds, check per-class mistakes for patterns like this, and, if possible, look at which pixels drove a prediction (tools like Grad-CAM highlight them).

Small data. 3,400 photos across ten categories is about 340 per class. Training from scratch on that is hard. The standard answer is transfer learning: start from a CNN already trained on millions of photos (torchvision ships several, such as ResNet), keep its learned edge, texture and shape detectors, and retrain only the final layer, or the last few, on your photos. With a few hundred photos per class, transfer learning nearly always beats training from scratch. You met the same idea with pretrained Hugging Face models in Module 11, and you will meet it again for LLMs in Module 33.

Near-duplicates leak. Staff often uploaded three almost identical shots of the same tray. If one lands in training and another in testing, the test score measures memory, not skill. Split by product or upload session, not by individual photo.

Things it has never seen. Upload a photo of a coffee cup and the model still picks one of its ten categories, often confidently. As in Module 27’s first lesson, a probability is not a promise. For the shop, Priya added a rule: if the top probability is below 0.6, ask a human.

Augmentations that lie. As the pause showed, every augmentation encodes a claim about what does not matter. Get one wrong and you damage the model quietly.

Try it yourself

1. Compute by hand the output shape and parameter count of nn.Conv2d(64, 128, kernel_size=3, padding=1) applied to a (64, 8, 8) input. Check with PyTorch.
2. Train the CNN without augmentation and with augmentation for 15 epochs each. Record train and test accuracy. Where is the overfitting gap larger?
3. From the confusion matrix, find the two classes most often confused with each other. Look at twenty of those mistakes. Would you have got them right at 32×32 pixels?

Show the answer

1. Padding 1 keeps 8×8, so the output is $(128,8,8)$. Parameters: $128\times (3\times 3\times 64)+128=73,856$.
2. Without augmentation, training accuracy races ahead of test accuracy (a big gap: memorisation). With augmentation, training accuracy is lower, because the task is harder, but test accuracy is higher and the gap is much smaller. Lower training accuracy with higher test accuracy is the signature of good regularisation.
3. On CIFAR-10 it is almost always cats and dogs. Many of those mistakes are genuinely hard at 32×32. That is a useful calibration: before blaming the model, check what a human could do with the same input. For Crumb, the equivalent pair turned out to be croissants and almond croissants.

Common mistakes

• Flattening images into dense networks. It throws away the spatial structure and multiplies parameters. Use convolutions.
• Augmenting the test set. Augmentation is for training only. Evaluate on clean, unaltered images.
• Augmentations that change the label. Think about each one for your task: flips for pastries yes, for digits no; colour jitter for categories yes, for bake quality no.
• Reporting only overall accuracy. Look at per-class accuracy and the confusion matrix; the mistakes are rarely evenly spread.
• Training from scratch on a few hundred photos per class. Start from a pretrained model.

Recap

• A convolution slides a small filter across the image, computing a dot product at each position; the result is a feature map showing where the pattern appears.
• Weight sharing means a filter’s few weights work everywhere, giving CNNs far fewer parameters and natural tolerance to shifts.
• Padding, stride and pooling control output sizes: $(n+2p-k)/s+1$, and 2×2 max pooling halves height and width.
• Stacked conv blocks build a hierarchy from edges to textures to objects; a small classifier on top produces class scores.
• Augment training data with label-preserving changes, evaluate with per-class accuracy and a confusion matrix, and reach for transfer learning when photos are few.

At the end of the month Meera asked for the photo-sorter numbers. Arjun showed her the confusion matrix for the shop photos instead of a single accuracy figure. “It checks every new upload. When it disagrees with the category the staff chose, or isn’t sure, it asks a person. Most of what it flags really is miscategorised. Its weak spot is croissants versus almond croissants, which, honestly, staff mix up too.”

“And the cheesecake in croissants?”

“Flagged within a second of upload.”

Priya had moved on to a different screen: minute-by-minute temperature logs from the proofing cabinets, and eighteen months of daily sales. “Images are about where things are. Next month is about when. Sequences, memory and time.”

Go deeper

• PyTorch tutorials: the CIFAR-10 classifier tutorial and the transfer learning tutorial are the natural next steps.
• Hugging Face documentation for pretrained image classification models you can fine-tune on your own photos.