Generative adversarial networks

Two of Crumb’s oldest stores, Camp and Deccan, still take phone orders on paper slips. A clerk types them in at the end of each shift. Last month Arjun built a digit reader (the convolutional network from Module 28) to read the quantity and phone-number boxes automatically, and it worked well, except on the slips that mattered. It read one handwritten 1 as a 7, and a customer who ordered one Christmas plum cake was sent seven.

“It’s the data,” Arjun told Priya. “I’ve only got about 900 labelled digits in our clerks’ handwriting. The reader needs more examples of messy, real-world digits.”

“Labelling more costs clerks’ time,” said Meera. “Can the computer not make more?”

Priya smiled. “That is a real research question with a famous answer: two neural networks playing a game against each other. Practise on MNIST first, the public set of 70,000 handwritten digits. If you can make convincing fake digits there, we’ll talk about Crumb’s slips.”

What “generating” actually means

So far in this course, models have mostly judged things: is this email a refund request, which digit is this, what is the next token. A generative model does something different: it produces brand-new examples that look as if they came from the same source as its training data. New handwritten 7s that no clerk ever wrote. New faces of people who do not exist.

The usual recipe starts with randomness. You pick a list of random numbers, called a latent vector or noise vector $z$ (64 numbers, say), and a network turns it into an image. Different random numbers, different images. The hard part is training that network so its outputs look real.

Arjun’s first attempt: copy the real digits

Arjun’s plan seemed straightforward. Build a generator, a network that takes 64 random numbers and outputs a 28 × 28 image. To train it, pair each random input with a random real MNIST digit, and use mean squared error (the average of the squared pixel differences) to push the output towards that digit.

# Arjun's first generator: match random real digits pixel by pixel
G = nn.Sequential(nn.Linear(64, 256), nn.LeakyReLU(0.2), nn.Linear(256, 784), nn.Tanh())
opt = torch.optim.Adam(G.parameters(), lr=1e-3)
for real, _ in loader:                       # batches of real MNIST digits
    real = real.view(-1, 784)
    fake = G(torch.randn(real.size(0), 64))  # random noise in, image out
    loss = ((fake - real) ** 2).mean()       # how far from the real digit?
    opt.zero_grad(); loss.backward(); opt.step()

After one pass over the data, every output looked the same: a grey, ghostly smudge, something between a 0, a 3 and an 8. Sixteen different random inputs produced sixteen near-identical smudges.

“I asked for digits and it gave me fog,” he said.

“Think about what you asked it to do,” Priya said. “The random input has nothing to do with which digit you paired it with. So for any input, the target could be a 1, a 4, a 9, anything. What single picture is closest, on average, to all of them at once?”

Arjun worked it out. “The average of all the digits. A grey blob.”

“Squared error rewards being close on average. When the right answer is uncertain, the safest bet is a blur. You don’t want close on average. You want convincing. And who decides what’s convincing?”

Priya’s mithai test

“Picture an apprentice at Crumb,” said Priya, “learning to make kaju katli. And picture Meera as head taster. Every morning, Meera gets a tray with some pieces from the master confectioner and some from the apprentice, unlabelled. She has to say which is which.”

“And the apprentice?”

“Never sees the master’s recipe. The only feedback they get is Meera’s verdict on their pieces: ‘clearly fake, too dry’ or ‘I couldn’t tell’. They adjust and try again tomorrow.”

“And Meera gets better too,” said Arjun. “Each day she learns new tells.”

“So both improve together. The apprentice can’t get away with sloppy work, because the taster keeps learning. The taster can’t relax, because the apprentice keeps learning. If it all goes well, one day Meera is right only half the time: the apprentice’s katli is indistinguishable from the master’s.”

That is a generative adversarial network, or GAN, introduced by Ian Goodfellow and colleagues in 2014:

• The generator $G$ is the apprentice. It takes random noise $z$ and produces a fake image $G(z)$.
• The discriminator $D$ is the taster. It looks at an image and outputs a probability between 0 and 1 that the image is real.
• They train in turns. $D$ learns to say 1 for real images and 0 for fakes. $G$ learns to make $D$ say 1 for its fakes.

The generator never sees a real image directly. Everything it learns comes through the discriminator’s judgements, which is why it learns “convincing” rather than “average”.

Where the analogy breaks: Meera tastes for flavour, a quality she can name. A discriminator has no idea what a “7” is; it learns whatever statistical differences separate real from fake, which can include odd things like pixel speckle patterns. And unlike a kitchen, the two networks can fall into unhealthy loops, as you will see.

Following one training round with real numbers

Let us trace a single round. A batch has one real MNIST “3” and one fake made by the generator from a random $z$.

Step 1: the discriminator judges. It outputs:

• $D(\text{real 3})=0.80$: “80% sure this is real”. Good, it should be near 1.
• $D(G(z))=0.30$: “30% sure the fake is real”. Good, it should be near 0.

Step 2: the discriminator’s loss. We use binary cross-entropy, the standard loss for yes/no predictions: the penalty is $-\log$ of the probability given to the right answer. For the real image the right answer is “real” (probability 0.80); for the fake it is “fake” (probability $1-0.30=0.70$):

$${\mathcal{L}}_D=-\log (0.80)-\log (0.70)=0.223+0.357=0.580$$

The discriminator updates its weights to push this down: raise the real score, lower the fake one.

Step 3: the generator’s loss. The generator wants $D(G(z))$ to be 1. Its loss is $-\log$ of the probability the discriminator gave its fake of being real:

$${\mathcal{L}}_G=-\log (0.30)=1.204$$

Gradients flow backwards through the discriminator into the generator, telling it which pixels to change to look more real. The discriminator’s weights are left alone during this step.

Step 4: the next round. Suppose the generator improves, and the next fake gets $D(G(z))=0.50$. Its loss drops to $-\log (0.50)=0.693$.

Now the number to remember. If the generator became perfect, the best the discriminator could do is guess: $D=0.5$ for everything. Then:

• ${\mathcal{L}}_D=-\log (0.5)-\log (0.5)=1.386$
• ${\mathcal{L}}_G=-\log (0.5)=0.693$

So in a GAN, unlike every model you have trained before, the losses do not head to zero. A healthy GAN’s losses hover and wobble around values like these. A discriminator loss collapsing towards 0 is bad news: it means the taster is winning easily and the apprentice is getting no useful feedback.

Opening the hood: the minimax game

The original paper wrote the game as one value function that $D$ tries to maximise and $G$ tries to minimise, a minimax game:

$$\underset{G}{\min }\underset{D}{\max }{\mathbb{E}}_{x\sim \text{data}}[\log D(x)]+{\mathbb{E}}_{z\sim \mathcal{N}(0,I)}\left[\log \right(1-D(G(z))\left)\right]$$

Read it piece by piece. $\mathbb{E}$ means “average over many examples”. The first term is large when $D$ gives real images $x$ high scores. The second is large when $D$ gives fakes $G(z)$ low scores. $D$ wants both large. $G$ only affects the second term, and wants it small, meaning $D(G(z))$ close to 1.

The paper also proved something elegant: if both networks were perfectly flexible and perfectly trained, the game ends when the generator’s images are distributed exactly like the real data, and the discriminator outputs 0.5 everywhere. Real networks are neither, so this is a destination you approach, not one you reach.

The non-saturating trick

There is a practical problem with $G$ minimising $\log (1-D(G(z)))$ literally. Early in training the fakes are terrible, so the discriminator rejects them confidently, say $D(G(z))=0.01$. Right there, the curve of $\log (1-D)$ is almost flat, so the generator gets almost no gradient, almost no signal about how to improve, exactly when it needs one most. Measured at the discriminator’s final layer, the push it gets is about 0.01.

The fix, suggested in the same paper and used almost universally: the generator maximises $\log D(G(z))$ instead, which is the $-\log (0.30)$ loss you computed in Step 3. Same goal, “make D say real”, but at $D(G(z))=0.01$ the push is about 0.99, roughly 100 times stronger. This is called the non-saturating generator loss.

Building a digit GAN in PyTorch

Install with uv add torch torchvision. This runs on a laptop CPU: 20 passes over MNIST took about two minutes on an Apple-silicon MacBook. A free Colab GPU works too.

Step 1: the data. The generator will end with Tanh, which outputs values between -1 and 1, so we scale real images to the same range. If real and fake pixels lived on different scales, the discriminator could tell them apart by brightness alone.

import torch
import torch.nn as nn
from torch.utils.data import DataLoader
from torchvision import datasets, transforms
from torchvision.utils import save_image

torch.manual_seed(0)
device = "cuda" if torch.cuda.is_available() else "cpu"

# Pixels go from [0, 1] to [-1, 1]: (x - 0.5) / 0.5
tfm = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5,), (0.5,))])
mnist = datasets.MNIST("data", train=True, download=True, transform=tfm)
loader = DataLoader(mnist, batch_size=128, shuffle=True, drop_last=True)
Z_DIM = 64     # length of the random noise vector

Step 2: the two networks. Simple stacks of linear layers are enough for 28 × 28 digits. LeakyReLU lets a little signal through for negative inputs, which keeps gradients alive in both networks. The discriminator ends with a Sigmoid so its output is a probability.

G = nn.Sequential(                        # the apprentice: 64 numbers -> 784 pixels
    nn.Linear(Z_DIM, 256), nn.LeakyReLU(0.2),
    nn.Linear(256, 512), nn.LeakyReLU(0.2),
    nn.Linear(512, 784), nn.Tanh(),
).to(device)

D = nn.Sequential(                        # the taster: 784 pixels -> "probability real"
    nn.Linear(784, 512), nn.LeakyReLU(0.2),
    nn.Linear(512, 256), nn.LeakyReLU(0.2),
    nn.Linear(256, 1), nn.Sigmoid(),
).to(device)

Step 3: losses and optimisers. Each network gets its own optimiser. The settings lr=2e-4, betas=(0.5, 0.999) come from the DCGAN paper and are a well-worn starting point for GANs. fixed_z is a set of 64 noise vectors we keep forever, so we can watch the same inputs turn into digits over time.

bce = nn.BCELoss()                        # binary cross-entropy
opt_G = torch.optim.Adam(G.parameters(), lr=2e-4, betas=(0.5, 0.999))
opt_D = torch.optim.Adam(D.parameters(), lr=2e-4, betas=(0.5, 0.999))
fixed_z = torch.randn(64, Z_DIM, device=device)

Step 4: the training loop. This is the worked example, in code.

for epoch in range(1, 21):
    for real, _ in loader:
        real = real.view(-1, 784).to(device)        # flatten 28x28 images to 784 numbers
        n = real.size(0)
        ones = torch.ones(n, 1, device=device)      # label for "real"
        zeros = torch.zeros(n, 1, device=device)    # label for "fake"

        # 1) Train the taster: real -> 1, fake -> 0
        fake = G(torch.randn(n, Z_DIM, device=device))
        d_real, d_fake = D(real), D(fake.detach())  # detach: don't update G in this step
        loss_D = bce(d_real, ones) + bce(d_fake, zeros)
        opt_D.zero_grad(); loss_D.backward(); opt_D.step()

        # 2) Train the apprentice: make the taster say 1 for fakes (non-saturating loss)
        loss_G = bce(D(fake), ones)
        opt_G.zero_grad(); loss_G.backward(); opt_G.step()

    print(f"epoch {epoch:2d}  loss_D {loss_D.item():.2f}  loss_G {loss_G.item():.2f}  "
          f"D(real) {d_real.mean().item():.2f}  D(fake) {d_fake.mean().item():.2f}")
    with torch.no_grad():
        samples = G(fixed_z).view(-1, 1, 28, 28)
    save_image(samples * 0.5 + 0.5, f"gan_epoch_{epoch:02d}.png", nrow=8)  # back to [0, 1]

Two lines deserve a closer look. fake.detach() cuts the link back to the generator while the discriminator trains, so the taster’s lesson does not accidentally change the apprentice. And bce(D(fake), ones) is the non-saturating trick: the generator is graded on how close the taster came to saying “real”.

Here is a real run (your numbers will differ, since these are the last batch of each epoch and GAN training is noisy):

epoch  1  loss_D 0.67  loss_G 1.44  D(real) 0.76  D(fake) 0.30
epoch  2  loss_D 1.28  loss_G 2.48  D(real) 0.96  D(fake) 0.69
epoch  5  loss_D 0.50  loss_G 2.60  D(real) 0.86  D(fake) 0.26
epoch 10  loss_D 0.56  loss_G 3.46  D(real) 0.87  D(fake) 0.27
epoch 15  loss_D 0.71  loss_G 1.28  D(real) 0.72  D(fake) 0.20
epoch 20  loss_D 1.10  loss_G 2.54  D(real) 0.85  D(fake) 0.54

Arjun’s first reaction was alarm. “The generator loss went up from epoch 1 to epoch 10. Is it getting worse?”

“Open the pictures,” said Priya.

gan_epoch_01.png was 64 speckled grey blobs, most of them vaguely 0-shaped or 1-shaped. gan_epoch_20.png was a grid of real-looking handwritten digits: 7s, 9s, 3s, 1s, 4s, a bit speckled, some malformed, but unmistakably handwriting. The generator had improved enormously. Its loss went up because the discriminator had improved too. In a GAN, the losses measure the state of the contest, not the quality of the images.

Where GANs break

Mode collapse. Look closely at Arjun’s epoch-20 grid and something is off: plenty of 1s, 7s and 9s, very few 0s, 2s or 6s. The generator has found that some digits fool the discriminator more easily, and it leans on them. In the extreme, called mode collapse, a generator produces only one or two kinds of output (every sample a 1) because that single trick keeps fooling the current discriminator. It is the most famous GAN failure. Watching the fixed-noise grid over time is the simplest way to catch it; the next lesson shows how.

Instability. The two networks chase each other, so training can oscillate, stall or suddenly fall apart after looking fine for hours. Small changes in learning rate or architecture matter more than they do for ordinary networks. Save checkpoints often.

No honest score. A classifier has an accuracy. A GAN’s losses tell you about the contest, not the images. People use measures like the Fréchet Inception Distance (FID), which compares statistics of real and generated images, plus their own eyes. You will meet FID next lesson.

Better architectures help. Replacing the linear layers with convolutions (the DCGAN design of 2015) gives much cleaner images. Feeding the digit label to both networks (a conditional GAN) lets you ask for a specific digit, which is what you would want for filling gaps in Crumb’s slip data.

Fakes can deceive. The same technique, scaled up, produced the first wave of photo-realistic fake faces. That is worth remembering whenever you build a generator: decide in advance how its outputs will be labelled and used.

And the story’s ending: Arjun trained a conditional GAN on MNIST plus Crumb’s 900 slip digits and added 5,000 generated digits to his reader’s training set. The reader’s error rate on a held-out set of real slips dropped from 2.6% to 2.1%: a real but modest gain. The bigger improvement came the following week, when the clerks labelled 1,000 more real slips. Synthetic data helped; real data helped more. That is the usual story.

Try it yourself

1. Change Z_DIM to 2 and retrain for 10 epochs. Do the samples get better or worse? Why might that be?
2. Without running anything: in the first training round of a brand-new GAN, which network is likely to “win”, and what would you expect D(fake) to be?

Show the answer

1. Usually worse and less varied. With only two random numbers, the generator has very little room to describe the many ways a digit can look (slant, thickness, which digit it is), so samples tend to be blurrier and more repetitive. Too little noise limits variety; very large noise vectors are mostly harmless but wasteful.
2. The discriminator. Early fakes are random speckle, which is easy to tell from real digits, so D(fake) quickly drops to a small value, often below 0.1 within the first few hundred steps. This is exactly when the non-saturating loss earns its keep.

Common mistakes

• Expecting the losses to go to zero. A healthy GAN’s losses wobble around a balance. Judge progress by the images, not by the loss curve alone.
• Forgetting detach() when training the discriminator. The discriminator step then also pushes the generator, in the wrong direction. Training goes strange in ways that are hard to debug.
• Mismatched pixel ranges. If real images are in [0, 1] and fakes in [-1, 1], the discriminator wins on brightness alone. Normalise real data to match the generator’s output.
• Judging from a single random sample. Use a fixed grid of noise vectors so you see the same 64 inputs evolve, and so mode collapse is visible.
• Assuming synthetic data replaces real data. It can fill gaps; it rarely beats more real examples.

Recap

• A generative model produces new examples that look like its training data, usually starting from random noise.
• Matching real images pixel by pixel produces a blurry average; a GAN instead learns a judge of what looks real.
• The generator (apprentice) makes fakes from noise; the discriminator (taster) scores images as real or fake; they train in turns.
• The non-saturating generator loss, $-\log D(G(z))$, gives useful feedback even when early fakes are terrible.
• GAN losses do not head to zero; at a perfect balance the discriminator outputs 0.5, and the big risks are mode collapse and instability.

Meera looked at the grid of digits. “None of these were written by a person?”

“Not one,” said Arjun. “There are two networks. One forges handwriting from random numbers. The other is a detective trained on real handwriting, trying to catch the forgeries. They practise against each other. The forger never sees a real digit, only the detective’s verdicts, and still learns to write like a person. The catch is that it gets lazy: it likes writing the digits that are easiest to fake.”

“Could it make faces? For the Christmas posters?”

“Big GANs can,” said Priya. “But they’re temperamental to train. There’s a calmer cousin that builds a proper map of everything it has seen. Next time, Arjun takes it to a hackathon.”

Go deeper

• PyTorch tutorials, which include a DCGAN tutorial on face images.
• TensorFlow tutorials, which include a DCGAN on MNIST.