TensorBoard for generative models

Priya asked a simple question on Tuesday morning. “Of your fourteen face-VAE runs, which settings gave the best faces?”

Arjun opened his Colab notebook. The runtime had disconnected overnight, and with it went every printed number from the last six runs. The other eight were scattered across three notebooks, a text file called results_FINAL_v2.txt, and some screenshots on his phone. He knew run 9 “looked good”. He did not know whether run 9 used a learning rate of 0.001 or 0.0005, or whether that was the run where he had changed beta.

Meera, passing by, asked the manager’s version. “You’ve rented GPUs for two weeks. How do you know it’s getting better?”

“I look at the pictures,” Arjun said, and then, more honestly, “when I remember to save them.”

Why generative models need a dashboard more than most

When you train a classifier, one number tells most of the story: accuracy on held-out data. If it goes up, you are winning.

Generative models do not give you that comfort. In the GAN lesson you saw that the generator’s loss went up while its digits got dramatically better, because the loss measures the contest, not the quality. In the VAE lesson, a perfect-looking low loss produced the same beige face for every sample. The numbers you print can actively mislead you. You need several signals at once, plus the images themselves, over time, for every run.

That is what Arjun was missing. Not intelligence; instrumentation.

Arjun’s first attempt: print and squint

Arjun’s training loop already printed a line per epoch, like the GAN lesson’s:

epoch  1  loss_D 0.67  loss_G 1.44  D(real) 0.76  D(fake) 0.30
epoch  2  loss_D 1.28  loss_G 2.48  D(real) 0.96  D(fake) 0.69
epoch  3  loss_D 0.42  loss_G 2.30  D(real) 0.82  D(fake) 0.13

“Epoch 2 looks like a disaster,” Priya said. “The discriminator loss nearly doubled. Then epoch 3 is fine again. What happened?”

“I… don’t know.”

“Nothing happened. You’re printing the loss of the last batch of each epoch: 128 images out of 60,000. It’s one noisy sample, like judging a whole day’s baking by the last croissant out of the oven. The average over that epoch was steady. You were reading noise as news.”

He had also tried copying numbers into a spreadsheet after each run. It worked for three runs, then he forgot a column, then he forgot a run. And none of it showed him the images changing, which was the thing he cared about most.

Priya’s oven window

“How does Anil at the Camp store know the plum cakes are baking properly?” Priya asked.

“He looks through the oven window. And there’s that temperature chart on the wall that the oven writes automatically.”

“Does he open the door every five minutes?”

“No, the cakes would sink.”

“Does he write the temperature down by hand?”

“No. The oven logs it. He glances at the chart.”

“Then you want three things. A window: pictures of what the model is producing, at regular intervals, from the same inputs so you can compare. A chart that writes itself: averaged numbers logged automatically, every so often, not every second. And a label on every batch: which settings, which day, which recipe. That’s TensorBoard.”

Where the analogy breaks: an oven has one right temperature. A generative model has no single right number; you read several signals together, and the pictures settle arguments.

Following two GAN runs through the dashboard

To learn to read the dashboard, Arjun trained the MNIST GAN from two lessons ago twice, logging everything. The only difference was the learning rates:

• Run healthy: both networks at a learning rate of 0.0002, the standard DCGAN setting.
• Run strong_D: the discriminator at 0.001 (five times faster) and the generator at 0.0001 (half speed). A deliberately unfair contest.

Here are the per-epoch averages (over all 468 batches, not only the last one) from those real runs:

Epoch	healthy: loss D	loss G	D(fake)	strong_D: loss D	loss G	D(fake)
1	0.96	1.30	0.46	0.67	1.84	0.33
2	0.74	2.07	0.30	0.48	2.75	0.16
4	0.62	2.60	0.22	0.41	3.19	0.13
6	0.57	2.59	0.19	0.34	3.51	0.10
8	0.61	2.58	0.20	0.30	3.65	0.09

D(fake) is the discriminator’s average “probability real” for generated images. Read the two runs like a doctor reading two patients’ charts.

healthy settles. After epoch 4 every number levels off: the discriminator loss around 0.6, the generator loss around 2.6, D(fake) around 0.2. The discriminator is ahead, which is normal in practice, but the contest is stable. Neither side is running away with it.

strong_D drifts. Every epoch the discriminator loss falls further (0.67 to 0.30), the generator loss climbs (1.84 to 3.65), and D(fake) sinks towards zero (0.33 to 0.09). The taster is pulling away, and the apprentice’s feedback is getting weaker each epoch. That steady one-way drift, not any single value, is the warning sign. Left running, runs like this often stop improving or collapse.

Then the window: the fixed-noise image grids, side by side. By epoch 8 the healthy grid shows varied, readable digits. The strong_D grid is readable too, but blotchier and noticeably less varied. Arjun had also logged a crude “diversity” number, the average pixel spread across 256 random samples: 0.32 for healthy against 0.25 for strong_D. Fewer different-looking outputs is the early shadow of mode collapse.

Opening the hood: what TensorBoard actually is

TensorBoard started as TensorFlow’s visualisation tool and now works with PyTorch too. It has two halves:

• A writer in your training code. PyTorch’s SummaryWriter appends small records (“at step 1,500, loss/generator was 2.61”) to event files in a folder called the log directory. One folder per run.
• A viewer, a small local web server you start with tensorboard --logdir runs. It reads every run folder under runs/ and draws them, updating live as training writes more.

Each record has a tag (a name like loss/generator; the part before the slash groups charts into sections), a value, and a step (usually the training step or epoch, the x-axis). The main kinds:

What you log	Writer method	Use it for
A number	add_scalar(tag, value, step)	Losses, D(real), D(fake), KL, learning rate, FID
An image grid	add_image(tag, grid, step)	Fixed-noise samples, reconstructions
A histogram	add_histogram(tag, tensor, step)	Weights or gradients, to catch values exploding or dying
Settings plus results	add_hparams(hparams, metrics)	Comparing runs in a table

The viewer’s smoothing slider draws a moving average over noisy curves. It uses an exponential moving average: each smoothed point is mostly the previous smoothed point plus a little of the new value,

$$s_t=\alpha s_{t-1}+(1-\alpha )x_t$$

where $x_t$ is the raw value at step $t$ and $\alpha$ (between 0 and 1) is the slider. At $\alpha =0.9$, each new point moves the line only 10% of the way. That is the fix for Arjun’s last-batch problem, done for you, while the raw faint line stays visible underneath.

The metrics that matter for generative models

For a GAN: both losses; D(real) and D(fake) (the clearest view of who is winning); a fixed-noise image grid every epoch; and a diversity signal or FID.

For a VAE: the reconstruction loss and the KL term separately (their sum hides the tug of war); a grid of reconstructions next to their originals; a grid of random samples from the prior.

Across both: FID, the Fréchet Inception Distance. It runs real and generated images through a pretrained image network (Inception v3), summarises each set’s features by their average $\mu$ and spread $\Sigma$ (a covariance matrix, which records how features vary together), and measures how far apart the two summaries are:

$$\text{FID}=\Vert {\mu }_r-{\mu }_g{\Vert }^2+\mathrm{Tr}\left({\Sigma }_r+{\Sigma }_g-2({\Sigma }_r{\Sigma }_g{)}^{1/2}\right)$$

You do not need to compute this by hand. Read it as “how different the average generated image’s features are, plus how different the variety is”. Lower is better, and 0 means statistically identical. Because it measures variety too, FID rises when a GAN starts collapsing to a few outputs, even if each output looks sharp.

Adding TensorBoard to the GAN, one step at a time

Install with uv add tensorboard (plain pip install tensorboard in Colab, if it is not already there).

Step 1: create a writer per run, with a name that says what changed.

from torch.utils.tensorboard import SummaryWriter
from torchvision.utils import make_grid

run_name = "gan_lrG2e-4_lrD2e-4_z64"          # the settings, in the folder name
writer = SummaryWriter(f"runs/{run_name}")

Naming runs by their settings is the cheapest habit you will ever adopt. Arjun’s “run 9” problem disappears.

Step 2: log averaged scalars every N steps, not every step.

step = 0
for epoch in range(1, 21):
    for real, _ in loader:
        # ... the discriminator and generator updates from the GAN lesson ...
        if step % 50 == 0:                                   # every 50 batches
            writer.add_scalar("loss/discriminator", loss_D.item(), step)
            writer.add_scalar("loss/generator", loss_G.item(), step)
            writer.add_scalar("D/real", d_real.mean().item(), step)
            writer.add_scalar("D/fake", d_fake.mean().item(), step)
        step += 1

.item() turns a one-number tensor into a plain Python number. Every 50 steps is plenty: 9,000 steps of training give 180 points per curve, enough to see trends without slowing anything down.

Step 3: log the oven window, once per epoch, from fixed noise.

    # still inside the epoch loop, after the batches
    with torch.no_grad():
        samples = G(fixed_z).view(-1, 1, 28, 28) * 0.5 + 0.5   # back to [0, 1]
    writer.add_image("samples/fixed_noise", make_grid(samples, nrow=8), epoch)

make_grid tiles 64 images into one picture. Because fixed_z never changes, TensorBoard’s image tab gets a slider: drag it and watch the same 64 noise vectors turn from speckle into digits. If several cells start converging on the same digit, that is mode collapse, visible days before any number would tell you.

Step 4: close the writer, and record the settings with the results.

writer.add_hparams(
    {"lr_G": 2e-4, "lr_D": 2e-4, "z_dim": 64},
    {"final/loss_D": loss_D.item(), "final/D_fake": d_fake.mean().item()},
)
writer.close()     # flush everything to disk

The HParams tab in the viewer then shows a table of every run’s settings and results, sortable. That is the table Priya asked for on Tuesday.

Step 5: open the dashboard. Locally:

uv run tensorboard --logdir runs
# then open http://localhost:6006 in your browser

In Colab or Jupyter, run these in a cell instead; the dashboard appears inside the notebook:

%load_ext tensorboard
%tensorboard --logdir runs

You should see the Scalars tab with one coloured line per run, the Images tab with a slider per run, and the HParams tab with your table. Tick runs on and off in the left panel to compare them.

Logging a VAE: watch the tug of war

For the face VAE, log the two forces separately, plus two image grids.

writer = SummaryWriter("runs/vae_celeba_beta1_lat128")
fixed_faces, _ = next(iter(loader))            # the same 8 real faces every epoch
fixed_faces = fixed_faces[:8].to(device)
fixed_z = torch.randn(32, LATENT, device=device)

for epoch in range(1, 11):
    for i, (x, _) in enumerate(loader):
        # ... forward pass, vae_loss, backward, step, from the VAE lesson ...
        if i % 100 == 0:
            global_step = epoch * len(loader) + i
            writer.add_scalar("loss/reconstruction", rec.item(), global_step)
            writer.add_scalar("loss/kl", kl.item(), global_step)
    model.eval()
    with torch.no_grad():
        recon, _, _ = model(fixed_faces)
        writer.add_image("faces/original_vs_recon",
                         make_grid(torch.cat([fixed_faces, recon]), nrow=8), epoch)
        writer.add_image("faces/samples", make_grid(model.decode(fixed_z), nrow=8), epoch)
    model.train()
writer.close()

The reconstruction grid puts the 8 originals on the top row and their rebuilds underneath, so you can see exactly which details (earrings, backgrounds, hair parting) the model is losing.

Adding FID

FID needs a pretrained network and a few thousand images, so compute it occasionally (every few epochs), not every step. The torchmetrics library wraps it.

# uv add "torchmetrics[image]"
from torchmetrics.image.fid import FrechetInceptionDistance

fid = FrechetInceptionDistance(feature=2048, normalize=True)  # normalize: floats in [0, 1]

def to_rgb(imgs):                         # MNIST is greyscale; Inception expects 3 channels
    return imgs.repeat(1, 3, 1, 1)

real_imgs, _ = next(iter(DataLoader(mnist, batch_size=2000, shuffle=True)))
fid.update(to_rgb(real_imgs * 0.5 + 0.5), real=True)
with torch.no_grad():
    fake_imgs = G(torch.randn(2000, Z_DIM, device=device)).view(-1, 1, 28, 28).cpu() * 0.5 + 0.5
fid.update(to_rgb(fake_imgs), real=False)
writer.add_scalar("quality/fid", fid.compute().item(), epoch)
fid.reset()                               # clear the generated images before the next check

Library APIs change quickly. If a name has moved, check the docs linked below.

Where dashboards break

Logging too much. Arjun’s first instrumented run logged an image grid every step. After an hour the log folder was over 3 GB, the Colab disk was nearly full, and training had slowed by a third because writing images is not free. Scalars every 50 to 100 steps and images once per epoch is a sensible default.

Colab forgets. A Colab runtime’s disk is wiped when it disconnects, and your runs/ folder goes with it. Mount Google Drive and write logs there, or download the folder at the end of each session.

add_scalars creates phantom runs. The plural add_scalars (several lines on one chart) quietly writes each line into its own sub-folder, which then shows up as an extra “run” in the viewer. Prefer separate add_scalar calls with shared tag prefixes like D/real and D/fake.

Smoothing hides spikes. A heavy smoothing slider makes a sudden collapse look like a gentle bend. When something looks odd, turn smoothing down and look at the raw line.

One run is an anecdote. GANs in particular vary a lot between random seeds. If run_A beats run_B by a little, run each with two or three seeds before believing it.

FID has fine print. It needs thousands of images to be stable (with a few hundred it is biased upwards), and numbers are only comparable when computed the same way: same image count, same resizing, same library. Never compare your FID with a number from a paper unless you know they match.

Try it yourself

Run the GAN twice for 5 epochs with TensorBoard logging: once with lr_D = 2e-4, once with lr_D = 1e-3 (keep lr_G = 2e-4). Name the runs by their settings. In TensorBoard, which run’s D/fake curve drifts lower, and what does the image slider show at epoch 5?

Show the answer

The lr_D = 1e-3 run’s discriminator learns faster, so its D/fake curve sits lower and tends to keep sliding, while its generator loss climbs. The images at epoch 5 are usually recognisable in both runs, but the fast-discriminator run tends to look blotchier and less varied. The exact numbers depend on your seed; the direction of the drift is what you are looking for.

Common mistakes

• Printing last-batch numbers and reading them as trends. Average over many batches, or let TensorBoard’s smoothing do it.
• Logging random samples instead of fixed noise. New noise each epoch makes it impossible to tell whether the model improved or you got lucky.
• Logging only the total VAE loss. The reconstruction and KL terms tell you about the tug of war; their sum hides it.
• Vague run names. run9 means nothing in a week. Put the settings in the folder name.
• Trusting one number. For generative models, read losses, discriminator outputs, FID and images together.

Recap

• Generative models’ losses can mislead, so you need several signals over time, plus the images themselves.
• TensorBoard’s SummaryWriter writes scalars, image grids, histograms and settings into one folder per run; tensorboard --logdir runs shows them all.
• For GANs, watch both losses, D(real) and D(fake) and a fixed-noise grid; a steady one-way drift is the warning sign.
• For VAEs, log reconstruction and KL separately and show originals next to reconstructions; KL stuck at zero means posterior collapse.
• FID compares real and generated images’ feature statistics, catching both poor quality and poor variety; compute it occasionally and compare only like with like.

That afternoon, Arjun put the dashboard on the meeting-room screen. Meera watched him drag the image slider back and forth: 64 squares of static resolving into digits, epoch by epoch.

“So you can see it learning.”

“Every run, side by side, with its settings on the label,” he said. “The numbers tell me whether the two networks are still in a fair fight. The pictures tell me whether the fight is producing anything good. Before, I was opening the oven door every five minutes and forgetting what I’d seen.”

“And the fourteen face runs?”

“Gone, the ones on the Colab disk. Lesson learned. The new ones log to Drive.”

Priya was already looking ahead. “You’ve now trained models that make images. Next module, we go back to the models Crumb actually uses every day, the language models, and look inside them while they run: why they slow down, why they eat memory, and the tricks that make them fast.”

Go deeper

• PyTorch tutorials, which include a TensorBoard tutorial for PyTorch.
• TensorFlow tutorials, the home of TensorBoard’s own guides.
• Hugging Face docs, for generative-model libraries that log to TensorBoard.