Diffusion, Vision Transformers and CLIP

The Christmas campaign was three weeks away, and Meera came to the AI team with three problems on one sheet of paper.

1. Posters. The photographer’s quote was still ₹1.8 lakh. Could AI make campaign images of plum cakes in festive settings?
2. Damaged boxes. The packing line at the central kitchen shipped about 1,200 cake boxes a day in December. Last year, roughly 1 in 60 left with a crushed corner or smudged icing, and each one cost a refund and a bad review.
3. The photo library. Crumb had 4,000 product photos in a shared drive, named things like IMG_4471.jpg. Nobody could find anything.

Arjun had a plan for the first one already. “I’ve built a GAN and a VAE in Module 31. I’ll train one on our 4,000 product photos.”

Priya let him run it overnight. The VAE produced soft brown smudges that were clearly cake-shaped and clearly not posters. The GAN produced a few sharp 64 × 64 images, then collapsed into the same chocolate-cake-ish blob over and over. Neither came close to a printable poster.

“Those were the state of the art about ten years ago,” said Priya. “Today’s image generators work differently. And the other two problems need transformers that see. Three ideas today: diffusion, vision transformers, and CLIP, which ties them together.”

Part 1: Diffusion, or how to un-steam a window

Why the old generators struggled

Recall the two generators from Module 31. A GAN learns by fooling a discriminator: sharp images, but unstable training and mode collapse. A VAE learns a smooth map of images: stable, but blurry, because its pixel-matching loss rewards averaging. Both try to produce a whole image in one step, from a random code.

Diffusion models split that one impossible jump into hundreds of easy ones.

Priya’s steamed-up window

“Picture the Camp store window on a cold December morning,” Priya said, “fully steamed up. You can’t see anything. Now someone wipes it, a little. A vague shape appears. Another little wipe. Now it’s clearly a display case. Another. Cakes.”

“So each wipe removes a bit of steam.”

“Now imagine a model that has learned, from millions of photos, how to do one small wipe on any steamed-up window: look at a slightly foggy picture and guess which part is fog. That’s an easy, well-defined job. Then start from a window that’s entirely fog, with nothing behind it, and keep wiping.”

“There’s nothing behind it. What appears?”

“Whatever the model believes a window ought to look like. Each wipe makes the image a little more like a real photo. After fifty wipes, you have a photo of a bakery window that never existed.”

Where the analogy breaks: real fog hides a real scene. In diffusion, there is no hidden scene; the model invents one, wipe by wipe, guided by what it learned and by your prompt.

Following one pixel through the noise

Training needs the opposite direction first: take real images and add noise to them, in a controlled schedule, so the model can practise removing it. This is the forward process.

Take one pixel of a real plum cake photo, with value $x_0=0.8$ (pixels scaled to the range -1 to 1). Draw one random noise value from a bell curve, say $ϵ=-1.2$. The noisy pixel at step $t$ is:

$$x_t=\sqrt{{\bar{\alpha }}_t}{x}_0+\sqrt{1-{\bar{\alpha }}_t}ϵ$$

${\bar{\alpha }}_t$ (alpha-bar) is a number from the noise schedule that starts near 1 (almost all signal) and falls towards 0 (almost all noise) as $t$ grows. Follow the pixel:

Step	${\bar{\alpha }}_t$	Calculation	Noisy pixel $x_t$
Early	0.99	$0.995\times 0.8+0.1\times (-1.2)$	0.676
Middle	0.50	$0.707\times 0.8+0.707\times (-1.2)$	-0.283
Late	0.01	$0.1\times 0.8+0.995\times (-1.2)$	-1.114

Early on, the pixel is still mostly cake. By the end, it is almost pure noise; the 0.8 has nearly vanished.

The training task: show the model the noisy pixel (in practice, the whole noisy image) and the step number $t$, and ask it to predict the noise that was added, $ϵ=-1.2$. If it predicts $-1.0$, the error is 0.2. The loss is the squared error between true and predicted noise, averaged over every pixel:

$$\mathcal{L}=\Vert ϵ-{ϵ}_{\theta }(x_t,t){\Vert }^2$$

Here ${ϵ}_{\theta }$ is the model (with learned parameters $\theta$). Notice the trick that makes training cheap: thanks to the formula, you can jump straight to any step $t$ in one calculation. No need to add noise 999 times to practise on step 999.

Generating: running the film backwards

To make a new image, start with pure random noise, set $t$ to the last step, and repeat:

1. Ask the model to predict the noise in the current image.
2. Subtract a scaled portion of that prediction, giving a slightly cleaner image for step $t-1$ (sampling methods add back a little fresh randomness here too).
3. Move to the next step.

The original 2020 DDPM method used 1,000 steps. Better samplers soon cut this to 20 to 50 steps, and distilled models such as SD-Turbo produce a usable image in 1 to 4 steps by training a fast student to imitate the slow sampler: the same distillation idea from Module 32.

Making it practical: latent diffusion and text prompts

Running hundreds of steps on full-resolution pixels is expensive. Latent diffusion, the idea behind Stable Diffusion (2022), uses a VAE, the very model you built in Module 31, to compress each image first. A $512\times 512$ colour image has $512\times 512\times 3=786,432$ numbers; Stable Diffusion’s VAE compresses it to a $64\times 64\times 4$ latent of 16,384 numbers, 48 times fewer. All the noising and denoising happens on the small latent; the VAE’s decoder turns the final latent back into pixels. The VAE’s blurriness matters much less here, because the decoder only has to render detail that the diffusion process has already decided.

How does “plum cake on a festive table, warm light” steer the result? A text encoder turns the prompt into embeddings, and the denoising network attends to them through cross-attention, the same mechanism a translation decoder uses to look at its encoder’s output (Module 30). Stable Diffusion 1.x used CLIP’s text encoder for this, which is where Part 3 of this lesson connects.

One more trick, classifier-free guidance, makes images follow prompts more strongly. At each step the model predicts the noise twice, once with the prompt and once without, and pushes further in the direction the prompt suggests:

$$\widehat{ϵ}={ϵ}_{\text{no prompt}}+w\cdot ({ϵ}_{\text{prompt}}-{ϵ}_{\text{no prompt}})$$

A guidance weight $w$ around 7 is a common default for Stable Diffusion 1.x. Higher values follow the prompt more literally but can look overcooked.

The denoising network itself was originally a convolutional U-Net. Many newer image models, such as Stable Diffusion 3 and FLUX, use transformers instead: the architecture from Module 30, applied to patches of the latent image. Which brings us to Part 2.

Diffusion in code

The forward process and one training step fit in a few lines. The TinyDenoiser stands in for a real U-Net or transformer:

import torch
import torch.nn as nn
import torch.nn.functional as F

T = 1000
betas = torch.linspace(1e-4, 0.02, T)          # how much noise each step adds (DDPM schedule)
alpha_bar = torch.cumprod(1 - betas, dim=0)    # how much of the original survives by step t

def add_noise(x0, t, noise):
    """Jump straight to step t: x_t = sqrt(a_bar) * x0 + sqrt(1 - a_bar) * noise."""
    a = alpha_bar[t].view(-1, 1, 1, 1)
    return a.sqrt() * x0 + (1 - a).sqrt() * noise

for t in (0, 50, 250, 500, 999):
    print(f"t={t:3d}  signal kept {alpha_bar[t].sqrt():.3f}  noise {(1 - alpha_bar[t]).sqrt():.3f}")

class TinyDenoiser(nn.Module):
    """Stand-in for a U-Net: takes a noisy image and the step, predicts the noise."""
    def __init__(self):
        super().__init__()
        self.net = nn.Sequential(nn.Conv2d(2, 32, 3, padding=1), nn.SiLU(),
                                 nn.Conv2d(32, 32, 3, padding=1), nn.SiLU(),
                                 nn.Conv2d(32, 1, 3, padding=1))
    def forward(self, x, t):
        t_map = (t.float() / T).view(-1, 1, 1, 1).expand_as(x)   # tell it how noisy x is
        return self.net(torch.cat([x, t_map], dim=1))

model = TinyDenoiser()
opt = torch.optim.Adam(model.parameters(), lr=1e-3)
x0 = torch.rand(16, 1, 28, 28) * 2 - 1        # a batch of "images" in [-1, 1]
t = torch.randint(0, T, (16,))                # a random noise level for each image
noise = torch.randn_like(x0)
loss = F.mse_loss(model(add_noise(x0, t, noise), t), noise)   # learn to predict the noise
opt.zero_grad(); loss.backward(); opt.step()

t=  0  signal kept 1.000  noise 0.010
t= 50  signal kept 0.985  noise 0.173
t=250  signal kept 0.722  noise 0.692
t=500  signal kept 0.279  noise 0.960
t=999  signal kept 0.006  noise 1.000

cumprod multiplies the per-step survival rates together, giving ${\bar{\alpha }}_t$ for every step at once. By step 500, only 28% of the original signal remains; by step 999, essentially none.

To use a real pretrained model, Hugging Face’s Diffusers library loads a whole pipeline. On a GPU (Colab’s T4 works):

# uv add diffusers transformers accelerate torch   (NVIDIA GPU recommended)
import torch
from diffusers import AutoPipelineForText2Image

pipe = AutoPipelineForText2Image.from_pretrained(
    "stabilityai/sd-turbo", torch_dtype=torch.float16).to("cuda")
image = pipe(prompt="a plum cake on a festive table, warm light, photo",
             num_inference_steps=1, guidance_scale=0.0).images[0]   # turbo models: 1 step, no guidance
image.save("plum_cake.png")

Library APIs change quickly. If a name has moved, check the docs linked below.

Meera’s poster plan ended up as a hybrid: generated backgrounds and festive scenes from a commercially licensed model, with Crumb’s real plum cake photographed on a plain table and composited in. The cake customers would receive was the cake in the poster.

Part 2: Vision transformers, or reading an image as a sentence

The packing-line problem

For the damaged boxes, Arjun’s first instinct was the convolutional network (CNN) from Module 28, and it was a reasonable one: CNNs are excellent at images. But Priya wanted him to understand the architecture that now dominates image models too.

“You’ve spent a module learning that transformers read sequences of tokens,” she said. “An image isn’t a sequence. How would you make it one?”

Arjun thought. “Read it pixel by pixel, left to right, row by row?”

“A $224\times 224$ photo has 50,176 pixels. With attention’s cost growing with the square of the length?”

“Two and a half billion attention scores per layer. No.” He looked at the photo on his screen. “Then group the pixels. Cut the image into tiles, and treat each tile as a word.”

Patches as tokens

That is exactly the Vision Transformer (ViT), from a 2020 Google paper titled “An Image is Worth 16x16 Words”. Follow one $224\times 224$ photo of a cake box:

1. Cut into patches. Tiles of $16\times 16$ pixels give $14\times 14=196$ patches.
2. Flatten and project. Each patch has $16\times 16\times 3=768$ numbers (three colour channels). A learned linear layer maps each to a 768-number embedding: one “token” per patch.
3. Add a [CLS] token and positions. A special learned token goes at the front, as in BERT (Module 30), making 197 tokens. Learned position embeddings record where each patch came from, because attention on its own ignores order.
4. Run a transformer encoder. Standard blocks of multi-head self-attention and feed-forward layers. Every patch can attend to every other: a crushed corner can be compared with the rest of the box in a single layer.
5. Classify. The [CLS] token’s final vector goes into a small classifier: “ok” or “damaged”.

import torch
import torch.nn as nn

# A Conv2d with kernel = stride = 16 is the standard trick for "cut into 16x16 patches and project"
patchify = nn.Conv2d(in_channels=3, out_channels=768, kernel_size=16, stride=16)
image = torch.randn(1, 3, 224, 224)              # one RGB image
patches = patchify(image)                        # (1, 768, 14, 14): a 14x14 grid of embeddings
tokens = patches.flatten(2).transpose(1, 2)      # (1, 196, 768): a sequence of 196 tokens
print(patches.shape, tokens.shape)

torch.Size([1, 768, 14, 14]) torch.Size([1, 196, 768])

A convolution whose window and step are both 16 visits each patch exactly once and applies the same learned projection to it, which is precisely “flatten each patch and multiply by one matrix”. From here on, it is the encoder you built in Module 30.

The practical upshot for Crumb: never train a ViT from scratch on 800 photos of cake boxes. Start from a ViT pretrained on millions of images and fine-tune it, exactly the fine-tuning logic of this module. To try a pretrained one:

# uv add transformers torch pillow
from transformers import pipeline

classify = pipeline("image-classification", model="google/vit-base-patch16-224")
print(classify("cake.jpg")[:3])     # top 3 of the 1,000 everyday categories it was trained on

That model knows 1,000 everyday categories, not “damaged Crumb box”. Arjun fine-tuned it on 800 labelled photos from the packing line (650 fine, 150 damaged, with the damaged ones photographed from several angles). On 200 held-out photos it caught 46 of 50 damaged boxes while flagging 9 good ones for a human to double-check: not perfect, but a quick human check on flagged boxes was far cheaper than refunds.

Part 3: CLIP, the bridge between pictures and words

The photo library problem

The third problem was search: find “chocolate truffle cake with gold leaf” among 4,000 files named IMG_4471.jpg. Tagging them by hand would take a week. Arjun had used CLIP briefly in Module 12; now he could see how it works inside.

How CLIP learns

CLIP (contrastive language-image pretraining), from OpenAI in 2021, has two encoders: an image encoder (often a ViT, like the one you met in Part 2) and a text encoder (a transformer). Both output vectors in the same embedding space. It was trained on about 400 million image-caption pairs gathered from the internet with one goal: an image’s vector should be close to its own caption’s vector and far from every other caption’s.

Follow one training batch of three pairs: a plum cake photo with “a plum cake”, a croissant photo with “a croissant”, a delivery scooter with “a delivery scooter”. CLIP computes the similarity of every image with every caption: a $3\times 3$ grid. Suppose the similarities, after CLIP’s scaling, are:

	“a plum cake”	“a croissant”	“a delivery scooter”
plum cake photo	3.2	1.8	0.5
croissant photo	1.5	3.0	0.8
scooter photo	0.4	0.9	2.8

The matching pairs sit on the diagonal. Softmax along each row turns them into “which caption is mine?” probabilities: the plum cake photo gives 0.76 to its own caption, the croissant photo 0.75, the scooter photo 0.81. The loss is the cross-entropy of picking the right caption (about 0.27 for the first row), averaged over rows, plus the same calculation down the columns (each caption picking its image). Training pushes the diagonal up and everything else down. This is contrastive learning: learning by comparing matches against non-matches. With large batches, each pair is contrasted against many thousands of wrong ones.

The result is the embedding idea from Module 07, extended across two kinds of data: “photos of plum cakes” and the words “plum cake” land in the same neighbourhood of the map.

Using CLIP: zero-shot labels and photo search

Because images and text share a space, you can classify an image against labels you invent on the spot, with no training. Arjun tried it on a crude drawing of a brown cake shape:

import torch
from PIL import Image
from transformers import CLIPModel, CLIPProcessor

model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32")
processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")

labels = ["a photo of a chocolate cake", "a photo of a croissant", "a photo of a delivery scooter"]
inputs = processor(text=labels, images=Image.open("cake.jpg"), return_tensors="pt", padding=True)
with torch.no_grad():
    probs = model(**inputs).logits_per_image.softmax(dim=-1)   # image vs each label
for label, p in zip(labels, probs[0]):
    print(f"{p:.2f}  {label}")

0.57  a photo of a chocolate cake
0.43  a photo of a croissant
0.00  a photo of a delivery scooter

Right answer, but barely: on a crude drawing, “brown baked thing” was about as far as CLIP could get. On real product photos, the margins are much larger.

For the photo library, encode every photo once, store the vectors, and compare each search query against them. The Sentence Transformers library wraps CLIP neatly:

# uv add sentence-transformers pillow
from pathlib import Path
from PIL import Image
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("clip-ViT-B-32")                 # CLIP, wrapped for easy encoding

paths = sorted(Path("photos").glob("*.jpg"))                # Crumb's product photo library
img_vecs = model.encode([Image.open(p) for p in paths])      # one 512-number vector per photo

def search(query, k=3):
    scores = model.similarity(model.encode([query]), img_vecs)[0]   # cosine similarity
    best = scores.topk(min(k, len(paths)))
    return [(paths[i].name, round(s.item(), 3)) for s, i in zip(best.values, best.indices)]

print(search("chocolate truffle cake with gold leaf"))

For 4,000 photos, encoding takes a few minutes on a laptop, once; each search is then instant. Store the vectors in the vector database from Module 08 and the library becomes searchable by anyone on the team.

Where these models break

Diffusion models get details wrong. Text in images, hands, counts (“exactly six candles”), and brand-accurate products are classic failures, though newer models are much better. Never let a generated image stand in for your actual product.

ViTs are hungry. Without large-scale pretraining they underperform CNNs; with too few fine-tuning examples of the rare class (damaged boxes), they miss it. Collect examples of the failures you care about, and measure the misses, not only overall accuracy.

CLIP has blind spots. It is weak at counting, at reading small text, and at fine distinctions it rarely saw captioned (“eggless” versus “regular” sponge looks identical). It also absorbed biases from internet captions. Check results on your own photos before trusting it.

All three inherit their data. Each was trained on huge collections of internet images whose licences and consent are still debated. For a business, the licence of the specific model you deploy is what matters; read it.

Try it yourself

1. Using the noise formula, what is the noisy value of a pixel $x_0=-0.5$ with noise $ϵ=0.6$ at a step where ${\bar{\alpha }}_t=0.75$?
2. A ViT uses $32\times 32$ patches on a $224\times 224$ image. How many patch tokens does it get, and why would it be cheaper but possibly less accurate than $16\times 16$ patches?

Show the answer

1. $\sqrt{0.75}\times (-0.5)+\sqrt{0.25}\times 0.6=0.866\times (-0.5)+0.5\times 0.6=-0.433+0.3=-0.133$.
2. $224/32=7$, so $7\times 7=49$ tokens (plus [CLS]), against 196. Attention cost falls roughly with the square of the token count, so about 16 times fewer attention scores. But each token now summarises a bigger area, so fine details such as a small crushed corner are squeezed into fewer numbers, which can hurt accuracy.

Common mistakes

• Treating generated images as product photos. Generate scenes and backgrounds; photograph the real product.
• Ignoring model licences. Image models vary widely in what commercial use they allow.
• Training a ViT from scratch on a small dataset. Fine-tune a pretrained one.
• Measuring only overall accuracy on rare defects. With 1 damaged box in 60, a model that always says “fine” is 98% accurate and useless. Count how many damaged boxes it catches.
• Assuming CLIP understands everything in a caption. Test it on the distinctions your business actually needs.

Recap

• Diffusion models learn to predict the noise added to images at every noise level, then generate by starting from pure noise and removing it step by step.
• Latent diffusion runs this process on a VAE’s compressed latent, steered by text through cross-attention and classifier-free guidance; distillation cuts the steps from hundreds to a handful.
• Vision transformers cut an image into patches, embed each as a token, add positions and a [CLS] token, and run a standard transformer encoder; they need large-scale pretraining, then fine-tune well.
• CLIP trains an image encoder and a text encoder together so matching images and captions land close in one embedding space, enabling zero-shot labels and search by words.
• All three inherit the strengths, gaps and licences of their training data; check them before you rely on them.

At the end of December, Arjun walked Meera through the results. “Posters: the scenes are generated with a properly licensed model, and the plum cake in them is our real plum cake, photographed by Anil on a plain table. The packing line: a camera and a vision transformer flag about four in five damaged boxes for a second look. The photo library: type ‘gold leaf truffle’ and the right photos appear, because the model learned where pictures and words sit on the same map.”

“And you understand how all of it works?”

Arjun thought about the module: the transformer he had built by hand, the GAN that drew too many sevens, the KV cache that crashed the Diwali server, the 4-bit model on the shop PC, the fine-tuned assistant that sounded like Shanti. “Well enough to know where each one breaks,” he said. “Which, Priya keeps telling me, is the part that matters.”

Go deeper

• Hugging Face docs, see Diffusers for image generation and Transformers for ViT and CLIP.
• Hugging Face LLM course
• PyTorch tutorials
• Attention Is All You Need, the architecture ViT and modern diffusion transformers reuse.