Zero-shot classification with CLIP and CLAP

Crumb’s website was getting filters. Customers would tick “eggless”, “festive”, “birthday” or “savoury” and see matching recipes and products. The recipe archive held 6,000 photos going back eleven years, and not one of them was tagged.

“How long would it take to tag them by hand?” Meera asked.

Arjun did the sum. Five seconds a photo, if nobody gets bored: 6,000 × 5 = 30,000 seconds, more than eight hours of clicking, and the next 500 photos would need it all again. Training his own classifier would need hundreds of hand-labelled examples for every tag first.

“There must be a model for this,” he said, and opened the image classifier from the computer vision lesson.

Arjun’s dictionary of labels

The ImageNet classifier knew 1,000 labels. None of them were “eggless”, “festive”, “birthday” or “savoury”. Arjun’s idea: translate. He ran 50 photos through it and started writing a dictionary from its labels to Crumb’s tags:

IMAGENET_TO_CRUMB = {
    "candle": "birthday",
    "trifle": "festive",
    "chocolate sauce": "birthday",
    "pretzel": "savoury",
    "bagel": "savoury",
    # ... and 40 more guesses
}

By photo 50 the dictionary had 45 entries and was mostly wrong. A Diwali box of kaju katli came back as “packet”; a birthday cake with sparklers came back as “plate”. A Christmas plum cake came back as “French loaf”, which his dictionary filed under “savoury”.

Priya pulled up a chair. “You’re trying to force someone else’s labels into yours. Different question. Imagine teaching a child what ‘festive’ looks like, with no labelled examples at all. How would you do it?”

Arjun thought. “Show them lots of pictures with captions people wrote? ‘Our Diwali table.’ ‘Christmas morning.’ ‘Grandma’s birthday.’ Eventually they’d connect the words with what’s in the pictures.”

“That is almost exactly how CLIP was trained. And once it has made those connections, you can write any caption, and ask which photos match it.”

A matching game played in one space

CLIP stands for Contrastive Language–Image Pre-training. OpenAI trained it on about 400 million image-and-caption pairs collected from the internet. It has two halves:

• an image encoder that turns a picture into a list of 512 numbers, and
• a text encoder that turns a sentence into a list of 512 numbers.

Training taught both halves to put a photo and its true caption close together in the same 512-number space, and to push mismatched captions far away. It worked in huge batches: 32,768 images and their 32,768 captions at a time, so each image had to pick out its one true caption from tens of thousands. That is the contrastive part.

You met this kind of space in Module 07, where similar sentences ended up close together. The new idea is that pictures and sentences share one space. A photo of a birthday cake lands near the sentence “a photo of a birthday cake”, and far from “a photo of bread”.

That makes zero-shot classification possible: classifying into labels the model was never trained on as categories. You write each label as a short caption, encode all of them, encode the image, and pick the caption that lands closest. You first met zero-shot classification for text in Module 11, sorting Crumb’s emails; CLIP does the same for pictures.

Where the analogy breaks: the child learns from a family photo album. CLIP learned from the internet, with all its biases. It saw far more Christmas than Diwali, far more English captions than Marathi ones. Keep that in mind; it will matter.

Following one photo through CLIP

Before trusting it with 6,000 recipe photos, Priya made Arjun test it on a photo where he knew the answer for certain: his flatmate’s two cats, asleep on a pink blanket on the sofa. They wrote four candidate captions.

Step 1: encode the image. The photo goes through CLIP’s image processor (resized to 224 × 224, like the vision models last lesson) and its image encoder. Out come 512 numbers.

Step 2: encode the captions. Each caption goes through the text encoder. Four lists of 512 numbers.

Step 3: compare. Cosine similarity scores how closely two lists point in the same direction: 1 means identical direction, 0 means unrelated. Here are the real numbers Arjun got:

Caption	Cosine similarity
“a photo of cats”	0.275
“a photo of a sofa”	0.231
“a photo of a birthday cake”	0.196
“a photo of a dog”	0.193

Arjun frowned. “0.275 versus 0.231. That’s barely different. Is it even sure?”

Step 4: scale and softmax. CLIP multiplies every similarity by a learned temperature of 100, giving 27.5, 23.1, 19.6 and 19.3. Then softmax (Module 11: raise e to each number and divide by the total) turns them into probabilities. Because the scores were multiplied by 100, a small gap becomes a big one:

Caption	Scaled score	Probability
“a photo of cats”	27.5	0.987
“a photo of a sofa”	23.1	0.012
“a photo of a birthday cake”	19.6	0.0004
“a photo of a dog”	19.3	0.0003

Show the answer

No. The photo does show a sofa; the cats are just more prominent. Softmax forces the labels to compete for one prize, so “sofa” loses even though it is also true. The probabilities answer “which of these captions fits best?”, not “is each of these true?”. For tagging, where one photo can be both “birthday” and “eggless”, you need a different approach. You will build it below.

Arjun tried one more caption: “two cats sleeping on a pink blanket”. Its similarity was 0.306, higher than any of the others. More specific, accurate captions land closer. That is a hint about how to write labels.

Step 5: a Crumb photo. Then they tried one from the archive: a chocolate cake with candles on a cake stand. With labels written as captions (“a photo of a birthday cake”, “a photo of festive Indian sweets”, “a photo of savoury snacks”, “a photo of bread”), the birthday caption won clearly. Encoding one photo took a fraction of a second on the laptop.

Opening the hood: why it works, and what it cannot see

Three details explain most of CLIP’s behaviour.

Captions, not categories. CLIP’s text encoder was trained on sentences people write under photos, so it understands “a photo of a birthday cake” better than the bare word “birthday”. That is why tools wrap labels in a template. The Hugging Face pipeline does it for you: by default it turns each label into “This is a photo of {label}.” So you pass plain labels like “birthday cake” to the pipeline, and full captions when you compute similarities yourself.

It only knows what is visible. CLIP compares pixels with words. It can see candles, sparklers, Christmas holly, a box of sweets. It cannot see ingredients.

Show the answer

Not reliably, and not for a good reason. An eggless chocolate cake and a regular chocolate cake look identical. Any “eggless” tag CLIP gives is a guess from incidental features, and a wrong one could hurt a customer who avoids eggs. That tag must come from the recipe’s ingredient list in the database, not from the photo. Knowing what a model cannot know is as important as knowing what it can.

It can read. CLIP saw millions of images containing text, so it partly reads words in pictures. A photo of an apple with a paper label saying “iPod” stuck on it gets classified as an iPod. This typographic attack was famous when CLIP came out, and it will bite Crumb shortly.

The real code

uv add transformers torch pillow sentence-transformers

Zero-shot classification with a pipeline. The quickest way to pick one best label:

from PIL import Image
from transformers import pipeline

classify = pipeline("zero-shot-image-classification", model="openai/clip-vit-base-patch32")

photo = Image.open("archive/0412.jpg")
results = classify(photo, candidate_labels=[
    "a birthday cake", "festive Indian sweets", "a Christmas cake", "savoury snacks", "bread",
])
for r in results:
    print(f'{r["label"]:25} {r["score"]:.3f}')

The pipeline wraps each label in its template, encodes the image and labels, and returns them sorted by probability. You can change the template with hypothesis_template="A bakery photo of {}.".

Multi-label tagging with raw similarities. For tags that can all be true at once, skip softmax. Compare the raw similarity of each tag’s caption with a threshold that you choose per tag, by checking a few dozen hand-tagged photos. The sentence-transformers library wraps CLIP so that one encode call handles both images and text:

from sentence_transformers import SentenceTransformer, util

clip = SentenceTransformer("clip-ViT-B-32")   # CLIP, packaged for easy image and text embeddings

TAGS = {
    "birthday": ("a photo of a birthday cake with candles", 0.27),
    "festive":  ("a photo of festive Indian sweets for Diwali", 0.26),
    "christmas": ("a photo of a Christmas cake", 0.27),
    "savoury":  ("a photo of savoury baked snacks", 0.26),
}
tag_vecs = clip.encode([caption for caption, _ in TAGS.values()])

def tag_photo(path: str) -> list[str]:
    img_vec = clip.encode(Image.open(path))
    sims = util.cos_sim(img_vec, tag_vecs)[0]
    return [tag for (tag, (_, threshold)), s in zip(TAGS.items(), sims) if s >= threshold]

print(tag_photo("archive/0412.jpg"))   # e.g. ['birthday']

Those thresholds are starting guesses. The honest way to set them: hand-tag 100 photos, then pick the threshold for each tag that balances missed tags against wrong ones.

Searching the archive with words. The same embeddings give you image search for free. Encode the 6,000 photos once (a few minutes on a laptop), save the vectors, and search with any sentence:

from pathlib import Path

paths = sorted(Path("archive").glob("*.jpg"))
photo_vecs = clip.encode([Image.open(p) for p in paths], batch_size=32, show_progress_bar=True)

def search(query: str, k: int = 5):
    hits = util.semantic_search(clip.encode(query), photo_vecs, top_k=k)[0]
    return [(paths[h["corpus_id"]].name, round(h["score"], 3)) for h in hits]

print(search("a plum cake with holly on a red plate"))

The marketing team loved this one. “Find me last year’s Diwali hamper photos with marigolds” now took seconds.

CLAP: the same trick for sound

CLAP (Contrastive Language–Audio Pre-training) applies the same idea to audio: an audio encoder and a text encoder trained on hundreds of thousands of sound clips with descriptions, sharing one space. That means zero-shot classification of sounds with labels you write.

Crumb had a real use. In the central kitchen, oven timers sometimes beeped for minutes while everyone was busy, and trays of cookies burned. A cheap microphone could listen, and alert the shift lead when it heard beeping that did not stop.

import librosa
from transformers import pipeline

listen = pipeline("zero-shot-audio-classification", model="laion/clap-htsat-unfused")

clip, _ = librosa.load("kitchen_10s.wav", sr=48000)   # this CLAP model expects 48 kHz audio
for r in listen(clip, candidate_labels=[
    "an oven timer beeping", "a person speaking", "a mixer running", "a dog barking",
]):
    print(f'{r["label"]:25} {r["score"]:.3f}')

Note the sampling rate. Last lesson’s speech models wanted 16 kHz; this CLAP model was trained on 48 kHz audio. Check the model card every time.

Arjun’s first tests were encouraging. A recording of a synthetic two-tone beep scored 0.996 for “an oven timer beeping”. The shop announcement from last lesson scored 0.998 for “a person speaking”. Real kitchens are messier, with beeps, talking and a mixer all at once, which is the same softmax problem as before: one winner when several things are true. For the alert, he used the raw similarity for the beeping label only, checked every five seconds, and alerted after a minute of continuous beeping.

Video: a pile of photos with a clock

Customers sometimes send videos of unboxing a damaged cake. Can CLIP judge a video? A video is just a sequence of frames: still images, 25 or 30 every second. The simplest zero-shot approach is to sample a frame every second, run CLIP on each, and average.

# uv add opencv-python
import cv2
import numpy as np

def video_scores(path: str, labels: list[str], every_s: float = 1.0):
    video = cv2.VideoCapture(path)
    fps = video.get(cv2.CAP_PROP_FPS) or 25
    step, frames, i = int(fps * every_s), [], 0
    ok, frame = video.read()
    while ok:
        if i % step == 0:
            frames.append(Image.fromarray(cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)))  # OpenCV uses BGR order
        ok, frame = video.read()
        i += 1
    frame_vecs = clip.encode(frames)
    label_vecs = clip.encode(labels)
    sims = util.cos_sim(frame_vecs, label_vecs).numpy()      # one row per frame, one column per label
    return dict(zip(labels, sims.mean(axis=0).round(3))), dict(zip(labels, sims.max(axis=0).round(3)))

avg, peak = video_scores("unboxing.mp4", ["a photo of a squashed cake", "a photo of a perfect cake in a box"])
print("average:", avg, "peak:", peak)

Show the answer

The average gets diluted by 16 seconds of a closed box and a kitchen counter, so “squashed” might lose. The peak (the highest frame score) catches the 4 seconds that matter. For “did X ever happen in this video?”, use the peak or the top few frames; for “what is this video mostly about?”, use the average. True video models, such as X-CLIP for zero-shot or VideoMAE behind the video-classification pipeline, also understand motion across frames, which frame sampling cannot.

Library APIs change quickly. If a name has moved, check the docs linked below.

Where it breaks: the birthday plum cake

Arjun tagged all 6,000 photos overnight and spot-checked 200 the next morning. About 9 in 10 were right. Then Meera found a problem on the new website: the Christmas plum cake was showing up under “birthday”.

Priya opened the photo. A plum cake, holly, a red plate… and in the background, a “Happy Birthday” banner from another order being packed on the same table.

“It read the banner,” she said. “The typographic attack, in a bakery.”

They fixed it the practical way: tags with low margins between labels went to a human review queue, and the product photos were reshot (on white, using last lesson’s background removal). Here is the wider list of things to watch for:

• Softmax competition. The pipeline picks one winner. For multi-label tags, use raw similarities with per-tag thresholds.
• Invisible properties. Ingredients, allergens, price and freshness are not in the pixels. Take them from your data, never from CLIP.
• Reading text in images. Banners, labels and stickers can override what is actually pictured.
• Cultural and data bias. CLIP saw far more Christmas than Diwali. In Arjun’s test, “festive” photos of Christmas scored higher than equally festive Diwali sweets. Write culturally specific captions (“festive Indian sweets for Diwali”) and test each tag separately.
• Counting and fine detail. “Three cupcakes” versus “four cupcakes” is close to a coin toss. CLIP is about overall content.
• Wrong sampling rate for CLAP. 48 kHz for this model, not 16 kHz.

Try it yourself

Pick 20 photos from your own phone: food, pets, places, documents. Write five captions that each describe a category, such as “a photo of food”, “a photo of a pet”, “a screenshot of text”.

1. Run the pipeline on all 20 and count how many top labels are right.
2. Rewrite the two worst-performing captions to be more specific, and run it again.
3. Use search() to find photos with a sentence like “a sunset over water”.

Show the answer

Typical results: 15 to 18 of 20 correct on the first try, with errors on photos that genuinely fit two categories (a pet on a plate of food) or on screenshots. More specific captions, like “a photo of a dog or cat” instead of “a pet”, usually fix a couple. Search should return relevant photos in the top five even though you never labelled anything. That is the power of a shared space: once images and words live in it, any sentence becomes a query.

Common mistakes

• Using softmax probabilities for multi-label tags. They make true labels compete. Use raw similarities and thresholds.
• One-word labels with your own similarity code. Write captions (“a photo of a birthday cake”); the pipeline adds a template, raw similarity code does not.
• Asking CLIP about invisible facts. Eggless, gluten-free and price come from your database.
• Averaging over a whole video. A short event disappears; look at the peak frames.
• Skipping the spot-check. Tag a sample by hand and measure before publishing.

Recap

• CLIP has an image encoder and a text encoder trained so that photos and their captions land close together in one 512-number space; CLAP does the same for sounds.
• Zero-shot classification: write each label as a caption, embed everything, pick the closest. No training needed.
• Similarities are scaled by about 100 before softmax, so small gaps become confident-looking probabilities that make labels compete.
• The same embeddings give you image search and multi-label tagging with per-tag thresholds; videos can be handled frame by frame.
• CLIP cannot see ingredients, reads text in images, and carries the biases of its internet training data.

Arjun explained it to Meera with the archive search open on his laptop. “This model learned from 400 million photos with captions, so it puts pictures and sentences on the same map. I write each tag as a sentence, and it checks which sentences sit near each photo. No training, no hand-labelling. It’s right about nine times in ten, it can’t know which cakes are eggless, so that tag comes from the recipe database, and sometimes it reads a banner in the background and gets confused, so unsure ones go to a person.”

Meera typed “Diwali hamper with marigolds” into the search box and watched last year’s photos appear. Then she asked a question CLIP could not answer. “Can I ask it something about a photo? Like a delivery photo: ‘Is the box sealed? What’s the order number on the sticker?’ And can it make pictures for our Diwali posters?”

Go deeper

• Hugging Face documentation (zero-shot image and audio classification): https://huggingface.co/docs
• Hugging Face LLM course: https://huggingface.co/learn/llm-course