Computer vision: classification, detection, background removal

Crumb’s online shop had 340 product photos. Most had been taken on the kitchen counter by whoever was free: a chocolate truffle cake next to a dirty mixing bowl, a box of Diwali kaju katli with a phone charger poking into the corner, a Christmas plum cake on a grey granite slab.

“A photographer quoted eighty thousand rupees to reshoot everything on white,” Meera said. “And there’s a second problem. Last Saturday the Viman Nagar store ran out of glazed doughnuts at five o’clock, and nobody noticed until seven. Two hours of customers walking out. There’s already a camera over the display case. Can a computer count doughnuts?”

Arjun had only ever fed models text. But he knew images were just numbers. How hard could it be?

Arjun’s colour rule

For the product photos, Arjun reasoned like this: the background is light, the cake is dark, so make every light pixel transparent. Ten lines of Pillow, Python’s standard image library:

from PIL import Image

img = Image.open("truffle_cake.jpg").convert("RGBA")   # RGBA: red, green, blue, alpha (transparency)
new_pixels = [
    (r, g, b, 0) if r > 200 and g > 200 and b > 200 else (r, g, b, a)   # near-white -> invisible
    for r, g, b, a in img.getdata()
]
img.putdata(new_pixels)
img.save("truffle_cake_cutout.png")

The result was a disaster. The grey granite counter was not “near-white”, so it stayed. The white cream rosettes on top of the cake were near-white, so they vanished, leaving holes. The white cake board under the cake disappeared too, so the cake floated in mid-air. On the plum cake photo, nothing happened at all.

For the doughnuts, he tried a ready-made image classifier from the Hub on a frame from the display-case camera. It replied with one word, “bagel”, and a score. Not a count. Not even the right food.

Priya looked at both results. “When you look at the cake photo, how do you know the cream is part of the cake and the counter isn’t?”

“I just… know. It’s on the cake. It’s cake-shaped.”

“Exactly. You don’t decide by colour. You recognise things, and their shapes, and what usually goes with what. Your rule only knows colour. That’s what a vision model learns: things, not colours.”

Three questions about one display case

Priya drew Crumb’s glass display case on the whiteboard, with six doughnuts and a cake inside, and wrote three questions under it.

flowchart LR
  A["Photo of display case"] --> B["Classification: what is this a photo of?"]
  A --> C["Detection: what objects, and where?"]
  A --> D["Segmentation: which exact pixels?"]

• Image classification gives one label (or a ranked list of labels) for the whole picture. “This is a photo of a bakery.” It cannot count, and it cannot point.
• Object detection finds each object separately, names it, and draws a bounding box around it: a rectangle described by its left, top, right and bottom pixel positions. “Doughnut at (112, 40)–(260, 190), doughnut at (300, 52)–(441, 201)…” Count the boxes and you have a stock count.
• Image segmentation labels every single pixel. Its output is a mask: a black-and-white picture the same size as the photo, where white means “this pixel belongs to the object”. Background removal is segmentation: keep the cake’s pixels, drop the rest.

Each is more precise than the last, and slower. Arjun had used classification for a detection problem and a colour rule for a segmentation problem.

All three belong to computer vision: the part of AI that turns images into meaning. And the models that handle more than one kind of input, such as images and text together, are called multimodal. This whole module is about them.

Following one photo through all three

Priya took a single frame from the Viman Nagar camera at 4:55 p.m. last Saturday, a phone-quality photo of 4,032 × 3,024 pixels, and they sent it through all three tasks.

How a computer sees it. A pixel is one tiny coloured square. Each pixel is three numbers between 0 and 255: how much red, green and blue. So this one photo is 4,032 × 3,024 × 3 ≈ 36.6 million numbers.

Classification. The classifier, google/vit-base-patch16-224, first sends the photo through an image processor, which plays the same role a tokenizer plays for text. It shrinks the photo to 224 × 224 pixels and rescales the numbers into the range the model was trained on. Now the photo is a grid of 3 × 224 × 224 = 150,528 numbers.

Show the answer

The overall scene survives: colours, big shapes, the general layout. Small things suffer most. A doughnut that covered 300 × 300 pixels in the original now covers about 17 × 17. Text on a price tag becomes a smudge. This is why classifiers work on “what is this photo of” and struggle with “is there a small chip on the third cake”. When small details matter, use a model that works at higher resolution, or crop the region first.

The model then cuts the 224 × 224 image into small squares called patches, each 16 × 16 pixels. That is 14 × 14 = 196 patches. Each patch becomes a list of numbers, much like each word becomes a token. Add one special summary token, as BERT had [CLS] in the last module, and the model, a transformer (the same kind of neural network behind chat models), now looks at 197 “tokens” of picture. Out comes one score for each of the 1,000 categories it was trained on, and softmax turns those into probabilities. For Arjun’s frame:

Label	Score
bakery, bakeshop, bakehouse	0.38
bagel	0.21
French loaf	0.07

Arjun laughed. “So ‘bakery’ was right all along, I just looked at the wrong line.”

“And ‘doughnut’?” Priya asked.

He searched the label list. There was no doughnut among the 1,000. This model was trained on ImageNet, a famous dataset of everyday categories: it knows “bagel”, “pretzel” and “trifle”, but not “doughnut”, “croissant” or “kaju katli”. A classifier can only answer with labels it was trained on. Show it a croissant and it may confidently say “French loaf”.

Detection. The detector, facebook/detr-resnet-50, resizes the photo so its shorter side is about 800 pixels: much more detail than 224. DETR (DEtection TRansformer) then fills exactly 100 “slots”. Each slot holds either an object (one of the 80 everyday categories in the COCO dataset it was trained on, which happily include “donut”, “cake” and “cup”) with a box, or the answer “no object”. For the 4:55 p.m. frame, after dropping slots below 70% confidence, it returned:

donut 0.99  box (412, 1210)-(866, 1592)
donut 0.98  box (930, 1188)-(1381, 1566)
donut 0.97  box (1455, 1240)-(1890, 1611)
cake  0.95  box (2210, 980)-(3302, 1720)
donut 0.91  box (402, 2010)-(840, 2390)
donut 0.74  box (2950, 2140)-(3390, 2500)

Five doughnuts, one cake. It took about two seconds on a laptop processor.

Show the answer

Because it cannot know in advance how many objects a photo will contain. The designers picked a number comfortably above what a normal photo holds and trained the model to fill unused slots with “no object”. The threshold then throws away slots the model is not confident about. A photo with more than 100 objects (a tray of 150 cookies) would be undercounted: a real limit to know about.

Segmentation. For the product photos, they used facebook/detr-resnet-50-panoptic, a sibling of the detector that also outputs a mask for every object it finds. Panoptic means “everything visible”: every pixel gets assigned to some object or background region. For the truffle cake photo, the “cake” mask was a black-and-white image exactly the size of the photo. Using it as the transparency layer kept the cream rosettes (they are part of the cake shape) and dropped the granite (it is not). About three seconds per photo.

Opening the hood: what the model actually learned

Why does a vision model succeed where the colour rule failed? During training, a model like ViT or DETR saw hundreds of thousands or millions of labelled images. It adjusted its parameters so that patches with certain textures, edges and arrangements push the scores toward certain labels. Early layers end up responding to simple patterns: edges, corners, colour blobs. Later layers combine them into parts (a ring shape, a glossy glaze) and then whole objects (“donut”). Nobody wrote a rule for “doughnut”. The model learned one from examples. You will build these layers yourself in Module 28.

That is also why the model’s knowledge has edges. It recognises what it saw in training, in roughly the conditions it saw it. Photos taken through a glass case with shop lights reflecting off it are not quite the same as the clean photos in COCO.

The three tasks differ mainly in their output head (the last layer that shapes the answer, as in Module 11): one score per label for the whole image, a label and four box coordinates per slot, or a label and a full mask per object. The body underneath can be very similar.

The real code, one task at a time

Install the pieces once:

uv add transformers torch pillow timm

timm is a library of image models; DETR uses one of its networks as a backbone, the part that first extracts visual features from the pixels.

Classification.

from PIL import Image
from transformers import pipeline

image = Image.open("display_case.jpg")
classify = pipeline("image-classification", model="google/vit-base-patch16-224")
for guess in classify(image, top_k=3):
    print(f'{guess["label"]:35} {guess["score"]:.2f}')

Pipelines also accept a file path or an image URL, so classify("display_case.jpg") works too. top_k=3 asks for the three best labels.

Detection, with a count and a picture to check it.

from PIL import ImageDraw

detect = pipeline("object-detection", model="facebook/detr-resnet-50")
found = detect(image, threshold=0.7)   # ignore guesses below 70% confidence

draw = ImageDraw.Draw(image)
for obj in found:
    b = obj["box"]   # {'xmin': ..., 'ymin': ..., 'xmax': ..., 'ymax': ...} in pixels
    draw.rectangle([b["xmin"], b["ymin"], b["xmax"], b["ymax"]], outline="red", width=6)
    draw.text((b["xmin"], b["ymin"] - 14), f'{obj["label"]} {obj["score"]:.2f}', fill="red")
image.save("display_case_boxes.jpg")

doughnuts = sum(1 for obj in found if obj["label"] == "donut")
print("Doughnuts on the shelf:", doughnuts)

Each result is a dictionary with a label, a score and a box. Always save the boxed image while you are building. Looking at the boxes is the fastest way to understand a wrong count. Note the label is spelled "donut", the way COCO spells it; a check for "doughnut" would always return zero.

Segmentation, for white-background product photos.

segment = pipeline("image-segmentation", model="facebook/detr-resnet-50-panoptic")

photo = Image.open("truffle_cake.jpg")
regions = segment(photo)   # one dict per region: {'label', 'score', 'mask'}
for r in regions:
    print(r["label"], r["mask"].size)

cake_mask = next(r["mask"] for r in regions if r["label"] == "cake")   # black-and-white image

cutout = photo.convert("RGBA")
cutout.putalpha(cake_mask)                     # white in the mask = visible, black = transparent
cutout.save("truffle_cake_cutout.png")         # PNG keeps transparency; JPEG cannot

white = Image.new("RGB", photo.size, "white")
white.paste(cutout, mask=cake_mask)            # paste only the cake pixels onto white
white.save("truffle_cake_white.jpg")

The alpha channel is a fourth number per pixel saying how see-through it is. putalpha uses the mask as that channel: cake pixels opaque, everything else invisible.

General models like this one give good but slightly rough edges. For catalogue-quality cut-outs (fine chocolate shavings, sugar-dusted edges), the Hub has models trained only for background removal, such as BRIA’s RMBG family and BiRefNet. Many of them ship their own Python code, which you load with trust_remote_code=True.

Be careful with remote code. trust_remote_code=True downloads and runs Python code from the model’s repo on your machine, with your permissions. Only use it for publishers you trust, read the code first, and pin a revision for production. Check the licence too: several popular background-removal models allow only non-commercial use unless you make an agreement with the publisher.

Library APIs change quickly. If a name has moved, check the docs linked below.

Where it breaks: six doughnuts on the wall

Arjun set the counter to check the camera every five minutes and send a WhatsApp alert when doughnuts dropped below three. For two days it worked beautifully. Then, one quiet Tuesday afternoon, with the case completely empty, the counter reported “Doughnuts on the shelf: 6”.

Priya opened the boxed image. Six neat red boxes, all on the back wall, around a large poster advertising the new glazed doughnut range.

“The model is right,” she said. “Those are doughnuts. It just doesn’t know which ones are for sale.”

The fix was to look only where it matters. They measured the display case’s position in the camera frame once and cropped to it before detecting:

CASE_AREA = (300, 900, 3600, 2700)   # left, top, right, bottom, measured once from a camera frame
shelf = Image.open("display_case.jpg").crop(CASE_AREA)
found = detect(shelf, threshold=0.7)

That pattern, a region of interest, fixes a surprising number of real vision bugs. Here is what else experienced engineers check for:

• Labels the model never learned. An ImageNet classifier cannot say “doughnut”; a COCO detector cannot say “kaju katli”. Print model.config.id2label before you plan anything.
• Conditions unlike the training photos. Glare on glass, low evening light and steep camera angles all lower scores. Test on frames from your camera, at different times of day.
• The threshold. Too high and half-hidden doughnuts at the edge are missed; too low and reflections become doughnuts. The trade-off has names: recall (finding everything that is there) versus precision (being right about what you found).
• Overlapping objects. Doughnuts stacked in a pile are often counted as one. Ask staff to display them in a single layer, or count from above.
• Rough masks. Fine edges such as piped cream or shavings come out jagged with general models. Check every cut-out before it goes on the website.
• Licences and remote code. Vision models vary widely in what they allow. Read the card.

Show the answer

First detection, to find each doughnut and crop it out. Then classification of each crop into “fresh” or “stale”. No off-the-shelf model knows those labels, so you would need a few hundred labelled photos of Crumb’s own doughnuts to fine-tune a classifier, or you could try a zero-shot model like CLIP (next lesson but one) with descriptive labels and measure how well it does. Combining tasks like this, detect then classify the crops, is extremely common.

Try it yourself

Take a photo of a table with several objects on it: cups, fruit, a phone, a book. Then:

1. Run the image classifier and note its top label.
2. Run object detection with threshold=0.7, save the boxed image and count the objects.
3. Run it again with threshold=0.3 and compare.
4. Cut out one object with the panoptic segmentation model and save it as a PNG.

Show the answer

The classifier gives one label for the whole scene, usually the biggest or most distinctive thing (“dining table”, “desk” or one of the items). Detection at 0.7 finds the obvious objects with tight boxes. At 0.3 you get more boxes: some real objects it was unsure about, plus mistakes such as a shadow labelled “cell phone” or one cup boxed twice. The cut-out should show the object on a transparent background (most image viewers draw transparency as a checkerboard). If the mask includes a bit of table, that is the rough-edges limitation in action.

Common mistakes

• Using classification to count. Classification gives one answer per photo. Counting needs detection.
• Using colour rules for shapes. Pixels do not know what object they belong to. Use segmentation.
• Checking the wrong label string. COCO says donut. Print the labels you actually get.
• Saving cut-outs as JPEG. JPEG has no transparency, so the background comes back. Save PNG.
• Testing only on nice photos. Your camera’s glare, angle and lighting are what matter.

Recap

• An image is a grid of pixels; an image processor resizes and normalises it, as a tokenizer does for text, and many models then split it into patches that act like tokens.
• Classification answers “what is this a photo of”, detection answers “what and where” with bounding boxes, segmentation answers “which pixels” with masks.
• pipeline("image-classification"), pipeline("object-detection") and pipeline("image-segmentation") cover all three.
• Models only know their training labels and conditions: ImageNet has no doughnut; COCO has “donut”; neither has kaju katli.
• Crop to a region of interest, tune the threshold on your own camera’s photos, and always save boxes or masks so you can see what the model saw.

That evening Arjun showed Meera the new product photos, every one of them on clean white, and the WhatsApp alert that had just fired: “Viman Nagar: 2 doughnuts left in case.”

“The models don’t look at colours,” he explained. “They learned what things look like from millions of photos. One kind says what a photo is of. Another kind finds each object and draws a box, so we can count them. A third kind colours in exactly which pixels belong to the cake, so we can cut it out. The poster on the wall fooled it once, so now it only looks inside the case.”

Meera was delighted for about ten seconds. Then her phone buzzed: a two-minute WhatsApp voice note from the Baner store manager, half in Marathi, about the oven. Then another. She had twenty-three unplayed.

“Can it hear too?”

Go deeper

• Hugging Face documentation (Transformers tasks, image pipelines): https://huggingface.co/docs
• PyTorch tutorials (images as tensors): https://pytorch.org/tutorials/