Visual Q&A, diffusion, inpainting and video

Three requests reached Arjun on the same Monday.

From operations: delivery partners upload a “proof of delivery” photo for every order, about 300 a day. Rohan, the delivery manager, wanted each photo checked. Is the box sealed? Is it at a door? Which order number is on the sticker? Right now, nobody looked at them unless a customer complained.

From accounts: suppliers sent invoices as phone photos on WhatsApp. Someone typed the total, the GST number and the due date into the accounts sheet by hand. Forty invoices a week.

From marketing: Diwali was five weeks away. They needed poster images of gift hampers, and one beautiful product photo had a delivery partner’s hand reaching into the corner. “Can AI just remove the hand? And make us a short video for Instagram?”

Arjun’s multiple-choice exam

Arjun started with the delivery photos, using CLIP from the last lesson. He wrote two captions, “a photo of a sealed cardboard box” and “a photo of an open box”, and it picked the right one most of the time. Promising.

Then he reached the third question: which order number is on the sticker? CLIP can only choose between captions you have already written. He stared at his screen, then typed, half joking:

labels = [f"a box with an order sticker that says {n}" for n in range(1000, 10000)]

Nine thousand captions, for one photo. And it still could not read the GST number off an invoice, or say where the box was.

Priya laughed. “You’ve built a multiple-choice exam with nine thousand options. What you want is a colleague you can just ask. What would a model need, to answer any question about a photo in its own words?”

Arjun thought it through. “It needs to understand the picture, like CLIP’s image half. And it needs to write, like a chat model.”

“So bolt them together.”

Bolting eyes onto a language model

A vision-language model (VLM) is exactly that: an image encoder joined to a language model. Answering free-form questions about images is called visual question answering (VQA).

The design, in three parts:

1. A vision encoder (often a CLIP-like model) turns the image into a grid of feature vectors, one per patch.
2. A small projector (a little neural network) translates those vectors into the language model’s own “token language”. The results are called image tokens.
3. A language model reads the image tokens followed by your question’s text tokens, as if the picture were part of the prompt, and generates the answer one token at a time.

This is also how Claude, GPT and Gemini see images, which you will use in Module 17. On the Hugging Face Hub you can run small open ones yourself, such as SmolVLM, Qwen-VL or the larger Gemma models from Module 11.

Where the analogy of “a colleague who looks at the photo” breaks: the language model part is still a text generator. If the picture does not settle a question, it fills the gap with the most plausible text, not the true text. Hold onto that.

Following one photo through a VLM

Priya’s rule: test on something you can check before anything that matters. Arjun reused the photo from the CLIP lesson, his flatmate’s two cats asleep on a pink blanket, with two TV remotes beside them. He asked a question that CLIP could never answer:

“How many remote controls are in this picture, and what colour is the blanket?”

1. Image in. The pipeline’s processor resized the photo and cut it into patches.

2. Image tokens. The vision encoder and projector turned those patches into image tokens. Small models like SmolVLM compress hard, to tens or a few hundred tokens per image; big hosted models use more, often around a thousand for a typical photo. Those tokens fill up the context window and, on paid APIs, the bill.

3. Prompt assembly. The chat template placed the image tokens, then the question, then the “assistant’s turn” marker, exactly as in Module 11.

4. Generation. The language model wrote, token by token:

There are two remote controls in the image, and the blanket is pink.

Both correct. That was HuggingFaceTB/SmolVLM-256M-Instruct, a model with only 256 million parameters, and it took about 4.6 seconds on Arjun’s laptop.

Then they tried a real delivery photo: a cake box on a doormat, with an order sticker. “Is the box sealed? Is it at a door? What is the order number on the sticker?” The answer came back fluent and confident: “Yes, the box is sealed with tape. It is placed on a doormat in front of a door. The order number is 1012.”

The sticker said 1042.

Show the answer

The image was shrunk before encoding, so a sticker that was readable in the original became a few blurry patches, and “4” and “1” can look alike when blurry. Then the language model did what text generators do: it produced a plausible-looking number. There is no built-in “I’m not sure” for each detail. Never trust a VLM’s reading of IDs, prices or dates on its own: check it against your database (does order 1042 or 1012 belong to this delivery partner today?), or use OCR built for text.

Document VQA: questions about paperwork

Supplier invoices are a special kind of image: mostly text, where position carries meaning. “₹18,450” next to the word “Total” means something different from “₹18,450” in a list of items. Document visual question answering (document VQA) answers questions about images like that.

There are two generations of approach.

OCR plus layout. First, OCR (optical character recognition, Module 11) reads every word and its position on the page. Then a model like LayoutLM, trained on words and their positions, answers by pointing at the right words, just like the extractive reader from the handbook lesson. For one invoice, the trace looks like this: OCR finds 212 words, each with a box; the question “What is the total amount?” goes in with them; the model scores each word as the start or end of the answer and returns the span ₹18,450 with a score of 0.9-something. Because it points at OCR’d words, it cannot invent a number that is not on the page. But if OCR misreads, the answer is wrong.

Vision-language models. Newer VLMs read the document image directly and write the answer, and they handle messy layouts, stamps and handwriting better. They can also invent.

For Crumb’s accounts, Priya picked the VLM for convenience, plus a check: the extracted total must equal the sum of the line items, and the GST number must match the supplier’s record. Anything that fails goes to a person.

The real code: asking questions about images

uv add transformers torch pillow accelerate

Visual question answering with a small VLM. Current versions of transformers use the image-text-to-text task for this. (Older versions also had pipeline("visual-question-answering"), with small models like ViLT; you will see it in older tutorials.)

from transformers import pipeline

vlm = pipeline("image-text-to-text", model="HuggingFaceTB/SmolVLM-256M-Instruct")

messages = [{
    "role": "user",
    "content": [
        {"type": "image", "url": "delivery_1042.jpg"},   # a local path or an https URL
        {"type": "text", "text": "Is the box sealed? Is it in front of a door? Answer yes or no for each."},
    ],
}]
out = vlm(text=messages, max_new_tokens=60)
print(out[0]["generated_text"][-1]["content"])

The message format is the chat format from Module 02, except that content is a list of parts: an image part and a text part. Asking for short, structured answers (“yes or no for each”) makes the output easier to check in code. The order-number question is deliberately left out: that comes from the order database.

Document question answering with OCR and layout.

# Needs the Tesseract OCR program installed (brew install tesseract / apt install tesseract-ocr)
# uv add pytesseract
from transformers import pipeline

doc_qa = pipeline("document-question-answering", model="impira/layoutlm-document-qa")
for q in ["What is the total amount?", "What is the invoice date?", "What is the GST number?"]:
    best = doc_qa(image="invoice_flour_oct.jpg", question=q)[0]
    print(f"{q:30} {best['answer']:20} {best['score']:.2f}")

Each answer comes with a score, like the handbook reader. Low scores go to a person.

Library APIs change quickly. If a name has moved, check the docs linked below.

Turning it around: making images from words

Marketing’s request runs in the opposite direction: words in, picture out. That is the job of diffusion models, the technology behind Stable Diffusion, FLUX and most image generators.

Priya explained it with a photo held behind frosted glass. “Imagine I add a little fog to a photo, then a little more, then more, a thousand times, until it’s pure grey static. Now I train a model on millions of photos to do one small thing: look at a foggy image and guess what fog was added in the last step, so it can be removed. Just one step of de-fogging.”

“And to make a new picture…?”

“Start from pure static. Ask the model to remove one step of fog. Then another. Then another. Each step, the model is also shown your text, so it de-fogs towards ‘a Diwali gift hamper with marigolds’. After enough steps, a sharp image appears that never existed before.”

That is diffusion. Training: add noise to real images, teach a model to predict the noise. Generation: start from noise, remove predicted noise step by step, steered by the text. You will build a tiny one from scratch in Module 31.

Following one prompt through a diffusion model

The prompt: “product photo of a Diwali gift hamper with kaju katli, brass diyas and marigold flowers, soft studio light”.

1. Encode the text. A text encoder (often CLIP’s text half, from the last lesson) turns the prompt into vectors that will steer every step.

2. Start from noise in a small space. Working on full pixels would be slow: a 512 × 512 colour image is 786,432 numbers. So most modern models work in a compressed latent space. A separate network called a VAE squeezes images 8 times smaller in each direction, so the model works on a 64 × 64 × 4 grid: 16,384 numbers, 48 times fewer. Step 2 fills that grid with random noise. A seed number decides exactly which noise, so the same seed and prompt give the same image.

3. Denoise, step by step. Classic models take 25 to 50 steps. “Turbo” and “schnell” variants were trained to need only 1 to 4. Each step is a full pass through a large network, which is why diffusion wants a GPU.

4. Guidance. At each step the model can compare “denoise with the prompt” against “denoise without it” and push harder in the prompt’s direction. The guidance scale sets how hard. Higher follows the prompt more literally, and too high looks over-cooked. (Turbo models are trained to work without it, so they use 0.)

5. Decode. The VAE expands the final 64 × 64 × 4 latent back into a 512 × 512 image.

On a decent GPU, a one-step turbo model produces an image in well under a second. On a laptop processor, the same job can take a minute or more. Through a hosted Inference Provider, it takes a few seconds and costs a small amount per image.

Show the answer

Speed and cost. Fifty steps means fifty passes through a big network for every image. A turbo model trades a little detail and variety for being dozens of times faster, which matters for live previews, for generating 200 poster variations to choose from, and for running on cheaper hardware. It is the same quality-versus-speed trade you saw with Whisper’s sizes and quantized models.

Inpainting: repainting only the hand

Inpainting is diffusion with a stencil. You give an existing image, a mask (white where you want new content, black where everything must stay), and a prompt describing what should fill the white area. Only the masked region is denoised; the rest is kept. For the hamper photo: paint the delivery partner’s hand white in the mask, and prompt “an empty cream-coloured table surface”. The hand disappears; the hamper is untouched.

You already know how to make masks: the segmentation models from the computer vision lesson produce exactly this kind of black-and-white image.

The real code: generating and repainting

uv add diffusers transformers accelerate torch pillow

Text to image, on your own GPU.

import torch
from diffusers import AutoPipelineForText2Image

pipe = AutoPipelineForText2Image.from_pretrained(
    "stabilityai/sdxl-turbo", torch_dtype=torch.float16, variant="fp16"
).to("cuda")   # "mps" on Apple Silicon; a CPU works but is very slow

prompt = "product photo of a Diwali gift hamper with kaju katli, brass diyas and marigold flowers, soft studio light"
image = pipe(
    prompt=prompt,
    num_inference_steps=1,    # turbo models need 1 to 4 steps
    guidance_scale=0.0,       # turbo models are trained without guidance
    generator=torch.Generator("cuda").manual_seed(42),   # same seed + same prompt = same image
).images[0]
image.save("hamper_concept.png")

torch_dtype=torch.float16 loads the weights in 16-bit to halve memory, and variant="fp16" downloads the 16-bit files directly. AutoPipelineForText2Image reads the repo’s configuration and builds the right pipeline, the same “Auto” idea as Module 11.

Text to image, with no GPU. Call a hosted model through an Inference Provider:

from huggingface_hub import InferenceClient

client = InferenceClient()   # reads HF_TOKEN
image = client.text_to_image(prompt, model="black-forest-labs/FLUX.1-schnell")   # returns a PIL image
image.save("hamper_concept_hosted.png")

Inpainting.

from diffusers import AutoPipelineForInpainting
from diffusers.utils import load_image

inpaint = AutoPipelineForInpainting.from_pretrained(
    "diffusers/stable-diffusion-xl-1.0-inpainting-0.1", torch_dtype=torch.float16, variant="fp16"
).to("cuda")

photo = load_image("hamper_with_hand.png").resize((1024, 1024))
mask = load_image("hand_mask.png").resize((1024, 1024))   # white = repaint, black = keep

fixed = inpaint(prompt="an empty cream-coloured table surface, soft studio light",
                image=photo, mask_image=mask, strength=0.99).images[0]
fixed.save("hamper_clean.png")

strength controls how much of the masked area is re-imagined: near 1.0 replaces it completely, lower values keep some of the original.

Library APIs change quickly. If a name has moved, check the docs linked below.

Video: the heaviest modality

Video is many images plus time. Two jobs, very different in cost:

• Understanding video. Classify it (pipeline("video-classification") with models like VideoMAE), sample frames and use CLIP (last lesson), or give sampled frames to a VLM and ask questions. Practical on a laptop for short clips.
• Generating video. Text-to-video diffusion models (available in diffusers and through hosted providers) denoise a whole block of frames together so that motion stays consistent. A few seconds of video means generating dozens of images that agree with each other. Expect to need a large GPU with tens of gigabytes of memory, or a hosted service, and minutes per clip.

For the Instagram reel, Priya was blunt: generate still concepts, pick the best, and let the marketing team animate a real product photo with a hosted tool. Fully generated product video was too slow, too expensive and too unpredictable for a five-week deadline.

Where it breaks

The delivery checker ran for a week in “advise only” mode. It flagged eleven photos as “box open”. Ten really were. The eleventh was a closed box with a white sticker across the seal that looked, at low resolution, like a gap. Rohan called that a good week. Then the order-number reader matched a photo to the wrong customer, and he asked for it to be switched off. Arjun replaced it with a database check: the system knows which orders each partner is carrying, so it only asks the VLM to confirm “Does the sticker show 1042? Answer yes or no”, a far easier question than “read the number”.

Show the answer

It turns open-ended generation into verification. The model no longer has to produce four exact digits from a blurry patch; it only has to judge whether the image is consistent with one candidate the database already trusts. A wrong “yes” is still possible, but the answer can only ever be an order that partner was actually carrying, so a misread can no longer silently attach a photo to a stranger’s order.

On the diffusion side, the first batch of Diwali posters had three kinds of problems: text inside the image came out garbled (“Happy Diwlai”), the “brass diyas” looked like Western church candles, and one hamper contained sweets Crumb does not sell. Meera then asked the most important question of the week: “Can we put a generated hamper on the website if it doesn’t look exactly like what we ship?”

“Not as a product photo,” Priya said. “A customer would be right to complain. Use generated images for mood and concept posters, label them as illustrations, and photograph the real hamper for the shop.”

Everything experienced engineers watch for here:

• VLMs invent details. Especially numbers, small text, counts and anything partly hidden. Verify against your data; ask yes/no confirmation questions.
• Image tokens cost money. On paid APIs, each image is roughly as expensive as a page or more of text. Resize to what the question needs.
• Text in generated images. Most diffusion models still misspell words. Add text afterwards in a design tool.
• Cultural accuracy. Models trained mostly on Western images get Indian festivals, sweets and objects subtly wrong. Review every image with someone who knows.
• Honesty. Generated images must not pretend to be real products. Label illustrations.
• Consent and likeness. Never generate or inpaint images of real, identifiable people without their consent.
• Licences. Image-generation models carry very different licences; some restrict commercial use. Check before marketing uses an image.
• Hardware. Diffusion and especially video need a GPU or a paid hosted service. Budget for it.

Try it yourself

1. Take a photo of your desk. Ask a small VLM three questions: one it should answer easily (“What colour is the mug?”), one about counting (“How many pens are there?”), and one about small text (a label or book title). Note which ones it gets right.
2. Generate an image of a dish from your region with a hosted model through InferenceClient.text_to_image. Then add the words “Happy Birthday” to your prompt and look closely at the lettering.

Show the answer

Typical results: colours and large objects are right; counts are right for two or three items and shaky beyond; small text is often partly wrong but confidently stated. For generation, regional dishes often come out as a generic or Westernised version, and requested text frequently has missing or swapped letters. Both are exactly the failure modes in this lesson. Neither is a reason not to use these models; they are reasons to keep a check or a human in the loop.

Common mistakes

• Using CLIP for open questions. CLIP chooses between captions you wrote; a VLM writes answers.
• Trusting VLM-read numbers. Check them against your own records, or ask yes/no confirmation questions.
• Expecting a laptop to run diffusion quickly. Use a GPU, a turbo model, or a hosted provider.
• Asking diffusion models for text. Add text in a design tool.
• Publishing generated images as product photos. Label them, or photograph the real thing.

Recap

• A vision-language model is an image encoder plus a projector plus a language model; it turns images into tokens and answers questions in words. pipeline("image-text-to-text") runs small ones locally.
• Document VQA answers questions about paperwork, either by pointing at OCR’d words (LayoutLM) or by reading the image directly (VLMs).
• Diffusion models learn to remove noise step by step, steered by text; turbo variants need 1 to 4 steps, classic ones 25 to 50.
• Inpainting repaints only the white area of a mask; video generation is diffusion over many frames and needs serious hardware.
• All of them can invent details: verify numbers, review images, label illustrations and respect licences and consent.

On Friday, Arjun summed it up for Meera at the end of the week’s meeting. “For photos and paperwork, we use a model that has a camera bolted onto a language model, so we can ask it questions. It’s good at ‘is the box sealed’, bad at reading small numbers, so it only confirms what our database already says. For posters, we use a model that sculpts a picture out of static, guided by our words. It’s great for ideas, but it misspells words and gets Diwali slightly wrong, so a person reviews every image and we never pass one off as a real product.”

Meera nodded. “So it can see, read, listen and paint. Here’s what I actually want. A customer sends a photo of a squashed cake on WhatsApp. Can something look at it, check the order, apply our refund rules, and reply, without a person in the middle? And later… could it answer the phone?”

Priya smiled. “That isn’t one model any more. That’s an agent. And agents that see and speak are a module of their own.”

Go deeper

• Hugging Face documentation (Transformers image-text-to-text, document QA; Diffusers): https://huggingface.co/docs
• Hugging Face LLM course: https://huggingface.co/learn/llm-course