Module 12
Multi-Modal AI with Hugging Face
Models can see, hear, speak and paint, not only read. In this module you run open Hugging Face models that count objects, cut products out of photos, transcribe and translate voice notes, tag photos and sounds with labels you write in plain words, answer questions about images and documents, and generate or repair images with diffusion.
By the end of this module you can
- choose between image classification, object detection and segmentation for a real problem, and run each with a pipeline
- transcribe and translate speech with Whisper, generate speech with text-to-speech, and avoid the sampling-rate and silence traps
- tag, search and classify images, sounds and videos zero-shot with CLIP and CLAP, including multi-label tagging with thresholds
- ask questions about photos and documents with a vision-language model, and design checks for the details it invents
- explain how diffusion turns noise into an image, generate and inpaint images, and judge when generated media is honest to use
Lessons
4 in this module- 01Computer vision: classification, detection, background removalYou can tell which vision task a problem needs, run image classification, object detection and segmentation pipelines, count objects in a region of interest, and cut a product out onto a white background.
- 02Speech recognition, denoising and text-to-speechYou can load audio at the right sampling rate, transcribe and translate speech with Whisper, guard against invented transcripts, clean noisy recordings, compare clips with audio embeddings, and generate spoken announcements with text-to-speech.
- 03Zero-shot classification with CLIP and CLAPYou can classify, tag and search images, sounds and videos with labels you write in plain words using CLIP and CLAP, choose thresholds for multi-label tags, and recognise what these models cannot see.
- 04Visual Q&A, diffusion, inpainting and videoYou can ask free-form questions about photos and documents with vision-language models, generate and inpaint images with diffusion, explain why video is the most expensive modality, and design checks for the details these models invent.