Module 17
Multi-Modal Agents and Voice Agents
Agents become far more useful when they can look at photos and hold a spoken conversation. In this module you build an agent that reasons over customer and delivery photos with tools, learn why voice agents live or die by latency, turn-taking and interruptions, compare speech-to-speech models with chained speech-to-text, LLM and text-to-speech pipelines, and ship a streaming voice agent that takes bakery orders and doubles as a hands-free coding assistant.
By the end of this module you can
- send images to Claude, OpenAI and Gemini, estimate their token cost, and build agents whose tools return images
- defend a multimodal agent against prompt injection hidden in images and against reused or misleading photos
- explain a voice latency budget, tune turn detection, and implement barge-in that keeps the model's memory honest
- choose between speech-to-speech and chained voice architectures for a real product, and justify the trade-offs
- build a streaming STT to LLM to TTS agent with tools, read-back confirmation and per-stage latency logging
Lessons
4 in this module- 01Multi-modal agents and image reasoningYou can build an agent that looks at customer photos, fetches more images and records through tools, applies a policy within limits enforced in code, and resists prompt injection hidden inside images.
- 02Speech-to-speech voice agentsYou can explain how speech-to-speech models and realtime APIs keep a phone conversation under about a second per turn, tune turn detection with a VAD, and handle barge-in so the agent stops and remembers only what the caller heard.
- 03Chained voice agents: STT to LLM to TTSYou can build a chained voice agent that streams between speech-to-text, Claude and text-to-speech, speaks the first sentence while the rest is still being written, writes for the ear, and measures latency at every stage.
- 04Project: Voice-based AI agentYou ship a streaming chained voice agent that holds natural, interruptible conversations in about a second per turn, takes bakery orders through tools with read-back confirmation, and switches toolkits to work as a hands-free voice coding assistant.