Speech-to-speech voice agents

Last Diwali week, Crumb’s order line rang about 600 times a day. Three people answered it, between serving customers at the counter. Meera’s own estimate was that they picked up six calls in ten. The other 240 went to voicemail, and most of those callers simply ordered somewhere else. At an average phone order of about ₹1,200, that was close to ₹3 lakh of orders a day ringing out.

“I don’t want a robot that says ‘press one for cakes’,” Meera said. “I want something people can just talk to. It asks what they want, checks we have it, takes the order. And if they’re confused or angry, it hands over to a person.”

Arjun had every piece already. Whisper could hear. Claude could think. A text-to-speech model could talk. He glued them together over a weekend.

The nine-second silence

Arjun’s weekend build worked like a relay race with one runner at a time:

1. Record the caller until there has been 1.5 seconds of silence.
2. Send the whole recording to Whisper (Module 12) on the office laptop.
3. Send the transcript to Claude and wait for the complete reply.
4. Send the complete reply to a text-to-speech model, and wait for the whole audio file.
5. Play it.

He called it from his phone, with Priya listening on speaker. He asked: “Hi, do you have the Christmas plum cake in 500 grams?” Then he waited. He timed it later from the recording:

Stage	Time
Waiting for 1.5 s of silence	1.5 s
Whisper transcribing the whole recording on a laptop processor	2.4 s
Claude writing the complete reply	3.1 s
Text-to-speech for the complete reply	1.9 s
Total silence before the first word	8.9 s

At about the five-second mark, Priya leaned towards the phone. “Hello? … Hello? Is anyone there?” Those words were recorded too, so when the bot finally answered the plum cake question, it immediately started processing “Hello? Is anyone there?” as a new question. Then, while it was reading out the plum cake prices, Arjun tried to say “Actually, one kilo.” The bot kept talking right over him, to the end of its paragraph.

“Every piece works,” Arjun said. “So why does it feel so broken?”

A conversation is a rally, not a relay

Priya picked up a tennis ball from her desk.

“In a relay, each runner waits for the baton. In a tennis rally, you’re already moving towards where the ball will land while it’s still on the other side of the net. Conversation is a rally. How long can you stay silent on a phone call before the other person says ‘hello?’”

Arjun thought about his own calls. “A second? Two at most.”

“Researchers have measured this across many languages. The typical gap between one person finishing and the other starting is around 200 milliseconds. That’s faster than anyone could hear a whole sentence, understand it and plan a reply. So how do people do it?”

“They must start planning before the other person has finished.”

“Exactly. And there’s a second rule. In a rally, either player can change the game at any moment. On a call, either person can interrupt, and the other one stops. Your bot breaks both rules. It waits for the whole sentence before doing anything, and it can’t hear while it speaks.”

Three ideas came out of that conversation, and they shape everything in this lesson:

• The latency budget. Latency is the waiting time between a request and its response. For voice, the number that matters is voice-to-voice latency: from the moment the caller stops speaking to the moment they hear the first sound of the reply. Under about one second feels natural. Over two seconds feels broken. That one second is a budget to be shared between every stage.
• Turn-taking. The system must decide when the caller has finished, quickly and correctly. Too eager and it cuts people off mid-thought; too cautious and it feels sluggish.
• Barge-in. When the caller starts speaking over the bot, the bot must stop talking at once, and must remember that the caller only heard part of what it was going to say.

One model that listens and speaks

The most direct way to beat the relay is to remove the relay. A speech-to-speech model takes audio in and produces audio out, in a single model, continuously.

Under the hood, it works much like the vision-language models from Module 12. Incoming speech is cut into short slices and turned into audio tokens, which sit in the model’s context alongside any text such as your instructions. The model generates its reply as audio tokens too, which are turned straight back into sound. There is no separate transcription step and no separate voice step. That has some lovely side effects: the model hears how things are said (hesitation, frustration, a laugh), and it can respond with tone, not just words.

Providers offer these models through realtime APIs: services you connect to once at the start of a call and keep open, streaming audio in both directions. OpenAI’s Realtime API and Google’s Gemini Live API are two examples; more appear every few months.

The connection itself is usually one of:

• A WebSocket: a single connection that stays open so both sides can send messages at any time, instead of the one-question-one-answer pattern of normal web requests. Typical between your server and the provider.
• WebRTC: the technology behind browser video calls. It is built for real-time audio over unreliable networks and includes echo cancellation. Typical between a browser or phone app and the provider.
• Telephony: for real phone numbers, a provider such as Twilio connects the phone network to your server, which then streams the call audio to the model. Phone audio is low quality, usually 8,000 samples per second.

Following one caller turn, millisecond by millisecond

Priya rebuilt Arjun’s test call on the whiteboard as a timeline, this time with a speech-to-speech model over a realtime API. The caller says the same sentence: “Hi, do you have the Christmas plum cake in 500 grams?”

sequenceDiagram
  participant C as Caller
  participant S as Your server
  participant M as Speech-to-speech model
  C->>S: Audio in 20 ms slices, while speaking
  S->>M: Same slices, streamed straight through
  Note over M: 0 to 2.4 s: caller speaking, model already listening
  Note over M: 2.4 to 2.9 s: silence, turn detector waits
  M-->>S: Speech stopped at 2.9 s
  M-->>S: First reply audio at about 3.2 s
  S-->>C: Caller hears the reply at about 3.3 s

0 to 2,400 ms. The caller speaks. Their phone sends audio in tiny slices, 20 milliseconds each, about 50 per second, and your server forwards each slice to the model the moment it arrives. By the time the caller finishes, the model has already heard every word. There is no “upload the recording” step.

2,400 to 2,900 ms. Silence. Is the caller done, or just pausing for breath? A voice activity detector (VAD, a small model that labels each slice of audio as speech or not-speech) waits for a set amount of silence, here 500 ms, before declaring the turn over. At 2,900 ms it fires a “speech stopped” event.

2,900 to about 3,200 ms. The model starts generating its reply. Because it already understood the question while listening, the first chunk of reply audio is ready in roughly 300 ms.

About 3,300 ms. That first chunk crosses the network and starts playing: “Yes! Our Christmas plum cake comes in 500 grams and one kilo…” The rest streams in behind it while it plays.

Voice-to-voice latency: about 900 ms. Down from 8.9 seconds. And look at where those 900 ms went: more than half of it, 500 ms, was spent waiting to be sure the caller had finished.

Show the answer

Because people pause in the middle of sentences all the time: “My order number is… um… one zero four two.” With a 100 ms threshold, the bot would jump in at every hesitation and talk over the caller. Arjun tested this on a recording with a natural pause: with a 300 ms threshold, one spoken sentence was split into three separate turns; with 800 ms it stayed as one. Every voice system tunes this trade-off between snappy and polite, and many now add a smarter check, described below.

Then the caller interrupts. At 4,100 ms, while the bot is mid-sentence, the caller says: “Actually, make it one kilo.” Now the barge-in sequence runs:

1. Detect (about 4,300 ms). The VAD hears speech from the caller while the bot is speaking.
2. Stop the sound. Your code immediately stops playing the bot’s audio and throws away any audio that has arrived but not yet been played.
3. Cancel the response. It tells the model to stop generating the rest of that reply.
4. Fix the memory. The model generated “Yes! Our Christmas plum cake comes in 500 grams and one kilo, and the 500-gram one is ₹650.” But the caller only heard up to “…500 grams and”. Your code tells the API to cut the stored reply at the point playback stopped.
5. Listen. The caller’s new sentence is processed as the next turn.

Show the answer

It generated them, but the caller never heard them. If the conversation history still contains “the 500-gram one is ₹650”, the model will later behave as if the caller knows the price: “As I mentioned, that’s ₹650”, or worse, consider the price agreed. Truncating the stored reply to what was actually played keeps the model’s memory matching the caller’s experience. It is one of the most commonly forgotten steps in voice agents.

Opening the hood: the latency budget and turn-taking

Here is the full budget for one turn, with typical ranges. The exact numbers depend on the provider, the network and the settings, so measure your own.

Stage	Typical time	Notes
Capture and encode a slice	20 to 40 ms	Smaller slices mean lower delay, more overhead
Network, caller to model	30 to 100 ms	Lower if servers are near the caller; Pune to a far-away region adds more
Turn detection	200 to 800 ms	Usually the biggest single item
Model to first audio	200 to 600 ms	Bigger models and long instructions are slower
Network, model to caller	30 to 100 ms
Playback buffer	50 to 150 ms	A small cushion so audio does not stutter
Total	roughly 0.6 to 1.8 s	Aim for under 1 s on typical turns

Turn detection deserves a closer look, since it is usually the biggest item. There are two broad approaches:

• Silence-based. A VAD watches for N milliseconds of silence. Simple, fast, language-independent, and fooled by hesitations.
• Semantic. A small model also looks at what was said. “I’d like to order…” is clearly unfinished; “That’s all, thanks.” is clearly finished. Semantic detection can end a turn quickly after a complete sentence and wait patiently after an incomplete one. Several realtime APIs offer it as an option, and open-source turn-detection models exist too.

Barge-in has its own traps. The bot’s own voice, coming out of a speakerphone, can leak back into the microphone and trigger the VAD: the bot interrupts itself. Echo cancellation (built into WebRTC and phone systems) removes the bot’s own audio from the microphone signal. And not every sound from the caller is an interruption: “mm-hmm” and “okay” while the bot speaks are backchannels, signs of listening. Requiring a few hundred milliseconds of speech, or checking the words, before stopping avoids halting at every “mm”.

Tools work in speech-to-speech models much as in text agents (Module 06). The model emits a tool call, such as check_stock(item="plum cake", size="500g"), your server runs it and sends the result back, and the model continues speaking. A tool that takes two seconds is two seconds of dead air, so good voice agents say something first, “Let me check that for you”, exactly as a person would.

The real code: exploring turn detection

Realtime APIs are young and their event names and settings change often, so this lesson shows one piece you can run today (turn detection) and the shape of a realtime session loop. Next lesson, you will build a complete voice agent you control end to end.

Running a VAD on a recording. Silero VAD is a small, free, widely used voice activity detector. Record yourself saying a sentence with a deliberate pause in the middle, then:

# uv add silero-vad librosa torch
import librosa
import torch
from silero_vad import load_silero_vad, get_speech_timestamps

vad = load_silero_vad()
audio, _ = librosa.load("my_order.wav", sr=16000)     # VAD models expect 16 kHz here
wav = torch.from_numpy(audio)

for silence_ms in [300, 800]:
    turns = get_speech_timestamps(wav, vad, sampling_rate=16000,
                                  min_silence_duration_ms=silence_ms, return_seconds=True)
    print(f"{silence_ms} ms threshold -> {len(turns)} turn(s): {turns}")

Arjun tested it on Module 12’s shop announcement, “Order 1042, two Diwali hampers, is ready for pickup at the counter”, with a 0.6-second pause added in the middle, as a hesitant caller might leave. The output was:

300 ms threshold -> 3 turn(s): [{'start': 0.6, 'end': 1.9}, {'start': 2.4, 'end': 3.0}, {'start': 3.7, 'end': 6.2}]
800 ms threshold -> 1 turn(s): [{'start': 0.6, 'end': 6.2}]

min_silence_duration_ms is the “how long a pause ends a turn” setting. At 300 ms, one sentence was split into three turns: a brief natural pause in the announcer’s voice and the added hesitation were both treated as “finished”. A voice agent would have started answering before hearing the rest of the sentence. That is the trade-off from the pause-and-think box, in real numbers.

The shape of a realtime session. This is pseudocode: plain Python showing the logic, with made-up function names standing in for your provider’s SDK. Event names differ between providers; read their realtime docs for the real ones.

# PSEUDOCODE: the logic of a speech-to-speech session, not a real SDK
session = connect_realtime(model=REALTIME_MODEL, instructions=CRUMB_PHONE_PROMPT,
                           tools=[check_stock, place_order, transfer_to_human],
                           turn_detection={"type": "semantic"})   # or silence-based, with a threshold

played_ms = 0            # how much of the current reply the caller has actually heard

async for event in session.events():                  # one open connection, events flowing both ways
    if event.type == "caller_audio":                  # 20 ms slices from the phone
        session.send_audio(event.audio)
    elif event.type == "reply_audio":                 # slices of the model's spoken reply
        played_ms += speaker.play(event.audio)
    elif event.type == "caller_started_speaking" and speaker.is_playing():
        speaker.stop_and_clear()                      # barge-in: silence the bot at once
        session.cancel_reply()                        # stop generating the rest
        session.truncate_reply(at_ms=played_ms)       # keep only what the caller heard
    elif event.type == "tool_call":
        result = run_tool(event.name, event.arguments)
        session.send_tool_result(event.call_id, result)
    elif event.type == "reply_done":
        played_ms = 0

The loop is short because the hard parts (understanding speech, deciding when the caller has finished, generating audio) happen inside the model. Your code’s job is to move audio quickly, run tools, and handle barge-in correctly.

Library APIs change quickly. If a name has moved, check the docs linked below.

Where it breaks: the Diwali dry run

Before going live, Crumb ran a dry run: staff phoned the agent pretending to be customers, from the shop floor, during a busy Saturday. The speech-to-speech agent felt wonderful. It replied in under a second, laughed politely at jokes, and stopped instantly when interrupted. Then the problems surfaced.

• The shop was loud. A mixer starting up in the background triggered barge-in, so the bot kept stopping mid-sentence. Raising the minimum speech length before an interruption fixed most of it.
• “Ek kilo, na?” Callers switched between English, Hindi and Marathi mid-sentence. The model handled it better than Whisper had, but still misheard sizes and names sometimes. Every order now ends with a read-back: “So that’s one kilo plum cake, for pickup at Baner at six. Is that right?”
• Disputes. A customer later insisted she had ordered two cakes, not one. Meera asked for the transcript. Speech-to-speech models can optionally produce text transcripts alongside audio, but Arjun had not turned that on. Now every call is stored with a transcript, and callers are told at the start that they are speaking with an AI assistant and the call is recorded.
• Swapping the brain. Meera wanted the agent to use the same Claude-based logic and refund rules as the WhatsApp agent from the last lesson. With speech-to-speech, the “brain” is whichever model the realtime provider offers. You cannot drop in a different LLM in the middle.
• Cost per minute. Audio tokens cost more than text tokens. At the time of writing, a minute of speech-to-speech conversation cost in the region of a few US cents to over ten cents, depending on provider and model, and long system prompts are paid for again on every turn. Check the current pricing page before promising Meera a number, then measure a week of real calls.

Here is the honest comparison Priya drew for Meera:

	Speech-to-speech	Chained (STT → LLM → TTS)
Typical voice-to-voice latency	Lowest, often under 1 s	Higher; under about 1.5 s with careful streaming
Hears tone and emotion	Yes	Mostly lost in the transcript
Choice of LLM	The provider’s model	Any model, including Claude
Transcripts and debugging	Optional, extra	Built in: every stage is text you can log
Control over the voice	The provider’s voices	Any TTS voice
Cost	Usually higher per minute	Usually lower, varies with choices

Try it yourself

1. Run the Silero VAD script on three recordings of yourself: one fast sentence, one with an “um” pause, and one where you read out a ten-digit phone number slowly. For each, find the smallest min_silence_duration_ms that keeps it as a single turn.
2. If you have access to a realtime voice playground from any provider, have a 2-minute conversation with it. With a stopwatch app, estimate the voice-to-voice latency on five turns, then interrupt it three times and note how quickly it stops.

Show the answer

Typical results: the fast sentence stays whole even at 300 ms; the “um” pause needs around 600 to 800 ms; the slowly read phone number can need over a second, because people pause between digit groups. That is why many voice agents use semantic turn detection, or temporarily lengthen the threshold when they have just asked for a number. In playgrounds, good speech-to-speech systems usually answer in under a second and stop within a few hundred milliseconds of being interrupted; the stopwatch makes that “feels natural” quality concrete.

Common mistakes

• Waiting for complete recordings. Stream audio from the first millisecond; the model should hear the caller while they speak.
• Ignoring turn detection in the budget. It is often half the latency. Tune it, and consider semantic detection.
• Forgetting to truncate on barge-in. The model will think the caller heard things they never heard.
• No echo cancellation. On speakerphone, the bot interrupts itself.
• Dead air during tools. Say “let me check” before slow tool calls.
• No transcripts. You will need them for disputes, debugging and evaluation.

Recap

• Natural conversation leaves about 200 ms between turns; a voice agent should aim for under about one second of voice-to-voice latency.
• Speech-to-speech models take audio in and give audio out in one model, over a realtime connection (WebSocket, WebRTC or telephony), hearing the caller while they speak.
• Turn detection (silence-based or semantic) decides when the caller has finished and is often the biggest slice of the latency budget.
• Barge-in means stop playback, cancel the reply, and truncate the stored reply to what the caller actually heard.
• Speech-to-speech wins on latency and expressiveness; chained pipelines win on control, choice of LLM and transcripts.

Arjun explained it to Meera with Priya’s tennis ball in his hand. “My first version was a relay: record everything, then transcribe, then think, then speak. Nine seconds of silence. The new kind of model is a tennis player: it listens while you talk, waits about half a second to be sure you’ve finished, and starts answering in under a second. If you interrupt, it stops, and it remembers you only heard half its sentence.”

“So we use that one,” Meera said.

“Maybe,” Priya said. “But you also want our own Claude-based ordering logic, the same refund rules as WhatsApp, any voice we like, and a full transcript of every call. For that, we build the relay again, properly this time, with every runner moving at once.”

Go deeper

• OpenAI documentation (Realtime API and voice agents): https://platform.openai.com/docs
• Gemini API documentation (Live API): https://ai.google.dev/gemini-api/docs
• Anthropic documentation (tool use): https://docs.anthropic.com
• Building effective agents: https://www.anthropic.com/research/building-effective-agents