Chained voice agents: STT to LLM to TTS

Meera listened to the speech-to-speech demo twice and loved it. Then she wrote her real requirements on the whiteboard, one line at a time:

1. Use the same brain as the WhatsApp agent. Same Claude model, same refund rules, same tools.
2. Keep a written transcript of every call. For disputes, and so she can read a day’s calls over chai.
3. Choose our own voice. Warm, Indian English, eventually Marathi.
4. Feel quick. “Nobody should say ‘hello?’ to my phone line.”

Priya underlined the first three. “Every one of those is easier if the call goes through text in the middle. Speech to text, then Claude, then text to speech. A chained agent.”

“That’s what I built first,” Arjun said. “It took nine seconds.”

“You built a relay. Today we build an assembly line.”

Faster runners, same relay

Arjun’s first instinct was to swap every slow part for a fast hosted one. He replaced laptop Whisper with a hosted transcription API, kept Claude but switched it to low effort, and replaced the open TTS model with a hosted voice. Then he timed the same test question again: “Do you have eggless chocolate cake for tomorrow?”

Stage	Weekend build	Faster parts
Wait for 1.5 s of silence	1.5 s	1.5 s
Transcribe the whole recording	2.4 s	0.6 s
Claude writes the whole reply	3.1 s	1.6 s
Synthesize the whole reply	1.9 s	0.9 s
Silence before the first word	8.9 s	4.6 s

Half the time, and still twice too slow. Worse, the reply was four sentences long, so the caller waited 4.6 seconds and then got a speech.

Show the answer

Stop waiting for each stage to finish before the next one starts. In a relay, the total is the sum of every stage’s full time. If each stage passes along small pieces as soon as it has them, the stages overlap, and what the caller waits for is only the time until the first piece makes it all the way through: the first words transcribed, the first sentence written, the first syllable spoken. The rest happens while the caller is already listening.

The Diwali hamper line

Priya used Crumb’s own Diwali production line to explain.

“Two hundred corporate hampers. Do we bake all two hundred boxes of kaju katli, then start packing hampers, then start labelling?”

“No,” Arjun said. “The first tray comes out of the kitchen, packing starts on it. The first packed hamper goes straight to labelling. All three stations work at once, on different hampers.”

“And how soon does the first finished hamper leave the building?”

“After one tray’s worth of time at each station. Not after two hundred.”

“That’s the whole lesson. In a voice agent, each station is a model, and what moves along the line is not hampers but small pieces: slices of audio, tokens of text, sentences, chunks of sound. Engineers call this streaming, sending output piece by piece as it is produced, which you met in Module 19, and pipelining, letting every stage work at the same time on different pieces.”

“So what’s the smallest piece each station can work on?”

That question turned out to be the whole design.

• Speech-to-text (STT) can work on slices of audio as small as 20 to 100 milliseconds. A streaming STT service sends back partial transcripts while the caller is still speaking (“do you have”, “do you have eggless”, “do you have eggless chocolate cake”) and a final transcript a moment after they stop.
• The language model produces tokens (small chunks of text) one at a time, and with streaming you receive each one as soon as it exists.
• Text-to-speech (TTS) needs enough text to say something with natural rhythm. One word at a time sounds robotic and choppy. A sentence, or at least a clause, is the practical smallest piece.

So the trick in the middle of the line is a small piece of your own code: a sentence splitter that collects the language model’s tokens and hands each complete sentence to TTS the moment its full stop arrives.

Following one turn down the line

Priya traced the same question through the streaming version, measuring from the moment the caller stops speaking.

flowchart LR
  A["Caller stops speaking"] --> B["Turn detection: 300 to 500 ms"]
  B --> C["STT final transcript: 150 to 300 ms"]
  C --> D["LLM first token: 300 to 600 ms"]
  D --> E["First full sentence: 100 to 200 ms"]
  E --> F["TTS first audio: 100 to 250 ms"]
  F --> G["Network and playback: 50 to 150 ms"]
  G --> H["Caller hears reply: about 1 to 2 s total"]

0 ms. The caller finishes: “Do you have eggless chocolate cake for tomorrow?” While they were speaking, the streaming STT service was already sending partial transcripts, so the words are almost all known.

500 ms. The turn detector (a voice activity detector waiting for 500 ms of silence, as in the last lesson) declares the turn over.

700 ms. The STT service sends the final transcript, about 200 ms after the endpoint. Your code appends it to the conversation history and calls Claude with streaming switched on.

1,100 ms. Claude’s first token arrives, about 400 ms later. This delay is called time to first token (TTFT): how long a language model takes to start answering. It grows with the length of the prompt and with how much thinking the model does first.

1,250 ms. The first sentence is complete: “Yes, we do!” That is only a handful of tokens. The sentence splitter sees the “!” and sends it to TTS straight away, while Claude keeps writing.

1,400 ms. The first chunk of audio comes back from a streaming TTS service.

About 1,500 ms. After the network and a small playback buffer, the caller hears “Yes, we do!”. Meanwhile Claude has finished its second sentence (“Our eggless chocolate truffle needs 24 hours’ notice, so tomorrow evening works.”), and it is already being synthesized, ready to play the moment the first sentence ends.

Voice-to-voice latency: about 1.5 seconds, down from 4.6, with the same components. Then Arjun made three more changes:

• He switched to semantic turn detection (from the last lesson), which ends a clearly finished question after about 300 ms instead of 500.
• He shortened the system prompt and switched on prompt caching (Module 21), so Claude reuses its work on the unchanging part of the prompt; together with low effort, TTFT fell to about 300 ms.
• He moved his server to a cloud region close to Pune and to the providers’ servers.

The typical turn dropped to about 1.0 to 1.1 seconds. Not quite speech-to-speech territory, but well inside “feels natural” for a phone order line.

Show the answer

Because the caller waits only for the first sentence to be generated and spoken; everything after it streams while they listen. A first sentence of 3 words reaches TTS after a few tokens; a first sentence of 30 words takes ten times longer to complete. You ask for it in the system prompt: “Start every reply with a short sentence of five words or fewer that directly answers or acknowledges the caller.” People do this naturally (“Sure!”, “Let me check.”), which is part of why it sounds right.

What does a call cost? Arjun estimated a typical three-minute order call with about ten turns. The figures move often, so treat them as an example of how to estimate, and check current price pages:

Piece	How it is billed	Rough cost for one 3-minute call
Hosted STT	Per minute of audio	About 1 US cent or less
Claude (10 turns)	Per token; history resent each turn, much of it cached	Several US cents
Hosted TTS	Per character or per minute of speech	Around 1 to 2 US cents
Total		Roughly 5 to 10 US cents, about ₹4 to ₹9

Against the value of an order that would otherwise ring out, that is easy for Meera to approve. It is still worth logging, per call, so you notice if a change doubles it.

Opening the hood: what changes when the output is spoken

A chained pipeline has a hidden benefit and a hidden trap. The benefit: text sits in the middle, so you can log it, search it, evaluate it, and swap any stage. The trap: everything the language model writes will be spoken, and most models have learned to write for screens.

Write for the ear. A reply full of bullet points, bold text, emoji and “see the table below” turns into nonsense when spoken. A voice system prompt needs different rules:

• Plain sentences only: no lists, no markdown, no emoji.
• One idea per sentence, and short replies: two or three sentences, then let the caller speak.
• Ask one question at a time.
• Confirm important details by reading them back.

Normalize text before speaking. TTS models read what they see. “₹1,450” may come out as “rupee symbol one comma four five zero”. “1 kg” may become “one kay-gee”. “6pm” may become “six pee-em” or “sixty-pm”. Either instruct the model to write numbers the way they should be spoken, or run a small text normalization step before TTS that rewrites prices, sizes and times into words.

Memory is a transcript. The conversation history is just text: what the caller said (from STT) and what the agent said (from the LLM). That makes barge-in in a chained agent a little more work than in speech-to-speech. When the caller interrupts, you must stop playback, cancel the TTS requests still in flight, stop the LLM stream, and record in the history only the sentences that were actually played. Tracking which sentences reached the speaker makes that easy.

Tools take time. If Claude needs to call check_stock("eggless chocolate truffle", "2026-11-05") and it takes 800 ms, the caller hears silence unless something fills it. Two common patterns: let the model say a short sentence before its tool call (“Let me check that for you”), which streams to TTS like any other sentence, or have your code play a short pre-recorded acknowledgement whenever a tool call starts.

The real code: from relay to assembly line

uv add anthropic openai sounddevice soundfile numpy python-dotenv

sounddevice records from your microphone and plays audio through your speakers.

Step 1: one turn, as a relay, with a stopwatch. Start simple, push-to-talk style, and time every stage. You cannot improve latency you have not measured.

import io, os, time
import numpy as np
import sounddevice as sd
import soundfile as sf
import anthropic
from openai import OpenAI
from dotenv import load_dotenv

load_dotenv()
claude = anthropic.Anthropic()
oa = OpenAI()
# Model names change; the current list is on each provider's models page
STT_MODEL = os.getenv("OPENAI_STT_MODEL", "gpt-4o-mini-transcribe")
TTS_MODEL = os.getenv("OPENAI_TTS_MODEL", "gpt-4o-mini-tts")
RATE = 16000

VOICE_PROMPT = """You answer the phone for Crumb & Co., a bakery in Pune.
Speak in plain sentences: no lists, no markdown, no emoji.
Start every reply with a short sentence of five words or fewer.
Keep replies to two or three sentences, and ask one question at a time.
Write prices and sizes as they are spoken, like "six hundred and fifty rupees" and "one kilo"."""

def record(seconds: float = 5.0) -> np.ndarray:
    audio = sd.rec(int(seconds * RATE), samplerate=RATE, channels=1, dtype="float32")
    sd.wait()                                           # block until recording finishes
    return audio[:, 0]

def transcribe(audio: np.ndarray) -> str:
    buf = io.BytesIO()
    sf.write(buf, audio, RATE, format="WAV")
    buf.name = "turn.wav"                               # the API uses the name to detect the format
    buf.seek(0)
    result = oa.audio.transcriptions.create(model=STT_MODEL, file=buf,
                                            prompt="Crumb, eggless, kaju katli, plum cake")  # hint tricky words
    return result.text

def synthesize(text: str) -> tuple[np.ndarray, int]:
    speech = oa.audio.speech.create(model=TTS_MODEL, voice="alloy", input=text, response_format="wav")
    audio, rate = sf.read(io.BytesIO(speech.content))
    return audio, rate

history = []
t0 = time.perf_counter()
heard = transcribe(record())
t1 = time.perf_counter()
history.append({"role": "user", "content": heard})
reply = claude.messages.create(model="claude-opus-5-5", max_tokens=300, system=VOICE_PROMPT,
                               output_config={"effort": "low"}, messages=history)
text = next(b.text for b in reply.content if b.type == "text")
t2 = time.perf_counter()
audio, rate = synthesize(text)
t3 = time.perf_counter()
sd.play(audio, rate); sd.wait()
history.append({"role": "assistant", "content": text})
print(f"heard: {heard!r}\nSTT {t1-t0:.2f}s (includes recording) | LLM {t2-t1:.2f}s | TTS {t3-t2:.2f}s")

A few new pieces. The transcription prompt gives the STT model a hint about unusual words it should expect, which helps with “eggless” and “kaju katli”. output_config={"effort": "low"} asks Claude to think less before answering, which lowers time to first token; for short phone replies, that trade is usually right. The printout is your first latency budget.

Step 2: stream Claude and speak sentence by sentence. Now the assembly line. A background thread speaks sentences from a queue while the main thread is still receiving tokens:

import queue, re, threading

SENTENCE_END = re.compile(r"(?<=[.!?])\s+")              # split after . ! or ? followed by a space
ABBREVIATIONS = ("Rs.", "Mr.", "Mrs.", "Dr.", "approx.")

def speaker(q: "queue.Queue[str | None]", spoken: list[str]):
    while (sentence := q.get()) is not None:              # None means "reply finished"
        audio, rate = synthesize(sentence)
        sd.play(audio, rate); sd.wait()
        spoken.append(sentence)                           # remember what the caller actually heard

def reply_streaming(history: list[dict]) -> str:
    q, spoken = queue.Queue(), []
    worker = threading.Thread(target=speaker, args=(q, spoken))
    worker.start()
    buffer, first, start = "", True, time.perf_counter()
    with claude.messages.stream(model="claude-opus-5-5", max_tokens=300, system=VOICE_PROMPT,
                                output_config={"effort": "low"}, messages=history) as stream:
        for chunk in stream.text_stream:
            buffer += chunk
            *complete, buffer = SENTENCE_END.split(buffer)  # keep the unfinished tail in the buffer
            pending = ""
            for piece in complete:
                piece = pending + piece
                if piece.endswith(ABBREVIATIONS):           # "Rs." is not the end of a sentence
                    pending = piece + " "                    # glue it onto the next piece
                    continue
                pending = ""
                if first:
                    print(f"first sentence ready after {time.perf_counter() - start:.2f}s")
                    first = False
                q.put(piece)
            buffer = pending + buffer
    if buffer.strip():
        q.put(buffer.strip())
    q.put(None)
    worker.join()
    return " ".join(spoken)

Line by line, the new parts: SENTENCE_END splits text after sentence-ending punctuation followed by a space. *complete, buffer = ... keeps everything except the last, unfinished fragment, which waits in buffer for more tokens. The pending logic glues a piece ending in an abbreviation onto the next piece, so “Rs. 650” is never split after “Rs.”, and the order of sentences is preserved. The speaker thread synthesizes and plays each sentence in order, so while it is speaking sentence one, Claude is already writing sentence two. spoken records exactly what reached the speaker, which is what belongs in the history if the caller interrupts.

Arjun’s run of the eggless question printed:

first sentence ready after 0.46s

and he heard “Yes, we do!” about a quarter of a second later, while the rest of the reply was still being written. Swap the hosted STT for a streaming one, add a VAD from the last lesson in place of push-to-talk, and you have the full assembly line, which is exactly what you will build in this module’s project.

To run fully locally instead, swap transcribe for the Whisper pipeline and synthesize for an open TTS model from Module 12. The structure does not change, which is the point of a chained design.

Library APIs change quickly. If a name has moved, check the docs linked below.

Where it breaks: “egg list”

In the first week of testing with staff, one transcript read: “Do you have an egg list chocolate cake?” Claude, trying to be helpful, replied: “Sure! Here’s a list of our egg-based chocolate cakes…” The caller had asked for the opposite of what she got.

That is the core weakness of a chain: errors compound. A small mistake in the first stage becomes a confident wrong answer in the second and a fluent, believable sentence in the third. Speech-to-speech models can be fooled too, but a chain throws away the audio and passes on only the words, so the language model has no way to notice that “egg list” sounded odd. Arjun attacked it from three sides:

• Better hearing. The transcription prompt listing Crumb’s vocabulary (“eggless”, “kaju katli”, “plum cake”) cut these errors sharply.
• Smarter reading. The system prompt now says: “The caller’s words come from speech recognition and may contain mistakes. If something sounds odd for a bakery, ask.”
• Confirmation. Every order ends with a read-back: “So that’s one eggless chocolate truffle, one kilo, for pickup tomorrow at six. Is that right?”

Other things that go wrong in chained voice agents:

• Echo. On speakerphone, the agent’s own voice reaches the microphone, the STT transcribes it, and the agent replies to itself. Use echo cancellation (WebRTC and phone systems provide it) or a headset.
• Barge-in across three stages. Stopping the speaker is not enough; you must also cancel the LLM stream and any TTS requests in flight, then store only the spoken sentences.
• Splitting in the wrong place. Abbreviations, decimal prices (“₹6.50”) and URLs confuse simple sentence splitters. Test the splitter on real replies.
• Markdown in speech. One stray asterisk is read out as “asterisk”. Keep the voice prompt strict, and strip formatting in code before TTS.
• Silent tool calls. Fill any wait over about half a second.
• Costs that creep. The history grows every turn, and Claude reads it all every time. Cache the stable prefix and keep calls short.

Show the answer

The sentence splitter. Without the abbreviation check, SENTENCE_END sees “Rs.” followed by a space as the end of a sentence, sends “Rs.” to TTS on its own, and the rest becomes a second, separately synthesized sentence. The lesson: voice systems fail in ways that are invisible in text logs. Always listen to a sample of real replies, not just read the transcripts.

Try it yourself

1. Run Step 1 and write down the three timings for five different questions. Which stage is biggest?
2. Run Step 2 with the same five questions and note the “first sentence ready” time for each.
3. Change the system prompt to remove “Start every reply with a short sentence” and run again. What happens to the first-sentence time?

Show the answer

In step 1, the LLM and TTS stages usually dominate, because each waits for a complete reply; recording time is fixed by your five seconds. In step 2, the first sentence is typically ready in under a second, often around half a second, and you hear speech while Claude is still writing. In step 3, the first sentence usually gets longer (“Yes, we do have eggless chocolate cakes available, and we can certainly prepare one for you tomorrow.”), so its time roughly doubles or triples. Prompt wording is a latency tool.

Common mistakes

• Optimizing stages instead of overlapping them. Streaming between stages beats making any one stage faster.
• Measuring nothing. Log a timestamp at every stage boundary for every turn.
• Screen-style prompts. Markdown, lists and long paragraphs are wrong for speech.
• Speaking raw numbers and symbols. Normalize prices, sizes and times into spoken words.
• Trusting the transcript. Hint the STT with your vocabulary, tell the LLM the transcript may be wrong, and read back important details.
• Forgetting what was actually heard. On interruption, the history must hold only the spoken sentences.

Recap

• A chained voice agent is STT → LLM → TTS, with text in the middle that you can log, test and swap: any LLM, any voice, full transcripts.
• Run as a relay, the stages add up to many seconds; streamed and pipelined, the caller waits only for the first sentence to get through every stage.
• A typical streamed turn: turn detection 300 to 500 ms, final transcript 150 to 300 ms, first token 300 to 600 ms, first sentence 100 to 200 ms, first audio 100 to 250 ms, network 50 to 150 ms: about 1 to 1.5 seconds.
• Prompt for the ear: short first sentence, plain words, numbers written as spoken, one question at a time.
• Errors compound down the chain: hint the STT, warn the LLM, confirm by reading back, and handle barge-in across all three stages.

Arjun explained the new design to Meera with the Diwali production line still drawn on the whiteboard. “It’s like our hamper line. The first tray of sweets goes to packing before the rest are made. Here, the moment the caller stops talking, the transcript goes to Claude, and the moment Claude finishes its first short sentence, the voice starts saying it, while Claude writes the next one. About a second, our own Claude logic, our own voice, and a transcript of every call.”

“About eight rupees a call?” Meera asked.

“Give or take.”

“Then build it properly. The real phone line, with the order tools.” She turned to leave, then stopped. “And Arjun, I saw you talking to your laptop yesterday, asking it to run the tests. What was that?”

Arjun went slightly red. “A side project. A voice assistant for coding.”

Priya grinned. “Build both. Same engine, two toolkits. That’s your project.”

Go deeper

• Anthropic documentation (streaming, tool use, prompt caching): https://docs.anthropic.com
• OpenAI documentation (speech to text, text to speech): https://platform.openai.com/docs
• Hugging Face documentation (open STT and TTS models): https://huggingface.co/docs
• Building effective agents: https://www.anthropic.com/research/building-effective-agents