Project: Voice-based AI agent

Diwali was four weeks away. Last year, Crumb’s order line rang about 600 times a day and staff answered six calls in ten. Meera had approved the plan: a voice agent that answers the phone, talks naturally, checks the menu and stock, takes the order, reads it back, and hands over to a person whenever it is unsure.

Arjun had a second, smaller motive. Every evening he worked on Crumb’s code with his hands full: a cup of chai in one, the other scrolling a failing test. He kept wishing he could just say “run the cart tests” and “why is the discount test failing?” and hear the answer, the way people talk to Cursor or Claude Code but out loud.

“Same engine, two toolkits,” Priya said. “The ears, the voice, the turn-taking, the streaming: all shared. Only the tools and the system prompt change. Build the engine once, properly, and give it two jobs.”

The brief

You are building a voice agent that runs on your laptop, using your microphone and speakers. (Connecting it to a real phone number is a stretch goal.) It must:

• Listen hands-free and decide when the caller has finished speaking.
• Reply in roughly one second on typical turns, starting with a short first sentence.
• Stop talking within a few hundred milliseconds when interrupted, and remember only what the caller actually heard.
• Use tools: look up the menu, check availability, draft an order, and confirm it only after the caller says yes.
• Keep a written transcript and a latency log of every turn.
• Switch, with one setting, to a coding mode that can search code, read files, run tests and propose edits in a repository, speaking short summaries instead of reading code aloud.

Who it is for: Crumb’s customers (bakery mode) and Arjun himself (coding mode). Use the same engine for both.

What you will build

Must have

• Push-to-talk first, then hands-free turn detection with a voice activity detector (VAD).
• Speech-to-text, Claude with streaming and tools, text-to-speech, sentence-by-sentence.
• A voice-specific system prompt: plain sentences, short first sentence, one question at a time, numbers written as spoken.
• Bakery tools: get_menu, check_availability, prepare_order, confirm_order, transfer_to_human.
• Barge-in: stop playback, cancel generation, store only spoken sentences.
• A JSONL log: one line per turn with the transcript, the reply, tool calls and per-stage timings.
• Coding mode with tools: search_code, read_file, run_tests, propose_edit, apply_edit, all sandboxed to one repository folder.

Nice to have

• Semantic turn detection or an adaptive silence threshold (longer while the caller reads out a phone number).
• A filler phrase played when a tool takes longer than 700 ms.
• Text normalization for prices, weights and times.
• A local mode using Whisper and an open TTS model from Module 12.
• A real phone number through a telephony provider.

Architecture

flowchart LR
  MIC["Microphone"] --> VAD["VAD turn detection: 300 to 500 ms"]
  VAD --> STT["Speech-to-text: 150 to 300 ms"]
  STT --> AG["Agent core: Claude, first token 300 to 600 ms"]
  AG <--> TOOLS["Toolkit: bakery or coding"]
  AG --> SPL["Sentence splitter: 100 to 200 ms"]
  SPL --> TTS["Text-to-speech: 100 to 250 ms"]
  TTS --> SPK["Speaker"]
  VAD -->|"barge-in: stop"| SPK
  AG --> LOG["Transcript and latency log"]

Microphone and VAD. A background audio stream delivers 32 ms slices of 16 kHz audio. Silero VAD (from the speech-to-speech lesson) labels each slice as speech or silence and emits “speech started” and “speech ended” events. “Ended” closes the caller’s turn; “started” while the agent is talking means barge-in.

Speech-to-text. The caller’s turn audio goes to a transcription model, with a vocabulary hint (“eggless”, “kaju katli”, or your repo’s module names in coding mode). The transcript is appended to the conversation history.

Agent core. Claude, called with streaming and the active toolkit. Text tokens flow out to the sentence splitter as they arrive. When Claude asks for a tool, the core runs it, appends the result, and continues. Any short sentence Claude says before a tool call (“Let me check that”) is spoken while the tool runs.

Toolkit. A list of tool definitions plus a function that runs them. Bakery mode and coding mode each provide one. The core does not care which.

Sentence splitter. Collects tokens and releases each complete sentence, handling abbreviations like “Rs.”, exactly as in the chained voice agents lesson.

Text-to-speech and speaker. A speaker thread synthesizes sentences in order and plays them, recording which sentences were actually heard.

Log. One JSON line per turn, so you can measure latency, read transcripts, and build an evaluation set later (Module 23).

Tech stack

Piece	Tool	Why
Audio in and out	sounddevice, soundfile, numpy	Simple, cross-platform microphone and speaker access from Python
Turn detection	silero-vad	Small, fast, free VAD with a streaming iterator
Speech-to-text	OpenAI transcription API (or Whisper via transformers)	Fast hosted option with vocabulary hints; Whisper for fully local
Language model	Claude claude-opus-5-5 via the anthropic SDK	Strong tool use; streaming; effort: "low" for quick replies
Text-to-speech	OpenAI speech API (or an open TTS model)	Natural voices, WAV output; swap freely in a chained design
Concurrency	threading, queue	Lets listening, thinking and speaking overlap without async complexity
Logs	JSON Lines files	Easy to grep, load into pandas, or turn into evals
Config	python-dotenv	Keeps API keys out of code

uv init voice-agent && cd voice-agent
uv add anthropic openai sounddevice soundfile numpy silero-vad torch python-dotenv

Put ANTHROPIC_API_KEY and OPENAI_API_KEY in .env. Wear headphones while developing: without echo cancellation, your agent will hear itself through the speakers.

Build it, milestone by milestone

1. Record a turn and transcribe it

Why. Everything downstream depends on hearing correctly, and you need to see the failure modes (wrong words, numbers, silence) before you build on top of them.

Start with push-to-talk: record for a fixed time, then transcribe.

# voice/io.py
import io, os
import numpy as np, sounddevice as sd, soundfile as sf
from openai import OpenAI
from dotenv import load_dotenv

load_dotenv()
oa = OpenAI()
RATE = 16000
STT_MODEL = os.getenv("OPENAI_STT_MODEL", "gpt-4o-mini-transcribe")   # model names change; check the docs

def record(seconds: float = 5.0) -> np.ndarray:
    audio = sd.rec(int(seconds * RATE), samplerate=RATE, channels=1, dtype="float32")
    sd.wait()
    return audio[:, 0]

def transcribe(audio: np.ndarray, hint: str = "") -> str:
    buf = io.BytesIO(); sf.write(buf, audio, RATE, format="WAV"); buf.name = "turn.wav"; buf.seek(0)
    return oa.audio.transcriptions.create(model=STT_MODEL, file=buf, prompt=hint).text

if __name__ == "__main__":
    print(transcribe(record(), hint="Crumb, eggless, kaju katli, plum cake, Baner, Koregaon Park"))

Working looks like: you say “Do you have eggless chocolate cake for tomorrow?” and see that sentence printed within about a second of recording ending. Try ten phrases, including an order number, a phone number and a Hindi or Marathi word, and write down what goes wrong.

Most likely bug: you hear nothing, or the transcript is empty. Check sd.query_devices() for the right input device, and print np.abs(audio).max(): a value near 0 means the microphone is muted or not permitted (on macOS, grant your terminal microphone access).

Stuck? Hint

If the transcript is nonsense rather than empty, check the sampling rate: you recorded at 16,000 Hz and wrote the WAV at 16,000 Hz, so they must match. If you prefer to run locally, replace transcribe with the Whisper pipeline from Module 12 and pass {"raw": audio, "sampling_rate": 16000}.

2. Reply with Claude and speak it back

Why. This closes the loop for the first time: you speak, it speaks. It is slow on purpose, so you can measure the relay before you replace it.

# voice/turn.py
import io, os, time
import anthropic, sounddevice as sd, soundfile as sf
from voice.io import record, transcribe, oa

claude = anthropic.Anthropic()
TTS_MODEL = os.getenv("OPENAI_TTS_MODEL", "gpt-4o-mini-tts")

def synthesize(text: str):
    speech = oa.audio.speech.create(model=TTS_MODEL, voice="alloy", input=text, response_format="wav")
    return sf.read(io.BytesIO(speech.content))           # returns (audio, rate)

t0 = time.perf_counter(); heard = transcribe(record()); t1 = time.perf_counter()
reply = claude.messages.create(model="claude-opus-5-5", max_tokens=300, output_config={"effort": "low"},
                               system="You answer the phone for Crumb & Co., a bakery in Pune. Two sentences, plain speech.",
                               messages=[{"role": "user", "content": heard}])
text = next(b.text for b in reply.content if b.type == "text"); t2 = time.perf_counter()
audio, rate = synthesize(text); t3 = time.perf_counter()
sd.play(audio, rate); sd.wait()
print(f"heard={heard!r}\nsaid={text!r}\nSTT+rec {t1-t0:.2f}s  LLM {t2-t1:.2f}s  TTS {t3-t2:.2f}s")

Working looks like: a spoken two-sentence answer, and a timing line. Expect the LLM and TTS stages together to take a couple of seconds or more. Write the numbers down: this is your “before” picture.

Most likely bug: the reply plays too fast, too slow, or as static. You played audio at a different rate than the one returned. Always use the rate that sf.read gives you.

Stuck? Hint

If speech.content fails, print type(speech). Older and newer SDK versions expose binary responses slightly differently; the speech docs show the current way to get the bytes or stream them to a file.

3. Hold a conversation with a voice-friendly prompt

Why. A single turn is a demo; a conversation needs memory and a prompt written for the ear. This is where the agent starts to feel like a person on the phone rather than a chatbot reading aloud.

# voice/prompts.py
BAKERY_PROMPT = """You answer the phone for Crumb & Co., a family bakery with six stores in Pune.
You are talking, not writing: use plain sentences, never lists, markdown or emoji.
Start every reply with a short sentence of five words or fewer.
Keep replies to two or three sentences and ask one question at a time.
Write prices, sizes and times as spoken: "six hundred and fifty rupees", "one kilo", "six in the evening".
The caller's words come from speech recognition and may contain mistakes.
If something sounds odd for a bakery, ask them to repeat it.
Tell callers at the start that you are Crumb's AI assistant and that calls are recorded."""

Keep a history list across turns: append {"role": "user", "content": heard} after each transcription and {"role": "assistant", "content": text} after each reply. Loop: record, transcribe, reply, speak, repeat until the caller says goodbye.

Working looks like: a five-turn conversation where the agent remembers what you said two turns ago (“You mentioned eggless; this one is eggless too”), never reads out an asterisk, and asks one question at a time.

Most likely bug: the agent reads out formatting (“asterisk asterisk”) or long lists. Strengthen the prompt, and add a tiny cleaner before TTS: text.replace("*", "").replace("#", "").

Stuck? Hint

If replies are still long, give one example exchange in the prompt showing the length you want. Models copy the shape of examples more reliably than they follow length instructions.

4. Add order tools with read-back confirmation

Why. Without tools, the agent invents menu items and prices (Module 11 showed how small models fill gaps; large ones do it too, more convincingly). Tools ground it in Crumb’s real data, and a two-step order design stops it placing orders the caller never agreed to.

Define the toolkit as data plus a runner:

# voice/bakery_tools.py
import json, uuid

MENU = {
    "plum cake": {"sizes": {"500 g": 650, "1 kg": 1200}, "eggless": False},
    "chocolate truffle": {"sizes": {"500 g": 700, "1 kg": 1350}, "eggless": True, "notice_hours": 24},
    "kaju katli box": {"sizes": {"250 g": 450, "500 g": 880}, "eggless": True},
}
DRAFTS, ORDERS = {}, []

TOOLS = [
    {"name": "get_menu", "description": "List items, sizes, prices in rupees, eggless flag and notice needed.",
     "input_schema": {"type": "object", "properties": {}}},
    {"name": "check_availability", "description": "Check whether an item and size can be ready at a store at a time.",
     "input_schema": {"type": "object", "properties": {
         "item": {"type": "string"}, "size": {"type": "string"},
         "store": {"type": "string"}, "pickup_time": {"type": "string"}},
         "required": ["item", "size", "store", "pickup_time"]}},
    {"name": "prepare_order", "description": "Draft an order and return a summary to READ BACK to the caller. Does not place it.",
     "input_schema": {"type": "object", "properties": {
         "items": {"type": "array", "items": {"type": "object", "properties": {
             "item": {"type": "string"}, "size": {"type": "string"}, "qty": {"type": "integer"}},
             "required": ["item", "size", "qty"]}},
         "name": {"type": "string"}, "store": {"type": "string"}, "pickup_time": {"type": "string"}},
         "required": ["items", "name", "store", "pickup_time"]}},
    {"name": "confirm_order", "description": "Place a drafted order. Only call after the caller clearly said yes to the read-back.",
     "input_schema": {"type": "object", "properties": {"draft_id": {"type": "string"}}, "required": ["draft_id"]}},
    {"name": "transfer_to_human", "description": "Hand the call to a staff member, with a one-line summary.",
     "input_schema": {"type": "object", "properties": {"summary": {"type": "string"}}, "required": ["summary"]}},
]

def run_tool(name: str, args: dict) -> str:
    if name == "get_menu":
        return json.dumps(MENU)
    if name == "check_availability":
        item = MENU.get(args["item"].lower())
        if not item or args["size"] not in item["sizes"]:
            return "Not on the menu in that size."
        return "Available." + (f" Needs {item['notice_hours']} hours' notice." if "notice_hours" in item else "")
    if name == "prepare_order":
        total = sum(MENU[i["item"].lower()]["sizes"][i["size"]] * i["qty"] for i in args["items"])
        draft_id = uuid.uuid4().hex[:6]
        DRAFTS[draft_id] = args | {"total_inr": total}
        return json.dumps({"draft_id": draft_id, "total_inr": total, "read_back": args})
    if name == "confirm_order":
        draft = DRAFTS.pop(args["draft_id"], None)        # code enforces: no draft, no order
        if not draft:
            return "No such draft. Prepare and read back the order first."
        ORDERS.append(draft)
        return f"Order placed. Reference {args['draft_id']}."
    if name == "transfer_to_human":
        return "Transferring now."
    return f"Unknown tool {name}"

Then the agent core: a streaming tool loop. It speaks any text Claude produces (including “Let me check that” before a tool call) and keeps looping while Claude asks for tools:

# voice/core.py
def agent_turn(history: list, system: str, tools: list, run_tool, say) -> None:
    """Run one caller turn: stream Claude, speak text via say(), run tools until Claude is done."""
    while True:
        with claude.messages.stream(model="claude-opus-5-5", max_tokens=600, system=system, tools=tools,
                                    output_config={"effort": "low"}, messages=history) as stream:
            for chunk in stream.text_stream:
                say(chunk)                                  # goes to the sentence splitter
            final = stream.get_final_message()
        history.append({"role": "assistant", "content": final.content})
        if final.stop_reason != "tool_use":
            return
        results = [{"type": "tool_result", "tool_use_id": b.id, "content": run_tool(b.name, b.input)}
                   for b in final.content if b.type == "tool_use"]
        history.append({"role": "user", "content": results})   # all results in one message

stream.text_stream yields text as it arrives; get_final_message() returns the complete message, including any tool calls, once the stream ends. This is the manual tool loop from Module 06 with streaming added.

Working looks like: you say “I’d like a one kilo plum cake for pickup at Baner tomorrow at six, my name is Ritu.” The agent checks availability, drafts the order, reads it back (“So that’s one plum cake, one kilo, twelve hundred rupees, for Ritu at Baner tomorrow at six. Shall I place it?”), and only after you say yes calls confirm_order. ORDERS contains exactly one order.

Most likely bug: the agent calls confirm_order without a read-back, or names an item that does not exist. The draft design catches the first in code; for the second, make sure get_menu is available and the prompt says “Only offer items from get_menu.”

Stuck? Hint

If prepare_order crashes with a KeyError, the model passed a size like “1kg” instead of “1 kg”. Either normalize sizes in run_tool (strip spaces, lower-case) or list the exact size strings in the tool description. Return errors as text instead of raising, so the model can correct itself.

5. Stream sentences to TTS and log latency per stage

Why. This is the step from relay to assembly line, from several seconds to about one. And from now on, every change you make must be measured, or you are guessing.

Connect say() to the sentence splitter and speaker thread from the chained voice agents lesson. Then add a timing log:

# voice/log.py
import json, time

class TurnLog:
    def __init__(self, path="turns.jsonl"):
        self.path, self.t = path, {}
    def mark(self, stage: str):
        self.t[stage] = time.perf_counter()
    def write(self, **fields):
        base = self.t.get("speech_end")
        ms = {k: round((v - base) * 1000) for k, v in self.t.items() if base}
        with open(self.path, "a") as f:
            f.write(json.dumps({"ms_after_speech_end": ms, **fields}, ensure_ascii=False) + "\n")
        self.t = {}

Call log.mark("speech_end") when the turn ends, log.mark("transcript") when STT returns, log.mark("first_token") on the first chunk from Claude, log.mark("first_sentence") when the splitter releases one, and log.mark("first_audio") just before the speaker starts. At the end of the turn, log.write(heard=..., spoken=..., tools=[...]).

Working looks like: a line per turn like {"ms_after_speech_end": {"speech_end": 0, "transcript": 640, "first_token": 1010, "first_sentence": 1150, "first_audio": 1390}, ...}. With push-to-talk, the first audio should now start around 1 to 1.5 seconds after you stop, for replies that need no tools.

Most likely bug: sentences play out of order or overlap. Only one thread may call sd.play, and it must wait (sd.wait()) before the next sentence. The speaker thread owns playback; nothing else touches it.

Stuck? Hint

If first_token is slow (over a second), shorten the system prompt, keep effort low, and check you are not resending huge tool results every turn. If first_audio lags far behind first_sentence, your first sentence is too long; enforce the five-word opener in the prompt.

6. Go hands-free with VAD turn detection and barge-in

Why. Push-to-talk is not a phone call. Real callers just talk, pause, and interrupt. This milestone is where most of the “feels human” quality comes from, and where most of the bugs live.

Stream microphone audio in 32 ms slices (512 samples at 16 kHz, the size Silero expects) into Silero’s VADIterator:

# voice/listen.py
import queue, threading
import numpy as np, sounddevice as sd, torch
from silero_vad import load_silero_vad, VADIterator

vad = VADIterator(load_silero_vad(), sampling_rate=16000, min_silence_duration_ms=500)
frames: "queue.Queue[np.ndarray]" = queue.Queue()
speaking = threading.Event()        # set while the agent's voice is playing
cancel = threading.Event()          # set on barge-in

def on_audio(indata, n_frames, time_info, status):
    frames.put(indata[:, 0].copy())

def listen_for_turn() -> np.ndarray:
    """Block until the caller finishes a turn; return its audio. Fires barge-in if they talk over us."""
    turn, in_speech = [], False
    while True:
        chunk = frames.get()
        event = vad(torch.from_numpy(chunk))
        if event and "start" in event:
            in_speech = True
            if speaking.is_set():
                cancel.set(); sd.stop()          # barge-in: silence the agent immediately
        if in_speech:
            turn.append(chunk)
        if event and "end" in event and in_speech:
            return np.concatenate(turn)

stream = sd.InputStream(samplerate=16000, channels=1, dtype="float32", blocksize=512, callback=on_audio)
stream.start()

Then make the rest of the engine respect cancel: the LLM loop in agent_turn breaks out of text_stream when cancel.is_set() (leaving the with block closes the stream), the splitter stops releasing sentences, the speaker thread clears its queue, and the history gets only the sentences in spoken, not everything Claude generated. Set speaking while playing and clear it after. Clear cancel at the start of each new turn.

Working looks like: you talk without pressing anything; the agent answers about a second after you stop. Halfway through a long answer, you say “Actually, make it two”, and the agent stops within a few hundred milliseconds and responds to the correction. The logged history shows the interrupted reply cut off where it was cut off in your ears.

Most likely bug: the agent interrupts itself. Its own voice from the speakers triggers the VAD. Use headphones, or ignore VAD “start” events during the first 300 ms of playback and require at least 200 ms of continuous speech before treating it as barge-in.

Stuck? Hint

If turns end too early when you pause mid-sentence, raise min_silence_duration_ms to 700 or 800 and measure the latency cost in your log. A neat trick: when the agent has just asked for a phone number or address, temporarily use a longer threshold, because people pause between digit groups.

7. Swap in the coding toolset for a voice coding assistant

Why. This proves the architecture: same ears, voice, turn-taking and streaming, completely different job. It is also genuinely useful, and the safety rules you add here (a sandbox, no arbitrary shell, confirm before writing) are the same ones production coding agents use.

# voice/code_tools.py
import subprocess
from pathlib import Path

REPO = Path("~/code/crumb-shop").expanduser().resolve()
PENDING = {}

def safe(path: str) -> Path:
    p = (REPO / path).resolve()
    if not p.is_relative_to(REPO):
        raise ValueError("Path outside the repository.")
    return p

def run_code_tool(name: str, args: dict) -> str:
    try:
        if name == "search_code":
            r = subprocess.run(["git", "grep", "-n", args["pattern"]], cwd=REPO, capture_output=True, text=True, timeout=10)
            return r.stdout[:3000] or "No matches."
        if name == "read_file":
            return safe(args["path"]).read_text()[:6000]
        if name == "run_tests":
            target = [str(safe(args["path"]))] if args.get("path") else []   # one test file, or all tests
            r = subprocess.run(["uv", "run", "pytest", "-q", *target], cwd=REPO,
                               capture_output=True, text=True, timeout=180)
            return (r.stdout + r.stderr)[-3000:]       # the end of the output holds the summary
        if name == "propose_edit":
            PENDING[args["path"]] = (args["old"], args["new"])
            print(f"\n--- proposed edit to {args['path']} ---\n- {args['old']}\n+ {args['new']}\n")
            return "Edit shown on screen. Ask the user to confirm before applying."
        if name == "apply_edit":
            old, new = PENDING.pop(args["path"])
            f = safe(args["path"]); text = f.read_text()
            if text.count(old) != 1:
                return "The old text must appear exactly once. Propose a more specific edit."
            f.write_text(text.replace(old, new)); return "Applied."
    except Exception as e:                                # errors go back to the model as text
        return f"Error: {e}"
    return f"Unknown tool {name}"

CODING_PROMPT = """You are a hands-free coding assistant for the repository you can search and read.
The user is listening, not reading: never read code aloud. Summarise in one or two spoken sentences,
mention file names and line numbers, and say "I've put the details on screen" when you print them.
Never apply an edit until the user clearly says yes after you propose it."""

Write CODE_TOOLS definitions for these five tools in the same format as the bakery ones, and choose the toolkit and prompt with one setting: MODE=coding uv run python -m voice.main. Give the STT a coding vocabulary hint: your module names, “pytest”, “refactor”.

Working looks like: you say “Run the cart tests.” Within a second you hear “Running them now,” then about the time pytest takes, then “Two passed, one failed: test discount in test cart, line forty-one, expected ninety, got one hundred.” You say “Why?”, and it reads the test and the code, and explains in two sentences. You say “Fix it.” It prints a proposed edit and asks; you say “Yes, apply it”, and it applies the edit and offers to re-run the tests.

Most likely bug: the assistant reads code aloud character by character (“def underscore apply underscore discount open bracket…”). Strengthen the “never read code aloud” rule, and keep printed details on screen.

Stuck? Hint

Path.is_relative_to needs Python 3.9 or newer. If tests take long, the caller hears silence: speak “Running them now” first (the prompt’s short-first-sentence rule usually does this), and consider streaming a “still running” sentence every ten seconds from your tool runner.

Done means done

• Push-to-talk and hands-free modes both work.
• Median time from end of speech to first audio is under 1.5 seconds over 20 logged no-tool turns (aim for about 1 second).
• Interrupting the agent stops its voice within about half a second, and the stored history contains only the spoken part of the interrupted reply.
• The agent never reads out markdown, lists or symbols; prices and sizes are spoken naturally.
• Bakery mode places an order only through prepare_order followed by confirm_order Bakery mode places an order only through `prepare_order` followed by `confirm_order` after a spoken "yes".
• Asking for an item that is not on the menu gets an honest "we don't make that" and an alternative, not an invented product.
• Saying “I want to talk to a person” triggers transfer_to_human Saying "I want to talk to a person" triggers `transfer_to_human` with a summary.
• Coding mode can search, read, run tests, propose and apply an edit, and refuses paths outside the repository.
• Every turn appears in turns.jsonl Every turn appears in `turns.jsonl` with transcript, reply, tool calls and per-stage timings.
• The README explains how to run both modes and shows a latency table from your logs.

Stretch goals

• A real phone number. Connect a telephony provider’s media stream to your server; resample 8 kHz phone audio to 16 kHz for the VAD and STT. Phone lines add echo and noise: test with real phones.
• Semantic turn detection. Add a small check on the partial transcript (“does this sound finished?”) to end complete turns faster and wait patiently on incomplete ones.
• Streaming STT. Use a streaming transcription service so the transcript is ready almost as soon as the turn ends.
• Fully local mode. Whisper (Module 12), a local LLM through Ollama (Module 20) and an open TTS voice. Compare latency, quality and cost in your README.
• Marathi and Hindi. Try multilingual STT and TTS; measure accuracy on ten recorded calls per language.
• An eval set. Record 30 test calls, store their expected outcomes (order placed, transfer, refusal), and re-run them after every prompt change (Module 23).
• Speech-to-speech comparison. Rebuild bakery mode on a realtime speech-to-speech API and compare latency, naturalness and cost per call.

Common pitfalls

• Echo. Without headphones or echo cancellation, the agent hears itself and barges in on itself.
• Wrong sampling rates. Microphone, VAD, STT and TTS may all expect different rates. Log the rate at every boundary.
• Measuring by feel. Use the latency log. Your sense of “fast” changes after a hundred test calls.
• Storing what was generated instead of what was heard. After barge-in, the model’s memory must match the caller’s.
• Letting the model act without confirmation. Orders and file edits go through a propose-then-confirm step that your code enforces.
• Silent tools. Anything over about 700 ms needs a spoken acknowledgement.
• Shell access. Never give a voice coding assistant a general “run any command” tool. Mishearing “run the tests” as “remove the tests” should be impossible, not just unlikely.
• Forgetting the AI disclosure. Tell callers they are talking to an AI and that calls are recorded.

Show it off

README. Open with a one-paragraph description and a 60-second demo video: a bakery call with an interruption and an order read-back, then a coding session where you ask it to run tests and fix a failure by voice. Include the architecture diagram, setup steps for both modes, and a latency table from your own logs (median and 90th percentile for transcript, first token, first sentence and first audio).

Demo video. Record your screen with the terminal visible, so viewers can see the transcript and latency log scroll by while they hear the conversation. Show one thing going wrong and being handled: an interruption, or a request for something not on the menu.

LinkedIn post. Something like: “I built a voice agent that takes bakery orders and doubles as a hands-free coding assistant. Speech-to-text, Claude with tools and text-to-speech, streamed sentence by sentence so replies start in about a second. The hardest parts weren’t the models: they were turn-taking, interruptions, and making sure the agent only remembers what the caller actually heard. Code and latency numbers in the repo.” Then tag the tools you used and link the repo.

Diwali week came. The agent answered 3,100 calls in seven days. It took about half of the orders end to end, handed the rest to staff with a one-line summary, and Meera read the transcripts over chai every night. Arjun, meanwhile, fixed the discount bug by voice with a cup of chai in each hand. Priya made him promise to stop doing that.