Latency & Feeling Human — Building Voicebots — Telephonic & Browser, the Real Engineering

Two things make a voicebot feel like talking to a person rather than a machine: it responds quickly, and you can interrupt it. Both are about the same underlying thing — respecting the rhythm of human conversation — and both are where voicebots most often fail. This lesson makes latency concrete with a worked budget (so you know where to spend effort), and covers barge-in, the interruption handling that a natural conversation demands.

The Latency Budget of One Turn

“Make it fast” is vague. Let’s make it specific: here’s where the time goes in one turn, from the user finishing their sentence to the bot starting to speak. (These numbers are illustrative — the point is the shape, and you should measure your own stack.)

The stages and their illustrative costs:

Endpointing / VAD (~250 ms) — the time to be sure the user has stopped (Lesson 4). Often the single biggest chunk, because you must wait out a pause to confirm they’re done.
Final STT transcript (~150 ms) — settling the final text after the endpoint.
Network to LLM (~60 ms) — sending the transcript, queueing.
LLM time-to-first-token (~350 ms) — how long until the model starts producing its reply. The other big chunk.
TTS time-to-first-byte (~150 ms) — until the first audio comes out (Lesson 5).
Network / playout buffer (~90 ms) — getting that audio to the user’s ear.

Add them up naively and you get ~1050 ms — right at the “this feels laggy” threshold (Lesson 1). So how do real voicebots feel snappy? Two moves.

Beating the Naive Sum: Stream and Overlap

The naive sum assumes the stages run one after another, which they don’t have to. The core trick (Lesson 1) is to stream and overlap:

Feed partial STT to the LLM so it can start as soon as the final arrives (and sometimes act on partials for simple cases).
Feed partial LLM output to the TTS sentence-by-sentence, so TTS starts speaking while the LLM is still generating the rest of the reply (Lesson 5).

With overlap, the TTS’s time-to-first-byte starts during the LLM’s generation, not after it — so the stages’ times don’t fully add; they telescope. That’s how a ~1-second naive pipeline becomes a sub-second perceived response. This is why streaming STT (Lesson 4) and streaming TTS (Lesson 5) aren’t optional niceties — they’re what make the budget work.

Then attack the two big bars directly:

Tune endpointing (the ~250 ms). A smarter endpointer — using the content of the transcript to detect completeness, not just raw silence — can shave this meaningfully without cutting people off. This is high-value tuning (Lesson 4).
Pick a fast LLM for time-to-first-token (the ~350 ms). A smaller, faster model often beats a bigger one for the snappy feel voice needs — and it’s cheaper too (Lessons 6, 11). Reserve big models for genuinely hard turns.

And hide what’s left: for a slow operation (a tool call that takes a moment, Lesson 6), play a tiny natural filler — “let me check that for you…” or a brief thinking sound — so the user hears something immediately instead of dead air. Humans do this constantly (“um, let me see”); a well-placed filler buys you time without feeling laggy.

Finally, the rule that governs all of it: measure the real end-to-end number on your stack. Provider spec sheets lie by omission (they quote their stage, not your whole loop). Instrument every stage (Lesson 12) and optimise the one that’s actually slow — usually endpointing, the LLM, or a slow tool call.

Barge-in: Let People Interrupt

Speed is half of feeling human; the other half is barge-in — letting the user interrupt the bot, exactly as they would a person. A bot that ploughs on while you’re trying to correct it feels broken and infuriating.

When the user starts talking while the bot is speaking, four things must happen fast:

VAD detects the user talking over the bot. You’re listening even while speaking.
Immediately stop and flush the TTS — stop the speaker, discard queued audio. Don’t talk over the user; yield the floor instantly (this is why TTS must be interruptible, Lesson 5).
Discard the half-spoken reply and start listening to / transcribing the new input.
The LLM takes the interruption as the next turn — often it needs the earlier context too (what it was saying, what the user is now correcting).

Why this is genuinely hard: on a phone call the bot hears its own voice (echo), so without echo cancellation it would detect “speech” that’s actually itself and “interrupt itself” constantly. And the timing is tight and real-time. This is a major reason to use a framework (Lesson 7): Pipecat handles VAD + flush + restart (and relies on the transport’s echo cancellation, Lesson 3) so you don’t hand-build this delicate machinery. Get barge-in right and the bot feels like a patient, attentive person; get it wrong and it feels like a hold-music menu.

Pitfalls to Avoid

Running stages sequentially. The naive sum lands at the laggy threshold. Stream and overlap so TTS starts before the LLM finishes (Lessons 4–5).
Not measuring end-to-end. Spec sheets quote one stage; optimise your whole loop. Instrument every stage (Lesson 12) and fix the real bottleneck.
Ignoring endpointing and LLM TTFT. They’re the two biggest bars. Tune endpointing and pick a fast model — don’t optimise the small stages first.
Dead air during slow steps. A tool call with silence feels broken. Play a tiny filler (“let me check…”) to cover it.
No barge-in. A bot you can’t interrupt feels like a machine. Detect the user over the bot, stop/flush TTS instantly, take the interruption as the next turn.
Forgetting echo cancellation. Without it the bot interrupts itself on a phone call. Rely on WebRTC/the provider/your pipeline (Lessons 3, 7).

Quick Recap

The illustrative turn budget (~1050 ms naive) is dominated by endpointing (~250 ms) and LLM time-to-first-token (~350 ms); the naive sum sits at the laggy threshold.
Beat it by streaming and overlapping (TTS speaks while the LLM still generates), tuning endpointing, picking a fast LLM, and hiding remaining delay with a filler. Then measure your real stack.
Barge-in: when the user talks over the bot, VAD detects it → stop/flush TTS → discard the reply → take the interruption as the next turn; echo cancellation stops the bot interrupting itself.
A framework (Pipecat) handles barge-in and overlap for you (Lesson 7). All numbers illustrative — measure yours.

Next lesson: giving the bot knowledge the right way — when you need RAG, when a plain prompt or a structured lookup is better and cheaper, and how to store your information.