Text-to-Speech (TTS)

STT gave your bot ears; TTS (Text-to-Speech) gives it a voice. It turns the LLM’s reply text back into audio the user hears. It’s the stage users judge the bot on most directly — a robotic or mispronouncing voice undermines everything — and, as you’ll see, it’s usually where most of your money goes. Two things dominate this lesson: making TTS fast (which is about streaming and time-to-first-byte), and choosing a provider that balances quality against cost for Indian voices.

Time-to-First-Byte Is Your Latency

The instinct is to think of TTS speed as “how long to generate the audio for this sentence.” For a voicebot, that’s the wrong number. What the user experiences as latency is time-to-first-byte (TTFB) — how long until the first audio starts playing. Once speech is flowing, the user is happy; the total length of the reply doesn’t add to the perceived delay (they’re listening to it).

So the key technique is streaming TTS fed from a streaming LLM: as the LLM produces its reply token by token, you send completed chunks (a sentence, or even a clause) to the TTS immediately, and play the audio as it comes back. The bot starts speaking the first sentence while the LLM is still generating the rest. This is the TTS half of the “stream everything” principle (Lesson 1): combined with streaming STT (Lesson 4) and a fast LLM (Lesson 6), it’s how you get a sub-second turn (Lesson 8). A TTS that only returns audio after the whole reply is written adds its entire generation time to your latency — avoid it, or use it only for short replies.

Quality, Pronunciation, and Controls

Beyond speed, the things that make a voicebot’s voice good:

• Naturalness and prosody. Does it sound like a person — with natural rhythm, intonation, and (where appropriate) warmth — or like a robot reading? This is where premium TTS (ElevenLabs) shines, and where cheap older TTS sounds obviously synthetic.
• Correct pronunciation, especially of Indian names, places, and domain terms (scheme names, towns, people). A voice that mangles “Pradhan Mantri Awas Yojana” or a caller’s town sounds careless and erodes trust. Test your actual vocabulary.
• SSML and controls. SSML (Speech Synthesis Markup Language) lets you control pauses, emphasis, and how things are read. Crucially, it handles numbers, dates and currency correctly: “₹2,000” should be spoken “do hazaar rupaye,” not “rupee two comma zero zero zero.” Getting numbers and dates right is a surprisingly common failure and matters a lot for schemes, amounts and appointments.
• Multilingual voices. For India you need natural voices in the user’s language (Lesson 10) — a Hindi reply should sound like a Hindi speaker, not English TTS reading Hindi words.

Barge-in: You Must Be Able to Stop It

One requirement is easy to forget until it bites: your TTS must be interruptible. When the user barges in (Lesson 8) — starts talking while the bot is speaking — you have to stop the audio instantly and flush whatever’s queued. A TTS integration that can’t be cut off mid-sentence makes barge-in impossible, and a bot you can’t interrupt feels broken. So when you integrate TTS, make sure you control playback at a fine grain (you can stop and discard buffered audio immediately), not just “fire and forget a whole clip.” This is another reason a framework like Pipecat (Lesson 7), which handles flush-on-interrupt for you, is valuable.

The Big One: TTS Is Usually Your Biggest Cost

Here’s the fact that reshapes how you think about TTS: it’s typically the single largest line item in your per-minute cost (Lesson 11), and provider choice swings it enormously.

The trade-off:

• ElevenLabs sets the bar for naturalness and emotion — it sounds genuinely human. It’s also premium-priced; illustratively it can cost ~10×+ per minute of speech versus cheaper options. For a high-touch, low-volume use case where voice quality is the product, it can be worth it.
• Sarvam TTS gives natural Indian-language voices much cheaper — a strong default for Indian multilingual voicebots where you need good Indic voices at scale.
• Google/Azure neural TTS offers many languages at low per-character cost, reliably.
• Open/self-hosted (e.g. Coqui-style models) is cheapest at scale but you operate it.

Because TTS cost scales with how much your bot talks, two levers follow directly (both in Lesson 11): pick a cheaper TTS (Sarvam/Google over ElevenLabs is often the single biggest saving), and cache TTS audio (many replies repeat — “please hold,” common answers — so synthesize once and reuse). The honest guidance: use the voice quality your use case actually needs. A citizen helpline at huge volume should probably use a cost-effective Indic TTS; a premium concierge bot might justify ElevenLabs. Don’t pay for premium voice you don’t need, and don’t ship a robotic voice for something user-facing and important — match it to the job.

Pitfalls to Avoid

1. Optimising total generation time, not TTFB. The user feels time-to-first-byte. Stream TTS from the LLM sentence-by-sentence so speech starts fast.
2. Non-streaming TTS for long replies. Waiting for the whole reply to synthesize adds its full time to latency. Stream, or keep replies short (Lesson 6).
3. Mangling numbers, dates and names. “₹2,000” read as digits, or a mispronounced scheme/town name, sounds careless. Use SSML and test your real vocabulary.
4. TTS you can’t interrupt. Breaks barge-in (Lesson 8). Ensure you can stop and flush audio instantly.
5. Defaulting to the most expensive voice. TTS is usually your biggest cost; ElevenLabs can be ~10×+ Sarvam. Match voice quality to the use case and cache repeated audio (Lesson 11).
6. Using English TTS for Indian-language replies. It sounds wrong. Use natural voices in the user’s actual language.

Quick Recap

• The latency that matters is time-to-first-byte — stream text from the LLM into TTS sentence-by-sentence so the bot speaks while the LLM is still generating.
• Quality means naturalness, correct pronunciation of Indian names/numbers/dates (SSML), and multilingual voices; and TTS must be interruptible for barge-in (Lesson 8).
• TTS is usually your biggest cost: ElevenLabs sounds superb but is premium (illustratively ~10×+ Sarvam/min); Sarvam and others give good Indian voices far cheaper.
• Levers: pick a cost-effective Indic TTS and cache repeated audio (Lesson 11). Match voice quality to the use case.

Next lesson: the brain between the ears and the voice — the LLM — and why prompting it for a spoken conversation is a different craft from prompting it for chat.