Building Voicebots — Telephonic & Browser, the Real Engineering
A practical course on building production voicebots for India: the STT→LLM→TTS loop, phone (PSTN/SIP) vs browser (WebRTC) channels, WebSocket vs WebRTC and when to use each, streaming STT and TTS (Sarvam, ElevenLabs, Deepgram, Whisper), prompting an LLM for voice, orchestrating with Pipecat, latency and barge-in, RAG vs structured data vs plain prompts, real use cases (government schemes helpline, visitor management/gatepass, support, outbound), cost engineering to make it cheap (with worked ₹/minute math), production operations, and privacy/compliance (DPDP, call recordings).
Voicebots have quietly become good enough to be genuinely useful — a citizen can call a number and ask about a government scheme in Marathi, a visitor can get a gatepass by talking to a kiosk, a customer can check an order by voice. But almost everything written about them is either a vendor demo or a toy tutorial that falls apart the moment you need two languages, a phone line, low latency, and a budget. This course is the opposite: the real engineering of building a voicebot you can put in production in India.
We cover both channels you’ll actually need — telephonic (a phone call, which reaches everyone, including rural and feature-phone users) and browser/app (WebRTC, cheaper and higher quality) — and the transport question underneath them (WebSocket vs WebRTC, and when to use each). We go through the pipeline piece by piece — speech-to-text, the LLM brain, text-to-speech — with the Indian reality of 20+ languages and code-mixing front and centre, using tools like Sarvam, ElevenLabs, Deepgram and Whisper. We orchestrate it with Pipecat, make it feel human by attacking latency and barge-in, and give it knowledge the right way — knowing when you need RAG and when a plain prompt or a structured lookup is better and cheaper.
Then the parts people skip: real use cases worked end to end (a government-schemes helpline per state, visitor management/gatepass, support, outbound reminders), cost engineering to make it cheap (with worked ₹-per-minute arithmetic showing a ~6× difference between a naive and an optimised stack), production operations (testing, monitoring, scaling, failure handling), and privacy and compliance (DPDP, consent, PII in call recordings) — because a voicebot is a data-collection system that happens to talk.
It’s written in plain Indian English, grounded in illustrative-but-honest cost and latency numbers (clearly labelled, because provider pricing shifts), and every lesson ends with pitfalls and a quick recap. A note on the numbers: pricing and provider features change fast — treat the figures as a model to compute your own, not gospel.
What This Course Covers
| # | Lesson | What you will learn |
|---|---|---|
| 1 | Anatomy of a Voicebot | The STT→LLM→TTS loop, turn-taking, the latency target |
| 2 | Two Channels: Phone vs Browser | Telephony (PSTN/SIP) vs browser (WebRTC); same brain, swap transport |
| 3 | WebSocket vs WebRTC | Both transports explained, NAT/STUN/TURN, when to use each |
| 4 | Speech-to-Text (STT) | Streaming, VAD/endpointing, Indian languages; Sarvam, Deepgram, Whisper |
| 5 | Text-to-Speech (TTS) | Streaming, time-to-first-byte; Sarvam vs ElevenLabs vs others |
| 6 | The LLM Brain for Voice | Spoken-style prompting, function calling, handling ASR errors |
| 7 | Orchestrating with Pipecat | The frame pipeline, interruptions, swappable transports |
| 8 | Latency & Feeling Human | The ~800 ms budget (worked), streaming, barge-in |
| 9 | Knowledge: RAG vs Without RAG | When to RAG, structured data vs prompt, how to store info |
| 10 | Use Cases: Schemes, Gatepass & More | Govt-schemes deep-dive + a use-case gallery |
| 11 | Cost Engineering: Make It Cheap | The ₹/min stack (worked), where to cut, ~6× savings |
| 12 | Production: Test, Monitor, Scale | Evals, monitoring, failure handling, concurrency |
| 13 | Security, Privacy & Compliance | Consent, DPDP, PII in recordings, abuse, TRAI rules |
Who Is This For?
- Engineers who want to build a real voicebot (phone and/or browser), not just run a demo
- Indian startups and teams building multilingual voice for citizens, customers or employees
- Anyone confused by WebSocket vs WebRTC, Pipecat, STT/TTS choices, or RAG-or-not for voice
- Product folks who need to understand what a voicebot costs and how to make it cheap
How to Use It
Read it in order the first time — it builds from the audio loop up to a cost-optimised production system. Then use it as a reference: when you’re choosing a transport (Lesson 3), picking STT/TTS (Lessons 4–5), designing knowledge (Lesson 9), building a specific use case (Lesson 10), or defending a budget (Lesson 11), come back to that lesson. Keep the pitfalls sections handy — voicebots fail in the unhappy paths (silence, noise, wrong language), and those lists are where the hard-won lessons live.
Let’s start with the loop that every voicebot is: you speak, it listens, it thinks, it speaks back.