Speech-to-Text (STT)

Everything your voicebot does depends on correctly hearing what the user said. Speech-to-Text (STT) — also called ASR (Automatic Speech Recognition) — is the first real component in the pipeline, and getting it right, for voice specifically and for India specifically, is harder and more important than people expect. This lesson covers the three things that matter: streaming, endpointing, and the Indian-language reality that drives your provider choice.

Streaming, Not Batch

The STT you may have used for transcribing a recording is batch: give it a whole audio file, get back the full transcript. That’s useless for a live voicebot, because you’d have to wait for the user to completely finish and then wait for the whole transcription — seconds of delay (Lesson 8). Voice STT must be streaming.

A streaming STT takes audio chunks as they arrive and continuously emits its best guess so far:

• Partial transcripts update live as the user speaks: “what is the…” → “what is the sch…” → and so on. These are tentative and can change as more audio arrives.
• A final transcript is emitted when the user has stopped (or a phrase is settled): “what is the scheme.” This is the stable text you act on.

Why this matters so much: partial transcripts let your pipeline start working before the user even finishes. You can begin preparing, and the instant the final arrives you’re ready to send it to the LLM — no “wait for the whole utterance, then start transcribing” delay. Streaming STT is a prerequisite for the sub-second turn (Lesson 1, Lesson 8); a batch STT alone makes a snappy voicebot impossible.

Endpointing: Knowing When to Reply

Here’s a problem that sounds trivial and absolutely isn’t: how does the bot know the user has finished talking so it can respond? This is endpointing, and it rests on VAD (Voice Activity Detection) — detecting speech vs silence in the audio.

Get it wrong in either direction and the bot feels broken:

• Too eager (declares “done” after a short pause): the bot cuts people off mid-sentence, especially someone who pauses to think (“I want to ask about… the housing scheme”). Infuriating.
• Too slow (waits a long time after speech stops): the bot feels dead, and people repeat themselves or hang up.

Tuning endpointing — the silence threshold, how it handles filler pauses, whether it uses the content of the partial transcript to guess completeness — is some of the highest-value work in a voicebot. And it has a direct cost in latency: the time you wait to be sure the user stopped is pure delay added to every turn. In the Lesson 8 budget, endpointing is often the single biggest chunk (~250 ms in the illustrative example). So it’s worth real effort: a well-tuned endpointer makes the bot feel both responsive and polite, which is most of “feeling human.”

The Indian Reality: Languages, Accents, Code-Mixing

Generic, English-first STT struggles badly with Indian voice, and this is where provider choice earns its keep. The challenges stack up:

• Many languages. India has 20+ major languages; your users may speak Hindi, Marathi, Tamil, Telugu, Bengali, and more — and you often don’t know which until they start talking (so language detection matters too).
• Strong, varied accents even within a language.
• Code-mixing — the completely normal Indian habit of mixing languages in one sentence: “mujhe scheme ke baare mein batao,” “mera order kahan hai.” An STT that assumes one pure language per utterance mis-hears this constantly.
• Phone audio is narrowband and noisy. On a telephony call (Lesson 2), the audio is lower fidelity and often has background noise, which is harder than a clean browser mic.

The consequence: for an Indian voicebot, pick an STT trained for Indian speech. This is exactly where Sarvam is strong — it’s built for Indian languages and code-mixing — and it’s a big reason it comes up throughout this course. Other good options depending on your needs: Deepgram (fast streaming, strong English), Google/Azure STT (broad language coverage, mature), and self-hosted Whisper (open, cheapest at scale once you have steady volume — Lesson 11). The right choice depends on your languages, latency needs and budget; there’s no universal winner.

Picking a Provider

Judge an STT on more than one axis:

• Accuracy on your languages and audio — measured by Word Error Rate (WER), but test it on your actual audio (your languages, phone-quality, your domain words like scheme names), not a vendor benchmark.
• Latency — streaming responsiveness and how fast finals arrive. For a voicebot this matters as much as raw accuracy.
• Endpointing quality — some providers give you better VAD/endpointing out of the box.
• Languages and code-mixing support — non-negotiable for Indian multilingual use.
• Cost model — per-minute pricing vs self-hosting (Lesson 11).

For most Indian multilingual voicebots, an Indic-first STT like Sarvam usually beats a generic English-first one on the metrics that matter (accuracy on Indian speech, code-mixing). But if you’re English-only and latency-obsessed, Deepgram might win; if you have huge steady volume, self-hosted Whisper may be cheapest. Decide by testing on your real audio.

A final reframe: for a voicebot, WER is not the whole story. A transcriber that’s 2% more accurate but adds 300 ms of latency or has clumsy endpointing can make a worse voicebot than a slightly-less-accurate but faster, better-endpointing one. Optimise for the conversation, not the benchmark.

Pitfalls to Avoid

1. Using batch STT for live voice. It forces a wait-then-transcribe delay that kills your latency budget. Use streaming STT with partials and finals.
2. Neglecting endpointing. Bad VAD either cuts users off or feels dead, and it’s a big latency cost. Tune it carefully — it’s high-value work (Lesson 8).
3. Using English-first STT for Indian users. It mis-hears accents and code-mixing. Pick an Indic-trained STT (Sarvam) for multilingual Indian voice.
4. Testing on clean audio only. Phone audio is narrowband and noisy; benchmark on your real channel and domain words, not a vendor demo.
5. Optimising WER alone. Latency and endpointing matter as much for a voicebot. Optimise the conversation experience, not just accuracy.
6. Ignoring language detection. You often don’t know the caller’s language upfront; detect it (or ask) early so STT and TTS match (Lesson 10).

Quick Recap

• STT must stream: emit partial transcripts as the user talks, then a final — so the text is ready the instant they stop, enabling the sub-second turn.
• Endpointing / VAD (knowing when the user stopped) is critical and often the biggest latency cost; tune it so the bot is responsive and doesn’t cut people off.
• The Indian reality — many languages, accents, code-mixing, noisy phone audio — means you should pick an Indic-trained STT (Sarvam is strong; also Deepgram, Google, self-hosted Whisper).
• Judge STT on accuracy on your real audio, latency, endpointing, language support, and cost — not WER alone, and not a vendor benchmark.

Next lesson: the bot’s voice — text-to-speech — where streaming (time-to-first-byte) is everything, and where your provider choice (Sarvam vs ElevenLabs) quietly sets your biggest cost.