AI voice agents for Indian languages: Hindi, Kannada, Tamil, Telugu

Most "multilingual" voice AI demos are English with a Hindi menu. Real Indian calls are different: a caller starts in Kannada, drops an English product name, switches to Hindi for the address and finishes with a Kannada thank-you, on a mobile connection from a market. Agents that handle this are not the result of a better model in the abstract; they are the result of benchmarking speech engines per language on your recordings, designing for code-switching, and giving front-line staff the tools to fix pronunciations. This guide sets out how we do it and what to expect per language.

Why one engine never wins

Speech-to-text quality varies by language, accent, audio quality and even by whether numbers are spoken in English or the local language. In our benchmarks one engine leads on Hindi, another on Kannada and Tamil, a third on Telugu with heavy English mixing, and the ranking shifts with background noise. Text-to-speech is the same story: naturalness and pronunciation of Indian names differ widely. A production agent therefore detects language in the first seconds and routes each call to the best engine for it, rather than committing to a vendor.

Language readiness in 2026: a practical view

Treat this as a starting map; the benchmark on your recordings decides. Indian-focused providers and the large cloud engines both belong in the test; we do not rank vendors publicly because the answer changes per language and per year. Sarvam AI and Deepgram are among the engines commonly benchmarked; the method matters more than the list.

The benchmark method

• Collect a consented sample of real calls per language, including noisy ones and code-switched ones
• Transcribe a reference set by native speakers
• Run each candidate engine; measure word error rate overall and on the words that matter (names, numbers, places, product names)
• Score synthesis with listening tests by front-line staff, not by engineers
• Measure latency per engine, because a slightly more accurate engine that adds 400 milliseconds may lose the conversation
• Choose per language; document the routing rule

Designing for code-switching

Hinglish, Kanglish and Tanglish are not edge cases; they are the norm for urban callers. The agent must accept a sentence that mixes scripts and vocabularies, respond in the caller's dominant language, and keep English terms (product names, medical terms) intact rather than translating them. Prompts and evals are written with mixed-language examples from your own calls, and the dialogue model is chosen for how it handles them, not for a benchmark in a single language.

Pronunciation is a product feature

Doctor names, branch names, localities and product names are where synthesis fails first and where callers notice most. We give the front-line team a settings screen with a pronunciation dictionary: they hear a name spoken wrongly, they fix it, and the next call is right. In the hospital network rollout most week-one flags were exactly this, and the staff fixing them themselves is what turned sceptics into owners.

What to build first

Start with one site, one default language and the three or four intents that dominate volume, appointment booking, order status, reminders. Add languages by benchmark, not by ambition. A pilot that handles Kannada and English well at one hospital is worth more than one that handles six languages badly across the network. Expansion is then a configuration change per site: default language, dictionary, hours.

Cost implications

Speech engines are billed per minute and rates differ by provider; Indian-language routing may use two or three engines across a day's calls. Budget speech at $0.02–0.06 per minute on top of telephony and model costs, and measure per language on the dashboard so the routing rule can be revisited. The full breakdown is in How much does an AI voice agent cost per minute?.

A worked example

A clinic chain in Karnataka wanted booking calls handled in Kannada, Hindi and English. The benchmark on 400 consented calls showed one engine clearly ahead on Kannada and code-switched calls, another on Hindi. Routing by detected language, a dictionary of 180 doctor and branch names maintained by reception staff, and a hand-off phrase list in all three languages went live at one site with a third of calls. Week one produced pronunciation fixes; week three expanded to all routine calls; the remaining sites followed one per week with their own dictionaries. Resolution without a human, per language, is reported weekly.

Team and timeline

Six to ten weeks: two weeks for the language benchmark and integration wrapper, three to four for the agent and dictionary tooling, and a staged pilot. The team is a conversational AI engineer, a real-time media engineer and a delivery lead who sits with reception during the pilot. Your side contributes recordings, native-speaker reviewers for the reference transcripts and staff who will listen daily for the first weeks.

Before you start: a checklist

• Consented call recordings per language, including noisy and mixed-language calls
• Native speakers to produce reference transcripts
• The list of names and places the agent will have to say
• Default language per site and the hand-off phrases in each language
• Agreement on which engines may process voice data and where it is stored
• A pilot site and a starting share of calls

Text-to-speech quality: what to listen for

Naturalness is only part of it. Listen for correct stress on Indian names, for numbers spoken in the form callers expect, for pauses at commas and clause boundaries, and for how the voice handles English words inside a Kannada or Hindi sentence. Have three native-speaking staff score each candidate voice on a short script of real confirmations, blind to the vendor. The winner per language goes into the routing table; the dictionary handles the exceptions.

Glossary

• Code-switching: mixing languages within a sentence, e.g. Hinglish or Kanglish
• Word error rate (WER): the standard recognition accuracy measure; we weight it toward names and numbers
• Language detection: identifying the caller's language from the first seconds, continuously updated
• Engine routing: sending each call to the best speech engine for its language
• Pronunciation dictionary: staff-maintained list of how names and places are spoken
• Reference transcript: a native speaker's transcription used to score engines

Mistakes we see

Common mistakes: benchmarking on clean studio audio instead of real calls; measuring overall WER when the words that matter are names and numbers; choosing one vendor for all languages; ignoring code-switching until the pilot; and asking engineers rather than native-speaking staff to judge synthesis. The fix in every case is the same discipline: your recordings, your reviewers, per language.

Questions clients ask

• Can the agent switch language mid-call? Yes; detection runs continuously and the reply follows the caller.
• What about regional accents within a language? They are part of the benchmark sample; if an accent underperforms, the routing rule or the engine changes for that site.
• Do we need a different agent per language? No. One agent, one set of intents, with speech engines and phrasing per language.
• How do numbers and dates work? Both local and English forms are tested and accepted; the confirmation reads back in the caller's preferred form.
• Can it handle English-only callers? Of course; English is one of the routed languages.

What good looks like after 90 days

At ninety days you should see per-language resolution rates converging, a dictionary that reception updates weekly without tickets to engineering, and a routing table that has been revisited at least once as engines improved. If one language lags, the benchmark is rerun for it alone rather than the whole system.

Related reading

What is an AI voice agent? covers the architecture; Voice AI compliance in India covers consent and recording; the voice agent service and healthcare pages show where it is used, with starting prices on the pricing page.

Multilingual voice is where Indian businesses gain most from AI and where most vendors overpromise. Benchmark, route, let staff own pronunciation, and expand by evidence.