Engineering
The AI Voice Agent Technology Stack Explained: Speech, Models, Latency and Telephony
Updated Muhammad Taha8 min read
An AI voice agent is built from five parts working in real time: telephony that carries the call, speech-to-text that turns the caller's voice into words, a language model that decides what to say and do, text-to-speech that speaks the reply, and integrations that act on your systems. The quality of the call depends mostly on how fast and how well these parts work together.
Key takeaways
- The five parts are telephony, speech-to-text, language model, text-to-speech and tools or integrations.
- Latency, the pause between the caller finishing and the agent replying, is the biggest factor in how natural a call feels.
- Handling interruptions, background noise and mixed languages separates good voice agents from poor ones.
- You can use a voice-agent platform or build the pipeline yourself. Each has trade-offs in cost and control.
- Test with real calls on real phone lines, not only with clean recordings.
The pipeline at a glance
| Stage | What it does | What to look for |
|---|---|---|
| Telephony | Connects the phone call (a number, SIP or a cloud telephony provider) to your software | Reliable lines, local number support, call routing and transfer |
| Speech-to-text (STT) | Transcribes the caller's speech into text as they talk | Accuracy on Urdu, English and mixed speech, accents and noise |
| Language model (LLM) | Understands the request, chooses the response and any action | Speed, reasoning quality, tool use and cost |
| Text-to-speech (TTS) | Converts the reply into natural speech | Voice naturalness, language support and speed |
| Tools and integrations | Check calendars, update CRM, look up orders | Reliable APIs, permissions and error handling |
| Logging and analytics | Stores transcripts, summaries and outcomes | Searchable records and privacy controls |
For the business view of voice agents, read AI voice agents for business and AI call agents in Pakistan.
Why latency matters most
In a human conversation a reply arrives within a fraction of a second. If a voice agent takes two or three seconds to respond, callers talk over it, repeat themselves or hang up. Latency is the sum of several steps: detecting that the caller has stopped speaking, transcribing, the model thinking, generating speech and sending audio back. Voice systems reduce it by streaming (each stage starts before the previous finishes), using fast models, keeping prompts short and placing servers close to the caller.
When you evaluate a voice agent, judge it on a real phone call, not a website demo. A fast and slightly less clever agent often sounds better than a slow and brilliant one.
Interruptions, silence and noise
- Barge-in: callers interrupt. The agent should stop speaking when it hears the caller and listen.
- End-of-turn detection: deciding when the caller has really finished, not just paused. Poor detection causes the agent to cut in or wait too long.
- Background noise: traffic, shops and speakerphones are normal in many calls. Choose speech recognition that copes with them.
- Silence and filler words: the agent should handle "umm", pauses and "hello? are you there?" gracefully.
- Repeating and confirming: for names, numbers and times, confirm by reading back.
Language: Urdu, English and mixed speech
Callers in Pakistan often mix Urdu and English in one sentence. Support varies between speech providers, and quality differs by accent and phone line quality. The practical approach is to pilot two or three speech and voice options with recordings or live calls from your own customers, compare transcription accuracy on names, numbers and place names and choose accordingly. The language model usually copes well with Urdu and Roman Urdu text, so the weakest link is typically speech recognition and the naturalness of the synthetic voice.
Platform or custom build?
| Option | Good when | Trade-offs |
|---|---|---|
| Voice-agent platform (such as Vapi or Retell) | You want to launch quickly and standard features are enough | Per-minute platform fees, less control over fine details |
| Custom pipeline on provider APIs | You need control of cost, data handling or special logic | More engineering and ongoing maintenance |
| Hybrid | You want fast voice plumbing and custom business logic | Needs a clean boundary between the two |
Platform names and features change quickly, so check current documentation. For reasoning about build versus buy, see build vs buy AI agents.
Connecting the agent to your business systems
A voice agent becomes useful when it can do things: check availability in a calendar, create a booking, look up an order, update the CRM or send a WhatsApp confirmation. These are tools the model can call during the conversation. Keep tools narrow and safe: a booking tool that can only create appointments in allowed slots is safer than open database access. See how AI agents use tools. Add a transfer tool so the agent can hand the call to a person with a summary.
Testing and monitoring
- Make test calls from real mobile phones in noisy places and on weak signals.
- Test accents, speaking speeds, interruptions and callers who change their mind.
- Review transcripts and recordings every week at first.
- Track latency, transfer rate, resolution rate and caller feedback.
- Re-test after changing the model, voice or prompts.
The general approach is in how to test and evaluate an AI agent and AI agent monitoring.
A latency budget in plain terms
Think of the delay between the caller finishing a sentence and hearing the first word of the reply as a budget to be spent across the pipeline. A little time goes to deciding the caller has finished. More goes to getting the final transcript. The model needs time to begin its reply. The speech system needs time to produce the first audio. And the network adds delay on each hop. Each part may be small, but together they decide whether the call feels natural. Practical ways to protect the budget are: choose speech and model providers with streaming support, keep the prompt and tool results short, avoid calling slow external systems in the middle of a sentence without a filler phrase such as "one moment while I check", and run services in a region close to your callers.
Choosing voices and personas
- Pick a voice that suits your brand and callers, such as calm and clear rather than theatrical.
- Set a pace slightly slower than conversation for clarity, especially for names and numbers.
- Test how the voice pronounces local names, areas and brand terms, and adjust with pronunciation settings where available.
- Decide on a simple persona and keep it consistent: the same greeting, the same way of confirming details.
- Always disclose that the caller is speaking to an AI assistant.
Security and privacy in voice systems
- Know where audio goes: telephony, speech and model providers may each process the call. List them and read their data terms.
- Minimise retention: decide how long recordings and transcripts are kept, and delete them on schedule.
- Protect payment and ID details: do not let the agent collect card numbers or national ID numbers by voice unless the system is designed and certified for it.
- Verify identity before sharing account-specific information.
- Control who can listen: restrict access to recordings and keep an audit trail.
- Disclose recording and AI use according to the rules where your callers are.
Voice data can be as sensitive as any data you hold. Treating it carefully from the start is easier than fixing it after an incident.
Related guides
- How to Set Up an AI Call Agent Step by Step: A Practical Guide
- AI Call Agent vs Call Centre: A Cost and Quality Comparison
- AI Call Agents in Pakistan: How They Work, Use Cases and Cost Drivers
- AI SaaS Pricing Models and Unit Economics: Keeping Margins When AI Costs Money
- Service: AI Call Agents
Frequently asked questions
How does an AI voice agent work?
It receives the call through telephony, converts speech to text, lets a language model decide the reply and actions, converts the reply to speech and uses integrations to act on your systems, all in real time.
What causes delays on AI phone calls?
Delays come from the sum of speech recognition, model processing, speech generation and network distance. Streaming and faster models reduce them.
Can AI voice agents handle accents?
Modern speech recognition handles many accents, but accuracy varies. Test with your own callers.
Do I need a special phone number?
You can use a new number from a telephony provider or forward calls from your existing number. The right approach depends on your phone system.
Is it better to use a platform or build custom?
Platforms are faster to launch. Custom builds offer more control. Many teams start with a platform and customise the logic.
Can Growmerz build a voice agent for me?
Yes. Growmerz builds AI call agents with English and Urdu support and integrations to your systems, starting with a free AI audit.
Can a voice agent be built in-house without a platform?
Yes, using speech, model and telephony APIs, but it requires real-time engineering and ongoing maintenance. Many teams use a platform for the audio plumbing and build their own logic on top.
Sources and further reading
Primary documentation and research behind this guide.
A good voice agent is a well-tuned pipeline, tested on real calls and connected to useful tools. See our AI call agents service or book a free AI audit.