
The first thing people try to fix when their voice agent feels bad is the prompt. Usually the prompt is fine. What is wrong is that the caller finished speaking and then sat in silence for a beat too long, and a human brain reads that pause as this thing is not listening to me.
Text chat forgives latency because you can see a typing indicator and you set your own pace. A phone call does not forgive it. Conversation has a rhythm, and callers notice when you break it long before they notice a slightly worse answer.
Where the time goes
A single conversational turn is a chain, and every link bills you:
| Stage | What happens |
|---|---|
| Endpointing | Deciding the caller actually stopped talking |
| Transcription | Speech to text |
| Inference | The model composes a reply |
| Synthesis | Text back to speech |
| Transport | Audio travels to the caller |
The total is what the caller experiences. Optimising the model alone moves one link in a chain of five, which is why "we switched to a faster model and it still feels laggy" is such a common report.
Endpointing is the underrated one
Endpointing decides when the caller has stopped speaking, and it sits before every other stage. Tune it aggressively and the agent interrupts people mid-sentence. Tune it conservatively and every single turn carries dead air.
It is also the stage most teams never touch, because it is not where the interesting AI is. That is precisely why it is usually the cheapest win available.
Worse, it interacts with how people actually talk. Someone reciting a phone number pauses between digit groups. Someone thinking says "uhh" and stops. A fixed silence threshold treats both as end-of-turn and talks over them.
Streaming is not optional
If any stage waits for the previous stage to finish, you have already lost. The fix is that every stage starts working on partial output:
- Transcription emits partial hypotheses while the caller is still talking
- Inference starts on the partial transcript
- Synthesis begins on the first clause, not the finished sentence
The reply starts arriving before the pipeline has finished thinking, which is what makes the agent feel conversational instead of like a form submission.
Concretely, the difference is whether you await a whole stage or iterate it:
// Slow: each stage waits for the previous one to finish.
const transcript = await transcribe(audio)
const reply = await complete(transcript)
await speak(reply)
// Fast: every stage consumes a stream and emits one.
for await (const clause of complete(transcribe(audio))) {
speaker.push(await synthesise(clause))
}The second version starts producing audio while the model is still writing. The first cannot start until everything upstream is done, so its best case is the sum of every stage, and the caller hears that sum as silence.
One caveat worth planning for: streaming makes barge-in harder. If the caller starts talking while your agent is mid-sentence, you have already queued audio that now needs discarding. Decide early where that queue lives, because retrofitting a cancel path through three streaming stages is genuinely unpleasant.
Cover the gap honestly
Some latency is irreducible, but you control what fills it. Silence reads as broken; a short acknowledgement, the verbal equivalent of a nod, reads as listening and buys a few hundred milliseconds of goodwill.
Used carelessly this becomes a tic. The agent says "got it" forty times per call and sounds like a hold recording. Use it where a human would naturally use it, and nowhere else.
What to measure
Instrument each stage separately, in production, on real calls. Aggregate latency tells you something is wrong; per-stage latency tells you which link to fix. And measure at the tail, not the mean: the 95th percentile turn is the one that makes a caller decide your agent is broken.
I build the phone agents I write about.
If calls are going unanswered, or booking still needs a person on the line, I can tell you whether a voice agent fits, and what it would take to get one answering on a real number.