I'm taking on new voice AI and automation builds this quarter. See what I build →
Blog

A voice agent is a latency budget wearing a trench coat

Azzaz SiddiquiVoice AI4 min read
A turn-taking timeline: the caller's waveform, a latency bar split into three stages with the model call highlighted in orange, then the agent's reply.

The first thing people try to fix when their voice agent feels bad is the prompt. Usually the prompt is fine. What is wrong is that the caller finished speaking and then sat in silence for a beat too long, and a human brain reads that pause as this thing is not listening to me.

Text chat forgives latency because you can see a typing indicator and you set your own pace. A phone call does not forgive it. Conversation has a rhythm, and callers notice when you break it long before they notice a slightly worse answer.

Where the time goes

A single conversational turn is a chain, and every link bills you:

StageWhat happens
EndpointingDeciding the caller actually stopped talking
TranscriptionSpeech to text
InferenceThe model composes a reply
SynthesisText back to speech
TransportAudio travels to the caller

The total is what the caller experiences. Optimising the model alone moves one link in a chain of five, which is why "we switched to a faster model and it still feels laggy" is such a common report.

Endpointing is the underrated one

Endpointing decides when the caller has stopped speaking, and it sits before every other stage. Tune it aggressively and the agent interrupts people mid-sentence. Tune it conservatively and every single turn carries dead air.

It is also the stage most teams never touch, because it is not where the interesting AI is. That is precisely why it is usually the cheapest win available.

Worse, it interacts with how people actually talk. Someone reciting a phone number pauses between digit groups. Someone thinking says "uhh" and stops. A fixed silence threshold treats both as end-of-turn and talks over them.

Streaming is not optional

If any stage waits for the previous stage to finish, you have already lost. The fix is that every stage starts working on partial output:

  • Transcription emits partial hypotheses while the caller is still talking
  • Inference starts on the partial transcript
  • Synthesis begins on the first clause, not the finished sentence

The reply starts arriving before the pipeline has finished thinking, which is what makes the agent feel conversational instead of like a form submission.

Concretely, the difference is whether you await a whole stage or iterate it:

[ ts ]
// Slow: each stage waits for the previous one to finish.
const transcript = await transcribe(audio)
const reply = await complete(transcript)
await speak(reply)

// Fast: every stage consumes a stream and emits one.
for await (const clause of complete(transcribe(audio))) {
    speaker.push(await synthesise(clause))
}

The second version starts producing audio while the model is still writing. The first cannot start until everything upstream is done, so its best case is the sum of every stage, and the caller hears that sum as silence.

One caveat worth planning for: streaming makes barge-in harder. If the caller starts talking while your agent is mid-sentence, you have already queued audio that now needs discarding. Decide early where that queue lives, because retrofitting a cancel path through three streaming stages is genuinely unpleasant.

Cover the gap honestly

Some latency is irreducible, but you control what fills it. Silence reads as broken; a short acknowledgement, the verbal equivalent of a nod, reads as listening and buys a few hundred milliseconds of goodwill.

Used carelessly this becomes a tic. The agent says "got it" forty times per call and sounds like a hold recording. Use it where a human would naturally use it, and nowhere else.

What to measure

Instrument each stage separately, in production, on real calls. Aggregate latency tells you something is wrong; per-stage latency tells you which link to fix. And measure at the tail, not the mean: the 95th percentile turn is the one that makes a caller decide your agent is broken.

[ Voice AI ]

I build the phone agents I write about.

If calls are going unanswered, or booking still needs a person on the line, I can tell you whether a voice agent fits, and what it would take to get one answering on a real number.