Why the pause matters: latency in AI phone calls

You ask a question on the phone and nothing happens. One second. Two. You say "hello?" and the agent starts talking over you. This is the most common complaint about AI calls, and it is rarely about intelligence. It is about time.

What the caller experiences

People expect a reply almost immediately in conversation. A short gap reads as thinking; a long one reads as a dropped call or a machine. Callers fill silence by repeating themselves, which then collides with the agent's late answer and makes everything worse.

The delay a caller feels runs from the moment they stop speaking to the moment they hear the first sound of the reply. Everything below happens inside that gap.

Where the time goes in one turn

In a chained agent, a single turn passes through several stages in order.

1. Deciding the caller has finished. The agent has to wait long enough to be sure it is not a mid-sentence pause. This is often the largest single piece and it is a setting, not a technical limit. See barge-in and turn-taking.
2. Speech recognition. Turning the final words into text. Streaming recognition does most of this while the caller is still talking.
3. The language model's first token. The time before the model emits the first word of its reply. Longer prompts, bigger models and extra "thinking" make it slower.
4. The voice's first audio. The time before the voice service returns the first sound. The agent can start speaking once the first sentence is ready; it need not wait for the whole reply.
5. The phone network. Audio travels to your software and back. Distance between the caller, the carrier and the servers adds delay, and mobile networks vary.

Each hand-off between services also adds a little. They look small individually and add up.

Some published numbers, labelled

Twilio, a telephony vendor, published a guide in November 2025 on latency for voice agents built as a chain. These are the vendor's own benchmark targets, not independent measurements, and the guide says its component figures are starting points rather than best-in-class. It gives a target of about 1,115 milliseconds from the caller stopping to the caller hearing a reply, with an upper limit of 1,400 ms. Its component targets are about 350 ms for speech recognition, 375 ms for the language model's first token, and 100 ms for the voice's first audio, with upper limits of 500, 750 and 250 ms. The same guide notes at least ten network traversals per response in a typical chain.

Treat these as a rough shape rather than a promise. Real calls vary with the vendors you pick, the prompt, the language and the network. The useful lesson is the proportion: no single stage is the whole problem, and the model's first token and the end-of-speech wait are usually big.

Why chained and realtime differ

A chain has separate recognition, model and voice stages, so there are more hand-offs and each must finish something before the next can start (though good set-ups overlap them by streaming). The benefit is control: you choose each vendor, read the text between steps and fix the stage that is slow.

A realtime model accepts audio and returns audio directly, removing some hand-offs. Many people find it feels quicker and more fluid in conversation. The trade is that you have less choice of voice and recognition, and less visibility inside the turn. Which one is right depends on languages, voices, cost and how much you want to tune. We did not find an independent, like-for-like comparison we could cite, so test both on your own calls.

Practical levers

• Shorten the prompt. Move long reference material out of the main prompt and fetch it only when needed.
• Pick a faster model for simple turns. Greetings and confirmations do not need the biggest model.
• Tune the end-of-speech wait. A little shorter feels brisker; too short interrupts people mid-thought.
• Keep replies short. A one-sentence answer starts sooner and ends sooner than a paragraph.
• Speak the first sentence early. Let the voice start while the rest of the reply is still being written.
• Choose servers near your callers, and avoid unnecessary extra hops between services.
• Cover tool waits. If the agent must check a calendar, a short "one moment" is better than silence.

Reading latency from a recording

You do not need special tools to see where the time goes. Open a call recording and its transcript side by side and mark three moments for a slow turn: where the caller's last word ends, where the agent's first sound begins, and what the agent said first. If the first thing the agent says is a filler such as "let me check", the gap before it is your base latency. If the first thing is the real answer, the gap includes everything the model had to work out.

Repeat for a dozen turns across several calls. A pattern usually appears quickly. Gaps that are long only after questions about prices or dates point to a tool call or a long reference section in the prompt. Gaps that are long after every turn point to the end-of-speech wait or the model choice. Gaps that are long only on certain calls point to the caller's network or language, which you can check by comparing the same script from a different phone.

A pause is not always a fault

Some delay is the right amount. If a caller reads out a ten-digit phone number, a quick reply after the fourth digit would be wrong, and a slightly longer wait is kinder. If the agent is about to quote a fee, a half-second of apparent care does no harm. The aim is not the smallest possible gap on every turn. It is a gap that feels natural for the kind of turn it is: instant for "yes", slightly longer for a considered answer, and a spoken "one moment" when real work is happening behind the scenes.

Callers also forgive a short, honest delay more readily than a fast wrong answer. If a trade-off must be made between speed and accuracy, for example choosing a smaller model that answers quicker but misreads dates, check the transcripts for errors before celebrating the faster time.

Language matters too. Recognition and voice quality, and therefore their speed, can differ between English and a regional language, so measure each language you plan to serve rather than assuming a result from one carries over.

Measure it yourself

Do not trust any figure, including the ones above, over your own test. Make ten calls from a normal mobile in your own city, time the gap with a stopwatch or by reading the recording, and note the slowest three. A single slow turn matters more to callers than a good average. When transcripts show people saying "hello?" or repeating themselves, you have found a latency problem.

Where to start

1. Place ten test calls and write down the gap after each question.
2. If the gaps are long on every turn, look at the end-of-speech wait and the model first.
3. If only some turns are slow, look at tool calls and long answers.
4. Change one thing at a time and repeat the test.

Try the speed yourself in the live demo.

Sources

1. Twilio: Core latency in AI voice agents, published 17 November 2025, accessed October 2026. Figures are Twilio's own benchmark targets.