Why Your Voice AI Agent Keeps Interrupting

There are two completely independent decisions happening on every call, and they point in opposite directions:

• Endpointing decides when the caller has finished so the agent may start. Vapi calls this startSpeakingPlan. Retell calls it responsiveness.
• Barge-in decides when the caller has started so the agent must stop. Vapi calls this stopSpeakingPlan. Retell calls it interruption_sensitivity.

When somebody says "the agent keeps cutting me off," that is always barge-in. And almost every time I open somebody's config after that complaint, the setting they have been dragging around for a week is the endpointing one. The result is predictable: the agent now feels sluggish on every single turn, and it still talks over people.

Fix the detector that is actually firing. Leave the other one alone until you have.

A barge-in is not one event, it is four, and they all have to unwind before the caller gets a usable turn:

1. The VAD crosses its threshold and declares "user speaking."
2. The in-flight TTS audio is flushed — whatever was already buffered gets thrown away mid-word.
3. The in-flight LLM request is cancelled. You paid for those tokens.
4. Transcription restarts on the new audio.

Nowhere in that chain does anything ask what the caller said. The whole sequence completes on sound alone. That is the entire bug, and it is a design choice, not a defect — reacting to energy is what makes a fast agent feel fast.

So the question is never "how do I make barge-in smarter." It is "how much evidence do I want before I let it fire." That is a dial, it sits in your config, and the two platforms expose it differently.

Retell gives you two floats between 0 and 1, both defaulting to 1, and confusing them is the single most common voice-agent config mistake I see.

{
  "interruption_sensitivity": 0.6,   // how easily the caller stops the agent
  "responsiveness": 1,               // how fast the agent starts after the caller
  "enable_backchannel": true,
  "backchannel_frequency": 0.8,
  "reminder_trigger_ms": 10000,
  "reminder_max_count": 1
}

• interruption_sensitivity — lower is harder to interrupt. At 0 the agent can never be interrupted at all. That absolute floor is genuinely right for a compliance disclosure it has to finish reading, and genuinely wrong for anything conversational, because a caller correcting a wrong address now has no way to stop it.
• responsiveness — this is the pause before the agent speaks, and Retell's own guidance is that dropping it by 0.1 adds roughly half a second of wait. Useful when your callers are elderly or the line is bad. Useless against interruptions.

The rest of the conversational surface lives in the same object: enable_backchannel defaults to false, backchannel_frequency to 0.8, reminder_trigger_ms to 10000 with one reminder, and end_call_after_silence_ms to 600000 with a hard floor of 10000. Ten minutes of silence before hangup is far too generous for a lead-qualification call; I pull that down every time. Check the current values against the Retell create-agent reference before you paste anything, because these defaults have moved before.

If you are standing one of these up from scratch, the wiring comes first — my Retell setup guide covers that, and this post is what you do on day three when real callers start complaining.

Vapi splits the same decision into two plans, and it exposes the one knob I wish every platform had.

{
  "stopSpeakingPlan": {
    "numWords": 3,          // 0 = pure VAD, cuts on any sound
    "voiceSeconds": 0.2,    // how long voice must persist
    "backoffSeconds": 1     // silence after being interrupted
  },
  "startSpeakingPlan": {
    "waitSeconds": 0.4,
    "smartEndpointingPlan": { "provider": "livekit" }
  }
}

numWords is the evidence bar. At 0 you get pure VAD: fastest possible reaction, and every cough wins. At 2 or 3, the agent waits for the transcriber to confirm actual words before it yields. You pay roughly a quarter second on genuine interruptions and you buy near-total immunity to background noise, TV audio, and the person talking in the next cubicle.

voiceSeconds (default 0.2) is the cruder version of the same idea — how long detected voice has to persist before it counts. backoffSeconds (default 1) is how long the assistant stays quiet after being cut off, and it is worth raising if your agent has the habit of jumping back in while the caller is still mid-thought.

On the other side, startSpeakingPlan.waitSeconds defaults to 0.4, and smartEndpointingPlan can run on Krisp, Deepgram Flux, Assembly, LiveKit or Vapi's own model instead of a flat timer. The full field list is in Vapi's speech configuration docs. If you are still choosing between the two platforms, I compared them here — and turn-taking control is one of the places the gap is real rather than cosmetic.

Half the false cuts I have chased down were not interruptions at all. They were backchannels — the noises humans make to signal I am still here, keep going. Yeah. Right. Okay. Mm-hmm.

A VAD cannot tell those apart from "wait, that's wrong." A word counter can, and the separation is almost embarrassingly clean: a backchannel is one word, an objection is three or more. That is the entire reason numWords: 3 works as well as it does. It is not a noise filter, it is a grammar filter.

There is a second move here that people skip. Turn on the agent's own backchannels — enable_backchannel on Retell, off by default. When the agent murmurs while the caller talks, callers stop feeling the need to test whether the line is still alive, and the interruption rate drops on its own. Set backchannel_frequency conservatively; the default of 0.8 is chatty enough to get distracting on longer answers.

Do these in order. Skipping to step three is why most agents end up both slow and rude.

1. Reproduce it on a recording. Find a real call where the agent talked over somebody, note the timestamp, and listen to what the caller actually made — a word, a backchannel, a breath, a door. You cannot tune a detector you have never watched fire. This step gets skipped constantly and it is the one that tells you which of the other four you need.
2. Move the evidence bar from energy to words. Vapi: numWords to 2 or 3. Retell: interruption_sensitivity down from 1 in steps of 0.2. This alone fixes the large majority of complaints.
3. Only then touch endpointing. If slower callers are still getting clipped, raise waitSeconds past 0.4 or drop responsiveness by 0.1. Remember that each 0.1 costs you about half a second on every turn. This is the setting that makes an agent feel dead on the phone.
4. Add backchannels. Last, not first. They change how the call feels, not whether the detector is correct.
5. Re-listen to the same recording after every single change. One variable at a time. These settings interact, and if you move two at once you will never know which one worked — you will just have a config nobody on your team can explain in six months.

Here is the part the tuning guides tend to skip. Every setting that makes an agent harder to interrupt also makes it slower to yield when someone genuinely needs it to stop. There is no configuration that wins both. There is only a configuration that matches the call.

• Inbound support and sales. Bias toward yielding. An agent that talks over an objection loses the call outright. numWords: 2, normal endpointing.
• Noisy environments — car, warehouse, jobsite. Bias hard toward evidence. numWords: 3, raised voiceSeconds, and accept the extra quarter second.
• Disclosures and confirmations. The script has to finish. Retell's interruption_sensitivity: 0 exists for exactly this and nothing else.
• Long explanatory answers. Raise backoffSeconds so the agent does not leap back in the moment the caller pauses to breathe.

I run an inbound agent on a real phone number, and moving it off pure VAD onto word-count gating did more for call quality than any model or voice change I have made since. The calls did not get faster. They got less annoying, which turns out to be the thing callers actually notice — and it is the same lesson as what voice AI can and cannot do in 2026: the wins are in conversational mechanics, not in raw speed. If you want to see the whole thing wired up for a real business, here is a full Retell build for a roofing company.