Voice Agent Interruption Handling: VAD and Endpointing Settings That Work

Switching barge-in on gave 34% more completed conversations and 41% fewer hang-ups in the first 30 seconds, on a 2,000-call sample of the same outbound qualification agent. Voice agent interruption handling was the only variable: version A had barge-in enabled (VAD threshold 0.55, endpointing 380ms) and version B spoke its full turn before listening. The script, the voice and the LLM were identical.

It is the single largest UX lever in voice agent design, and almost always the last thing teams tune.

What barge-in actually is (and what it isn't)

Barge-in means the agent stops speaking when it detects the caller talking and begins listening immediately. Without barge-in, the agent speaks its full turn (potentially a 6-second response) before it processes what the caller said. The caller experiences this as being talked at by a system that ignores them.

Barge-in is not the same as interruption tolerance. An agent with barge-in enabled will stop speaking. What it does next, whether it correctly interprets a mid-sentence interruption, preserves its prior context, and generates a sensible response, depends on the endpointing and context-preservation implementation.

Teams often enable barge-in, see the agent stop speaking, and move on. The harder problem is what happens in the 400ms after the interruption is detected.

How the VAD stack detects an interruption

Voice AI barge-in starts with Voice Activity Detection (VAD), the component that decides whether the audio signal contains speech. VAD runs continuously during the agent's turn; when it detects speech above its threshold, it signals an interruption.

The production standard for local VAD is Silero VAD, a compact PyTorch model (~1MB) that runs at under 1ms per 30ms audio chunk on CPU. Most managed voice platforms (Retell, VAPI) use Silero or an equivalent internally, but expose it through configuration parameters rather than direct model access.

The core parameters you control:

vad_config:
  model: silero_v4
  threshold: 0.55          # 0–1; probability above which audio is classified as speech
  min_speech_ms: 200       # minimum speech duration to trigger barge-in
  min_silence_ms: 400      # silence duration required before speech event ends
  noise_suppression: true  # pre-process with NSNET2 before VAD
  window_size_ms: 30       # VAD inference frequency

The threshold is the primary tuning variable. The min_speech_ms guard prevents very short sounds (a cough, background TV) from triggering barge-in. The min_silence_ms determines when the caller has finished speaking.

If you are choosing a voice AI agent platform, this is how barge-in configuration compares:

For teams on managed platforms, Retell's configuration surface is currently the most complete. For teams who need custom VAD models or corpus-specific tuning, the Pipecat open-source framework provides full control. Our Twilio, Retell and VAPI comparison covers the wider platform trade-offs.

Voice agent endpointing settings that matter

Endpointing is the complement of VAD: where VAD detects speech starting, endpointing detects speech ending. Too sensitive and the agent cuts the caller off mid-sentence; too slow and it pauses awkwardly before responding.

The endpointing delay (how long to wait after the last detected speech before treating the turn as complete) should be calibrated to your call type:

• Outbound qualification calls: 380–420ms. Callers on outbound calls tend to give shorter, more decisive answers.
• Inbound service calls: 480–550ms. Callers describing a problem often pause mid-sentence while collecting their thoughts.
• Complex questions: 600–800ms with semantic endpointing. If the caller is likely to give multi-sentence responses, volume-based endpointing alone is insufficient.

Semantic endpointing, using a small model to classify whether the caller has completed their thought rather than relying purely on silence duration, is now practical in production. ElevenLabs' turn detection uses a transformer-based turn-completion classifier; it reduces premature cut-offs by approximately 30% compared to pure silence-based endpointing on conversational queries.

Context preservation when the agent is interrupted mid-thought

The hardest part of barge-in is deciding what to do with the context that was interrupted.

When a caller interrupts at second 2 of a 6-second TTS response, the agent has committed to a conversational direction. AI voice agent interruptions come in three kinds:

• Aligned ("yes, exactly" or completing the agent's point): the agent should acknowledge and continue along the same path
• Redirecting ("actually, can we go back to price?"): the agent should abandon the current thread and switch to the redirected topic
• Clarifying ("sorry, what does that mean?"): the agent should pause, answer the clarification, then return to its thread

The naive implementation: stop speaking, process the interruption from scratch as a new turn, lose the prior context. This produces an agent that feels forgetful: it was mid-sentence explaining something and now acts as if that explanation never started.

The better implementation: maintain an in-memory conversation state object that records both the agent's intent at the point of interruption and the content that had been delivered. Pass this context to the LLM alongside the caller's interruption:

{
  "system": "You are continuing a conversation. You were interrupted while saying: '{{interrupted_text}}'. The caller said: '{{caller_utterance}}'. Respond to the caller's input, and if appropriate return to the point you were making.",
  "conversation_history": [...],
  "interrupted_context": {
    "agent_intent": "explaining pricing structure",
    "delivered_text": "Our setup fee is £2,400, which covers",
    "remaining_text": "the integration, the agent training, and the first 30 days of optimisation."
  }
}

Without that state object the LLM sees only the caller's interruption, so the explanation the agent had started is lost.

False barge-in: background noise, music, and multi-speaker environments

The failure mode that erodes caller trust most quickly: the agent stopping mid-sentence because a car radio triggered the VAD. Two to three false barge-ins in a call and callers disengage.

False barge-in sources, ranked by frequency in UK outbound campaigns: 1. Background music or TV in a home office 2. Car radio during a hands-free call 3. A second person speaking in the caller's environment 4. HVAC noise in a call centre (constant low-frequency noise can trigger poorly-calibrated VAD) 5. Network artefacts on poor mobile connections (packet bursts that sound like brief speech)

To stop an AI voice interrupting itself on noise, apply these three layers:

Noise suppression pre-processing: run the incoming audio through a noise suppression model (NSNET2, RNNoise) before the VAD stage. This removes stationary background noise before the VAD sees the signal. Available as a pre-built component in most managed platforms.

min_speech_ms guard: require speech to persist for at least 200ms before triggering barge-in. A radio jingle or a single syllable from background TV rarely sustains for 200ms at a level that would also pass the VAD threshold.

Post-interruption validation: after the VAD triggers and TTS stops, run a second classification pass on the audio that triggered it. If the classifier returns low confidence that the audio was directed at the agent (as opposed to background noise), continue the TTS from where it was interrupted rather than switching to listen mode.

What changed in 2025–2026: turn-taking models and semantic endpointing

The major shift in the past 12 months has been the move from energy-based VAD + silence-duration endpointing to semantic turn-taking models that predict whether a speaker has finished based on linguistic content, not just silence.

The practical difference: a caller who pauses mid-sentence while thinking, "so the price would be... [1.2 seconds]... about three grand a month?", triggers a silence-duration endpointer and causes the agent to jump in and miss the completion. A semantic model trained on conversational turn-taking recognises the mid-sentence construction and waits.

ElevenLabs' Conversational AI platform published a turn-taking architecture in early 2025 that combines VAD with a transformer-based completion classifier. Retell AI implemented a similar hybrid approach in their v3 release. The improvement in perceived conversational quality is significant enough that for any new deployment, semantic endpointing should be the default, not the advanced option.

Testing barge-in before going live

Barge-in behaviour is difficult to evaluate from call logs alone: you need to listen to calls during the first week and classify interruption handling manually. The automated metrics to track:

• False-barge-in rate: percentage of agent turns where TTS was interrupted but no meaningful caller speech followed within 2 seconds. A rate above 8% means your VAD threshold is too sensitive.
• Premature-cutoff rate: percentage of caller turns where the agent began speaking before the caller finished. A rate above 5% means your endpointing delay is too short, which is how an AI voice ends up interrupting callers.
• Context-preservation score: for calls with at least one barge-in, what percentage of agent responses after interruption acknowledged or continued from the pre-interruption context? This requires human annotation but should be tracked for a sample of 50 calls per week during optimisation.

Deepgram's endpointing documentation provides a useful reference for the relationship between utterance-end detection and barge-in; the STT layer's endpointing configuration interacts directly with the VAD layer above it, and misalignment between the two is a common source of double-cutoffs. Our Deepgram, Whisper and AssemblyAI comparison covers the STT side.

One good, one bad and one ugly barge-in deployment

Good: Retell AI agent, VAD threshold 0.55, min_speech_ms 200ms, semantic endpointing enabled, noise suppression pre-processing on, context preservation implemented in the LLM prompt. False barge-in rate: 3.2% of calls. Premature cut-offs: 1.8%. Caller-reported conversational quality on post-call survey: 4.1/5. Call completion rate: 71%.

Bad: VAD threshold at 0.3 (too sensitive) on an outbound campaign targeting car-commute hours. False barge-in rate: 22%. The agent stops speaking an average of 1.4 times per call due to car radio noise. Callers perceive the agent as broken. Campaign paused after day 3 with a 28% hang-up rate in the first 30 seconds.

Ugly: Barge-in completely disabled on an agent with average TTS turn length of 9 seconds. Callers who want to confirm a booking must wait for the full 9-second response, then speak. Most callers interrupt at second 4, find the agent continues speaking, and either give up or speak over it. The result is a dialogue where the caller and agent are often simultaneously talking, producing a call recording that is unintelligible. Completion rate: 34%. One client described their experience on a forum as "like being trapped in a bad phone tree from 2008."

For the full architecture that surrounds barge-in (STT model selection, LLM configuration, TTS streaming, and telephony stack), see Voice AI Architecture 2025: A Production Implementation Guide. For how TTS latency tuning interacts with barge-in UX, see TTS Caching for Voice Agents: Cutting Latency Below 200ms. The Voice AI and Document Analysis case study shows how barge-in was configured in a 10-agent production deployment.

We build and tune agents like this as part of our custom AI agents work. Book a 30-minute scoping call and we'll audit your current barge-in configuration and identify the tuning that's costing you calls.