Voice Activity Detection (VAD): Turn-Taking, Barge-In and Conversational Flow

A voice agent can have flawless transcription and a brilliant language model, and still feel painful to talk to. The reason is almost always the same: it does not know when to listen and when to speak. That timing — the heartbeat of every natural conversation — is governed by Voice Activity Detection.

Voice Activity Detection (VAD) is the quiet workhorse of conversational AI. It decides, dozens of times per second, whether the audio coming down the line is speech or just silence, breathing, traffic, or hold music. Get it right and the conversation flows. Get it wrong and the agent talks over people, leaves awkward gaps, or freezes while the caller waits.

This article is a practical tour of VAD for production phone agents: how it works, why turn-taking and barge-in depend on it, where it breaks, and how to make conversations feel genuinely human. If you want the upstream story of how audio becomes text, read our companion piece on real-time speech-to-text for voice agents.

What VAD Actually Does in a Live Call

At its core, VAD is a real-time binary decision made on tiny slices of audio — typically frames of 10 to 30 milliseconds. For each frame it answers one question: speech, or not speech?

That sounds trivial. In a live phone call it is anything but. The stream of speech/non-speech labels feeds three decisions that the caller feels immediately:

• When the user has started speaking — so the agent can stop its own audio and start listening
• When the user has finished their turn — so the agent can respond without cutting them off or stalling
• What deserves to be transcribed — so the speech-to-text engine is not wasting cycles on silence and noise

VAD is therefore not a side feature. It is the conductor that tells every other component when to act. A weak VAD will sabotage even the best transcription and the best language model.

The Two Failure Modes You Will Feel

Every VAD tuning sits between two opposite mistakes, and both are immediately audible:

• False positives (over-eager): noise gets labelled as speech. The agent thinks the caller is talking, so it stops and waits — or worse, it starts transcribing a cough and replies to nonsense.
• False negatives (too cautious): real speech gets labelled as silence. The agent misses the first syllables of a sentence, or never realises the caller is talking and keeps speaking over them.

The art of a good voice agent is balancing these two errors in the messy, unpredictable reality of telephony — not in a clean lab recording.

Turn-Taking: The Real Goal

Humans are astonishingly good at turn-taking. We predict when someone is about to finish, we slot our reply into the gap, and we overlap politely when we agree. We do all this in a fraction of a second, mostly subconsciously.

Replicating that on the phone is the central challenge of conversational voice AI, and VAD is the foundation. But raw VAD alone is not enough — it only tells you whether there is speech right now. Knowing whether a turn is over requires more.

VAD vs. Endpointing

It helps to separate two ideas that often get conflated:

• VAD answers: "Is there speech in this frame?" — a fast, local, acoustic decision.
• Endpointing answers: "Has the user finished their turn?" — a slower, contextual decision that builds on VAD.

A naive system declares the turn over the moment VAD reports a short silence. That is how you get an agent that interrupts people the instant they pause to think. Good endpointing instead asks: is this silence a real ending, or just a breath, a hesitation, or someone searching for a word?

What Robust Endpointing Considers

To decide when a turn is genuinely complete, a strong system blends several signals:

• Silence duration: a configurable threshold, but never a fixed magic number — it adapts to context.
• Prosody and intonation: a rising pitch often signals "I'm not done," while a falling pitch signals completion.
• Semantics: a sentence that is grammatically and logically complete is far more likely to be a real turn end than a trailing "and, uh...".
• Conversational role: a one-word "yes" answer and a long explanation deserve different patience.

This is exactly where conversational know-how meets raw signal processing. We go deeper into designing natural, context-aware dialogue in our article on conversational design know-how.

Barge-In: Letting Callers Interrupt

Nothing makes an automated system feel more robotic than being forced to listen to a long menu or a slow sentence you already understand. In a real conversation, you interrupt. You say "yes, transfer me" before the agent finishes its list of options.

Barge-in is the ability to honour exactly that instinct. The moment the caller starts speaking, the agent stops talking and starts listening. It is one of the single biggest contributors to a phone agent feeling alive rather than canned.

Why Barge-In Is Hard

Barge-in lives or dies on VAD quality, because of one nasty problem: echo. While the agent is speaking, its own voice may bleed back into the inbound audio stream. A careless VAD hears that and thinks the caller has started talking — so the agent stops itself mid-sentence for no reason.

Handling barge-in well therefore requires:

• Echo cancellation: separating the agent's own outgoing voice from the caller's incoming voice so the agent never interrupts itself.
• Fast, confident detection: the VAD must fire quickly when real speech begins, otherwise the interruption feels laggy and unresponsive.
• Graceful stopping: when a real interruption is detected, the agent must cut its audio cleanly and immediately, not finish the sentence first.

When all three work together, the caller experiences something rare in phone automation: the feeling of being heard the instant they speak.

Why the Phone Makes Everything Harder

Much of the VAD research and tooling assumes clean, wideband, single-speaker audio. The phone offers none of that comfort. Real calls are a hostile environment:

• Narrow bandwidth and compression: telephony squeezes audio into a limited frequency range and compresses it, stripping cues that VAD could otherwise rely on.
• Unpredictable background noise: a caller might be in a car, a busy office, a kitchen, or a windy street.
• Speech-like distractors: hold music, a TV in the background, a second person talking nearby, a barking dog — all can fool a simple detector.
• Echo and crosstalk: especially on speakerphone or poor connections.

This is why classic energy-threshold VAD — which simply flags any audio above a loudness cutoff as speech — falls apart in production. It cannot tell a loud cough from a spoken word. Modern, learning-based detectors that model the shape of speech, not just its volume, are far more robust to these distractors.

Latency: The Invisible Tax on Conversation

Everything above has to happen fast. Psychologically, humans expect a response within a few hundred milliseconds; beyond that, a conversation starts to feel laggy, and people begin to repeat themselves or talk over the gap.

VAD sits in the hot path of that budget. Every millisecond it spends deciding "speech or not" is a millisecond stolen from transcription, reasoning and speech synthesis. The detector must therefore be both accurate and fast — heavy models that need long audio context are a poor fit for a live call.

There is also a subtler trade-off. To declare a turn finished, the system waits for a silence of a certain length. Make that wait too short and the agent interrupts; make it too long and the agent feels slow and hesitant. Tuning this single parameter, in a way that adapts to context, is one of the most impactful things you can do for perceived conversational quality.

Designing VAD for Real Voice Agents

Bringing it together, a production-grade turn-taking stack is never a single off-the-shelf VAD. It is a coordinated system:

• A robust, low-latency VAD that holds up against telephone noise and speech-like distractors.
• Echo cancellation so the agent can talk and listen at the same time without interrupting itself.
• Context-aware endpointing that uses silence, prosody and semantics to decide when a turn is truly over.
• Instant barge-in so callers can interrupt naturally and feel heard.
• Adaptive thresholds tuned to the use case — a quick survey tolerates faster cut-offs than a customer pouring out a complaint.

This is precisely the layer LeetCall obsesses over. The goal is not a technically impressive demo in a quiet room; it is a phone conversation that, on a noisy line with a stressed caller, still feels like talking to a competent, attentive human.

FAQ

What is Voice Activity Detection (VAD)?

VAD is the component that decides, in real time, whether incoming audio contains speech or only silence and noise. In a voice agent it determines when the caller starts speaking, when they have finished, and when the agent can take its turn without cutting the person off.

What is the difference between VAD and endpointing?

VAD answers the frame-level question "is there speech right now?". Endpointing answers "has the user finished their turn?". Robust endpointing builds on VAD but adds context: silence duration, intonation, sentence semantics and hesitations.

What is barge-in and why does it matter?

Barge-in lets the caller interrupt the agent while it is speaking, just like in a real conversation. Without it, callers must wait for every sentence to finish, which feels rigid and frustrating. VAD is what makes barge-in possible and instantaneous.

Why is VAD harder over the phone?

Telephony introduces narrow bandwidth, compression, echo and unpredictable background noise. Non-speech sounds — coughs, hold music, background voices, clicks — can resemble speech. VAD has to stay accurate under these degraded conditions, exactly where naive energy thresholds break down.

How do you stop the agent from cutting users off?

By calibrating the end-of-turn silence threshold, accounting for natural pauses and hesitations, and combining VAD with sentence-level context to tell a genuine sentence ending from a breath. A good agent waits just the right amount: not too short (it interrupts), not too long (it feels sluggish).

How does LeetCall handle VAD and turn-taking?

LeetCall combines real-time voice activity detection with contextual endpointing and instant barge-in, so phone conversations feel like a human exchange. You can hear it directly through a demo or discuss your use case with our team.

Hear the Difference for Yourself

Turn-taking is the kind of thing you cannot judge on paper — you have to hear it. The pauses, the interruptions, the way an agent waits just long enough before replying. That is where a voice agent either earns trust or loses it in the first ten seconds.

If you are building phone automation that needs to feel genuinely conversational, book a live demo and listen to how natural barge-in and turn-taking sound on a real call. Curious about how it fits your volumes and channels? Explore our pricing or talk to our team — we are happy to dig into your specific use case.