Why do voice agents fail in noisy calls? Five failure modes, with numbers
Voice agents fail in noisy calls because the recogniser, the voice activity detector and the turn logic all treat every voice and every codec artefact as the caller. We list five failure modes and tie each to a number from our Null Test run.
Voice agents fail in noisy calls because every stage downstream of the microphone assumes the audio is one caller speaking cleanly. When it is not, the speech recogniser, the voice activity detector (VAD) and the turn logic each fail in their own way, and the failures compound.
The five failure modes
| Failure mode | Measured (Null Test, 2026-08-14) |
|---|---|
| STT insertions from other voices | 45 insertions in 190 words of two-voice audio at 0 dB (raw) |
| False barge-in | VAD false-alarm rate 57.8% on two voices at 10 dB (raw) |
| Wrong turn end | VAD false-alarm rate 97.7% under babble at 5 dB (raw) |
| Wrong-speaker intent | WER 84.74% on two voices at 0 dB (raw) |
| Telephony codec damage | WER 7.37% on A-law against 5.79% on the plain telephony condition (raw) |
These are the unprocessed control rows of Null Test: recogniser faster-whisper:base.en, Silero VAD at threshold 0.5, 10 speakers, 190 clips, 16 kHz, run of 2026-08-14. They describe what happens with no enhancement at all; the pooled control is 14.32% WER.
1. STT insertions from other voices
A recogniser has no notion of "the caller". A second voice at the same level produces words, which score as insertions. At 0 dB the raw audio gives 45 insertions and an 84.74% WER, because the two voices interleave.
2. False barge-in
Agents stop speaking when the VAD fires. Under two voices at 10 dB the VAD false-alarm rate is 57.8% on the raw audio, so a bystander can interrupt the agent as easily as the caller can.
3. Wrong turn end
End-of-turn logic waits for silence. Under babble at 5 dB the VAD false-alarm rate is 97.7%: the detector hears speech almost everywhere, so silence never arrives. Turn-detection models can be trained for noise (Daily describes retraining Smart Turn v3.2 with cafe and office noise, Daily blog, 2026), which helps this mode but not the others.
4. Wrong-speaker intent
When the transcript contains the bystander's words, the language model acts on them. The 84.74% WER at 0 dB is the extreme case; the intent error rate in your product depends on your prompts, which we do not measure.
5. Telephony codec damage
Narrowband codecs remove information before any of your code runs. A-law raises raw WER to 7.37% against 5.79% for the same clips through a clean telephony path. See 8 kHz is where voice AI breaks.
Does isolation fix them?
Partly, and not everywhere. On the same run our v8_2 model lowers WER on two voices at 0 dB from 84.74% to 74.21% and the VAD false-alarm rate at 10 dB from 57.8% to 44.4%, but it raises WER on two voices at 5 dB from 26.32% to 31.05%, and on A-law from 7.37% to 10.53%. Isolation addresses modes 1 to 4 in some conditions and does not address mode 5. Measure on your own calls first: how to evaluate speaker isolation.
FAQ
What is the most common reason a voice agent mishears in a noisy call?
In our data the largest effect is a second voice at a level close to the caller's: 84.74% WER at 0 dB against 14.32% pooled.
Does noise cancellation stop false barge-in?
Not for speech. A bystander's voice is speech, so a VAD still fires on it; see what is speaker isolation.
Should I always put an enhancer in front of the recogniser?
Our data says no. The untouched control had the lowest pooled WER of the shippable rows we measured, and ours was above it.