Live conversations: v8_2 vs ai-coustics Quail VF 2.2 S under two recognisers, and the condition decides who leads

Our main benchmark scores short, single utterances. A voice agent never hears an utterance on its own. It hears a call: someone talks, pauses, listens, talks again, while other people in the room come and go. So we built a second experiment shaped like a call, and ran our shipping model, v8_2, against a product built for the same job: ai-coustics Quail VF 2.2 S, a Voice Focus model whose job is also to keep the primary speaker and drop everyone else.

The short version, because it depends on who is listening:

• With faster-whisper base.en it is parity, and no engine beats raw on the pooled set.
• With Gemini Live input transcription, the transcript a live Gemini voice agent makes of the caller, v8_2 beats raw and leads VF S on the pooled set, by −6.47 pp [−9.89, −3.26].
• Under both, the condition decides who leads: v8_2 with a close-mic talker and a loud background, VF S with a distant talker at −10 dB.
• On turn-taking, VF S is better: under Gemini Live it triggers 52 false barge-ins against 125 for v8_2 and 97 for raw.

Every number in this post is on /benchmark and in benchmark.json under conversation.

What we built, and what it is not

The test set is constructed: real voices and real rooms, mixed by us. Every ingredient is a real recording, but the conversations were assembled digitally, not recorded end to end. A real call adds things a mix does not, such as codecs, automatic gain control and a talker who reacts to the room. Read every number below with that in mind.

• The target. One LibriSpeech test-clean speaker per conversation: 4 to 8 consecutive utterances from one chapter, separated by natural pauses of 0.3 to 2.5 s, with at least one 3 to 6 s "listening" gap where the caller waits for the agent.
• The background. 1 to 3 other real talkers (LibriSpeech dev-clean and MUSAN speech) who come and go with their own talk spurts and their own pauses. They are placed in a real room through measured BUT ReverbDB impulse responses, 1 to 4 m from the microphone. Half of the conversations also carry intermittent real ambience (DEMAND, MUSAN noise) at −20 dB relative to the target.
• The states. Every conversation contains all four states a voice agent has to handle: target alone, background voice alone inside the target's pauses, overlap, and no voice at all.
• The conditions. 40 conversation scripts, each mixed five ways, for 200 conversations of 30.4 to 59.8 s:
  • near: a close-mic target, as on a headset or a phone held to the ear, with each background talker at −10, −5 or 0 dB relative to the target;
  • room: the target itself 0.5 to 1.5 m from the microphone, through a measured room response, with the background at −10 or −5 dB.
• Streaming. Each conversation goes through the engine whole, with no reset inside it, the way a live call would. Engines are reset between conversations, and each output is delay-compensated by the engine's declared latency: 15 ms for v8_2, 30 ms for VF S.
• The competitor. ai-coustics Quail VF 2.2 S, run through the licensed SDK at its native 16 kHz with the SDK's default enhancement level (0.8). It is a benchmark reference and never a code path in anything we ship.
• Scoring, twice. faster-whisper base.en transcribes each whole conversation offline. Separately, each output is streamed in real time, as 100 ms PCM16 frames, into a Gemini Live session (gemini-live-2.5-flash-native-audio), and we score its input transcription and log when the agent replies and when it reports being interrupted. Both transcripts are scored against the target's own transcript. Every difference carries a 95 % confidence interval from a paired bootstrap over conversations.

Recogniser 1, faster-whisper base.en: parity, and raw is not beaten

All 200 conversations	raw	v8_2	VF S
WER %	37.82	38.46	38.59
insertions %	19.38	13.71	7.58
deletions %	3.20	5.54	12.51
background-word leak, per 100 words	4.06	1.74	0.94

v8_2 − VF S is −0.13 pp [−4.56, +4.47]. The interval sits squarely across zero, so this is parity. Against the unprocessed audio, v8_2 is +0.65 pp [−3.38, +4.73] and VF S is +0.78 pp [−3.60, +5.13]. With faster-whisper, no engine beats raw on the pooled set. This is the same lesson as our main benchmark, where the control also won on the pooled average: for this recogniser, averaged over everything, doing nothing is as good as either engine.

What the pooled line hides is the error mix. Raw audio fails by inventing words (the recogniser transcribes the background talkers). Both engines cut that, and VF S cuts it hardest. But VF S also drops the target's words far more often. The two engines reach the same total by different routes.

Per condition, faster-whisper base.en

Condition	raw	v8_2	VF S	v8_2 − VF S, pp [95 % CI]	Reading
Near, background −10 dB	14.70	10.86	8.89	+1.97 [−0.54, +4.63]	parity
Near, background −5 dB	21.65	24.00	27.04	−3.04 [−19.48, +14.99]	parity
Near, background 0 dB	38.70	37.19	55.94	−18.75 [−27.75, −9.90]	v8_2 ahead
Near, pooled	25.01	24.02	30.62	−6.60 [−13.36, −0.06]	v8_2 ahead
Room, background −10 dB	53.17	49.56	40.07	+9.50 [+4.31, +14.84]	VF S ahead
Room, background −5 dB	60.86	70.69	61.03	+9.66 [+3.07, +16.81]	VF S ahead
Room, pooled	57.02	60.13	50.55	+9.58 [+5.52, +13.90]	VF S ahead

WER in %, faster-whisper base.en. Negative differences mean fewer errors for v8_2.

Where v8_2 leads: a close-mic talker with a loud background

With the background talkers as loud as the target (near, 0 dB), v8_2 scores 37.19% WER and VF S 55.94%. Nearly all of that gap is deletions. VF S deletes 28.41% of the target's words there, against 4.57% for v8_2. When a second voice is as loud as the first, VF S gates out much of the target along with the background.

Pooled over the three near conditions, v8_2 leads by −6.60 pp [−13.36, −0.06]. State that carefully: the upper bound is barely below zero, and the lead comes from the 0 dB condition. At −10 dB VF S is slightly better on the point estimate (+1.97, interval across zero), and −5 dB is too noisy to call either way. The fair reading is "ahead when the background is as loud as the talker", not "ahead on every headset call".

Where VF S leads: a distant talker

Once the target itself is 0.5 to 1.5 m from the microphone, VF S leads by +9.58 pp [+5.52, +13.90] pooled, and in both room conditions. At room −5 dB v8_2 is worse than doing nothing: 70.69% against raw's 60.86%, a difference of +9.82 pp [+3.09, +17.40]. A reverberant target is a weak point of v8_2 today, and this run shows it plainly.

What the engines do to the audio

The transcript is one view. The signal is another, and on this one VF S does the thing a voice agent most wants during a pause much better: it makes the background go quiet.

Pooled, all conditions	raw	v8_2	VF S
background suppression in the target's pauses, dB	0.0	8.5	17.4
background words recognised in those pauses, per minute	115.86	68.05	41.79
onset change in the first 200 ms after a pause, dB	−0.30	−14.08	−23.03
SI-SDR on the target's speech, dB	5.29	7.49	3.91

In the cleanest near condition (−10 dB) the gap in pause suppression is widest: 10.6 dB for v8_2 against 30.1 dB for VF S. If your problem is an agent that answers the TV while the caller is silent, VF S holds the pause quieter in this test, and it lets fewer background words into the transcript (0.94 per 100 words, against 1.74 for v8_2 and 4.06 for raw).

The silence has a price. VF S attenuates the first 200 ms after the target starts speaking again by 23.03 dB on average, against 14.08 dB for v8_2, and it scores a lower SI-SDR on the target's own speech than the unprocessed input does. That is the mechanism behind its deletions. Neither engine is free here: v8_2 also attenuates onsets, by a lot.

Recogniser 2, Gemini Live: v8_2 beats raw, and leads VF S pooled

A voice agent built on Gemini Live does not run faster-whisper; it hears the stream itself. So we streamed all 600 outputs (200 conversations × 3 arms) into Gemini Live sessions in real time and scored the session's own input transcription. The run is complete.

All 200 conversations, Gemini Live	raw	v8_2	VF S
WER %	40.50	36.40	42.87
insertions %	3.17	3.66	1.54
deletions %	27.63	19.75	29.37
background-word leak, per 100 words	0.80	0.63	0.16

v8_2 − VF S is −6.47 pp [−9.89, −3.26], and v8_2 − raw is −4.10 pp [−6.28, −1.98]: under this recogniser v8_2 is the only engine that beats the unprocessed audio on the pooled set. VF S − raw is +2.37 pp [−0.92, +5.54], parity with raw.

Gemini Live fails differently from faster-whisper. It rarely invents the background talkers' words; it drops words when the target is hard to hear, so its errors are mostly deletions, and that is where v8_2 helps it: deletions fall from 27.63% raw to 19.75%, while VF S, which gates more of the target, sits at 29.37%.

Condition	raw	v8_2	VF S	v8_2 − VF S, pp [95 % CI]	Reading
Near, background −10 dB	21.76	14.40	17.43	−3.04 [−10.83, +5.06]	parity
Near, background −5 dB	23.97	18.94	24.82	−5.88 [−11.15, −1.18]	v8_2 ahead
Near, background 0 dB	39.71	33.06	60.95	−27.89 [−35.80, −19.85]	v8_2 ahead
Near, pooled	28.48	22.13	34.40	−12.27 [−16.84, −7.91]	v8_2 ahead
Room, background −10 dB	49.67	51.89	44.14	+7.74 [+2.48, +12.77]	VF S ahead
Room, background −5 dB	67.38	63.71	67.00	−3.28 [−8.07, +1.40]	parity
Room, pooled	58.52	57.80	55.57	+2.23 [−1.64, +6.08]	parity

WER in %, Gemini Live input transcription. Negative differences mean fewer errors for v8_2.

The shape matches faster-whisper with a different balance. The close-mic lead is larger and clearer: −12.27 pp [−16.84, −7.91] pooled over the near conditions, and at 0 dB VF S deletes 48.71% of the target's words against 16.53% for v8_2. The distant-talker side narrows: VF S still leads clearly at room −10 dB (+7.74 pp [+2.48, +12.77]), but the room pool is parity under Gemini Live (+2.23 pp [−1.64, +6.08]).

Turn-taking: here VF S is better

The same sessions show when the agent starts a reply and when Gemini Live reports that it was interrupted. A false response is a reply that starts with no target speech in the previous 3 s; a false barge-in is an interruption with no target speech in the previous 1.5 s, which in practice means the background talker cut the agent off.

All 200 conversations	raw	v8_2	VF S
agent responses	976	1001	970
false responses	49	56	37
barge-ins	939	1038	923
false barge-ins	97	125	52

This result goes against us, and we state it plainly. VF S triggers 52 false barge-ins against 125 for v8_2, and v8_2 triggers more than the unprocessed audio (97). False responses point the same way: per conversation, v8_2 − VF S is +0.10 [+0.01, +0.18]. It fits the signal metrics above: VF S holds the pauses much quieter, so a background voice in the caller's silence less often looks like the caller to the agent. v8_2 passes more of that background through, and Gemini Live hears it as speech. If false interruptions are your main failure, this run favours VF S.

Caveats

• Constructed, not recorded. Real voices and rooms, mixed by us. Not a field recording, and not a substitute for testing on your own calls.
• Two recognisers, two answers. faster-whisper base.en reads the pooled set as parity; Gemini Live reads it as a v8_2 lead. Neither is the truth for your stack: the recogniser you run decides which result applies to you.
• Forty speakers. The five conditions re-mix the same 40 scripts, so they are not independent samples, and one word is worth 0.005 pp pooled.
• Gemini Live is a hosted model. Its behaviour can change between versions; these numbers are for gemini-live-2.5-flash-native-audio as we ran it in October 2026.
• One setting for VF S. We ran the SDK default enhancement level. Another level would move its trade between suppression and deletions, in either direction.
• Leak is a lower bound. Only the transcribed background talker (LibriSpeech dev-clean) can be attributed word by word; the MUSAN talkers cannot.
• Licences. LibriSpeech (CC BY 4.0), MUSAN (CC BY 4.0), DEMAND (CC BY-SA 3.0), BUT ReverbDB (CC BY 4.0). We publish the numbers only, not the mixes or the audio.

Our main benchmark is unchanged by any of this: on the 19-condition matrix the control sits at 14.32% WER and our v8_2 row at 16.59%. This experiment adds a different question, about calls rather than clips, and its answer is not one-sided either.

If your users are on a headset or a phone in a loud place, and especially if your agent runs on Gemini Live, this run suggests v8_2 is worth trying. If they sit back from a laptop in a reverberant room, or false interruptions are what you most need to stop, it suggests VF S. Either way, the live demo runs v8_2 on your own microphone, and your own calls are the test that decides.

FAQ

Which is better, v8_2 or ai-coustics Quail VF 2.2 S?

It depends on the recogniser and the condition. With faster-whisper base.en the pooled result is parity: v8_2 − VF S is −0.13 pp [−4.56, +4.47]. With Gemini Live input transcription v8_2 leads on the pooled set by −6.47 pp [−9.89, −3.26]. Under both, v8_2 leads with a close-mic talker and a loud background, and VF S leads with a talker 0.5 to 1.5 m away at −10 dB. VF S also holds the pauses quieter and triggers fewer false barge-ins under Gemini Live (52 against 125).

Does either engine improve transcription over raw audio?

With faster-whisper base.en, not on the pooled set: raw scores 37.82% WER, v8_2 38.46% and VF S 38.59%, and both differences against raw have intervals across zero. With Gemini Live, v8_2 does: 36.40% against raw's 40.50%, a difference of −4.10 pp [−6.28, −1.98]. VF S against raw under Gemini Live is +2.37 pp [−0.92, +5.54], which is parity.

Does v8_2 reduce false interruptions in a Gemini Live agent?

No. In this run it increases them: 125 false barge-ins against 97 for the unprocessed audio and 52 for VF S, over 200 conversations. VF S's stronger suppression in the caller's pauses is the likely reason it does better here.

Are these real recordings?

They are built from real recordings: LibriSpeech and MUSAN voices, DEMAND ambience and measured BUT ReverbDB room responses. The conversations themselves are mixed digitally by us, not recorded end to end.

Can I download the audio?

No. We publish the numbers, the method and the per-condition tables in benchmark.json. The mixes and audio are not published.