Skip to content
anecho.ai
Get a key
Proof · nulltest.dev

Null TestEleven enhancers. One control. Nothing won on average.

One corpus, 19 conditions, one harness, every engine scored the same way. On pooled word error rate, no enhancer beat the unprocessed control. The null hypothesis won.

14.3% WER
Unprocessed control3,610 reference words across 19 conditions
15.2% WER
Best enhancerai-coustics Quail L — still worse than doing nothing
0of 11
Engines that beat raw, pooledBut at least one engine beat raw in 11 of 19 individual conditions
103→381
Insertions, raw → worstEnhancement produced more invented words, not fewer
Contrast · the published claims

The vendor numbers cannot all be true.

Four public answers to the same question. No shared corpus among them.

Published enhancement claims and their evidence
SourceClaimWhat shipped as evidence
Krisp~46% WER reductionMarketing figure. Corpus, conditions and transcriber unstated.
ai-coustics~43% WER reductionMarketing figure. Same shape, different model family.
DeepgramEnhancement hurts STT accuracyPublished position. No dataset we can find.
arXiv:2512.17562Raw beat enhanced in all 40 configurationsIndependent — but one research model, medical speech, semantic WER.

Here is a shared corpus. None of the four survives it. The reduction claims: the best engine on this bench moved pooled WER +0.9 points — a rise, not the advertised fall. The never-helps claims: at least one engine beat raw in 11 of 19 conditions. A number without a corpus is not evidence. This one you can download.

Why the published claims disagree — the full analysis

The best enhancer, ai-coustics Quail L: 15.2% pooled against the control’s 14.3%. The best you can legally ship, anecho.ai_focus_model_16khz_v8_2: 16.6%. Insertions — invented words — rose from 103 raw to 381 on DeepFilterNet3.

Pooled is not the whole result: at least one engine beat raw in 11 of 19 conditions, by up to −10.5 points on Competing speaker 0 dB. Enhancement pays where the audio is genuinely bad, costs accuracy where it is not; pooling hides the trade.

Methodology and caveatsThe full write-up
13 engines
Pooled results across every acoustic condition, by enhancement engine. The unprocessed control is the first row and every delta is measured against it.
LicenceBetter than raw
Raw (no processing)—14.32ref10360—0.94819.3—0 ms
Passthrough (control)—14.32±0.00103600/190.94819.30.0000 ms
ai-coustics Quail Lproprietary15.18+0.86128593/190.94320.60.12330 ms
anecho.ai_focus_model_16khz_v8_2proprietary — ours16.59+2.27119734/190.94118.50.06715 ms
ai-coustics Quail VF 2.2 Lproprietary17.59+3.27163765/190.94516.10.07530 ms
GTCRNMIT20.14+5.82223411/190.94517.90.05016 ms
FastEnhancer-SMIT20.33+6.01173723/190.94317.80.03416 ms
FastEnhancer-LMIT20.36+6.041711126/190.94115.60.37726 ms
FastEnhancer-BMIT20.64+6.32163803/190.94317.70.02116 ms
FastEnhancer-TMIT21.66+7.341461103/190.94017.50.01316 ms
FastEnhancer-MMIT22.13+7.81234845/190.94016.10.11022 ms
DeepFilterNet3 (96-frame warm-up)MIT OR Apache-2.030.69+16.373751300/190.91424.30.226100 ms
DeepFilterNet3MIT OR Apache-2.034.04+19.723812000/190.90123.90.133100 ms

Blue row is the unprocessed reference · red ΔWER means the engine made transcription worse · green VAD FA means it cut false triggers

Generated 2026-08-14T02:16:50Z · faster-whisper:base.en · silero-vad · Apple M1 Max · onnxruntime 1.28.0 · 1 thread

Route · what the data changed our mind about

Enhancement is not useless. It is pointed at the wrong consumer.

Wire it up

Voice activity F1 barely moves — but in the conditions that break barge-in, the false-alarm rate collapses.

Split pipeline · measuredSame audio, two consumers, opposite verdicts
Effect of enhancement on transcription versus on voice activity detection, per engine.
EngineTranscriptionTurn-taking (Onset)
WER raw → enhΔWERIns raw → enhF1 raw → enhFA Babble 5 dBFA Competing 5 dB
ai-coustics Quail L14.3→15.2+0.86103→1280.948→0.94398→9658→58
anecho.ai_focus_model_16khz_v8_214.3→16.6+2.27103→1190.948→0.94198→9258→53
ai-coustics Quail VF 2.2 L14.3→17.6+3.27103→1630.948→0.94598→7558→37
Resample-only 16→8→16 (control)14.8→18.6+3.86101→1730.950→0.94898→9758→55
GTCRN14.3→20.1+5.82103→2230.948→0.94598→9258→57
FastEnhancer-S14.3→20.3+6.01103→1730.948→0.94398→8558→56
FastEnhancer-L14.3→20.4+6.04103→1710.948→0.94198→6858→39
FastEnhancer-B14.3→20.6+6.32103→1630.948→0.94398→8758→56
FastEnhancer-T14.3→21.7+7.34103→1460.948→0.94098→8958→57
FastEnhancer-M14.3→22.1+7.81103→2340.948→0.94098→7558→48
DeepFilterNet3 (96-frame warm-up)14.3→30.7+16.37103→3750.948→0.91498→9358→57
DeepFilterNet314.3→34.0+19.72103→3810.948→0.90198→9258→57

Every engine we tested raised word error rate. Several of them cut the VAD false-alarm rate by twenty to thirty points in the conditions that actually break turn-taking. That is not a contradiction — it is two different consumers with two different requirements, and it is why the SDK emits two channels instead of picking a winner on your behalf.

Raw → your transcriberEnhanced → Onset

Meas · provenance

One machine, one recogniser, one test set.

Exactly which, and where that limits the conclusion.

Run configuration
Harness configuration for this run
RecogniserCTranslate2 int8 CPU, greedy, model=base.en
Voice activitysilero-vad · threshold 0.5 · 512-sample blocks
Test setv1 · 10 speakers · 190 clips · 1485 s
ConditionsClean, Noise -5 dB, Noise 0 dB, Noise 5 dB, Noise 10 dB, Noise 20 dB, Babble 5 dB, Competing speaker 0 dB, Competing speaker 5 dB, Reverb, Reverb + noise 10 dB, Telephony 8 kHz, Telephony + noise 10 dB, telephony_alaw, telephony_g722, telephony_opus12k, telephony_loss3, telephony_loss10, Competing speaker 10 dB
Seed20260101
HostApple M1 Max · macOS-26.5.2-arm64-arm-64bit
Runtimeonnxruntime 1.28.0 · 1 intra-op thread
The harness’s own caveats
  • DeepFilterNet3 is measured through a block-online adapter: every public ONNX export of it lacks recurrent state tensors, so per-frame streaming is impossible with the published weights. Its RTF here is an upper bound and its latency (100 ms) is far above the ~40 ms a native (Rust/tract) runtime achieves. Read the DFN3 rows as a lower bound on quality and an upper bound on cost.
  • ai-coustics rows are a proprietary baseline run through the licensed aic-sdk. They are a reference point, never a shipping code path.
  • WER is measured with faster-whisper:base.en (CTranslate2 int8 CPU, greedy, model=base.en). A different recogniser will give different absolute numbers; the harness supports Deepgram and AssemblyAI so the *ordering* can be checked on another engine.
  • DeepFilterNet3 runs at 48 kHz on 16 kHz source material upsampled to 48 kHz — there is no real content above 8 kHz for it to work with. That is the honest telephony/VoIP situation, but it is not the condition DFN3 was designed for.
  • The test set is 10 speakers / ~190 reference words per condition. That is enough to separate large effects and not enough to resolve differences of a point or two of WER.
  • Run-to-run repeatability of the recogniser was measured, not assumed — and the standing warning did not reproduce here. Elsewhere in this project faster-whisper on CPU has been seen to return different transcripts for byte-identical input (three runs of one aggregate gave 121.65 / 122.15 / 121.40%), and the standing guidance from that is to treat differences below roughly 6 pp on a small aggregate as decoder noise. On this host and this harness it did not happen: 25/25 scopes were bit-identical across repeated decodes (raw on 18 condition(s), 180 clips x 3 decodes; clearline on 5 condition(s), 50 clips x 3 decodes), including the arms where the recogniser is already hallucinating past 100% WER. Independently, re-measuring `raw` and `gtcrn` in a fresh process two days after the published run reproduced all 54 rows exactly (max ΔWER 0.0000 pp). Three repeats cannot prove determinism and a different host, venv or thread count may well behave differently, so the conservative 6 pp guidance still governs anything quoted from another machine — but on this table the resolution limit is set by the size of the test set, not by the decoder: one reference word is 0.53 pp in a single condition (190 words), and about 0.03 pp pooled across all 18. Differences of a point or two between adjacent rows are still not results; differences of tens of points are.
  • onnxruntime is pinned to 1 intra-op thread with sequential execution: the measured cost is one stream on one core, not a fan-out across the whole machine.
  • The `telephony*` conditions in this table are stored at 16 kHz after a full 8 kHz carrier leg (band-limit, codec, packet loss), so every backend sees them at its own native rate. The genuinely narrowband experiment — where nothing is resampled — is a separate artefact (`bench/out/telephony.json`).
  • Packet loss is modelled as bursty (2-state Gilbert-Elliott) erasure of 20 ms frames with attenuated-repeat concealment, not i.i.d. loss with silence fill. i.i.d. loss flatters every decoder and silence fill overstates the damage.
Rack · engines under test

What is on the bench, and what you are allowed to ship.

Licence is a first-class column: the engine that comes closest cannot legally go into your product — the closest one you can ship is ours.

Engines under test and their licences
EngineLicenceShippableRTFLatencyPooled WER
Raw (no processing)—permitted—0 ms14.32%
Passthrough (control)—permitted0.0000 ms14.32%
ai-coustics Quail Lproprietarybenchmark only0.12330 ms15.18%
anecho.ai_focus_model_16khz_v8_2proprietary — ourspermitted0.06715 ms16.59%
ai-coustics Quail VF 2.2 Lproprietarybenchmark only0.07530 ms17.59%
GTCRNMITpermitted0.05016 ms20.14%
FastEnhancer-SMITpermitted0.03416 ms20.33%
FastEnhancer-LMITpermitted0.37726 ms20.36%
FastEnhancer-BMITpermitted0.02116 ms20.64%
FastEnhancer-TMITpermitted0.01316 ms21.66%
FastEnhancer-MMITpermitted0.11022 ms22.13%
DeepFilterNet3 (96-frame warm-up)MIT OR Apache-2.0permitted0.226100 ms30.69%
DeepFilterNet3MIT OR Apache-2.0permitted0.133100 ms34.04%
Narrowband run · 8 kHz end to end · G.711 µ-law, 300–3400 Hz

What happens when the audio never leaves the phone line.

Wideband engines are measured here on narrowband input, which is what running one on a call actually requires.

11.1% WER
Untouched 8 kHz caller audioThe null hypothesis · 60 clips, 1,140 reference words · faster-whisper:base.en
−0.18pts
Best path that exists todayresample only (soxr_vhq) — 2 words out of 1,140. A tie, not a win.
+13.7pts
Worst path we measuredfastenhancer-t via soxr_vhq — a wideband model handed a 3.4 kHz signal through a resample wrapper.

Retracted · we used to say something else

We used to lead with a community report attributing a large fixed loss of every caller's audio to the default frame-processor resample pattern. Then we built the control, and our own measurement does not support it: 8→16→8 kHz with a deliberately broken sample-repeat resampler scores 150 dB SI-SDR against its input — at an exact 2:1 ratio it is algebraically the identity — while a correct polyphase resampler scores 38 dB, lower, because it actually filters. We are not making that claim any more.

If being the referee means anything, it means publishing the call that goes against you.

The control, the table, and what we got wrong
Live conversations · v8_2 vs ai-coustics Quail VF 2.2 S · constructed mixes

Live conversations: two recognisers, and the condition decides who leads.

200 constructed 30–60 s conversations — real voices and real rooms, mixed by us — each streamed whole through the engine with no reset inside it, then scored twice: by faster-whisper base.en, and by Gemini Live input transcription, the transcript a live Gemini voice agent produces of the caller.

With faster-whisper base.en: raw 37.82 %, v8_2 38.46 %, VF S 38.59 % WER pooled; v8_2 − VF S = −0.13 [−4.56, +4.47] pp (parity, CI spans 0); no engine beats raw on the pooled set.

With Gemini Live input transcription: raw 40.50 %, v8_2 36.40 %, VF S 42.87 % WER pooled; v8_2 − VF S = −6.47 [−9.89, −3.26] pp (v8_2 ahead); v8_2 beats raw on the pooled set. VF S still leads with a distant talker at −10 dB (+7.74 [+2.48, +12.77] pp).

On turn-taking the result goes against us: under Gemini Live input transcription, VF S triggers fewer false barge-ins — 52 against 125 for v8_2 and 97 for raw — and fewer false responses (37 / 56 / 49; v8_2 − VF S = +0.10 [+0.01, +0.18] per conversation). v8_2 triggers more false barge-ins than the unprocessed audio. VF S also holds the pauses quieter (17.4 dB of background suppression against 8.5 dB).

Recogniser 1 · faster-whisper base.en

faster-whisper base.en

Offline, open-weights, the recogniser of the main matrix.

−0.13pp WER
Pooled, v8_2 − VF S95 % CI [−4.56, +4.47] · parity (CI spans 0) · 200 conversations
−6.60pp WER
Close-mic target (headset / phone)95 % CI [−13.36, −0.06] · v8_2 ahead
+9.58pp WER
Target 0.5–1.5 m from the mic95 % CI [+5.52, +13.90] · VF S ahead
0of 2
Engines that beat raw, pooledraw 37.82 % · v8_2 38.46 % · VF S 38.59 % WER
Word error rate by condition · faster-whisper base.en

Near · background −10 dB

40 conv · 3,654 words · 1 word = 0.027 pp

Near · background −10 dB: WER, insertions, deletions and background-word leak per engine
EngineWER %Ins %Del %Leak /100
raw14.708.211.752.33
v8_210.863.422.160.71
ai-coustics Quail VF 2.2 S8.891.701.920.25

Near · background −5 dB

40 conv · 3,654 words · 1 word = 0.027 pp

Near · background −5 dB: WER, insertions, deletions and background-word leak per engine
EngineWER %Ins %Del %Leak /100
raw21.6510.733.562.85
v8_224.0012.103.071.01
ai-coustics Quail VF 2.2 S27.048.269.250.49

Near · background 0 dB

40 conv · 3,654 words · 1 word = 0.027 pp

Near · background 0 dB: WER, insertions, deletions and background-word leak per engine
EngineWER %Ins %Del %Leak /100
raw38.7020.422.574.16
v8_237.1916.614.572.63
ai-coustics Quail VF 2.2 S55.944.8428.411.01

Near, pooled (3 conditions)

120 conv · 10,962 words · 1 word = 0.009 pp

Near, pooled (3 conditions): WER, insertions, deletions and background-word leak per engine
EngineWER %Ins %Del %Leak /100
raw25.0113.122.633.11
v8_224.0210.713.271.45
ai-coustics Quail VF 2.2 S30.624.9413.190.58

Room · background −10 dB

40 conv · 3,654 words · 1 word = 0.027 pp

Room · background −10 dB: WER, insertions, deletions and background-word leak per engine
EngineWER %Ins %Del %Leak /100
raw53.1729.642.195.01
v8_249.5614.705.751.51
ai-coustics Quail VF 2.2 S40.0710.107.011.01

Room · background −5 dB

40 conv · 3,654 words · 1 word = 0.027 pp

Room · background −5 dB: WER, insertions, deletions and background-word leak per engine
EngineWER %Ins %Del %Leak /100
raw60.8627.915.915.94
v8_270.6921.7312.152.82
ai-coustics Quail VF 2.2 S61.0312.9715.961.94

Room, pooled (2 conditions)

80 conv · 7,308 words · 1 word = 0.014 pp

Room, pooled (2 conditions): WER, insertions, deletions and background-word leak per engine
EngineWER %Ins %Del %Leak /100
raw57.0228.784.055.47
v8_260.1318.218.952.16
ai-coustics Quail VF 2.2 S50.5511.5411.481.48

All, pooled (5 conditions)

200 conv · 18,270 words · 1 word = 0.005 pp

All, pooled (5 conditions): WER, insertions, deletions and background-word leak per engine
EngineWER %Ins %Del %Leak /100
raw37.8219.383.204.06
v8_238.4613.715.541.74
ai-coustics Quail VF 2.2 S38.597.5812.510.94
WER difference, pp · 95 % CI · faster-whisper base.en
Paired WER differences with 95 % confidence intervals, faster-whisper base.en
Conditionv8_2 − VF SReadingv8_2 − rawVF S − raw
Near · background −10 dB+1.97 [−0.54, +4.63]parity (CI spans 0)−3.83 [−7.47, −0.83]−5.80 [−10.22, −2.01]
Near · background −5 dB−3.04 [−19.48, +14.99]parity (CI spans 0)+2.35 [−5.92, +16.16]+5.39 [−4.56, +19.71]
Near · background 0 dB−18.75 [−27.75, −9.90]v8_2 ahead−1.51 [−9.31, +4.89]+17.24 [+6.01, +27.97]
Near, pooled (3 conditions)−6.60 [−13.36, −0.06]v8_2 ahead−0.99 [−5.24, +4.14]+5.61 [±0.00, +11.76]
Room · background −10 dB+9.50 [+4.31, +14.84]VF S ahead−3.61 [−17.42, +6.45]−13.11 [−26.20, −4.22]
Room · background −5 dB+9.66 [+3.07, +16.81]VF S ahead+9.82 [+3.09, +17.40]+0.16 [−5.86, +6.02]
Room, pooled (2 conditions)+9.58 [+5.52, +13.90]VF S ahead+3.11 [−4.55, +9.66]−6.47 [−13.60, −0.71]
All, pooled (5 conditions)−0.13 [−4.56, +4.47]parity (CI spans 0)+0.65 [−3.38, +4.73]+0.78 [−3.60, +5.13]
Recogniser 2 · Gemini Live input transcription

Gemini Live input transcription

The input transcription of a live voice agent: each output streamed (real time, 100 ms PCM16 frames) into a Gemini Live session, model gemini-live-2.5-flash-native-audio.

−6.47pp WER
Pooled, v8_2 − VF S95 % CI [−9.89, −3.26] · v8_2 ahead · 200 conversations
−12.27pp WER
Close-mic target (headset / phone)95 % CI [−16.84, −7.91] · v8_2 ahead
+2.23pp WER
Target 0.5–1.5 m from the mic95 % CI [−1.64, +6.08] · parity (CI spans 0)
1of 2
Engines that beat raw, pooledraw 40.50 % · v8_2 36.40 % · VF S 42.87 % WER · beats raw: v8_2
Word error rate by condition · Gemini Live input transcription

Near · background −10 dB

40 conv · 3,654 words · 1 word = 0.027 pp

Near · background −10 dB: WER, insertions, deletions and background-word leak per engine
EngineWER %Ins %Del %Leak /100
raw21.762.8513.360.82
v8_214.401.238.100.16
ai-coustics Quail VF 2.2 S17.430.7711.710.05

Near · background −5 dB

40 conv · 3,654 words · 1 word = 0.027 pp

Near · background −5 dB: WER, insertions, deletions and background-word leak per engine
EngineWER %Ins %Del %Leak /100
raw23.974.1913.271.23
v8_218.943.867.530.74
ai-coustics Quail VF 2.2 S24.820.8817.100.05

Near · background 0 dB

40 conv · 3,654 words · 1 word = 0.027 pp

Near · background 0 dB: WER, insertions, deletions and background-word leak per engine
EngineWER %Ins %Del %Leak /100
raw39.714.8423.041.07
v8_233.065.3416.531.34
ai-coustics Quail VF 2.2 S60.950.9348.710.08

Near, pooled (3 conditions)

120 conv · 10,962 words · 1 word = 0.009 pp

Near, pooled (3 conditions): WER, insertions, deletions and background-word leak per engine
EngineWER %Ins %Del %Leak /100
raw28.483.9616.561.04
v8_222.133.4810.720.75
ai-coustics Quail VF 2.2 S34.400.8625.840.06

Room · background −10 dB

40 conv · 3,654 words · 1 word = 0.027 pp

Room · background −10 dB: WER, insertions, deletions and background-word leak per engine
EngineWER %Ins %Del %Leak /100
raw49.671.8935.960.41
v8_251.893.0928.460.33
ai-coustics Quail VF 2.2 S44.142.6325.040.33

Room · background −5 dB

40 conv · 3,654 words · 1 word = 0.027 pp

Room · background −5 dB: WER, insertions, deletions and background-word leak per engine
EngineWER %Ins %Del %Leak /100
raw67.382.1152.520.47
v8_263.714.7638.150.57
ai-coustics Quail VF 2.2 S67.002.5244.250.30

Room, pooled (2 conditions)

80 conv · 7,308 words · 1 word = 0.014 pp

Room, pooled (2 conditions): WER, insertions, deletions and background-word leak per engine
EngineWER %Ins %Del %Leak /100
raw58.522.0044.240.44
v8_257.803.9333.310.45
ai-coustics Quail VF 2.2 S55.572.5734.650.31

All, pooled (5 conditions)

200 conv · 18,270 words · 1 word = 0.005 pp

All, pooled (5 conditions): WER, insertions, deletions and background-word leak per engine
EngineWER %Ins %Del %Leak /100
raw40.503.1727.630.80
v8_236.403.6619.750.63
ai-coustics Quail VF 2.2 S42.871.5429.370.16
WER difference, pp · 95 % CI · Gemini Live input transcription
Paired WER differences with 95 % confidence intervals, Gemini Live input transcription
Conditionv8_2 − VF SReadingv8_2 − rawVF S − raw
Near · background −10 dB−3.04 [−10.83, +5.06]parity (CI spans 0)−7.36 [−11.99, −3.02]−4.32 [−12.81, +3.82]
Near · background −5 dB−5.88 [−11.15, −1.18]v8_2 ahead−5.04 [−9.82, −0.61]+0.85 [−5.37, +7.43]
Near · background 0 dB−27.89 [−35.80, −19.85]v8_2 ahead−6.65 [−11.20, −2.00]+21.24 [+14.22, +28.70]
Near, pooled (3 conditions)−12.27 [−16.84, −7.91]v8_2 ahead−6.35 [−9.11, −3.74]+5.92 [+1.47, +10.57]
Room · background −10 dB+7.74 [+2.48, +12.77]VF S ahead+2.22 [−2.62, +7.30]−5.53 [−11.51, +0.06]
Room · background −5 dB−3.28 [−8.07, +1.40]parity (CI spans 0)−3.67 [−8.01, +0.83]−0.38 [−4.72, +3.89]
Room, pooled (2 conditions)+2.23 [−1.64, +6.08]parity (CI spans 0)−0.73 [−4.04, +2.72]−2.96 [−6.62, +0.63]
All, pooled (5 conditions)−6.47 [−9.89, −3.26]v8_2 ahead−4.10 [−6.28, −1.98]+2.37 [−0.92, +5.54]

WER, insertions and deletions as % of reference words. Leak: inserted words that also occur in the transcribed background talker's text (stop-words excluded, count-capped), per 100 reference words; untranscribed MUSAN talkers cannot be attributed, so this is a lower bound. CIs: paired bootstrap over conversations, 95 %. Negative means fewer errors for the first engine named; a difference whose interval spans zero is read as parity, never as a win.

Turn-taking · Gemini Live input transcription
Agent responses and barge-ins per engine
Conditionraw: responses / false · barge-ins / falsev8_2: responses / false · barge-ins / falseVF S: responses / false · barge-ins / false
Near · background −10 dB197 / 8 · 215 / 19210 / 8 · 214 / 24202 / 7 · 198 / 1
Near · background −5 dB208 / 12 · 228 / 23191 / 11 · 229 / 29204 / 4 · 204 / 7
Near · background 0 dB207 / 14 · 218 / 19185 / 10 · 218 / 22171 / 11 · 156 / 11
Near, pooled (3 conditions)612 / 34 · 661 / 61586 / 29 · 661 / 75577 / 22 · 558 / 19
Room · background −10 dB206 / 8 · 161 / 21217 / 13 · 203 / 28212 / 5 · 198 / 12
Room · background −5 dB158 / 7 · 117 / 15198 / 14 · 174 / 22181 / 10 · 167 / 21
Room, pooled (2 conditions)364 / 15 · 278 / 36415 / 27 · 377 / 50393 / 15 · 365 / 33
All, pooled (5 conditions)976 / 49 · 939 / 971001 / 56 · 1038 / 125970 / 37 · 923 / 52
False responses per conversation, paired differences
False responses / conversationv8_2 − VF Sv8_2 − rawVF S − raw
Near · background −10 dB+0.03 [−0.12, +0.17]±0.00 [−0.15, +0.12]−0.03 [−0.17, +0.15]
Near · background −5 dB+0.17 [−0.03, +0.38]−0.03 [−0.17, +0.15]−0.20 [−0.40, ±0.00]
Near · background 0 dB−0.03 [−0.17, +0.12]−0.10 [−0.30, +0.07]−0.07 [−0.25, +0.10]
Near, pooled (3 conditions)+0.06 [−0.05, +0.16]−0.04 [−0.13, +0.05]−0.10 [−0.21, +0.01]
Room · background −10 dB+0.20 [±0.00, +0.40]+0.12 [−0.07, +0.33]−0.07 [−0.23, +0.07]
Room · background −5 dB+0.10 [−0.12, +0.30]+0.17 [±0.00, +0.35]+0.07 [−0.12, +0.28]
Room, pooled (2 conditions)+0.15 [±0.00, +0.30]+0.15 [+0.01, +0.29]±0.00 [−0.12, +0.12]
All, pooled (5 conditions)+0.10 [+0.01, +0.18]+0.04 [−0.04, +0.12]−0.06 [−0.14, +0.02]
  • False response: the agent started a reply with no target speech in the previous 3 s.
  • False barge-in: Gemini Live reported `interrupted` with no target speech in the previous 1.5 s.
  • Counts are totals over the conversations in each row. Only false responses carry a paired CI in the report; the barge-in counts are reported as counts.
Signal metrics · what each engine does to the audio
Background suppression, target fidelity and onset loss
ConditionSupp. v8_2Supp. VF SSI-SDR rawSI-SDR v8_2SI-SDR VF SOnset v8_2Onset VF S
Near · background −10 dB10.6 dB30.1 dB12.0614.508.87−11.36 dB−22.82 dB
Near · background −5 dB7.7 dB23.7 dB7.1810.846.38−13.14 dB−24.84 dB
Near · background 0 dB6.9 dB11.6 dB2.226.74-0.24−13.60 dB−21.09 dB
Near, pooled (3 conditions)8.4 dB21.8 dB7.1510.695.00−12.70 dB−22.92 dB
Room · background −10 dB9.7 dB14.1 dB3.373.383.21−16.04 dB−23.17 dB
Room · background −5 dB7.6 dB7.5 dB1.601.991.35−16.26 dB−23.23 dB
Room, pooled (2 conditions)8.7 dB10.8 dB2.482.692.28−16.15 dB−23.20 dB
All, pooled (5 conditions)8.5 dB17.4 dB5.297.493.91−14.08 dB−23.03 dB
  • Supp.: raw minus output level on background-voice-alone segments inside the target's pauses (dB, mean over conversations).
  • SI-SDR: SI-SDR on target-active frames (mean dB); higher means less damage to the target.
  • Onset: energy change in the first 200 ms of the target's speech after a pause, relative to the rest of that run (dB; negative = onset attenuated).

The trade in one line: VF S silences more — 17.4 dB against 8.5 dB in the pauses — and pays for it at the target’s first syllable after a pause (−23.03 dB against −14.08 dB) and in target fidelity (SI-SDR 3.91 against 7.49; raw 5.29). Both engines attenuate onsets; neither is free.

Method
  • Constructed, not recorded end to end. Digitally constructed from real recordings: real voices and real measured rooms, mixed by us. Not recorded end to end. 40 conversation scripts, each mixed under 5 conditions = 200 conversations, 30.4–59.8 s each, seed 20261021.
  • Target: one LibriSpeech test-clean speaker per script, 4–8 consecutive utterances of one chapter with natural pauses (0.3–2.5 s) and at least one 3–6 s listening gap.
  • Background: 1–3 other real talkers (LibriSpeech dev-clean, MUSAN speech) who come and go with their own pauses, placed through measured BUT ReverbDB room responses at 1–4 m; in half of the scripts, intermittent real ambience (DEMAND, MUSAN noise) at -20 dB re the target.
  • Conditions: near — close-mic target (headset / phone), background talkers at −10, −5, 0 dB; room — target through a measured response at 0.5–1.5 m, background at −10, −5 dB.
  • Streaming: each conversation runs whole through the engine with no reset inside it; engines reset between conversations; output is delay-compensated by each engine’s declared latency (v8_2 15 ms, VF S 30 ms).
  • ai-coustics Quail VF 2.2 S: licensed SDK at its native 16 kHz, SDK default enhancement level (0.8). Benchmark only, never a code path.
  • Scoring: faster-whisper base.en on the whole conversation, and Gemini Live input transcription with each output streamed in real time into a live session; both against the target’s transcript; paired bootstrap over conversations, 95 %.
Caveats
  • The mixes are digitally constructed from real recordings. They are not recorded end to end, and a real call adds things a mix does not: codecs, AGC, the talker reacting to the room.
  • Two recognisers, two answers: faster-whisper base.en reads the pooled set as parity, Gemini Live input transcription as a v8_2 lead. Neither is the truth for your stack; your own recogniser decides.
  • 40 target speakers, one script each; the five conditions re-mix the same scripts, so they are not independent samples.
  • Raw WER is high by design (overlapping talkers). Insertions dominate raw; the engines trade insertions for deletions, and the pooled totals hide which error you get.
  • Leak counts only words attributable to the transcribed background talker — a lower bound.
  • ai-coustics Quail VF 2.2 S ran at one setting, the SDK default; another enhancement level would move its trade between suppression and deletions.
The full write-up
Verdict · what to do with this

Ignore pooled claims. Including any we might publish.

A pooled number is a property of the condition mix as much as of the engine, and no vendor shares a mix. Judge on your own voice.

Hear the shipping engine on your own voiceRaw results (JSON, CC-BY-4.0)
Take the data

The complete results: one JSON document at a stable, versioned URL, CC-BY-4.0./data/nulltest/v1/results.json

Challenge it

The harness is private because it embeds a licensed competitor SDK. Think a row is wrong, or your engine belongs on the bench? Write to [email protected] — we run it and publish the row, whichever way it lands.

Rules we hold ourselves to
The control is a row
Unprocessed audio is the first row of every table and every ΔWER is measured against it — here the null hypothesis won.
The same code scores everyone
One scoring path, nothing special-cased, and the result is published even when it reads “nothing won”.
Pooled and per-condition are reported separately
Both are published and neither may stand in for the other, whichever makes the better headline.
Error decomposition, not just a total
Substitutions, insertions and deletions are reported separately — inserted chatter and a dropped “not” are different failures.
A one-word difference is not a result
Any delta inside a single reference word is reported as a number and described as noise, never as a win.
Cost is reported next to accuracy
Real-time factor and latency sit in the same table as WER; an engine that misses the frame deadline has not won.
The proprietary baseline never ships
ai-coustics rows are a licensed measuring stick, never a code path in anything we sell.