Null Test v0.1: ten speech enhancers, eighteen conditions, and nothing beat raw on the pooled average

Updated 14 August 2026: the matrix was regenerated with five additional telephony carrier conditions, taking it from thirteen to eighteen. Absolute numbers moved; the ordering did not. Every figure below is quoted from the published benchmark.json.

Today we are publishing Null Test v0.1 — the harness, the dataset manifest, the configs, the rules, and the first full set of results. It measures what happens to speech-to-text accuracy when you put a speech enhancement engine in front of a recognizer, across eighteen acoustic conditions, against a null-hypothesis control that is allowed to win.

On the pooled numbers, it won — and the per-condition breakdown underneath that is where the actual engineering lives.

A null test is the canonical audio-engineering measurement: subtract the processed signal from the reference and whatever remains is exactly what the processor did — including the damage. That is the whole ambition of this project, so it is the name.

Why this exists

The public claims are irreconcilable. Krisp markets roughly a 46% WER reduction. ai-coustics markets roughly 43%. An independent study (arXiv:2512.17562) found enhanced audio scored worse than raw in all 40 configurations it tested — though only with MetricGAN+, on medical speech, in semantic WER. AssemblyAI, citing it, separately reported that Krisp's output roughly doubled WER when fed to its STT while cutting false VAD triggers about 3.5x. Deepgram has published the position that enhancement hurts accuracy, and we have not found accompanying data. Nobody shares a corpus, a condition set, a configuration, or a control. The long version of that disagreement is in Does noise suppression actually help speech-to-text?.

What was run

Recogniser	faster-whisper base.en, CTranslate2 int8 CPU, greedy decoding
VAD	Silero VAD, threshold 0.5, 512-sample blocks
Test set	10 speakers, 180 clips, roughly 1406 seconds of audio, seed 20260101
Reference words	about 190 per condition, 3420 total per backend
Conditions	18
Backends	12 (raw, passthrough control, and ten enhancers)
Host	Apple M1 Max, onnxruntime 1.28.0, pinned to 1 intra-op thread

The eighteen conditions: clean; broadband noise at -5, 0, 5, 10 and 20 dB SNR; babble_snr5; competing speaker at 0 and 5 dB; reverb and reverb_snr10; and seven telephony conditions — telephony (a G.711 µ-law round trip), telephony_snr10, plus A-law, G.722, Opus at 12 kbps, and bursty packet loss at 3% and 10%.

The ten enhancers: GTCRN (MIT), five FastEnhancer variants T/S/B/M/L (MIT), DeepFilterNet3 in two configurations (MIT or Apache-2.0), and two ai-coustics models, Quail L and Quail VF 2.2 L, run through the licensed SDK as a proprietary reference baseline.

Our own models are not in this matrix. Chamber and Clearline will be rows in it, under the same rules, against the same control that just beat everybody. Publishing the referee before the contestant was the point of the ordering.

The results

Pooled across all eighteen conditions:

Backend	WER	Insertions	Deletions	RTF	Latency
Raw (no processing)	14.77%	101	60	—	0 ms
ai-coustics Quail L	15.56%	126	57	0.124	30 ms
ai-coustics Quail VF 2.2 L	18.10%	161	75	0.076	30 ms
GTCRN (MIT)	20.26%	210	40	0.050	16 ms
FastEnhancer-L (MIT)	20.44%	159	107	0.376	25.8 ms
FastEnhancer-M (MIT)	22.40%	217	83	0.110	22 ms
FastEnhancer-T (MIT)	22.46%	145	109	0.013	16 ms
DeepFilterNet3	35.18%	380	197	0.133	100 ms

No enhancer beat raw on pooled WER. The best was ai-coustics Quail L at 15.56% against raw's 14.77% — about eight tenths of a point worse, which on this sample size we would call a tie rather than a loss. Everything below it lost by five points or more, and that is not a tie.

Pooled answers "should enhancement be on by default" and almost nothing else; the per-condition sections below are the ones to act on, and they say something different.

Insertions went up, not down. Raw produced 101. GTCRN produced 210. DeepFilterNet3 produced 380. This is the direct opposite of the vendor claim that enhancement removes hallucinated words, and it is worth dwelling on: an insertion is not a garbled word, it is the recogniser emitting a word that nobody said. Our reading is that what a suppressor leaves behind — musical noise, gated silence, smeared transients — is more decodable-as-speech than the noise it removed.

GTCRN's decomposition makes the point: 210 insertions against only 40 deletions, where raw had 101 and 60. It is not eating your function words, it is adding words nobody spoke. Opposite failure modes, and one WER number hides which one you bought. That is why every cell publishes S, D and I separately.

Where the vendor claim does hold

Stopping at the pooled number would be doing exactly what we criticise the vendors for. Enhancement wins real margins in specific conditions — in 11 of our 18, at least one engine beat raw — and they are the conditions you would predict: where noise or a competing voice is genuinely destroying phonetic cues.

The clearest is speaker isolation. ai-coustics Quail VF is a Voice Focus model: its job is to isolate the primary speaker and reject other voices. On competing speaker at 5 dB it cut insertions from 19 to 10 and WER from 26.3% to 20.5%. On babble at 5 dB, 24.7% to 20.5% with insertions down 10 to 7. Those are the strongest results any enhancer posted here. Low-SNR broadband noise shows the same shape — at 0 dB, eight of the ten engines beat raw, the best by 4.7 points.

The seven conditions where nothing beat raw are just as informative: the two reverberant ones, and the five telephony carrier conditions. In four of those five the best an engine manages is an exact tie with doing nothing, which is its own kind of answer.

What none of that supports is the general claim. Pooled, Quail VF is more than three points worse than raw, because it also runs on clean, reverberant and narrowband material where it can only remove information. Enhancement pays where the audio is genuinely bad and costs you where it is not, and a single scalar — ours included — hides which side of that line your traffic sits on.

VAD: the metric everyone quotes barely moves, the one that matters moves a lot

The same run scored Silero VAD on every stream. Pooled VAD F1: 0.950 raw against 0.947 for the best enhanced stream. Grade enhancement on VAD F1 and you conclude it does nothing.

The false-alarm rate is a different story, and a false alarm is what a spurious barge-in actually is:

Condition	False-alarm rate, raw	Enhanced	Engine
Babble at 5 dB	97.7%	68.2%	FastEnhancer-L
Competing speaker at 5 dB	58.1%	37.2%	ai-coustics Quail VF

Under babble at 5 dB the raw stream false-alarms on essentially every non-speech frame — a VAD that has stopped working as a gate. Enhancement takes that to 68.2%, still bad and thirty points better.

So the same run says enhancement costs you five points of pooled WER and buys you a third off your false-alarm rate. That is not a contradiction, it is an architecture: enhanced audio to the turn detector, raw audio to the recogniser. We shipped that as the SDK default and the reasoning is in The split pipeline.

Telephony: no enhancer helped, and several hurt badly

On the G.711 µ-law round trip, raw is 5.8% WER. Against that: ai-coustics Quail L +1.6 points, Quail VF +2.6, GTCRN +3.2, FastEnhancer-T +6.8, DeepFilterNet3 +9.5. On the same round trip with noise at 10 dB, raw is 10.5% and the spread runs from +0.5 to +9.5.

No enhancer produced a meaningful improvement on telephony audio, and several caused large degradations. Two rows come out fractionally under raw and neither is a result: FastEnhancer-S is -0.5 on the clean row — at 190 reference words, inside a single word of doing nothing — and +3.7 on the noisy one; Quail VF is -1.1 on the noisy row and +2.6 on the clean one. Ties, in both directions.

The five carrier conditions added in this update say the same thing more bluntly. On A-law, G.722, Opus at 12 kbps and 3% bursty loss, the best any engine manages is an exact tie with raw; on 10% loss nothing reaches it. The worst cells are severe — FastEnhancer-M posts 49.0% on A-law against raw's 7.4%, with 61 insertions where raw has 2. Companded quantisation noise is outside what that model expects and it responds by hallucinating.

That matters because telephony is where the call volume is, and the shape of it is not an accident: the DNS Challenge, the benchmark series most published denoisers are tuned against, never had a narrowband track. The field optimised for 16 and 48 kHz because that is what was scored.

It matters with a large asterisk. Raw WER of 5.8% on the base telephony condition means it is an easy one, and not the thing that breaks production phone deployments — see the caveats. The full narrowband story, including a separate 8 kHz end-to-end experiment with native narrowband engines and a resample round-trip control, is in 8 kHz is where voice AI actually breaks. Its short version: native 8 kHz beats resample-and-hope by 0.7 points for the same vendor, and nothing at all beats the untouched caller audio.

Caveats, in full

A result without its limits is an advertisement. These attach to every number above.

• Single recogniser. faster-whisper base.en. A different recogniser gives different absolute numbers. The harness supports Deepgram and AssemblyAI so the ordering can be re-checked, and we would like someone who does not trust us to do it.
• About 190 reference words per condition, 3420 pooled. Enough to separate large effects, not enough to resolve one or two points of WER: one word is worth roughly 0.5 points per condition and 0.03 pooled. Read the raw-versus-Quail-L gap as a tie; the five-to-twenty point gaps below it are not.
• DeepFilterNet3 is measured through a block-online adapter. No public ONNX export carries recurrent state tensors, so per-frame streaming is impossible with the published weights. Its 35.18% is a lower bound on quality and an upper bound on cost, and its 100 ms latency is far above what a native Rust runtime achieves. Never quote the number without this sentence. It also runs at 48 kHz over 16 kHz material upsampled to 48 kHz — the honest VoIP situation, not the condition it was designed for.
• ai-coustics rows are a proprietary baseline run through the licensed SDK. A reference point, never a shipping code path for us.
• Licensing constrains reuse. ESC-50 noise is CC-BY-NC-3.0, so those conditions are evaluation only. LibriSpeech is CC-BY-4.0, and the babble and competing-speaker conditions are built from LibriSpeech alone, so they carry no NC restriction.
• onnxruntime pinned to 1 intra-op thread, sequential execution. The RTF column is one stream on one core, not a fan-out.
• English only, batch scoring, no listening panel, no turn-level metrics. Enhancement is driven block by block with state carried across blocks, but the recogniser is scored on complete utterances, and VAD is frame-level. "Fewer false barge-ins" as a product metric needs labelled turn boundaries and an agreed definition.
• The telephony conditions are more realistic than they were, and still not a phone call. They now include A-law and G.722 legs, Opus at 12 kbps, and bursty two-state packet loss at 3% and 10% with attenuated-repeat concealment — rather than independent random loss with silence fill, which flatters decoders and overstates damage respectively. Still missing: acoustic echo with non-linear speaker distortion, handset-side suppression, and AMR-WB or EVS legs. Echo is the gap we care most about, because it is the impairment behind the part of our own roadmap that cannot be closed with a patch.
• These eighteen conditions are stored at 16 kHz after an 8 kHz carrier leg, so every backend sees them at its own native rate. The genuinely narrowband experiment — 60 clips, 8 kHz throughout, nothing resampled, native 8 kHz engines included — is a separate artefact with its own resolution limit of 1,140 reference words, where one word is 0.09 points.

Methodology, in enough detail to attack

Streaming is simulated honestly. Every enhancer is driven block by block at its declared block size, carrying state across blocks, with no lookahead beyond its declared algorithmic latency. Offline whole-file processing flatters an enhancer and is not what a live agent gets. Latency is reported as algorithmic latency — frame plus lookahead — a property of the model, not of our laptop.

RTF is measured, not quoted. Single pinned core, warmup passes discarded, hardware string recorded in every row. A model that wins on WER at 4x real time is not a shipping option, so the accuracy number and the cost number travel together.

Noise is mixed, not sourced pre-mixed. Clean speech plus noise at a target SNR computed over active speech regions only, so leading and trailing silence does not distort the ratio. Reverberation is convolution with measured room impulse responses. Telephony is a real codec round trip — resample to 8 kHz, G.711 µ-law encode and decode, resample back — rather than a low-pass filter standing in for a channel. A real codec leg, and still not a real phone call; the caveat above lists what it leaves out.

Text normalization is fixed and published. WER is (S + D + I) / N after one normalizer applied identically to every hypothesis and reference: case folding, punctuation removal, number expansion, contraction handling. Normalization can move WER by multiple points, so it is versioned in the repo and recorded in every row.

Determinism, and per-utterance outputs. Fixed seeds for noise selection, SNR draw and RIR selection; the dataset manifest is content-addressed with SHA-256 per file, verified on fetch, so the same config on the same commit produces the same rows. Every run also emits per-utterance hypotheses and references alongside the aggregates, so anyone can recompute, slice differently, or find the ten files carrying a result. Aggregate-only benchmarks cannot be checked, which is most of how the field got here.

The rules that make it a referee

1. The raw control is always in the matrix and is allowed to win. In v0.1 it won.
2. We score ourselves under the same rules, in the same tables. Chamber and Clearline will be rows, not a separate marketing page.
3. Every cell publishes, including the ones we lose. No condition gets dropped after we see the result.
4. Configs are in the repo. If a vendor believes we configured their engine badly, the fix is a pull request against a YAML file, and we rerun and republish.
5. Right of reply. Any maintainer or vendor can request a rerun at a different setting. We publish both.
6. Licensing is stated per row. Where a vendor's terms prohibit publishing benchmark results, we say the cell is blocked and name the reason rather than quietly omitting it.

The output format

The full document is at /benchmark. One backend's summary:

{
  "id": "gtcrn",
  "label": "GTCRN",
  "license": "MIT",
  "wer": 0.2026,
  "werSubstitutions": 443,
  "werInsertions": 210,
  "werDeletions": 40,
  "werRefWords": 3420,
  "vadF1": 0.9472,
  "vadFalseAlarm": 0.1638,
  "rtf": 0.0498,
  "latencyMs": 16.0
}

Each backend also carries a conditions object with the same decomposition per condition — where the interesting disagreements live.

Running it yourself

git clone https://github.com/anecho/nulltest
cd nulltest
uv sync

# fetch and verify the dataset against the content-addressed manifest
python -m nulltest.data fetch --manifest data/manifest.json

# one condition, two backends, one recogniser — a few minutes on a laptop
python -m nulltest.run \
  --condition babble_snr5 \
  --backend gtcrn --backend raw \
  --stt faster-whisper:base.en \
  --out out/

python -m nulltest.report out/ --format md

Adding an engine is one file. The benchmark only ever talks to this interface:

class Enhancer(Protocol):
    id: str
    sample_rate: int
    block_size: int              # samples per process() call
    algorithmic_latency_ms: float

    def reset(self) -> None: ...

    def process(self, block: np.ndarray) -> np.ndarray:
        """block: float32 mono, shape (block_size,), range -1 to 1. Returns same shape."""

If your engine implements reset and process, it can be in the matrix. We would particularly like the vendors whose published claims we quoted to submit their own adapters and configurations.

What happens next, and the commercial part

v0.2 adds turn-level endpointing metrics via Onset, streaming STT with latency-to-final, a second and third recogniser so the ordering can be checked rather than trusted, multi-language conditions, and an echo condition with non-linear speaker distortion. Chamber and Clearline enter the matrix when they have something worth scoring.

On that last point, plainly, because a benchmark run by a vendor is worth nothing if the vendor is coy about its own progress: Clearline's first proof of concept has trained and converged, and it is weak. It reaches +1.41 dB SI-SDR over the unprocessed mixture on 1,200 held-out validation mixtures, where competent target-speaker-extraction systems reach +8 to +12 dB. It has no WER number because it is not yet worth scoring through this harness, and it will not appear in any table above until it is. Architecture experiments are running.

The commercial part, stated once and without decoration: this benchmark is not the product. A benchmark has no buyer and no retention — nobody renews a leaderboard. What we intend to sell is Clearline, a telephony audio path: native 8 kHz primary-speaker isolation, residual echo suppression, per minute, with an instant API key and no sales call. Note exactly what v0.1 licenses us to say about that. It says nothing on the shelf helps on a narrowband channel, and that nobody ships speaker isolation at 8 kHz at all. It does not say ours will be better — that number does not exist until Clearline is a row here, on telephony conditions worth the name, against a control allowed to beat it. Null Test is how we found out what to build, and how you check whether we are lying about it.

Harness and issues: github.com/anecho. Full matrix: /benchmark. Quickstart: /docs. If you want a condition, an engine or a recogniser added, open an issue — including if you expect it to make us look bad.