How it was measured, and what that does not license you to say.

13 audio backends — 11 enhancement engines plus an unprocessed control and a passthrough sanity row — were scored across 18 acoustic conditions on a test set of 180 clips (10 speakers, 1406 seconds of audio, seed 20260101).

Transcription is CTranslate2 int8 CPU, greedy, model=base.en. Voice activity is silero-vad at threshold 0.5 on 512-sample blocks. Cost is measured on Apple M1 Max (macOS-26.5.2-arm64-arm-64bit) with onnxruntime 1.28.0 pinned to 1 intra-op thread and sequential execution, so every real-time factor here is one stream on one core, not a fan-out across the machine.

The control is a row, not a footnote. Unprocessed audio appears in every table and every delta is measured against it in the same condition. A benchmark without a null hypothesis is a brochure.

• Clean
• Noise -5 dB
• Noise 0 dB
• Noise 5 dB
• Noise 10 dB
• Noise 20 dB
• Babble 5 dB
• Competing speaker 0 dB
• Competing speaker 5 dB
• Reverb
• Reverb + noise 10 dB
• Telephony 8 kHz
• Telephony + noise 10 dB
• telephony_alaw
• telephony_g722
• telephony_opus12k
• telephony_loss3
• telephony_loss10

If you quote one number from this benchmark, quote it with the condition attached. Pooled WER across a fixed condition mix is a property of the mix as much as of the engine: weight the mix toward clean and lightly-noisy speech and enhancement looks like a tax; weight it toward competing speakers and low SNR and enhancement looks like a win. Both readings come from the same file.

The specific misreading we would like to head off is "enhancement never helps". That is not what 18 conditions showed. No engine beat raw on the pooled average; at least one engine beat raw in 11 of the 18 individual conditions. Both sentences are true, and the second one is the actionable half — it tells you the conditions in which turning enhancement on is a good trade.

The mirror-image misreading is also wrong. Enhancement is not a free win in the conditions where it helped: it moved word error rate by a few points on a test set whose resolution is about half a point per word, on one recogniser, and it raised insertions everywhere. Measure it on your own audio and your own recogniser before you commit — the SDK ships the same alignment code this harness uses, precisely so you can.

Read these against the resolution limit below before treating any of them as a result: several are inside one or two reference words.

• Clean: raw 4.7% → 4.2% (−0.53 points, FastEnhancer-M); 1 of 11 engines beat raw here.
• Noise -5 dB: raw 18.4% → 13.7% (−4.74 points, FastEnhancer-L); 3 of 11 engines beat raw here.
• Noise 0 dB: raw 11.1% → 6.3% (−4.74 points, ai-coustics Quail L); 8 of 11 engines beat raw here.
• Noise 5 dB: raw 8.9% → 6.8% (−2.11 points, FastEnhancer-L); 4 of 11 engines beat raw here.
• Noise 10 dB: raw 6.3% → 5.3% (−1.05 points, FastEnhancer-L); 5 of 11 engines beat raw here.
• Noise 20 dB: raw 4.2% → 3.2% (−1.05 points, FastEnhancer-M); 2 of 11 engines beat raw here.
• Babble 5 dB: raw 24.7% → 20.5% (−4.21 points, ai-coustics Quail VF 2.2 L); 1 of 11 engines beat raw here.
• Competing speaker 0 dB: raw 84.7% → 79.5% (−5.26 points, ai-coustics Quail L); 2 of 11 engines beat raw here.
• Competing speaker 5 dB: raw 26.3% → 20.5% (−5.79 points, ai-coustics Quail VF 2.2 L); 1 of 11 engines beat raw here.
• Telephony 8 kHz: raw 5.8% → 5.3% (−0.53 points, FastEnhancer-S); 1 of 11 engines beat raw here.
• Telephony + noise 10 dB: raw 10.5% → 9.5% (−1.05 points, ai-coustics Quail VF 2.2 L); 1 of 11 engines beat raw here.

Reverberation is the clean failure case. None of these models are dereverberation models, and on the two reverberant conditions the best engine still lost to doing nothing. SI-SDR goes sharply negative there — down to around −17 dB — which is the metric telling you the processed signal has been moved a long way from the reference, not merely denoised.

• Reverb: raw 5.8%, best engine 6.8% (+1.05 points, ai-coustics Quail L). Nothing beat doing nothing.
• Reverb + noise 10 dB: raw 13.7%, best engine 14.7% (+1.05 points, GTCRN). Nothing beat doing nothing.
• telephony_alaw: raw 7.4%, best engine 7.4% (±0.00 points, ai-coustics Quail L). Nothing beat doing nothing.
• telephony_g722: raw 5.3%, best engine 5.3% (±0.00 points, FastEnhancer-L). Nothing beat doing nothing.
• telephony_opus12k: raw 6.8%, best engine 6.8% (±0.00 points, FastEnhancer-S). Nothing beat doing nothing.
• telephony_loss3: raw 11.1%, best engine 11.1% (±0.00 points, ai-coustics Quail VF 2.2 L). Nothing beat doing nothing.
• telephony_loss10: raw 10.0%, best engine 10.5% (+0.53 points, ai-coustics Quail L). Nothing beat doing nothing.

Each condition carries roughly 190 reference words (3,420 across all 18 conditions), so one word is about 0.53 points of word error rate. Any per-condition delta smaller than that is a single word and is not a finding — including when it flatters us.

This is why the telephony column is reported as "no meaningful improvement" rather than as a win: the best engine there lands −0.53 points from the control, which is inside one reference word, while the worst costs +10.00 points, which is not.

The test set is deliberately small enough to say so plainly: 10 speakers is enough to separate large effects and not enough to resolve a point or two of WER. Treat rank orderings between adjacent engines as unresolved.

Every backend is scored at its own native rate, so a narrowband engine is resampled 16→8→16 to meet this matrix. To keep that handicap accountable rather than rhetorical, the table carries a resample-only control — the same round trip with no model in it. On the Telephony 8 kHz rows the control scores 5.8%, which makes it an easy condition, and easy conditions are exactly where enhancement has the least to win and the most to lose.

Anecho's own engine is now in this table, and it loses. Pooled it scores well behind doing nothing, and the control is what makes that readable: on the wideband half the band limit costs about six points of word error rate and our model adds roughly thirty more, while on the narrowband half the band limit costs under a point and our model still adds six. It loses at its own operating point, to raw audio and to a piece of wire — the one exception being plain Telephony 8 kHz, where it lands on the control and ties doing nothing. We publish it because a benchmark that exempts its author measures nothing.

• DeepFilterNet3 is measured through a block-online adapter: every public ONNX export of it lacks recurrent state tensors, so per-frame streaming is impossible with the published weights. Its RTF here is an upper bound and its latency (100 ms) is far above the ~40 ms a native (Rust/tract) runtime achieves. Read the DFN3 rows as a lower bound on quality and an upper bound on cost.
• ai-coustics rows are a proprietary baseline run through the licensed aic-sdk. They are a reference point, never a shipping code path.
• WER is measured with faster-whisper:base.en (CTranslate2 int8 CPU, greedy, model=base.en). A different recogniser will give different absolute numbers; the harness supports Deepgram and AssemblyAI so the *ordering* can be checked on another engine.
• Clearline is an 8 kHz model and these conditions are stored at 16 kHz, so its row is measured through the harness's resample path — 16 k -> 8 k -> model -> 8 k -> 16 k (soxr), the same path the native-8 kHz ai-coustics baselines take. On `clean`, `noise_*`, `babble_snr5`, `competing_*` and `reverb*` that is not its operating point: the 4-8 kHz band is discarded before the model ever sees it. The `resample8k` control row is that round trip with no model in it, so the band limit and the model can be told apart; and the telephony sub-aggregate below reports the regime the model was actually built for.
• The Clearline checkpoint benchmarked here is not shippable, and this row is not a claim that it is. Two gates it does not pass, both measured on its own eval (`runs/t4_300h_200ep_solo`, 1,800-clip test_v2) and neither of them visible in this table: (1) a solo regression — on clips with no competing speaker the model makes the audio worse, -0.874 dB SI-SDR with 37.1% of solo clips degraded (-0.595 dB / 33.0% on val_v2); (2) speaker conditioning contributes approximately nothing — the matched ablation scores FiLM +2.815 dB against no-conditioning +2.690 dB, a 0.125 dB difference against a 0.3 dB noise floor, so what is being measured here is a plain denoiser and not the speaker isolation the architecture is for. The row exists so Clearline can be *placed* next to the competition on identical audio, not to argue that it is ready.
• Clearline's own eval (`clearline/evaluate.py`, 1,800 clips, difficulty buckets) and this leaderboard are not comparable and were never meant to be: different clips, different conditions, different pooling. Only the recogniser is shared. This row exists precisely so that a comparable number exists.
• DeepFilterNet3 runs at 48 kHz on 16 kHz source material upsampled to 48 kHz — there is no real content above 8 kHz for it to work with. That is the honest telephony/VoIP situation, but it is not the condition DFN3 was designed for.
• The test set is 10 speakers / ~190 reference words per condition. That is enough to separate large effects and not enough to resolve differences of a point or two of WER.
• Run-to-run repeatability of the recogniser was measured, not assumed — and the standing warning did not reproduce here. Elsewhere in this project faster-whisper on CPU has been seen to return different transcripts for byte-identical input (three runs of one aggregate gave 121.65 / 122.15 / 121.40%), and the standing guidance from that is to treat differences below roughly 6 pp on a small aggregate as decoder noise. On this host and this harness it did not happen: 25/25 scopes were bit-identical across repeated decodes (raw on 18 condition(s), 180 clips x 3 decodes; clearline on 5 condition(s), 50 clips x 3 decodes), including the arms where the recogniser is already hallucinating past 100% WER. Independently, re-measuring `raw` and `gtcrn` in a fresh process two days after the published run reproduced all 54 rows exactly (max ΔWER 0.0000 pp). Three repeats cannot prove determinism and a different host, venv or thread count may well behave differently, so the conservative 6 pp guidance still governs anything quoted from another machine — but on this table the resolution limit is set by the size of the test set, not by the decoder: one reference word is 0.53 pp in a single condition (190 words), and about 0.03 pp pooled across all 18. Differences of a point or two between adjacent rows are still not results; differences of tens of points are.
• onnxruntime is pinned to 1 intra-op thread with sequential execution: the measured cost is one stream on one core, not a fan-out across the whole machine.
• The `telephony*` conditions in this table are stored at 16 kHz after a full 8 kHz carrier leg (band-limit, codec, packet loss), so every backend sees them at its own native rate. The genuinely narrowband experiment — where nothing is resampled — is a separate artefact (`bench/out/telephony.json`).
• Packet loss is modelled as bursty (2-state Gilbert-Elliott) erasure of 20 ms frames with attenuated-repeat concealment, not i.i.d. loss with silence fill. i.i.d. loss flatters every decoder and silence fill overstates the damage.

The measured results — everything in the results JSON — are published under CC-BY-4.0. Quote them, republish them, argue with them; attribution to Null Test by Anecho Audio, Inc. is requested, and a link to the methodology is appreciated because it carries the caveats.

The harness is Apache-2.0. The *derived audio* is a separate question and a stricter one: LibriSpeech is CC-BY-4.0. ESC-50 is CC-BY-NC-3.0 — the noise conditions are research/evaluation only. The competing-speaker and babble conditions are built from LibriSpeech alone and carry no NC restriction.

The ai-coustics rows are produced through their licensed SDK as a proprietary reference point. They are a measuring stick and never a shipping code path in anything we sell.

If you think we have measured your engine unfairly, the harness takes a new backend in one file — implement the four-method Enhancer protocol and it is picked up. Send a pull request and we will publish the corrected row.