We put our own model in our benchmark and it lost

When we published Null Test v0.1 we wrote down six rules. Rule 1 was that the raw control is always in the matrix and is allowed to win. Rule 2 was that we score ourselves under the same rules, in the same tables, rather than on a separate marketing page.

Clearline is now a row. It came last.

Pooled across all eighteen conditions, on the same 180 clips, through the same recogniser as every other backend: 39.6% word error rate against raw's 14.8%. Thirteen backends in the matrix and ours is the worst of them — behind DeepFilterNet3 measured through a block-online adapter, behind ai-coustics' Quail VF at 18.1%, and behind a resample-only control with no model in it at all at 18.6%.

That is the whole post in three sentences, and none of the rest of it is an excuse. What the rest of it is, is the decomposition: which part of that number is the band limit, which part is the model, where the model is merely bad versus catastrophic, and why the same checkpoint looked like the best thing this company had produced when we scored it on our own corpus a week earlier.

Validate the instrument before you trust the result

A benchmark that only produces bad news about competitors is a marketing asset. A benchmark that produces bad news about the vendor who runs it is only worth something if the vendor cannot quietly blame the harness — so the first thing we did with a result this bad was try to make it the harness's fault.

Two checks, both before we looked at Clearline's row.

The published rows still reproduce. Re-running raw and gtcrn through the new code path in a fresh process reproduced all 54 published rows exactly, max ΔWER 0.0000 pp. Whatever changed to admit a resampled 8 kHz backend into the matrix did not move anything already in it.

The decoder is not adding noise on this host. Elsewhere in this project faster-whisper on CPU has returned different transcripts for byte-identical input — three runs of one aggregate gave 121.65 / 122.15 / 121.40% — and the standing guidance from that is to treat differences under roughly 6 pp on a small aggregate as decoder noise. On this host it did not happen. 25 of 25 scopes were bit-identical across repeated decodes (raw on 18 conditions, 180 clips × 3 decodes; Clearline on 5 conditions, 50 clips × 3 decodes), including the arms where the recogniser is already hallucinating past 100% WER. So the resolution limit on this table is set by the size of the test set, not by the decoder: one reference word is 0.53 pp in a single condition (190 words), about 0.03 pp pooled.

Three repeats cannot prove determinism, and a different host, venv or thread count may well behave differently — the conservative 6 pp guidance still governs anything quoted from another machine. But on this table, a 24.9 pp gap is not a decoder artefact. It is the model.

The number, and the shape of it

Backend	Pooled WER	S	I	D	VAD F1	RTF	Latency
Raw (no processing)	14.77%	344	101	60	0.950	—	0 ms
ai-coustics Quail L	15.56%	349	126	57	0.945	0.124	30 ms
ai-coustics Quail VF 2.2 L	18.10%	383	161	75	0.945	0.076	30 ms
Resample-only 16→8→16 (control)	18.63%	413	173	51	0.948	0.00005	0 ms
GTCRN (MIT)	20.26%	443	210	40	0.947	0.050	16 ms
DeepFilterNet3	35.18%	626	380	197	0.901	0.133	100 ms
Clearline (8 kHz, resampled)	39.65%	776	410	170	0.835	0.048	16 ms

3420 reference words per backend. The rows omitted here — the passthrough control, the five FastEnhancer variants and the DFN3 warm-up variant — are unchanged from the published run and are in the full matrix.

It hallucinates rather than deletes. Insertions go from raw's 101 to 410. Deletions go from 60 to 170. An insertion is not a garbled word; it is the recogniser emitting a word that nobody said. This is the same failure mode we documented for every other suppressor in this matrix — insertions went up, not down — except that ours does four times as much of it as the unprocessed control, and roughly twice as much as GTCRN.

It is not a cost problem. RTF 0.048 on one pinned core, p99 block time 4.08 ms against its own 16 ms block period — 3.9x real-time headroom — and 16 ms of algorithmic latency. It is fast enough to ship and not good enough to ship, which is the least useful combination available.

The signal metrics disagree with each other, which is the point of this company. Against the same pooled audio: on PESQ-WB, Clearline scores 1.685 and the resample-only control scores 1.683 — indistinguishable, a model that reads as doing nothing. On STOI it is 0.744 against the control's 0.840. On SI-SDR it is +0.69 dB absolute where raw is +10.92 and the control is +2.89, so the model costs 2.20 dB beyond the band limit. On WER it is 21 points worse than the control. Four measurements, four different magnitudes of bad, and only one of them is the one a voice agent actually pays for. That is the objective mismatch argument, demonstrated on our own model rather than someone else's.

The VAD row is worse than it looks. F1 falls from 0.950 to 0.835 — by far the largest drop in the matrix — while the false-alarm rate improves, 17.2% to 15.1%. Fewer false alarms and much lower F1 means the losses moved into misses: it is gating out speech. The one branch where enhancement reliably earns its place in our own architecture, the turn detector, is a branch this checkpoint would also damage.

The control that separates the band limit from the model

Clearline is an 8 kHz model and these eighteen conditions are stored at 16 kHz, so its row is measured through the harness's resample path: 16 kHz → 8 kHz → model → 8 kHz → 16 kHz, soxr, the same path the native 8 kHz ai-coustics baselines take. On clean, the noise conditions, babble, competing and reverb that is not its operating point — the 4 to 8 kHz octave is discarded before the model ever sees it.

The obvious objection is therefore that we handicapped our own model, and the obvious answer is a control. resample8k is that exact round trip with no model in it: same decimation, same interpolation, same library, nothing in between. Whatever the band limit costs, it costs the control too. Whatever is left over is ours.

Regime	Raw	Resample-only control	Clearline	Band costs	Model adds
Wideband (11 conditions, 2090 ref words)	18.99%	24.88%	55.45%	+5.9 pp	+30.6 pp
Telephony (7 conditions, 1330 ref words)	8.12%	8.80%	14.81%	+0.7 pp	+6.0 pp

Read the wideband row first, because it is the one the objection was about. Throwing away the top octave of a 16 kHz recording costs 5.9 points. Running our model on what is left costs a further 30.6. The handicap is real and it is one sixth of the damage. Five sixths of it is the model.

Then read the telephony row, because that is the regime the model was built for. There the band limit is nearly free — 0.7 points, which is a bit over one reference word per condition — and the model still adds 6.0 points. It is not being punished by an unfair sample rate on the conditions that matter.

It is losing at its own operating point, and it loses to everybody there. On the same seven-condition telephony sub-aggregate: raw 8.12%, the resample-only control 8.80%, ai-coustics Quail L 9.02%, Quail VF 9.70%, Clearline 14.81%. Beaten by the competition, beaten by doing nothing, and beaten by a piece of wire.

Where it loses, condition by condition

The seven telephony conditions, which are the only ones we would ever quote for a narrowband model:

Condition	Raw	Resample-only control	Clearline	Δ vs raw
telephony (G.711 µ-law round trip)	5.79%	4.21%	5.79%	0.00
telephony_snr10	10.53%	7.89%	14.74%	+4.21
telephony_alaw	7.37%	8.95%	13.68%	+6.32
telephony_g722	5.26%	7.37%	9.47%	+4.21
telephony_opus12k	6.84%	11.05%	27.37%	+20.53
telephony_loss3	11.05%	11.58%	17.89%	+6.84
telephony_loss10	10.00%	10.53%	14.74%	+4.74

About 190 reference words per condition, so one word is 0.53 pp — read the telephony row as a genuine tie and everything below it as a genuine loss.

There is exactly one cell in this table that is not a loss. On plain telephony — a clean G.711 µ-law round trip with no added noise — Clearline scores 5.79%, identical to raw to the word. It ties doing nothing. It is still three words behind the resample-only control's 4.21%, which is to say a piece of wire beat it on the one condition where it drew.

telephony_opus12k is the cell to stare at: 27.37% against a raw baseline of 6.84%, with insertions at 8 where raw has 2. Opus at 12 kbps is a parametric codec doing aggressive things to the residual, and the model responds to that the way FastEnhancer-M responds to A-law companding — it hallucinates. Whatever this checkpoint learned about what speech looks like, low-bitrate Opus artefacts are outside it.

The wideband conditions are worse and less interesting, because they are out of regime. Two of them are worth naming anyway, because they show the failure mode at full size: babble_snr5 at 138.4% WER with 169 insertions where raw has 10, and reverb_snr10 at 116.3% with 117 insertions where raw has zero. A WER above 100% means the recogniser emitted more wrong words than there were words to get right. That is a model producing structured, speech-shaped output out of material it does not understand, and a recogniser dutifully decoding it.

Why it looked good a week earlier

Here is the part that is actually useful to anyone else, because the mistake is not specific to us.

Scored on our own evaluation corpus — 1,800 competing-speaker clips, same recogniser, faster-whisper base.en — the identical checkpoint beat raw audio:

Metric	Raw	Clearline	Δ
Word error rate	79.6%	70.9%	−8.70 pp
Insertions	5,737	3,269	−43%
VAD false alarms	62.8%	14.6%	−48.2 pp
SI-SDRi	—	+3.19 dB	—

That was, on its own terms, the best result this company had produced: the first engine anywhere in our own testing to beat raw audio on word error rate. It is also, in isolation, an advertisement — and the reason we are showing it here rather than on the landing page is that the two tables are not comparable and were never meant to be. Different clips, different conditions, different pooling, different difficulty distribution. Only the recogniser is shared. Null Test exists precisely so that a comparable number exists.

The comparable number is 39.6% and last place.

That contrast is the lesson, and it generalises. A model evaluated on the distribution it was trained on will report the performance of that distribution. Our corpus is competing-speaker mixtures rendered under a specific set of assumptions — a specific level, a specific onset order, a specific interference-to-target ratio range — and on material drawn from those assumptions the model does what it was taught. Null Test is eighteen conditions it never saw: reverberation, low-bitrate codecs, packet-loss concealment, broadband noise at five SNRs, clean speech with nothing to remove. It is overfit to its training distribution, and the only way to find that out was to score it against a corpus we did not build for it.

Every vendor claim in this category, including the two we quote most often, is a number from the vendor's own corpus. We now have a measured example of how far apart those two numbers can be for one checkpoint: 8.7 points better than raw on ours, 24.9 points worse on a neutral one. We do not know that anyone else's gap is that wide. We do know nobody has published theirs.

The two gates this checkpoint does not pass, neither of which is in the table

Both are measured on Clearline's own eval, runs/t4_300h_200ep_solo, and neither is visible anywhere in Null Test:

• A solo regression. On clips with no competing speaker at all, the model makes the audio worse: −0.874 dB SI-SDR with 37.1% of solo clips degraded (−0.595 dB / 33.0% on the val split). Most phone calls have exactly one person on the line. A lone caller attenuated as if they were interference is worse than doing nothing.
• The speaker conditioning contributes approximately nothing. The matched ablation scores FiLM conditioning at +2.815 dB against no conditioning at all at +2.690 dB — a 0.125 dB difference against a 0.3 dB noise floor. What is being measured in this table is therefore a plain denoiser, not the primary-speaker isolation the architecture is for, which is the entire product claim.

There is also no ONNX export, so it cannot run in a browser or through the Node SDK. Clearline is not shippable, this row is not a claim that it is, and nothing on this site sells it. The row exists so the model can be placed next to the competition on identical audio.

Caveats, in full

These attach to every number above.

• Single recogniser. faster-whisper base.en, CTranslate2 int8 CPU, greedy. A different recogniser gives different absolute numbers. The harness supports Deepgram and AssemblyAI so the ordering can be re-checked.
• About 190 reference words per condition, 3420 pooled. One reference word is 0.53 pp in a condition, about 0.03 pp pooled. Differences of a point or two between adjacent rows are not results; differences of tens of points are.
• Clearline's row is measured through a resample path, 16 kHz → 8 kHz → model → 8 kHz → 16 kHz, because these conditions are stored at 16 kHz after their carrier legs. The resample8k control is that round trip with no model in it, which is how the band limit and the model are told apart above.
• The ESC-50 noise conditions are CC-BY-NC-3.0 and therefore evaluation-only. LibriSpeech is CC-BY-4.0, and the babble and competing-speaker conditions are built from LibriSpeech alone.
• DeepFilterNet3 is measured through a block-online adapter because no public ONNX export carries recurrent state tensors. Its rows are a lower bound on quality and an upper bound on cost. Never quote its number without this sentence.
• The ai-coustics rows are a proprietary baseline run through the licensed SDK — a reference point, never a shipping code path for us.
• onnxruntime pinned to one intra-op thread, sequential execution. The RTF column is one stream on one core.
• Clearline's own eval and this leaderboard are not comparable. Different clips, different conditions, different pooling. That is the whole reason both appear above.
• Still missing from the condition set: acoustic echo with non-linear speaker distortion. It is the impairment behind the half of this product that cannot be closed with a patch, and until it is a condition, none of these tables measure the thing we most want measured.

What happens to the row

It stays. Rule 3 is that every cell publishes, including the ones we lose, and no condition gets dropped after we see the result. The next checkpoint gets scored under the same rules against the same control, and if it also loses, that publishes too.

The diagnosis we are working from is in the warm-up contract post: the corpus this checkpoint learned from always had the target speaker talk alone first, which is not what a phone call does, and the measured cost of that assumption is large enough to account for a lot of what is above. Fixing a corpus is slower than fixing a loss function and it is the actual work.

The full matrix, including every cell where we lose, is at /benchmark. The raw results file is at /data/nulltest/v1/results.json and the harness is at github.com/anecho. If you think we configured our own model badly, the fix is a pull request against a YAML file, and we will rerun and republish.