Our row is on the table: we scored our own engine on our own benchmark, and it did not beat the control either

Update 20 October 2026: a second experiment, live conversations, now scores v8_2 against ai-coustics Quail VF 2.2 S on 200 constructed 30–60 s calls under two recognisers. With faster-whisper base.en it is parity and neither engine beats raw audio on the pooled set; with Gemini Live input transcription v8_2 beats raw and leads VF S pooled. v8_2 leads with a close-mic talker and a loud background, VF S with a distant talker and on silencing the pauses, and VF S triggers fewer false barge-ins under Gemini Live. The row and the numbers in this post are unchanged.

When we published Null Test last August, the matrix carried one caveat in bold: our own models are not in this matrix. Publishing the referee before the contestant was the point of the ordering, and rule two of the benchmark — we score ourselves under the same rules, in the same tables — was a promise about the future.

The future arrived. The anecho.ai_focus_model_16khz_v8_2 row is on /benchmark today, measured by the same harness code as every other engine there, and this post is the honest account of what it says.

The result, before the reasoning

The matrix now scores eleven enhancers against the do-nothing control, across nineteen acoustic conditions — including a two-voice condition every engine is scored on, because a speaker-isolation engine that has never been measured against a second voice has not been measured at all.

Pooled across all nineteen conditions, the control — the untouched audio — sits at 14.32% WER. Our row sits at 16.59%.

Nothing on the table beat the control. Our engine included. We published the row anyway.

Why a losing row is the product working

Here is the uncomfortable economics of vendor benchmarks: most of them cannot lose. A number measured on a private corpus, under conditions the vendor chose, against no control, scored by a recogniser the vendor picked, is not a result — it is an advertisement with a percent sign. Whatever the model does, the page was always going to say the model won.

A benchmark is only worth reading if the vendor's own row can lose on it. The rules that make that possible were published before our row existed, precisely so we could not bend them once we had something to protect: the raw control is always in the matrix and is allowed to win; every cell publishes, including the ones we lose; the configs are in the repo; any vendor can demand a rerun at a different setting and we publish both. In the first run the control beat ten engines that were not ours. In this run it also beat the one that is. The page says so, in the same table, with the same decomposition.

That is not magnanimity. It is the only mechanism we know of that makes the rest of the page believable.

What the row actually says

Three things, and we will not decorate them.

First, the pooled loss. 16.59% against 14.32% is the answer to one question — "should this engine be on by default, for all material, in front of this recogniser" — and the answer is no, for our engine as for every other engine we have ever measured. On clean audio an isolation model can only remove information. Pooling averages its design conditions with the conditions where there was nothing useful to do. We have been writing that sentence about other people's engines since the first run; it would be remarkable if it stopped being true the day the engine was ours.

Second, where the row stands. Of everything on that table that you can legally ship, our row posts the closest pooled WER to raw. The one engine nearer to the control is a licensed proprietary reference we run as a baseline — its terms make it benchmark-only, and it is never a code path. We state this carefully because it is the kind of sentence that drifts into marketing if you let it: it is a relative standing on our own corpus, not a claim that the gap to raw is small enough for your traffic. The per-condition decomposition that answers the real question is on the page, and we are deliberately not quoting our own best cells here — selective quotation is the exact disease this benchmark exists to treat.

Third, the cost column. The row runs at 15 ms of algorithmic delay on one CPU core, at a measured real-time factor of 0.067. Accuracy numbers and cost numbers travel together in every row, because a model that wins at four times real time is not a shipping option, and ours has to live by the same rule.

The precedent that makes the loss credible

A vendor saying "we published a result against ourselves" invites a fair question: would you, again, if it were worse? We have one piece of evidence to offer, and it is not hypothetical. An earlier version of our telephony teardown repeated a community figure — that a default resample path discards roughly 16% of caller audio. We built the control, our measurement did not support the claim, and we retracted it in public, leaving the corrected section in place rather than deleting it. The retraction still sits on the benchmark page today, beside the data, where it is least comfortable for us.

Since then, the same verification suite that stops that retraction from quietly disappearing also pins every sentence the site says about our own row to the published artefact. The page cannot claim our engine beat the control unless benchmark.json says so, because the copy is derived from the file and the build fails otherwise. We built that guard after watching a refuted number survive on our own site for days. The lesson was never about the number; it was that prose unconnected to a measurement cannot be corrected by one.

Judge it on your own voice

Our corpus is ten speakers and a fixed condition set; the caveats are listed in the methodology and they are not small. The condition that actually decides a purchase is not in our matrix, because it is your audio — your handsets, your rooms, your second voices.

The engine behind the row runs in your browser, on your microphone, in the live demo. Record the room you actually take calls in, with the TV on, with someone talking behind you, and listen to what comes out. If it does not earn its 15 ms on your voice, the control costs nothing and our own table says that is a fair verdict.