How to evaluate speaker isolation: a six-question checklist for any engine, including ours

Update 20 October 2026: we put our own engine through these questions against ai-coustics Quail VF 2.2 S in a live-conversation benchmark: target-referenced WER, insertions and deletions reported separately, a raw control, paired confidence intervals. The answer depended on the recogniser: parity with faster-whisper base.en, a v8_2 lead with Gemini Live input transcription, fewer false barge-ins for VF S, and the condition deciding who leads.

Primary-speaker isolation stopped being a niche feature this year and became a line item: several vendors now market some form of primary-speaker focus, and most of the headline accuracy numbers around it are measured on private corpora, under conditions nobody outside the vendor can rerun. This post is the checklist we would use to buy such an engine — any engine, ours included. Six questions, each with a pass and a fail that an engineer can execute in an afternoon, none of which require trusting a demo clip.

1. Demand target-referenced WER on a corpus you can download

A WER-reduction claim is not a property of an engine. It is a property of at least four things together: the corpus, the acoustic conditions, the recogniser, and the text normalization applied before scoring. Change any one of them and the number moves, sometimes across zero. A vendor who quotes the number while keeping the other four private has not made a falsifiable statement — and unfalsifiable is worse than wrong, because wrong can at least be demonstrated.

Pass: you can fetch the corpus or its manifest, rerun the scoring, and reproduce the table to the digit. Fail: "on our internal dataset." The politeness of the phrasing does not change what it means, which is: you cannot check this.

2. Watch insertions specifically

An isolation engine has one defining failure mode: the second voice leaks through. A leaked voice does not show up in a transcript as a garbled word — it shows up as inserted words, the recogniser faithfully transcribing somebody who is not your caller. Aggregate WER is (S + D + I) / N, one number carrying three different failure mechanisms, and it will happily hide a doubling of insertions behind an improvement in substitutions. For a voice agent, inserted fluent text from the wrong speaker is the most dangerous error class there is, because a downstream LLM acts on it.

Pass: substitutions, deletions and insertions published separately, per condition. Fail: a single WER scalar. The longer argument about why the decomposition matters more than the total is in our write-up on error mechanisms.

3. Require the do-nothing control row

The question is never "does the processed audio score well." It is "does the processed audio beat the identical audio left untouched." That bar is much higher than it sounds: modern recognisers are trained on enormous amounts of noisy, real-world audio, and noise is in-distribution for them in a way that processing artifacts are not. "Beats raw" is the bar, and most engines fail it most of the time — on our own published matrix, pooled across all conditions, nothing has beaten the untouched control yet, our engine included.

Pass: the vendor's evaluation contains a raw, unprocessed row, scored identically, that was structurally allowed to win. Fail: every row in the table is some configuration of the vendor's product. A benchmark without a control is a leaderboard of one.

4. Measure latency as algorithmic delay plus block budget on one core

A single "X ms latency" figure conflates two numbers that behave completely differently in production. Algorithmic delay — frame size plus lookahead — is a property of the model: no hardware will ever make it smaller, and it sits on your conversation's critical path forever. Block budget is the compute each audio block costs, which on a pinned single core becomes a real-time factor and decides how many streams one core can serve. A marketing number that names neither the hardware, the thread count, nor which of the two it is measuring was produced on the best machine the vendor could find, and your capacity planning will inherit the difference.

Pass: algorithmic delay stated as a model property, plus a measured real-time factor with the hardware string and a one-thread discipline attached. Fail: "ultra-low latency."

5. Ask where the audio goes

There are two architectures and they are not interchangeable. An in-process library keeps the samples inside your media server, which means your existing data-processing story is unchanged. A cloud API adds a second processor of what is, in most jurisdictions, personal data — caller audio — which under GDPR means a data-processing agreement, a residency answer, and a line in your privacy documentation; it also adds a network round trip whose tail latencies will dwarf any algorithmic delay figure from question 4. Neither architecture is disqualifying. A vendor who cannot state, in one sentence, which process the samples ever leave, is disqualifying.

Pass: "the audio never leaves your process" or "the audio is processed in region X under DPA Y" — either, stated plainly. Fail: a diagram with a cloud in it and no sentence.

6. Price it against the public anchor

Per-minute pricing for this category is no longer hypothetical: LiveKit meters voice isolation at $0.0012 per minute past plan quotas, on a public pricing page anyone can cite. That makes it the ecosystem's anchor, whatever you think of it. Take whatever quote you were given — per-hour, per-seat, per-month with a committed floor — normalize it to one stream-minute, add the compute cost of running the engine from question 4's real-time factor, and put the result beside $0.0012. The comparison takes five minutes and vendors know whether their number survives it, which is why the quote so rarely arrives in that unit.

Pass: a price you can state per minute without a sales call. Fail: "contact us," plus a floor you discover in week three.

Where this checklist comes from

We did not invent these six questions for this post. They are, point by point, what our benchmark implements: target-referenced WER with substitutions, deletions and insertions in every cell; a content-addressed corpus manifest you can fetch; a do-nothing control that is structurally allowed to win — and currently does, at 14.32% pooled WER, with nothing under it, our own engine included; real-time factors measured on one pinned core with the hardware string in the row; and the results published under CC-BY-4.0 so you can quote them against us. The methodology carries the caveats, which are real.

If you sell a speaker-isolation engine and believe your row would look better than ours, write to [email protected]. Adding an engine is one adapter file. We run it under the published rules, and we publish the result whichever way it goes — the table above us is the proof of that.