We trained a speaker-isolation model on an assumption that does not hold in real calls

Target-speaker extraction has to answer a question that generic denoising never asks: of the two people talking, which one am I supposed to keep?

There are a few ways to tell a model the answer. You can enrol the speaker in advance and hand the network an embedding — accurate, and useless for an inbound call from a stranger. You can pick the loudest, or the nearest, which is not speaker isolation, it is a level meter. Or you can use the structure of the call itself: the primary speaker is the one talking alone at the start. Somebody says "hello" before the noise begins. Pool a speaker embedding over that opening window, condition every subsequent frame on it, and you have an enrolment-free system with a definition of "primary" that requires no configuration at all.

That is the warm-up contract, and it is what we built. Our model commits its embedding at 736 ms — 46 frames at a 128-sample hop — and every clip in every corpus we generated before this month rendered the target from t=0 with the interferers entering afterwards, because that is what makes the contract well defined.

Phone calls do not do that. The television was already on. The open-plan office was already loud. The other person on the speakerphone was already mid-sentence when the line connected. In a large fraction of real calls, the interference is running at t=0 and the caller starts talking into it.

We measured what that costs. It is the largest single number this project has produced, and it is not in our favour.

The experiment

Four arms, rendered from one room and one pair of sources. Same room, same simulated impulse responses, same utterances, same target-to-interference ratio, same codec leg. The arms differ in timing and nothing else, which is what makes this an ablation rather than four unrelated experiments.

A  trained contract   target ──────────────────────▶
                      bed         ┌────────────────▶      (target alone for 0.8 s)

B  reality            target      ┌────────────────▶
                      bed   ──────────────────────▶       (bed running at t=0)

C  bed only           target
                      bed   ──────────────────────▶       (no target — control)

D  B + lead-in        target ────┐ ┌───────────────▶
                      bed        └─────────────────▶      (0.8 s of B's own target, spliced on)

Arm D is the one that decides what the result means. It is arm B's audio with a genuine target-alone lead-in spliced onto the front — same speaker, same room, same recording. If D recovers arm A's performance, the defect is the warm-up contract. If D stays broken, the problem is simply that this material is out of distribution and timing has nothing to do with it.

48 items × three TIRs (0, 6 and 12 dB) = 144 matched pairs per arm, scored offline against the t4_300h_200ep_solo checkpoint at epoch 199 — the same checkpoint that appears, and comes last, in our public benchmark.

The result

Arm	In SI-SDR	Out SI-SDR	Δ	Clips degraded	cos → target	cos → background	Embedding picked the background
A trained contract	+6.75 dB	+11.76 dB	+5.01 dB	6.9%	0.948	0.862	9%
B reality (bed at t=0)	+4.71 dB	−9.92 dB	−14.63 dB	86.1%	0.866	0.971	96.5%
D B with 0.8 s lead-in	+5.67 dB	+10.58 dB	+4.91 dB	6.9%	0.948	0.862	9%
B with an oracle embedding	+4.71 dB	+6.52 dB	+1.81 dB	20.1%	—	—	—

Arm B is not "the model helps less". It is −14.63 dB: the output is fifteen decibels further from the target than the input was, and 86% of clips come out worse than they went in. A model that reliably damages six clips in seven is not underperforming, it is doing the wrong job confidently.

And arm D is the answer to the objection. Splice 0.8 seconds of the target's own voice onto the front of the identical mixture and the model recovers to +4.91 dB, with 6.9% degraded — arm A's numbers to two decimal places on the cosine columns. The audio in D and B is the same audio. The room is the same room. A 19.5 dB swing, from timing alone. The material is not out of distribution. The contract is.

The downstream number is the one a voice agent actually pays:

Arm	WER
raw 8 kHz, no model in the path	121.9%
A trained contract	64.5%
B reality	131.9%
D B with lead-in	76.5%

39 items, TIR +6 dB, faster-whisper base.en. Word error rates above 100% mean the recogniser emitted more wrong words than there were words to get right, which is what happens when the surviving signal is a second person talking fluently. On its trained contract the model takes 121.9% down to 64.5% — a large, real win, and the result that made us optimistic. On the case a phone call actually presents it lands at 131.9%, worse than not running the model at all.

The mechanism is visible in one number

The two cosine columns are the whole diagnosis.

In arm A the pooled embedding sits at cosine 0.948 to the target speaker and 0.862 to the background. In arm B those swap: 0.866 to the target, 0.971 to the background. The embedding is not confused, degraded or noisy. It is a clean, confident embedding of the wrong voice, and every downstream frame is conditioned on it. The model then does exactly what it was trained to do — keep the speaker matching the embedding and suppress everything else — which on arm B means keeping the television and suppressing the caller.

It does that in 96.5% of clips. This is not a tail failure to be fixed with more data of the same kind. It is deterministic behaviour following from a warm-up window that pools whatever is making noise when the window opens.

The oracle row bounds how much of the damage the embedding is responsible for. Hand the model the correct embedding and run it on arm B's audio anyway, and it recovers to +1.81 dB with 20.1% degraded — from a 14.6 dB hole to a modest positive. So the embedding is most of the defect and not all of it: even correctly conditioned, arm B material only reaches +1.81 dB against arm A's +5.01, because a mixture with no target-alone region anywhere is genuinely harder. Fixing the conditioning is necessary. It is not sufficient.

What we could and could not fix at inference

The corpus takes days to regenerate and a training run takes longer, so we asked what a front end could do in the meantime. The honest answer is: half of it.

There are two distinct versions of the warm-up defect, and they are not equally reachable.

The quiet start — nobody is talking yet, and the window pools room tone. A front-end gate can fix this, because there is a real acoustic onset to find. Holding the embedding until a sustained rise over a primed background estimate takes that arm from +1.83 to +4.03 dB, cuts degraded clips from 21% to 10%, and drops the rate at which the embedding lands on the wrong source from 69% to 4%. Cosine to target goes 0.882 → 0.966. That is a genuine fix and it shipped.

The bed running at t=0 — the arm B case — a front-end gate cannot reach, and the reason is structural rather than a tuning failure. The bed is speech. Any input-side speech detector fires at t=0, because something really is talking. The gate opens at 112 ms and the embedding still lands on the interferer. Measured: −14.11 → −13.98 dB, +0.13 dB against a 0.3 dB noise floor, with the wrong-source rate going up, 94% to 97%.

We also quantified the aggressive setting — never take the "something is already talking, open immediately" escape — because it looks like the obvious answer. It buys arm B +1.98 dB against a 14 dB hole and costs −1.03 dB on arm D, whose degraded rate goes 10% to 18%. With a background running continuously there is no clean onset to find, so the gate simply fires about 370 ms later on the background's own fluctuations. No front-end gate setting meaningfully fixes this. It is a training-data problem and it has to be paid for as one.

One more finding from that work, because it is the sort of thing that makes a metric lie. The same front end also normalises level, which is a large real win on the trained-contract arm — +9.65 dB at the quiet input level a real user reported, and a no-op at the level the corpus was normalised to. On arm B, level normalisation makes the number worse: −2.34 → −14.10 dB. It does not create the defect. It un-masks it. Arm B at the trained level was already −14.11 with the front end off, so being too quiet had been accidentally protecting users from a model acting confidently on an embedding of the wrong person. Restoring the correct operating point restored the damage. We shipped that as a stated regression, in the UI and in /health, rather than as a quiet improvement to an average.

The corpus fix, and the check that it did not invalidate the past

The corpus now renders all three onset orders instead of one:

Onset order	Share of eligible clips
Target first (the old contract)	45%
Interferer first	40%
Simultaneous	15%

Verified on 6,000 drawn specifications before any rendering: 45.5 / 40.1 / 14.4% over eligible clips, 30.6% interferer-first corpus-wide once solo clips and the same-distance control are counted — those are forced to target-first because their label is otherwise undefined.

One parameter matters more than the ratio. The interferer's head start is drawn from 200 to 2500 ms, deliberately longer than the 300–800 ms warm-up window. If the lead were always shorter than the window, "the background is already playing" would still resolve inside the pooling window every time, and the model could keep using the same shortcut while appearing to have learned the harder case. The point of the range is that it cannot.

Target-first keeps a plurality, because a lead-in is still the cleanest available definition of who to keep. It is no longer the only thing the model has ever seen.

And because changing a data generator silently invalidates every run that came before it: re-rendering 60 existing validation specifications through the revised generator reproduces the shards on disk bit for bit — worst absolute int16 delta 0, zero mismatched clips. All 60 old specifications deserialise under the new field defaults as target-first at the old level, so the revision cannot have changed any run trained before it, and old-versus-new comparisons stay valid. A data pipeline change that cannot be shown to be inert on old inputs is a change that quietly rewrites your history.

Caveats

• 144 matched pairs per arm, 48 items × three TIRs, one checkpoint, offline forward pass. Enough to establish a 19.5 dB effect; not a shipping validation.
• The WER figures come from a smaller sample and a noisier instrument. 39 items at TIR +6 dB. Running the identical items three times gave 121.65 / 122.15 / 121.40% on the unprocessed control, so the standing guidance from that measurement is to treat differences below roughly 6 pp on an aggregate this size as decoder noise. The 131.9% versus 121.9% gap is 10 pp and clears that bar; the 64.5% versus 76.5% gap between arms A and D is larger still. Do not read one-point differences in this table.
• These are simulated rooms, convolution with simulated impulse responses, one interferer, a codec leg. Not recorded calls. The direction of the effect is what we would defend; the exact decibel figure is corpus-specific.
• This is one architecture's warm-up window. A system that enrols speakers in advance does not have this failure mode, and has a different one.
• The checkpoint measured here is the same one that lost our public benchmark and it remains not shippable for two further reasons that this post does not fix: it degrades single-speaker clips, and its speaker conditioning contributes about 0.125 dB against a 0.3 dB noise floor, which means it is currently behaving as a plain denoiser rather than as speaker isolation.

The general version

The failure is not that the model is weak. On the distribution it was trained on it does the job well — +5.01 dB, 121.9% WER down to 64.5%. The failure is that the distribution encoded an assumption nobody wrote down as an assumption. "The target speaks first" entered the corpus as a convenience, because it makes the label well defined, and it left the corpus as a requirement, silently, with no test asserting it and no metric reporting when it was violated.

That is the shape to look for in your own data. Somewhere in every generated corpus there is a decision that was made to keep the labels clean, and if the world does not honour it, your evaluation will never tell you — because your evaluation was generated by the same code, under the same convenience.

The cheap detection is the one used above: build a matched pair that differs in the suspect variable and nothing else, and see how far apart they land. Ours were 19.5 dB apart. We would rather have known that before the training run than after it, and the only reason we know it now is that somebody asked what happens when the television is already on.

The public benchmark row for this checkpoint, including every condition where it loses, is at /benchmark.