The split pipeline: why your VAD and your STT want different audio

Updated 14 August 2026. When this post was first published, the evidence for the split was other people's — one narrow study and a stated vendor position. It is now ours and it is measured, and the numbers below have since been regenerated across eighteen acoustic conditions rather than thirteen. Absolute values moved, the ordering did not, and everything here is quoted from the published benchmark.json.

Your voice agent has one microphone stream and two consumers with opposite requirements. The turn detector wants the cleanest possible signal so it does not barge in on a television. The recognizer wants the audio it was actually trained on, artifacts and all. Almost every stack we have looked at hands both of them the same buffer, and then tunes one at the expense of the other.

This post is the engineering argument for splitting that stream, the measurements that now back it, and the parts that are annoying to get right: delay alignment, pre-roll, and where the split belongs in a LiveKit or Pipecat pipeline.

The two consumers want different things

Turn-taking wants clean. VAD and endpointing operate on energy and spectral cues over short frames. They are threshold machines. Drop the SNR and every threshold you calibrated moves: onset detection fires late because the speech-to-noise ratio needs longer to cross, offset detection fires early or never because the noise floor never drops below the hangover threshold. The highest-value failure mode in a voice agent is a barge-in false trigger — a second speaker in the room, a TV, café babble with an intelligible fragment in it — and that is exactly the case where a suppressor that isolates the primary speaker turns an unusable signal into a usable one.

Transcription wants raw. Modern STT models are trained on very large quantities of real, noisy audio. Noise is in-distribution. Neural suppression artifacts — spectral holes, musical noise, transient smearing, a noise floor that goes unnaturally dead between words — are not. An independent study (arXiv:2512.17562) found enhanced audio scored worse than raw in all 40 configurations tested, with the caveat that it used only MetricGAN+, on medical speech, in semantic WER. AssemblyAI, citing it, separately reported Krisp's noise cancellation roughly doubling WER when its output was fed to STT, while cutting false VAD triggers about 3.5x. That was public evidence, and thin. It is no longer the only evidence we have.

What our own run says about both halves

We ran ten enhancement engines across eighteen acoustic conditions against a raw control, scoring WER and VAD in the same pass. Provenance: single recogniser, faster-whisper base.en (CTranslate2 int8 CPU, greedy), Silero VAD at threshold 0.5, 10 speakers / 180 clips / roughly 1406 seconds, about 190 reference words per condition, Apple M1 Max, onnxruntime 1.28.0 pinned to one intra-op thread. Full method and caveats in the benchmark launch post.

On the transcription branch, enhancement lost the pooled average. Pooled WER was 14.77% raw against 15.56% for the best engine (ai-coustics Quail L) and 20.26% for GTCRN. Insertions — the recogniser inventing words nobody said — went from 101 raw to 210 for GTCRN and 380 for DeepFilterNet3. None of the ten beat doing nothing on that pooled average.

Per condition it is a different sentence, and the split pipeline depends on both of them being true. At least one engine beat raw in 11 of the 18 conditions, and the wins are large where they happen: at 0 dB broadband noise, eight of the ten engines beat raw, the best by 4.7 points. Enhancement pays where the audio is genuinely bad and costs you where it is not. The full table and its caveats are in Does noise suppression actually help speech-to-text?.

On the turn-taking branch, the picture inverts — but not in the metric people usually quote. Pooled VAD F1 barely moved: 0.950 raw against 0.947 for the best enhanced stream. If you were grading enhancement on VAD F1 you would conclude it does nothing.

The false-alarm rate tells a completely different story, and false alarms are what a barge-in actually is.

Condition	VAD false-alarm rate, raw	Enhanced	Engine
Babble at 5 dB	97.7%	68.2%	FastEnhancer-L
Competing speaker at 5 dB	58.1%	37.2%	ai-coustics Quail VF

Under babble at 5 dB, the raw stream false-alarms on essentially every non-speech frame the detector sees — 97.7% is a VAD that has stopped functioning as a gate. Enhancement takes that to 68.2%. That is still bad, and it is a thirty-point improvement in the one number that maps directly onto your agent interrupting a caller who has not spoken.

This is the measured basis for the split. Enhancement earns its place on the turn-taking branch unconditionally, and earns its place on the transcription branch only in specific conditions — in the same run, on the same audio, with the same recogniser. Those are not two opinions to be traded off. They are two branches that want two different signals, and one of them wants a different signal depending on the room.

Two honest limits on that. F1 barely moving means enhancement is trading false alarms against misses to some degree, and 190 reference words per condition separates large effects rather than small ones. And these are frame-level VAD statistics, not labelled turn boundaries: "fewer false barge-ins" as an end-to-end product metric still needs an agreed definition of a false barge-in, which we do not have yet.

The architecture

Single-stream is a compromise nobody chose deliberately:

                     ┌──────────────┐
  mic ──▶ enhance ──▶│  same buffer │──▶ VAD / endpointing   (happy)
                     └──────────────┘──▶ STT                 (five points worse, pooled)

The split pipeline is the same analysis, two outputs:

                 ┌─▶ enhanced ──▶ VAD / endpointing / barge-in
  mic ──▶ Chamber┤
                 └─▶ raw (delay-matched) ──▶ ring buffer ──▶ STT
                          ▲
                          └── turn boundaries index into HERE

One inference pass, two taps. The enhancer already computes a speech-presence estimate internally; the extra cost of emitting both streams is a copy and a delay line, not a second model.

The part everyone gets wrong: the raw stream is early

An enhancer has algorithmic latency — the analysis frame plus any lookahead. Across the models in our manifest that is 12 to 30 ms for everything except DeepFilterNet3, which we measure at 100 ms through a block-online adapter. The enhanced output therefore lags the raw input by that amount.

If you emit both streams naively, the turn detector produces boundaries in enhanced-stream time and you apply them to raw-stream indices, so every utterance you slice for STT is shifted by one frame. At 30 ms that is enough to clip a word-initial plosive, and a clipped /p/ is a substitution or a deletion in your transcript. The bug is subtle because it does not fail loudly; it just costs you a fraction of a point of WER forever.

The fix is a delay line on the raw path, matched to the enhancer's declared algorithmicLatencyMs, so both streams share a sample clock. Our SDK does this by default:

import { Anecho } from '@anecho/sdk';

const anecho = new Anecho({ apiKey: process.env.ANECHO_API_KEY });

const session = await anecho.createSession({
  model: 'chamber',
  sampleRate: 16000,
  splitPipeline: true,   // default
  alignRaw: true,        // delay raw by algorithmicLatencyMs, default
  prerollMs: 300,        // raw audio retained before onset, default
});

// 20 ms of float32 mono at 16 kHz = 320 samples
for await (const block of micBlocks) {
  const { enhanced, raw, speech, score } = session.process(block);

  turnDetector.push(enhanced, speech);  // clean stream drives boundaries
  rawRing.write(raw);                   // raw stream is what STT will read
}

chamber is our wideband enhancement model; on a phone leg you would pass clearline instead, which processes natively at 8 kHz. speech is Onset's frame-level decision computed on the enhanced signal. score is Nyquist's running call-quality estimate, which is the thing you alert on when a caller's line degrades mid-conversation.

Pre-roll, or why your first word disappears

A VAD declares speech after it has seen speech. Every detector has an onset delay — the frames it needed in order to be confident. If you start filling the STT buffer at the moment the VAD fires, you have already thrown away the attack of the first word.

This is why the ring buffer matters more than the split does. You want a continuously-written raw ring buffer with a few hundred milliseconds of history, and turn boundaries that index into it:

session.on('turn', async (turn) => {
  // turn.startSample / turn.endSample are in raw-stream sample indices
  const utterance = rawRing.slice(
    turn.startSample - session.prerollSamples,
    turn.endSample + session.hangoverSamples,
  );
  await stt.transcribe(utterance);
});

Two defaults worth stating explicitly, because they are the ones people tune first:

• Pre-roll 300 ms. Cheap insurance. At 16 kHz mono float32 that is 19.2 KB of memory per stream.
• Hangover past the offset. Trailing fricatives and unreleased final stops are low-energy and get cut by an eager endpointer. If your transcripts systematically lose plural /s/, this is why.

In Python the shape is the same:

import os
from anecho_sdk import Anecho

anecho = Anecho(api_key=os.environ["ANECHO_API_KEY"])
session = anecho.create_session(
    model="chamber",
    sample_rate=16000,
    split_pipeline=True,
)

for block in mic_blocks(320):          # 20 ms at 16 kHz
    out = session.process(block)
    turn_detector.push(out.enhanced, out.speech)
    raw_ring.write(out.raw)

Where the split lives in a real stack

In an agent framework, the split has to happen before the framework's own routing, because the framework assumes one audio track. Our plugins insert at that point and hand the two streams to the two consumers. The route parameter is the whole idea in one field:

from anecho_livekit import AnechoPlugin

session = AgentSession(
    stt=deepgram.STT(),
    vad=AnechoPlugin.vad(model="onset"),
    audio=AnechoPlugin.enhance(
        model="chamber",
        route="vad-only",     # STT receives the unprocessed, delay-matched stream
    ),
)

route takes vad-only (default), both, or stt-only. We ship vad-only as the default because that is where our own measurements point, on our corpus, with one recogniser — not because we have proven it for your traffic. anecho-pipecat exposes the same three values as a frame processor.

The routing matrix, stated plainly:

route	Turn-taking receives	STT receives	When to use it
vad-only	enhanced	raw, delay-matched	default; noisy environments, barge-in problems
both	enhanced	enhanced	competing-speaker traffic with a Voice Focus model, where our data says STT also wins
stt-only	raw	enhanced	rare; only if your VAD is already noise-robust and your STT is not
disabled	raw	raw	control, and what you should measure against

The both row is not hypothetical, and it is not rare either. Enhancement beat raw on transcription in 11 of our 18 conditions — most cleanly where a competing voice or low-SNR broadband noise is genuinely destroying phonetic cues. The largest single result is ai-coustics Quail VF on competing speaker at 5 dB, cutting insertions from 19 to 10 and WER from 26.3% to 20.5%. If your callers sit in rooms with another human talking, both is defensible on evidence. Pooled across all eighteen conditions the same model is more than three points worse than raw, because it also runs on clean, reverberant and carrier-degraded audio where it can only remove information — which is why it is not the default, and why this is a per-deployment measurement rather than a setting we can pick for you.

Cost, and what the split does not cost

The objection we hear is that this doubles the audio you are moving. It does not, if you keep the raw path local. Enhancement runs where the audio already is — in the browser worklet, in your media server, or in your Node process — and only the stream a given consumer needs crosses a network boundary. The enhanced stream feeds a VAD that is usually in-process. The raw stream goes to your STT vendor, which it was going to do anyway.

What it does cost:

• Memory. One ring buffer per session. Roughly 19 KB per 300 ms at 16 kHz mono float32; call it 64 KB per session with headroom.
• A delay line. One frame of enhancer latency added to the raw path so the clocks match. This does not add end-to-end response latency, because the raw path was ahead, not behind — you are aligning to the slower stream you already had.
• One more thing to reason about. Two streams means two places a bug can hide. The mitigation is that both are derived from one process() call with one sample clock, rather than two independent pipelines that can drift.

What is still unknown

• Some recognizers may be trained on enhanced audio. If a vendor pre-processes in their own ingest, feeding them raw is right for a different reason, or wrong for a subtle one. None of us can see those training sets.
• The SNR crossover. Enhancement clearly wins at low SNR and clearly loses on clean audio, but we cannot yet put a number on where it turns over. The density of winners peaks at 0 dB — eight of ten engines — and thins in both directions, with one engine still beating raw even on the clean condition. That is a slope, not a threshold, and it will differ per engine. Locating it properly is what would make conditional enhancement shippable.
• Turn-taking gains are frame-level, not turn-level. We report VAD F1 and false-alarm rate. "Fewer false barge-ins" needs labelled turn boundaries and an agreed metric. That is the next thing we owe the benchmark.
• Telephony is its own case. On an 8 kHz narrowband path the recogniser is out of distribution before you touch anything, no enhancer in our matrix produced a meaningful improvement there and several degraded it badly, and the reasoning changes; see 8 kHz is where voice AI actually breaks. We have since run a separate experiment at 8 kHz end to end, with three genuinely native narrowband models in it: native beats resample-and-hope by 0.7 points for the same vendor, and nothing at all beats the untouched caller audio.

The reason to encode the split in the SDK rather than in a blog post is that it makes the question testable per deployment. Flip route between vad-only, both, and disabled, run your own audio through the harness, and read the ΔWER against the passthrough control. If both wins on your traffic, use both — the SDK does not care which answer you get, and neither do we as long as the measurement is real.

Quickstart and the full createSession reference are at /docs. The harness that produces the ΔWER numbers is at github.com/anecho, and the published matrix is at /benchmark.