The resampler that ate 92% of a microphone

An agent we were building would connect, show a live session, stream audio for the whole call, and never once answer. The socket stayed open. Audio kept leaving the browser. Nothing came back.

The cause was four tokens of Python:

x8 = down(xin)
if not x8.size:
    continue

down is a stateful soxr.ResampleStream. The guard reads as "the polyphase filter has not primed yet, skip this block and come back next time." That is a reasonable thing to believe about a resampler, and it is wrong about this one. soxr.ResampleStream does not prime once and then emit steadily. It buffers internally and emits in bursts, for the entire session, and if you feed it small blocks it returns an empty array on the overwhelming majority of calls.

Before the diagnosis, one thing this post is not. We have previously retracted a claim that resampling per se costs you a fixed fraction of caller audio; we built the control and our measurement did not support it, and we are not quietly reintroducing it here. Nothing below is a property of resampling. It is a property of one library's emission schedule meeting one continue statement, and the audio was thrown away by our control flow, not by any filter.

The measurement

Four lines, reproducible on any machine with soxr installed. Feed a stream resampler the browser's own render quantum and count how often it gives you anything back.

import numpy as np, soxr

rs = soxr.ResampleStream(16000, 8000, 1, dtype="float32", quality="VHQ")
sizes = [rs.resample_chunk(np.zeros(128, np.float32)).size for _ in range(400)]
print(sum(s > 0 for s in sizes), "non-empty of", len(sizes))

On this host — soxr 1.1.0, macOS on Apple silicon — that prints 33 non-empty of 400. So 91.8% of calls return nothing at all, and the 33 that do return a fixed 758 samples each: 94.8 ms of 8 kHz audio, arriving in one lump, over and over, for as long as the session lasts.

The block size is a property of the resampler, not of your input. Vary the chunk you feed it and only the frequency of the bursts changes:

Input chunk	Chunk duration at 16 kHz	Calls returning nothing	Burst size out
128 samples (AudioWorklet quantum)	8 ms	91.8%	758 samples (94.8 ms)
256 samples	16 ms	83.2%	758 samples
512 samples	32 ms	66.5%	758 samples
1024 samples	64 ms	32.5%	758 samples
2048 samples	128 ms	0%	758 or 1516 samples

400 calls per row, one fresh ResampleStream each, 16 kHz to 8 kHz at VHQ.

Two things fall out. First, there is a threshold, and it is exactly the input needed to fill one output burst: at 2:1 decimation, 758 output samples take 1516 input samples, and feeding 1516 per call takes the empty returns down to the single priming call (0.5% of 200). At 2048 even that one comes back. Which is why nobody hits this with 20 ms server-side frames from a SIP leg and everybody hits it with a browser worklet. Second, the burst size tracks the filter, not the plumbing: at HQ it is 830 samples, at LQ 470, at 48 kHz → 16 kHz it is 1100. Those numbers are this version on this host and you should measure your own rather than trust ours; the shape is what transfers.

None of this is a defect in soxr. A polyphase resampler with a long anti-alias filter has to accumulate input before it can produce correctly-filtered output, and emitting in whole internal blocks is the efficient way to do that. The library is behaving exactly as a stream resampler should. The bug is entirely in what we concluded from an empty return.

Why an empty array meant something different in two places

We had the same shape of guard in two services, and it was harmless in one and severe in the other. The difference is worth stating precisely, because "we had this bug twice" is not the interesting part.

On the enhancement endpoint the code is:

x8 = down(pcm)
if not x8.size:
    continue          # harmless here
y8 = model.process(x8)
await ws.send_bytes(up(y8))

Everything downstream is derived from x8. An empty x8 genuinely means there is nothing to process yet, nothing has been dropped, and the audio is sitting inside the resampler's buffer waiting for its burst. The continue is correct.

On the agent endpoint the same three lines sat at the top of a function that had two parallel jobs: run the caller's audio through the model at 8 kHz, and separately keep an untouched 16 kHz copy for the control arm of an A/B. The continue returned before the second job ran. So with the filter switched off — the arm with no model in it at all, the arm that is supposed to be the honest baseline — the raw microphone buffer was discarded along with the empty x8 that had nothing to do with it.

The result, measured end to end against the real endpoint: 3.7 seconds of uplink across a 46 second call. About 8% of the caller. Zero turns, ever.

The rule we wrote into the code afterwards is narrower than "never skip a block", because sometimes skipping is right:

Nothing may condition one arm on the other arm's resampler having produced output.

It was in the telemetry the whole time

This is the part that should be uncomfortable, and it is the reason the post exists rather than a commit message.

The service already emitted a stats message roughly every sixteen blocks, and that message already carried a field called uplink_realtime — uplink seconds divided by wall-clock seconds since the session went live. On a healthy call it sits near 1.0. On the broken calls it sat at 0.08, in a JSON blob, in a browser console, for days.

Nobody read it, because the user-visible symptom ("the agent does not answer") pointed at the model, the prompt, the credentials and the API before it pointed at arithmetic. A number that says 8% of the audio you think you are sending is being sent was one scroll away from the person debugging, and it lost to a more interesting hypothesis every time.

Two changes came out of that, and they are cheap enough that we would recommend both to anyone running a live audio path:

• Publish a ratio, not a count. Bytes sent is unfalsifiable — it goes up on a broken call too. Seconds of audio delivered per second of wall clock has a known correct value, so a wrong value is legible without context. It now ships in the session and stats messages with a stated threshold: below roughly 0.9 means audio is being lost or arriving late.
• Say what the number implies, in the payload. The stats message now carries the sentence "Gemini's VAD cannot end a turn on audio it never received" next to the number, because the number alone did not connect to the symptom for anyone who had not already found the bug.

The second-order bug: the arms were not comparable

Fixing the discard exposed a subtler problem in the same code, and it is the one that would have quietly invalidated the experiment the endpoint exists for.

That endpoint is a single switch: same speaker, same background noise, filter on versus filter off, mid-conversation. For that comparison to mean anything the two arms must differ in the audio and in nothing else. They did not. The filtered arm inherited the resampler's ~95 ms burst cadence; the unfiltered arm passed the browser's raw 8 ms quantum straight through. Measured on the same call: 10.6 frames per second against 125.0 — an 11.8x difference in how one conversation was packetised.

Any behavioural difference a listener attributed to the filter was confounded with frame cadence, and 125 frames per second is far more than the API expects. Both arms are now coalesced to a fixed 100 ms frame before they are sent, so cadence is identical by construction, and the per-arm rate is reported rather than assumed — the health payload carries uplink_frame_hz for both arms, and they are supposed to read the same.

There is an interlock worth naming, because it is the kind of thing that turns one fix into a different bug. A part-built frame is audio that has not been sent. Coalescing is only safe because something else guarantees the buffer always has a producer — otherwise the tail of the caller's last sentence sits in a half-full frame at exactly the moment the server needs it in order to detect that the caller has stopped. That guarantee is the keep-alive described in the Gemini Live post, and the two changes are not separable.

Why use a stateful resampler at all

The obvious escape is to call the stateless soxr.resample() once per chunk and never see an empty return. Do not.

A stateless call restarts the polyphase filter at every chunk boundary, which stamps a discontinuity into the signal about thirty times a second on a 32 ms chunk. It is audible as a buzz at the chunk rate, and it is the kind of artefact a listener will quite reasonably blame on your model. The stateful resampler carries its filter state across calls precisely so that does not happen, and the bursty emission is the visible cost of that correctness.

That cost is honest and bounded: it is a fixed start-of-session delay, not jitter. We measure it rather than assume it — roundtrip_delay_ms() runs a correlation between input and output through a down/up resampler pair and reports the lag, and the result is published in /health as transport delay, kept separate from the model's own 16 ms algorithmic latency. Two different delays with two different owners, reported as two numbers.

The checklist

If you run a browser-to-server audio path with a sample-rate conversion in it:

1. Count what you actually send. Seconds of audio delivered over wall-clock seconds, per session, exposed to whoever is debugging. Not bytes.
2. Never let a continue cover more than the thing it is guarding. If a block does two independent jobs, an early return from one is a silent failure of the other. Split them.
3. Probe your resampler's emission schedule at your real block size, with the four lines above. If you feed it 128-sample quanta, expect nothing back most of the time; if you feed it 20 ms server frames, you will never see this at all.
4. Feeding it a full output block's worth of input per call makes the empty return almost disappear — but do not build on that number, because it is version- and ratio-dependent, and the priming call is still empty. Handle the empty case correctly instead.
5. Do not compare two arms with different packetisation. Coalesce to one frame size before the transport, and report the per-arm frame rate so a divergence is visible rather than inferred.

And the general one, which is the only part of this that is not about audio: the instrumentation that would have caught this already existed and already had the right value in it. Building the metric is the easy half. Making the metric say what it implies, next to the symptom the person is actually looking at, is the half we got wrong.