You can spend weeks on a voice agent.

The prompt. The flow. The voice. Shaving milliseconds off time-to-first-audio. You test it, it sounds beautiful, you are proud of it.

Then a customer picks up on a busy road.

A truck goes past. A TV in the next room. Music somewhere. A crowd outside.

And everything you optimised starts falling apart.

Because the agent is not hearing the customer. It is hearing everything.

Noise is not a transcription problem

This is the part that surprised us. Most teams treat background noise as an accuracy problem: the transcript gets a bit worse, the speech model does its best, you move on.

But the same audio feeds four different decisions in a realtime pipeline, and noise damages all four.

  • Speech recognition — words get dropped, substituted, or invented.
  • Voice activity detection — the pipeline cannot tell whether anyone is speaking.
  • Turn detection — the agent cannot tell when the customer has finished.
  • Barge-in — the agent cannot tell a real interruption from a passing horn.

Only the first one shows up in a transcript. The other three are what the customer actually feels.

The agent stops mid-sentence because a horn looked like the caller starting to speak.

It waits in awkward silence because VAD thinks someone is still talking.

It talks over the customer because turn detection missed that they had already started.

It ignores a real interruption, and the whole thing stops feeling like a conversation.

None of that is a model problem. Your speech model can be state of the art and your prompt can be perfect. If the audio arriving at the pipeline is noisy, every stage after it is working from bad input.

Garbage in, garbage out. The oldest rule there is. In voice AI it is brutal, because the garbage arrives sixty times a second and the agent has to answer in under a second.

The failure travels further than you think

Here is a real call from our staging environment. A caller on an inbound realtime agent, with a YouTube video playing in the room. Same audio, two tracks — before noise suppression and after — both transcribed with the same speech model.

Before:

Hello, can you hear me? वही पता कह रहा जिससे आपके साथ airport पर बैठ हुई Ok. तो actually I want to know about your service. Hello?

After:

Hello. Can you hear me? Ok. So actually I want to know about your service. Hello?

The Hindi sentence in the middle is the YouTube video. The caller never said it. The speech model heard it, transcribed it with 98% confidence, and attributed it to the customer.

That text does not stop at the transcript. It goes into the LLM context as something the customer said. The agent reasons over it. It shapes the next question. It ends up in the extracted fields, and from there it lands in the CRM as a fact about a real person.

That single pair of transcripts is the whole business case. Not the decibel numbers. The sentence the customer never said, sitting in your database.

What did not work

Our first attempt was a classical denoiser — MMSE spectral subtraction, the kind of thing that ships in every DSP library and runs in microseconds.

We built it, ran it, and deleted it.

Classical spectral subtraction estimates a noise profile and subtracts it. That works beautifully on stationary noise: a fan, a hum, an air conditioner, line hiss. It does almost nothing for babble, traffic, television or a crowd — which is the overwhelming majority of what an Indian phone call actually sounds like.

It was not a tuning problem. It was the wrong tool for the noise we have.

What worked: a small neural denoiser, on the CPU

What replaced it is a small streaming neural denoiser — roughly 48,000 parameters, half a megabyte on disk, running at 16 kHz on the CPU. No GPU. No third-party noise-suppression service in the media path.

That last part matters more than it sounds. Sending caller audio out to an external denoiser adds a network hop to the most latency-sensitive segment of the call, and puts customer audio through another vendor. Doing it in-process keeps both the milliseconds and the data where they belong.

The measured result on that same call:

  • Noise floor: −55.2 dBFS to −70.5 dBFS — a 15.2 dB reduction.
  • Signal-to-noise ratio: 28.4 dB to 39.5 dB — an 11.1 dB improvement.
  • CPU: about 2.5 ms per 16 ms frame, one streaming session per call.
  • Memory: about 45 MB per concurrent call.

The shape of the reduction matters as much as the size of it. On a control call where we switched the noise off midway, the denoiser removed 1–4 dB while the caller was speaking and 14–21 dB in the gaps between words. It is suppressing the background, not flattening the voice.

At fifty concurrent calls that is roughly two gigabytes and half a core. That is a cost you can actually carry.

Where it has to sit

Noise suppression only works if it runs on the caller uplink, before anything else consumes the audio. Before the speech model. Before the speech-to-speech vendor. Before VAD.

Put it anywhere else and you have cleaned the transcript while leaving turn detection and barge-in reading the raw noisy stream — which is how you end up with a clean transcript of a conversation that still feels broken.

The other half of that lesson is coverage. A production stack does not have one audio path, it has several: browser calls, telephony calls, cascaded pipelines, speech-to-speech pipelines, each with its own sample rate and its own entry point. We had five. The denoiser was wired into four of them.

The one it was missing was, of course, a path real customers use.

If you take one operational thing from this post, take that one: enumerate every ingress path and check each individually. "We shipped noise suppression" is not a true statement until all of them are covered, and the code will not tell you — a path with no denoiser looks exactly like a path with a denoiser that never triggered.

It has to fail open

A denoiser sits directly in the media path of a live phone call. If it stalls, the call stalls. If it crashes, the call is over.

So the requirement is not "clean the audio". It is never make the call worse than no denoiser at all.

In practice that means a few non-negotiables:

  • If the worker dies, in-flight audio and everything after it passes through unchanged, not dropped.
  • Output stays in caller order. Reordered frames are worse than noisy frames.
  • Restart on failure, with a bounded number of attempts — one garbage-collection pause should not silently bypass the rest of a five-minute call.
  • Report an honest per-call status: active, partial, bypassed, not confirmed. "Enabled in settings" is not evidence that it ran.

That last one is worth dwelling on. For a while we could not answer the question "what percentage of calls last week actually had suppression running?" — not because it was broken, but because nothing emitted the number. A feature you cannot measure in production is a feature you are trusting on faith.

The honest limits

This is the part that usually gets left out, so:

It removes noise, not other voices. In the call above it removed the YouTube speech only because a laptop speaker is muffled and reverberant enough that the model read it as noise. A real person speaking clearly next to your customer comes straight through. That is a genuinely different problem — target speaker extraction, where you enrol the caller's voice and keep only that — and no amount of denoising will solve it.

It dulls the top of speech. On the loudest frames it attenuates vowels by about 1 dB but consonants and sibilants by around 5 dB. It did not cost us accuracy here. On a soft-spoken caller it might.

The model is out of domain. It was trained on 16 kHz wideband read speech. Telephony audio arrives at 8 kHz and the top half of the spectrum is simply gone. It works anyway, which is a pleasant surprise rather than a design.

It is not echo cancellation and it is not gain control. A denoiser preserves the agent's own voice echoing back down the line, because that is speech. Those are two more problems, and they need their own answers.

Real calls are the only judge

We keep both tracks of every call — the raw caller audio and the cleaned caller audio — as separate recordings. Same call, same words, one variable.

That pairing is what turns "it sounds better" into something you can argue about. You can run both through the same speech model and compare word error rates. You can put both through voice evals and see whether turn-taking actually got smoother. You can find the call where it made things worse.

Everyone demos the perfect conversation in a quiet room. We did too, before we started measuring. Very few people show the customer calling from a street, an airport, a restaurant, or the driver's seat — which is where the calls actually come from.

Final thought

The next generation of voice agents will not be won by whoever has the smartest model or the cleverest prompt.

A lot of it will be won several layers below that, in the unglamorous work of making sure the audio reaching the pipeline is the customer's voice and nothing else.

Every stage downstream depends on what the agent hears. It is worth getting that part right first.You can spend weeks on a voice agent.