Bhargava: 500ms response delay is enough to make voice users hang up

Building a voice agent isn't just about choosing a smart model; it’s a race against human biology. Rishabh Bhargava, lead of the voice AI team at Together AI, notes that humans communicate with cues in roughly 300 milliseconds. If an AI agent takes more than 500 milliseconds to respond, the illusion of conversation shatters. At one second, users simply hang up. This reality makes latency the primary engineering constraint, dictating every architectural choice from model size to physical server location.

Cascading pipelines define the modern agent

The dominant production strategy remains the pipeline architecture, a sequence of specialized models working in a relay. It begins with Speech-to-Text (STT), acts through a Large Language Model (LLM), and concludes with Text-to-Speech (TTS). Each stage consumes a portion of the total latency budget. To succeed, developers must treat this as an "and" problem: the agent must be fast, smart, natural, and scalable simultaneously. If the STT engine fails to capture a name correctly, the subsequent models have no way to repair that error, leading to a cascading failure of user trust.

Balancing intelligence against the clock

In the middle of the pipeline sits the LLM, which Bhargava describes as the "brain." Here, size matters for all the wrong reasons. While a massive model might offer superior reasoning, it will likely burn through the latency budget. The sweet spot currently exists in the 8 to 30 billion parameter range. Models in this bracket can hit a Time to First Token (TTFT) of 200 to 300 milliseconds while maintaining enough intelligence for complex tool calling. To bridge the gap between speed and capability, some developers employ a thinker-talker pattern. A small, fast LLM handles the immediate verbal flow, while a larger, more capable model handles heavy-duty tool calls or complex reasoning in the background.

Why physical distance kills performance

Even with optimized code, the speed of light remains a bottleneck. Many developers overlook the hidden tax of network latency. Bhargava illustrates a scenario where a system with optimized engine latency still loses 75 milliseconds—roughly 30% of its performance—simply because the models sit in different data centers. Co-location is the necessary fix. Moving the STT, LLM, and TTS components into the same building or data center can drop network overhead from 75 milliseconds to just 5. For real-time voice, every 10 milliseconds is a hard-won victory.

The shift toward streaming native architectures

Traditional models like Whisper were designed for batch processing, often requiring 30-second audio clips to function effectively. This is incompatible with real-time needs. A new generation of streaming-native models, such as those from Nvidia, utilizes encoders that look ahead only 80 milliseconds. These models cache activations, ensuring heavy computation happens only once as the audio stream progresses. This shift reduces the need for complex, homegrown "chunking" logic that often introduces jitter and artifacts into the conversation.

Pure speech-to-speech is the next frontier

While pipelines are the current standard, the future points toward unified speech-to-speech models. By removing the intermediate text layer, these models can natively understand prosody—hesitation, tone, and emotion—that text-based LLMs miss. This architecture enables full-duplex communication, allowing the model to "back-channel" with small sounds like "aha" or "I see" while the user is still speaking. Although these models currently struggle with tool calling and strict instruction following compared to their pipeline counterparts, they represent the eventual evolution of more human-centric AI interfaces.