Opus 4.7

The brutal math of voice latency When you build a voice agent, silence kills the experience. If an artificial intelligence pauses for more than a second, the human brain assumes the system broke. Ornella Bahidika, a product manager at Microsoft, explains that the target latency budget is exactly 950 milliseconds. Large frontier models like Opus 4.7 lose the room because their reasoning steps take several seconds. Outsource the thinking to code To hit sub-second response times, Joel Allou stripped all logic out of the LLM. In their live AI voice tutor, Ace, a small model like Haiku 4.5 handles exactly one task: speaking. The model does not track student progress or plan the lesson. Instead, an external deterministic system coordinates the lesson and hands the model a tight, pre-digested summary at each turn. Why state machines beat raw inference The secret lies in a deterministic state machine built directly into the codebase. This scaffolding coordinates steps, tracks student mastery, and plans the conversation flow. Feeding structured instructions to Haiku 4.5 drops the response time to 900 milliseconds. Moving the cognitive heavy lifting from inference to local code saves time and cuts operational costs. The hidden tax of small models This design pattern requires upfront effort. Small models easily drift and require strict structural scaffolding to remain on track. You pay this engineering tax once during development, rather than paying a latency tax on every user interaction. For real-time applications, the model should be the smallest, fastest part of the entire system.