Susheem Koul

The illusion of LLM determinism When an AI agent fails in production, standard telemetry fails with it. You pull the raw prompt, run it locally, and watch it work perfectly ten times in a row. This non-deterministic behavior leaves engineering teams stranded. They cannot reproduce the bug, meaning they cannot guarantee a fix. Many developers try to force determinism by dropping the model temperature to absolute zero. This is a mistake. Temperature zero does not guarantee identical runs. Why temperature zero cannot fix your code Hardware-level non-determinism makes bitwise consistency impossible on GPUs. Tisha Chawla and Susheem Koul from Microsoft outline why this happens. First, floating-point addition is not associative; tiny changes in matrix operations flip the winning token. Second, batch invariance groups requests unpredictably, altering routing limits within mixture-of-experts (MoE) architectures. Chasing token-level consistency is a losing battle. You do not need the model to return the exact same characters. You need your system to execute the exact same state transitions. Replaying runs through execution boundaries Instead of freezing the model, developers must capture what the agent did. This shift moves focus from bitwise determinism to replayability. To achieve this, the engineers built Chronicle, a tool designed around the concept of a Boundary. A boundary wraps any node in an agentic workflow, such as an LLM call, a tool execution, or vector database retrieval. The boundary records every input and output pair along with metadata like model versions and code states. It freezes the exact context of a live run. Turning recorded failures into free test cases During debugging, developers can load this frozen trace and activate replay mode. This stubs every node except the one under review. By mock-running the LLMs with their cached outputs, engineers can test code changes locally. This deterministic testing costs nothing because it completely bypasses the hosted API.