Microsoft engineers build Chronicle to replay broken AI agent runs
The illusion of LLM determinism
When an AI agent fails in production, standard telemetry fails with it. You pull the raw prompt, run it locally, and watch it work perfectly ten times in a row. This non-deterministic behavior leaves engineering teams stranded. They cannot reproduce the bug, meaning they cannot guarantee a fix.
Many developers try to force determinism by dropping the model temperature to absolute zero. This is a mistake. Temperature zero does not guarantee identical runs.
Why temperature zero cannot fix your code
Hardware-level non-determinism makes bitwise consistency impossible on GPUs. Tisha Chawla and Susheem Koul from Microsoft outline why this happens. First, floating-point addition is not associative; tiny changes in matrix operations flip the winning token. Second, batch invariance groups requests unpredictably, altering routing limits within mixture-of-experts (MoE) architectures.
Chasing token-level consistency is a losing battle. You do not need the model to return the exact same characters. You need your system to execute the exact same state transitions.
Replaying runs through execution boundaries
Instead of freezing the model, developers must capture what the agent did. This shift moves focus from bitwise determinism to replayability. To achieve this, the engineers built Chronicle, a tool designed around the concept of a Boundary.
A boundary wraps any node in an agentic workflow, such as an LLM call, a tool execution, or vector database retrieval. The boundary records every input and output pair along with metadata like model versions and code states. It freezes the exact context of a live run.
Turning recorded failures into free test cases
During debugging, developers can load this frozen trace and activate replay mode. This stubs every node except the one under review. By mock-running the LLMs with their cached outputs, engineers can test code changes locally. This deterministic testing costs nothing because it completely bypasses the hosted API.
- Boundary
- 20%· products
- Chronicle
- 20%· products
- Microsoft
- 20%· companies
- Susheem Koul
- 20%· people
- Tisha Chawla
- 20%· people
Your Agent Failed in Prod. Good Luck Reproducing It. - Tisha Chawla & Susheem Koul, Microsoft
WatchAI Engineer // 14:10
We turn high signal in-person events for the top AI engineers, founders, leaders, and researchers in the world into the best free learning opportunities for millions around the world here on YouTube. Your subscribes, likes, comments, speaking, attendance, or sponsorships goes a long way toward making our biz model sustainable indefinitely. We strongly believe this industry deserves a better class of community and that we know how to do this well; we just need your support.