Witan Labs

The Hidden Complexity of the Grid Most software developers view spreadsheets as simple, flat grids. We assume an LLM that can write complex Python scripts will easily navigate a financial model. It turns out that spreadsheets are deceptively difficult for AI. When a human opens Microsoft Excel, they instantly grasp the visual layout. They spot the revenue tables, highlight assumptions, and track the flow of numbers. An AI agent sees none of this. To a model, a spreadsheet is a chaotic maze of coordinate pairs, hidden dependencies, and ambiguous labels. Nuno Campos and his team at Witan Labs spent four months teaching coding agents to interact with spreadsheets. They discovered that success requires moving beyond basic tool calls and building high-fidelity feedback loops. The REPL Breakthrough In early designs, the team built fifteen separate tools for the agent. The agent had to make sequential tool calls to read cells, write formulas, and check values. This approach created massive lag. High latency led to frequent timeouts. ```javascript // Instead of 15 separate tool calls, agents run a single script const data = sheet.getRange("A1:B10"); const total = data.filter(r => r[0] === "Revenue").reduce((sum, r) => sum + r[1], 0); ``` The breakthrough came when they replaced these fifteen tools with a single Node.js REPL. This interface allowed the agent to write a complete script to execute multiple actions in one round trip. Crucially, they chose a REPL with persistent state over a simple run-and-die code mode. With persistent state, the agent defines variables in one turn, analyzes the result, and uses those same variables in the next step. This design shortened the scripts the agent wrote, allowing more room for step-by-step reasoning. The Need for High-Fidelity Feedback An agent cannot write good code without a compiler, and it cannot build a spreadsheet without a calculation engine. To enable true self-correction, Witan Labs built two core validation engines: a formula engine to calculate cell values and a rendering engine to show the sheet as an image. They quickly learned a vital lesson about engine quality. If your custom calculation engine only supports half of Excel's formulas, you make the agent worse. The agent writes a valid formula, but the incomplete engine throws an error. The agent then gets confused, assumes its logic was wrong, and tries to fix code that was already correct. High-fidelity engines are non-negotiable for the validation loop. How to Measure True Progress When optimizing their system from 50% to 92% accuracy, the team had to overhaul their evaluation methods. Relying solely on an LLM-as-a-judge proved fragile. Changes in the evaluator's prompt would shift scores, making it impossible to tell if the agent had actually improved. They pivoted to deterministic testing. They took a golden spreadsheet with known inputs and outputs, fed the same inputs to the agent's generated sheet, and checked if the outputs matched. This black-box verification provided a rock-solid metric. Many apparent reasoning failures were actually simple framework bugs. By analyzing traces and fixing their plumbing, they cleared the path to production-grade reliability.