SWE-bench

The Trap of Static Agent Benchmarks Many software teams rely on public benchmarks to evaluate their AI agents. While tools like SWE-bench offer a standard baseline, they fall short when you ship to production. Public datasets measure general capabilities, not your company's proprietary database states, internal tool APIs, or strict compliance policies. Rustem Feyzkhanov, Head of AI Platform at Snorkel AI, warns that relying solely on general pass rates is a recipe for failure. Real production systems care about cost, latency, retries, and strict system behavior. To build a system that works, you must turn your raw production observability traces into repeatable, offline simulation benchmarks. Anatomy of a Simulation Task To move from static evaluation to dynamic simulation, you must build what looks like a specialized integration test. A simulation task requires three core components: * **The Environment:** A containerized setup, typically configured via Docker or Docker Compose, that mimics production databases, files, and mock API services. * **The Oracle Solution:** A pre-recorded, successful run of the task. If your oracle cannot solve the task within the simulated environment, your agent stands no chance. * **Verifiers and Evaluators:** Automated checks that evaluate the final state of the system, rather than just the final string output. By packaging these elements using standardized schemas like the Harbor format, developers can run identical tasks across different agent architectures, model versions, and system prompts. This setup guarantees an apples-to-apples comparison. Running Multi-Step, Long-Horizon Environments Production workflows rarely finish in a single turn. They represent long-running processes that span hours and multiple system interactions. To simulate these without runaway API costs, you need to structure your environments carefully. First, isolate external dependencies using sidecar containers for local database snapshots and mocked API endpoints. Second, replace real human interactions with simulated user profiles run by smaller, prompt-guided LLMs. Finally, break long-horizon tasks into discrete steps with independent verifiers. If your agent fails a critical step early on, kill the simulation immediately to save computational resources. Designing Smart Verifiers Over Output Checks Checking an agent's final text output is rarely enough. In a complex database or coding task, the final answer might look correct even if the agent corrupted underlying data along the way. Reliable verification requires a multi-tiered approach. Use deterministic code checks to verify file changes and database states. For qualitative tasks, use LLM-as-a-judge patterns or programmatic evaluators to assess the agent's actual reasoning path. When automated verifiers disagree or signal edge cases, route those specific traces to human subject matter experts. This keeps your automated evaluation loop accurate without drowning humans in manual code reviews. Integrating Simulations into the CI/CD Pipeline Your benchmark cannot remain a static dataset. It must act as a living software project governed by a continuous integration pipeline. When a production agent fails, capture that specific trace, package it as a new simulation task, and add it to your test suite. Run your entire test suite automatically before every agent deployment. This continuous feedback loop ensures that fixing a prompt error on Monday does not trigger a catastrophic regression on Friday. Treat your agent benchmarks with the same engineering rigor as your core codebase.