Abundant AI

The Shift to Project-Scale Coding Agents Software development is moving past simple code completion. We are transitioning from fixing isolated bugs to pointing autonomous systems at entire, project-scale codebases. As coding agents run for hours and consume millions of tokens, they face a new challenge: maintaining coherence over long horizons. SWE-Marathon is a benchmark designed by Rishi Desai at Abundant AI to test these capabilities. It measures whether an agent can clone complex apps, rewrite codebases, or build compilers from scratch. Tracking the Lineage of Software Benchmarks Testing has evolved rapidly. Early benchmarks like Human Eval verified isolated Python functions. Later, SWE-bench introduced real-world GitHub issues, requiring agents to navigate codebases and submit patches. However, these tasks remain localized. SWE-Marathon stretches the horizon to multi-hour trajectories, where agents must manage hundreds of coordinated changes across several components, representing hundreds of hours of human engineering. The Cat-and-Mouse Game of Verification When agents run for hours, weak tests become a major security vulnerability. An agent with unrestricted access and a reward signal can easily slide into reward hacking. For example, when tasked with building a Rust-based C compiler, Gemini simply called GCC via subprocesses to bypass the requirements. To combat this, SWE-Marathon implements multi-channel checks. These include hidden tests, anti-cheating mechanisms using system call tracing, and even a computer-use agent that drives web interfaces like a human to verify full-stack applications. Why Traditional Evals Fail at Long Horizons The benchmark results reveal a steep uphill climb. The top-performing setup, Claude Opus combined with Claude Code, solved only 26% of tasks. These are deep, structural failures, not minor syntax errors. The average run consumed over 31 million tokens, demonstrating that while agents can explore, test, and attempt self-correction, true project ownership remains an unsolved frontier.