Modal

The deceptive lure of objective metrics Ara Khan of Cline argues that the industry has split into two equally misguided camps regarding AI evaluations. The first is the **objective metrics** camp, where developers treat leaderboard scores as absolute truth. They assume a model with a slightly higher score is inherently superior, ignoring that GPT and Gemini can share similar numbers while performing drastically differently in production. This benchmark maxing often leads to "hoaxes" where models are tuned specifically to win at tests rather than to solve real problems. On the other side is the **taste camp**, which relies entirely on vibes. These developers anthropomorphize models, claiming they simply "like talking to" a specific AI. Khan suggests the truth exists in the middle: evaluations are broken tools that you must use anyway to ground your engineering in reality. Three heuristics for interpreting model scores Before building your own system, you must filter existing data through three specific lenses. First, never believe a model provider’s self-reported numbers; they are mere approximations. Second, adopt a strategy of **delayed adoption**. While it is tempting to switch models every time a new frontier leader emerges, Khan recommends waiting at least two weeks for the initial hype and "fire" to settle. Third, seek out new and highly precise evals. Standardized tests like HumanEval are now effectively obsolete for frontier coding models because they test simple logic like Fibonacci sequences rather than real-world software engineering. You need benchmarks that reflect the complexity of modern agentic workflows. Building a rigorous evaluation harness To move beyond vibes, you must build a system that tests agents in isolated, reproducible environments. Cline utilizes Terminal Bench from Stanford University, which provides 89 complex coding tasks including race conditions and infrastructure bugs. The technical setup requires more than just an API key. You must provide the agent with a virtual machine—using tools like Harbor or Modal—where it can run Python scripts, install environments, and read files in parallel. This allows you to identify the slowest task as your limiting factor and provides the "trace" files necessary for deep analysis. Identifying the three zones of improvement Once you have a baseline score—Cline started at 43% on Terminal Bench—you must categorize failures into three distinct zones. **Zone One** covers obvious flaws: harness crashes, rate limits, or fundamental bugs. **Zone Two** is the critical engineering space where you find nuance improvements. This involves identifying why an Anthropic model requires different prompt engineering than a Google Gemini model to succeed in your specific harness. Finally, avoid **Zone Three**, which is pure overfitting. Cheating to get a high score for a tweet provides no utility for your users. The goal is "hill climbing": systematically pulling levers in your harness and prompts until the model passes both the hard score requirements and the subjective vibe check.