Jones: AI complexity now requires agents to debug their own code

CLI tools bridge the agent context gap

Evals are essentially unit tests for prompts. At incident.io, these are stored in YAML files alongside Go code. However, as an AI system matures, these files often grow into multi-megabyte behemoths that exceed the context window of most LLMs. To solve this, the team built a specialized CLI called eval-tool. This allows a coding agent to query, edit, and append test cases without needing to ingest the entire file. This enables a robust red-green development cycle where an agent can programmatically verify that a prompt fix doesn't break existing behaviors.

File systems outperform custom debug UIs

While traditional dashboards help humans visualize traces, they are often useless for AI agents. The team discovered a massive unlock by serializing complex UI debugging views into downloadable, self-documenting file systems. By dropping these directories into a sandbox with Claude Code, the agent can use standard tools like grep to navigate the hierarchy of prompts and tool calls.

# Example agent workflow for debugging a failed trace
$ eval-tool get-case --id "incident-123"
$ claude-code "Analyze why the RCA in ./traces/123/ failed. Fix the prompt in ./prompts/analysis.go"

Parallel analysis at fleet scale

When tracking systemic performance across hundreds of customer accounts, individual debugging isn't enough. The team utilizes a "scrapbook" repository that runs 25 agents in parallel. Each agent performs a deep-dive analysis on a single investigation, storing its findings in Markdown files. A secondary clustering stage then aggregates these findings to identify cohort-level failure patterns. This structured pipeline transforms raw telemetry into actionable engineering tasks, allowing developers to focus on architectural fixes rather than data mining.

Tips for building agent-friendly internals

To replicate this success, prioritize plain-text formats over proprietary UIs. Use ASCII representations for complex traces to make them readable for LLMs. Finally, treat your internal runbooks as code; by defining analysis steps in structured Markdown, you provide the necessary guardrails for agents to perform repeatable, reliable work.