Etsy

Modern AI systems have reached a level of internal complexity where manual human debugging is no longer tractable. When an automated SRE tool like incident.io runs an investigation, it triggers hundreds of telemetry queries across logs, metrics, and traces. Founding engineer Lawrence Jones argues that when these systems fail, the resulting trace data is too vast for a human to parse. The solution isn't better UIs, but building internal tools specifically designed for Claude Code and other coding agents. CLI tools bridge the agent context gap Evals are essentially unit tests for prompts. At incident.io, these are stored in YAML files alongside Go code. However, as an AI system matures, these files often grow into multi-megabyte behemoths that exceed the context window of most LLMs. To solve this, the team built a specialized CLI called `eval-tool`. This allows a coding agent to query, edit, and append test cases without needing to ingest the entire file. This enables a robust red-green development cycle where an agent can programmatically verify that a prompt fix doesn't break existing behaviors. File systems outperform custom debug UIs While traditional dashboards help humans visualize traces, they are often useless for AI agents. The team discovered a massive unlock by serializing complex UI debugging views into downloadable, self-documenting file systems. By dropping these directories into a sandbox with Claude Code, the agent can use standard tools like `grep` to navigate the hierarchy of prompts and tool calls. ```bash Example agent workflow for debugging a failed trace $ eval-tool get-case --id "incident-123" $ claude-code "Analyze why the RCA in ./traces/123/ failed. Fix the prompt in ./prompts/analysis.go" ``` Parallel analysis at fleet scale When tracking systemic performance across hundreds of customer accounts, individual debugging isn't enough. The team utilizes a "scrapbook" repository that runs 25 agents in parallel. Each agent performs a deep-dive analysis on a single investigation, storing its findings in Markdown files. A secondary clustering stage then aggregates these findings to identify cohort-level failure patterns. This structured pipeline transforms raw telemetry into actionable engineering tasks, allowing developers to focus on architectural fixes rather than data mining. Tips for building agent-friendly internals To replicate this success, prioritize plain-text formats over proprietary UIs. Use ASCII representations for complex traces to make them readable for LLMs. Finally, treat your internal runbooks as code; by defining analysis steps in structured Markdown, you provide the necessary guardrails for agents to perform repeatable, reliable work.