Blue Sky

The hidden decay of static model training Most developers treat Large Language Models (LLMs) like static calculators, but Danilo Campos of PostHog warns that "model rot" is the primary silent killer of reliable code generation. Training a model is an immense capital expense for companies like Anthropic; consequently, these models are snapshots of a web that existed months or years ago. For fast-moving software projects, this 12-month lag means the agent is essentially guessing at APIs that have since evolved. Instead of relying on base model knowledge, the PostHog Wizard fights rot by treating documentation as a just-in-time injection. By serving fresh markdown files directly into the context window, you bypass the need for complex Retrieval-Augmented Generation (RAG) architectures. If the context window is large enough, simply shoving updated documentation into the prompt is the most effective way to ensure the agent doesn't invent non-existent keys or patterns. Building model airplanes for structural consistency When agents are left to their own devices, they often produce technically functional but architecturally bizarre code. To fix this, Campos advocates for "model airplanes"—thinned-down versions of production applications. These are not full boilerplate projects; they are simulacrums where complex logic like authentication is "off-shaped." For instance, a model airplane might allow any password to work, but it maintains the exact file structure and hook placement where event tracking should live. This provides a structural pattern for the agent to complete, ensuring the resulting code fits the intended architecture without wasting tokens on irrelevant production logic. Breadcrumbing and the end of clever code To prevent agents from finding 15,000 different ways to solve the same problem—a nightmare for support teams—you must sequence information through "breadcrumbing." Rather than telling the agent the final goal immediately, start by asking it to identify files with "business value." Once it identifies a login or Stripe interface, move to the next step: listing events without writing code. This incremental approach limits improvisation and keeps the agent on a predictable path. Perhaps the most jarring shift for veteran developers is the realization that prose is now more valuable than logic. The PostHog Wizard is composed of 90% markdown and only 8% actual tools. While code is a depreciating asset that rots the moment it's written, high-quality prose is an appreciating asset. As models improve, they will extract more utility from the same markdown instructions, whereas clever code remains static or becomes obsolete. Interrogating the robot for better tool control Human error remains the biggest threat to agent success. Developers often provide contradictory instructions or forget to grant tool permissions. To solve this, Campos uses "inference time interrogation." At the end of every run, the system asks the robot what could have been done better. This feedback loop revealed critical failures, such as agents receiving JavaScript instructions for Python projects. Furthermore, fine-grained tool control is essential for security. Instead of letting an agent read an entire `.env` file—which risks leaking secrets to cloud logs—give it specific tools that can only check for the presence of a key or write a new value, keeping the sensitive contents invisible to the LLM.