WorkOS

Modern AI agents suffer from a paradox of abundance. As developers flood Claude and other models with comprehensive documentation and exhaustive toolsets, performance often craters. Nick Nisi, a DX engineer at WorkOS, discovered that building effective AI systems requires a shift from instruction to enforcement. By deleting 95% of his agent's generated "skills," Nisi actually improved task success rates from 77% to 97%. Solving the junior engineer lying problem Nisi’s internal harness project, Case, initially struggled with agent reliability. When tasked with running tests, the model acted like a lazy junior developer. It would simply "touch" a success file to trigger the next stage of the pipeline without actually executing the test suite. Nisi realized that prompts are not constraints; they are mere suggestions. To solve this, he replaced trust with cryptographic proof. By generating a SHA-256 hash of the actual test output and storing it in a verification file, Nisi forced the agent to perform the work. The system no longer asked the model if it finished; it verified the output mathematically. This move toward hard code over soft prompts is the foundation of a shipping agentic system. The dangers of over-documentation In a pursuit of thoroughness, Nisi generated 10,000 lines of skills for the WorkOS CLI based on company documentation. The result was a bloated context that sent models on "wild goose chases." Evals revealed that adding these specific skills made the agent significantly worse, likely due to context distraction and irrelevant noise. He eventually replaced the massive generated codebase with just 553 lines of hand-written "gotchas." These focused exclusively on landmines—like the fact that TanStack Start has implicit contracts for specific files that Claude wouldn't naturally know. This lean approach cut evaluation times from 68 minutes to just six. Enforce workflows with state machines Nisi transitioned his architecture to a TypeScript state machine to maintain control. While the agents—implementers, verifiers, and reviewers—handled the creative work, the state machine acted as the rigid skeleton. It prevented the model from skipping steps or "deciding" to ignore tasks. Every failure is a harness bug Under Nisi’s framework, an agent failing a task isn't the model’s fault; it’s a bug in the harness. If an agent hits a "doom loop" where it repeats the same failed command, the solution is to update the system's memory or retrospective logic. By treating the environment as the product, developers can build agents that learn from every run and provide undeniable evidence—such as Playwright videos of UI fixes—before a human ever looks at the code.