Soheil Feizi says static production logs cannot fix failing AI agents

The Three Layers of Agent Optimization

Optimizing an agent is not a one-size-fits-all task. When an agent fails, developers can patch the system at three distinct layers.

First, the model layer allows developers to change the actual weights of the underlying LLM. While techniques like supervised fine-tuning (SFT) or reinforcement learning post-training—such as DPO or GRPO—are powerful, they require massive compute. Utilizing LoRA can make this cheaper, but it still demands structured benchmarks and explicit evaluators.

Second, the harness layer handles context engineering. This involves rewriting prompts, updating system code, or modifying tool integrations. Methods like prompt search mutate instructions to find winning candidates, but they also require explicit evaluation systems to verify changes.

Third, the memory layer stores facts and distilled skills. This is the fastest and cheapest layer to update. However, blindly writing corrections to memory is unverified. It risks introducing silent regressions where fixing one bug breaks a previously working behavior.

Four Core Pillars of Verifiable Continual Learning

To move past fragile, manual hotfixes, Feizi introduces Verifiable Continual Learning (VCL). This methodology relies on four non-negotiable principles.

• Replayability: You must transform static logs and feedback into an executable, simulated learning environment. This simulation mimics user interactions, mock tools, and success metrics, making the failure fully testable.
• Holisticness: A single failure can stem from multiple layers. A robust optimizer must analyze the root cause and route the fix to the precise layer—whether memory, harness, or model weights—with the smallest possible footprint.
• Lifelongness: Fixing a new bug must not introduce regressions. VCL integrates regression testing directly into the optimization loop rather than treating it as a post-hoc check.
• Efficiency: The learning loop must run quickly and frequently without scaling its computational cost linearly with the historical test suite size.

Driving Improvements in Practice

Implementing this paradigm shifts how developers maintain AI systems in production. Instead of manually writing custom test suites for every edge case, developers can generate simulated users and mock environments directly from a single feedback log.

In testing, this automated loop yielded a measurable 10% average score improvement in agent performance on a customer support benchmark. Rather than relying on guesswork, developers receive a clean pull request showing exactly which prompt, memory slot, or tool integration changed—and verified proof that existing features remain completely unbroken.