Sonam Pankaj blames static retrieval for 73 percent of agent failures

Most production AI agents start every single run with a blank slate. You track tool calls, log exceptions, and run evaluations, but that hard-earned performance data never actually reaches the agent. Sonam Pankaj, CEO of StarlightSearch, points out that the evaluation signal dies right inside your observability dashboard. An engineer must manually inspect the logs, rewrite prompts, and redeploy. This manual loop keeps agents failing at the exact same tasks.

Semantic similarity is not enough for complex reasoning

Traditional memory systems like LangChain or Mem0 focus heavily on user preferences and semantic similarity. They pull information that sounds similar to the query. But similarity does not equal utility. Just because a past run looks like the current task does not mean its execution steps actually succeeded. We need memory that records reasoning paths and knows which trajectories led to successful outcomes. If a customer support bot needs to process a refund, it does not need to know the user's favorite UI theme. It needs the exact sequence of system checks that prevented a double payout.

How utility scores rank memories like credit ratings

To bridge this gap, Agent RX introduces a utility score to re-rank retrieved memories. This system weights standard semantic similarity against historical success. Think of it as a credit score for agent memory. When a retrieved memory helps the agent pass its evaluation, its utility score goes up. If the run fails, the score drops. The agent learns from outcomes at runtime without requiring expensive model fine-tuning or manual prompt engineering.

Baking runtime lessons into permanent skills

Dynamic re-ranking prevents context stuffing by feeding only highly useful memories into the LLM context. Once a specific reasoning pattern proves its utility across multiple runs, Agent RX bakes that lesson directly into a "skill." This automatically updates the agent system prompt at runtime. If a database schema changes, the agent quickly learns to stop querying deprecated columns based on actual failures, bypassing the cold start challenge of static prompts. Benchmark results show this method boosts agentic task success rates from a baseline of 35.7% to 61.3%.