Mixedbread AI shuts down the Oracle Gap with custom-trained search agents

We are building incredibly smart models that act like cavemen when searching for data. Over the past few years, the reasoning capabilities of large language models have grown exponentially. Yet, retrieval has improved at a glacial pace. This massive disparity creates a massive bottleneck.

When we test top-tier models on complex deep-search benchmarks, their performance falls off a cliff when dropped into a noisy corpus. This gap between theoretical maximum performance with perfect documents and the actual results with standard search tools is what we call the Oracle Gap. It is not a reasoning failure. It is a retrieval failure.

Why Your Agents Write Gibberish Queries

If you inspect the raw queries generated by autonomous agents, you will find a mess of keyword-stuffed strings resembling old-school BM25 search patterns. There are three structural reasons for this behavior:

• Coding Agent Bias: Most models are trained on code-base exploration tasks, relying heavily on tools like grep to find exact regular expressions.
• Web Search Mimicry: Models try to mimic human search behavior on web tools, which are heavily optimized for simple keyword concatenation.
• Benchmark Distortions: Popular evaluation benchmarks structures favor exact entity overlaps, rewarding "caveman" style queries.

Because of this conditioning, agents struggle to utilize advanced semantic search systems effectively.

Designing a Custom Multi-Tool Search Harness

To break this cycle, the engineering team at Mixedbread AI built a specialized search harness that enforces better query behavior. This architecture runs on a short, fast execution loop constrained to four search rounds. It relies on four highly distinct search tools:

1. Overview Search: A broad semantic tool that retrieves summaries of up to 50 chunks to prevent context stuffing.
2. Main Semantic Search: A targeted tool returning the full payload of the top 10 relevant chunks.
3. Filter Chunks: A utility to sort and filter chunks based on metadata facets.
4. Grep Tool: Dedicated purely to exact keyword matching.

By instructing the agent to write a single, natural sentence describing its target rather than a "query," the framework tricks the model into leveraging true semantic searching.

Fine-Tuning and Reinforcement Learning with Custom Rewards

Teaching an agent to navigate this harness efficiently requires custom training. We trained a smaller, faster model using supervised fine-tuning backed by a teacher model, followed by on-policy reinforcement learning.

Our custom reward function balances a traditional retrieval score with trajectory evaluations. An LLM judge evaluates the trajectory, penalizing keyword-stuffed queries and rewarding natural phrasing, optimal tool selection, and smart parallel search exploration.

Early testing shows incredible promise. On the Oblique Congress benchmark, the agent achieved an NDCG@10 of 0.40, vastly outperforming the previous state-of-the-art multi-hop agent's score of 0.18. By co-designing the agent and the search interface, we can finally close the Oracle Gap.