ContextBench

Embedding-as-cache approach to code retrieval Code search isn't just about finding strings; it’s about managing compute costs and agent accuracy. In traditional agentic search, tools like Claude Code rely on "grepping" through the file system. Every time an agent asks a question, it reads files, filters metadata, and parses content from scratch. This repetitive process consumes thousands of tokens per session. Kuba Rogut from Turbo Puffer argues that embeddings should be viewed as "cached compute." By chunking, embedding, and indexing a codebase once into a vector database, you create a permanent semantic map. When an agent queries the system, it doesn't need to re-scan the entire directory; it pulls the exact semantic context it needs. For developers, this translates to faster response times and significant token savings across multiple agent sessions. Benchmarking precision with ContextBench To prove the efficacy of vector search, Rogut utilized ContextBench, a human-labeled dataset designed to measure retrieval quality. Unlike benchmarks that only look at the final answer, ContextBench tracks if the agent looked at the "golden" files, lines, and symbols required to solve a task. Results showed that raw Claude Code hits about 65% file precision—meaning one in every three file reads is a total waste. By introducing windowed reads and semantic search via Turbo Puffer, file precision jumped to 87%. This reduces the noise ratio to just one in eight files, allowing the LLM to focus on relevant logic rather than wading through irrelevant boilerplate. Code walkthrough for Turbo Grep integration Implementing semantic retrieval involves a pipeline that transforms raw source code into searchable vectors. The following logic illustrates how to parse and index a repository. ```javascript // Step 1: Parse and Chunk // Use a tree-sitter library to maintain code structure const chunks = codeSplitter.split(sourceFile); // Step 2: Generate Embeddings // Utilize Voyage AI's code-specific model const embeddings = await voyage.embed(chunks, { model: "voyage-code-3" }); // Step 3: Upsert to Vector DB await turbopuffer.upsert("my-repo-index", chunks.map((text, i) => ({ id: `chunk-${i}`, vector: embeddings[i], attributes: { text } }))); ``` This pipeline allows the agent to call a specialized search tool instead of a generic grep. The tool performs a similarity search against the user's natural language query, returning the most relevant code blocks instantly. Choosing between semantic search and grep Data from the ContextBench run reveals that neither tool is a silver bullet. Claude Code wins on file recall during exploratory tasks because it aggressively reads everything. However, semantic search excels at finding "behavior-adjacent" files—files that are functionally related but lack shared keywords. Conversely, standard grep remains superior for import tracing, where the specific name of a module is already known. The most effective systems, like Cursor, use a hybrid approach, knowing exactly when to trigger a vector lookup versus a literal string search.