Cloudflare engineer cuts AI token bloat by 99% with JavaScript execution

Traditional tool calling relies on a rigid back-and-forth exchange of JSON objects. While effective for simple tasks, this architectural pattern buckles under the weight of enterprise-scale systems. When you attempt to cram thousands of API endpoints into an AI's context window, you hit a wall of token exhaustion and latency. Sunil Pai argues that the industry is hitting a ceiling with the Model Context Protocol and standard tool-calling paradigms.

Instead of asking an AI to pick a tool from a massive list, Code Mode instructs the model to write and execute JavaScript directly. This approach treats the LLM as a programmer rather than a dispatcher. By generating executable scripts, agents can handle complex logic like looping, state management, and parallel execution in a single round trip, rather than waiting for multiple turns of JSON validation.

Solving the million-token API problem

The scale of modern platforms like Cloudflare illustrates the necessity of this shift. With over 2,600 API endpoints, exposing every function as a discrete tool would consume approximately 1.2 million tokens per request. This is economically and technically unfeasible for real-time agents.

By implementing two specific tools—search and execute—Cloudflare reduced this overhead to just 1,000 tokens. The model uses search to find the OpenAPI spec it needs, then writes a script to interact with the relevant endpoints. This 99.9% reduction in token usage enables agents to respond to complex crises, such as identifying and blocking offending IPs during a DDoS attack, in a fraction of the time.

Architecture of the execution harness

Running AI-generated code requires a specialized software architecture known as a "harness." This isn't a traditional container or virtual machine; it is a hardened environment designed for ephemeral, high-speed execution. Cloudflare utilizes V8 isolates to provide sub-millisecond startup times and a decade of security hardening.

// A conceptual look at capability-based execution
const harness = new SandboxHarness({
  permissions: ['network.cloudflare.api', 'state.read'],
  timeout: 500 // strict limits on execution
});

// The agent generates this logic to run inside the harness
const runAgentTask = async (api) => {
  const workers = await api.listWorkers();
  return workers.filter(w => w.status === 'error');
};

Key to this architecture is capability-based security. The sandbox begins with zero permissions. Developers must explicitly grant capabilities, such as specific network fetches or API access. This granular control ensures that even if an LLM generates hallucinations or malicious code, the damage is strictly contained within the observer-aware environment.

Emergent behavior and the end of fixed UIs

When models inhabit the state machine of an application rather than just generating static output, emergent behaviors appear. Kenton Varda demonstrated this by asking an AI to play tic-tac-toe on a canvas app. Instead of writing a specific game engine, the model inspected the raw array of coordinate strokes and "played" by adding its own vector points to the data structure.

This paves the way for Generative UI, where the software interface is no longer a static set of buttons but a custom-coded experience built on the fly for the user's specific intent. Whether returning shoes or debugging a server, the agent constructs the exact logic and interface needed for that moment, rendering the traditional "one-size-fits-all" dashboard obsolete.