Raj: poor web interfaces, not weak models, break today's browser agents

Many software teams assume building a better browser agent means waiting for a smarter foundation model. This assumption is wrong. Current models possess sufficient intelligence, yet existing agents consistently fail at basic, multi-step web workflows. Kushan Raj, former founding engineer at Sarvam AI, argues that the core bottleneck is the fragile interface we present to these models. Developers dump raw Document Object Model (DOM) data or rely solely on screenshots, forcing the agent to operate in a state of sensory overload.

The Math Behind Token Bloat

Raw web interfaces are incredibly noisy. A standard webpage DOM often consumes upwards of 20,000 tokens. This sheer volume introduces massive latency and high API costs. While a simple screenshot seems like an elegant alternative, it typically consumes about 1,100 tokens and offers only a partial, static snapshot.

To bypass this limitation, Raj designed a custom markdown representation that compresses an entire webpage down to just 1,800 tokens. This compact format preserves the semantic layout of the entire page, giving the model a highly efficient structural map without the overhead of raw HTML.

Fast Feedback Loops Beat Massive Models

Speed and execution reliability require active, step-by-step feedback loops. Traditional browser runtimes operate in a blind loop, sending a command and waiting for a final pass-or-fail state.

By tracking the runtime state continuously, the system can instantly alert the model when new elements appear, when pop-ups block targeted elements, or when a click action fails. This continuous environmental feedback enables cheap, lightweight models to recover from execution errors and complete complex workflows in seconds, outperforming larger models like Claude running on unoptimized runtimes.

Opening the Web Automation Pipeline

To make this runtime architecture accessible, the next step involves exposing this compressed page representation and execution engine to the public. Packaging this infrastructure as an API, a browser plugin, or an open-source framework allows developers to pass a simple URL and user intent, returning a completed execution sequence. Shifting the engineering focus from model scale to state representation is the key to reliable web automation.