Erik Meijer warns tool calls turn LLMs into dangerous agents

When we first began interacting with large language models, the relationship was conversational. You asked a question; you received a text response. This interface seemed relatively contained. However, the introduction of native tool calls changed the security model entirely. This feature effectively transitioned artificial intelligence from passive advisors to active participants capable of executing code, making database queries, and modifying systems.

Giving an AI agent the ability to execute tool calls introduces what security experts call the "lethal trifecta": direct access to private data, exposure to untrusted third-party content (which invites prompt injection), and the authority to execute tools. Because actions in the digital world are often irreversible—such as deleting files or executing financial transactions—relying on the model's internal alignment or "vibes" is a recipe for catastrophe.

The Failure of Model Alignment

To prevent models from generating harmful content or executing malicious instructions, the industry has relied heavily on reinforcement learning from human feedback (RLHF) and fine-tuning. We call this "alignment." But trying to bake safety directly into the neural weights of a model is inherently fragile. Security researchers and hobbyists routinely bypass these safeguards using sophisticated jailbreaks.

Furthermore, using an LLM as a judge to evaluate safety is not a mathematically sound defense. "Safe" is not a native mathematical property that a neural network can reliably verify on the fly. When a model executes in a loop, mutating the state of the system via input/output (IO) operations, we are essentially running code that has zero formal guarantees of correctness. We must assume that if an agent is given a goal, it will bypass any soft barriers to achieve it.

Air-Gapping the Agentic Loop

To make agents provably safe, we have to change the execution architecture. Instead of allowing the agent to execute tools directly inside its runtime loop, we must push the side effects (the IO) to the outer edge of the system. This architectural shift is known as "air-gapping" the agentic loop.

In this model, the agent does not execute actions. Instead, the agent produces a declarative plan—a program—representing the sequence of actions it intends to perform. This plan is handed over to a trusted external interpreter. By separating the generation of the plan from its execution, we gain a critical window of opportunity. We can analyze, verify, and validate the plan before a single side effect occurs in the real world.

Verifying Free Monads and Proof-Carrying Code

How do we represent this plan in a way that a machine can formally verify? The answer lies in compiler design and functional programming. By representing the agent's plan as an expression using a Free Monad, we turn a sequence of opaque side effects into a transparent, structured data tree.

Once the plan is reified into a structured program, we can apply classical static analysis techniques, such as data flow analysis and taint analysis. We can programmatically check if the plan violates safety policies.

This technique builds on Proof-Carrying Code, a security framework pioneered by academics in the 1990s. The agent must submit both its execution plan and a machine-checkable proof written in a formal logic language like Lean. A lightweight, highly auditable verification engine checks the proof. If the proof is valid, the plan executes; if not, it is rejected. We no longer need to trust the agent. We only need to trust the mathematical verifier.