Andrej Karpathy

Overview The landscape of Large Language Model (LLM) development is undergoing a fundamental shift away from "prompt engineering" toward a rigorous programming paradigm. DSPy represents this evolution, providing a declarative framework for building modular software where LLMs are treated as first-class citizens. Instead of manually tweaking strings to coax specific behaviors out of a model, developers define the **intent** of their program through typed interfaces and logical modules. Kevin Madura, a technical consultant at AlixPartners, argues that this transition is essential for enterprise-grade applications that require testability, robustness, and transferability across different models. This tutorial explores how to use DSPy to decompose complex business logic into maintainable Python code. We will examine the core primitives that allow you to separate the structure of your program from the implementation details of the underlying LLM. By the end of this guide, you will understand how to build a multi-stage pipeline that can classify, route, and process various document types using optimized prompting strategies that the system generates for you. Prerequisites To follow this tutorial, you should have a baseline understanding of the following concepts and tools: * **Python Programming**: Familiarity with classes, decorators, and asynchronous programming in Python. * **Pydantic**: Knowledge of Pydantic for data validation and settings management, as it underpins much of DSPy's type hinting. * **LLM Basics**: An understanding of how LLMs process tokens and the general concept of system prompts vs. user messages. * **Environment Setup**: A working Python environment with an API key for a provider like OpenAI, Anthropic, or Google Cloud (or an aggregator like OpenRouter). Key Libraries & Tools * **DSPy**: The core declarative framework used to structure and optimize LLM programs. * **LightLLM**: Used under the hood by DSPy to provide a unified interface for calling various model providers. * **Attachments**: A utility library that simplifies working with disparate file types (PDFs, images) and converting them into LLM-friendly formats. * **Phoenix**: An observability platform from Arize AI used for tracing and debugging LLM calls within the DSPy ecosystem. * **BAML**: A domain-specific language for extracting structured data from LLMs, which can be used as an adapter within DSPy for better token efficiency. Section 1: Signatures as Declarative Intent The heartbeat of any DSPy program is the **Signature**. A signature defines *what* a task should accomplish without specifying *how* it should be prompted. This is a critical distinction: you are defining the inputs and outputs, and DSPy handles the transformation into a prompt. Shorthand Signatures For simple tasks, you can use a shorthand string notation. This is ideal for rapid prototyping: ```python import dspy A simple sentiment classifier shorthand sentiment_predictor = dspy.Predict("text -> sentiment:int") response = sentiment_predictor(text="The service was absolute garbage.") print(response.sentiment) ``` In this example, `text -> sentiment:int` tells DSPy that the input field is named `text` and the output field is an integer named `sentiment`. Class-based Signatures For more complex enterprise logic, class-based signatures allow you to provide docstrings and field descriptions that the model uses to understand the context. These descriptions essentially function as "mini-prompts" embedded within your code structure. ```python class DocumentClassifier(dspy.Signature): """Classify the type of document based on visual and text content.""" document_images = dspy.InputField(desc="Images of the first few pages of the document") document_type = dspy.OutputField(desc="One of: SEC_FILING, PATENT, CONTRACT, OTHER") Usage classifier = dspy.Predict(DocumentClassifier) ``` Section 2: Building Logic with Modules **Modules** are the organizational units of DSPy, analogous to layers in a neural network. A module wraps one or more signatures and can include custom control flow, database calls, or other Python logic. Every module inherits from `dspy.Module` and implements an `__init__` method to define its components and a `forward` method for the execution logic. ```python class SupportAnalyzer(dspy.Module): def __init__(self): super().__init__() self.categorize = dspy.ChainOfThought("message -> category") self.sentiment = dspy.Predict("message -> sentiment:int") def forward(self, message): category = self.categorize(message=message).category sentiment = self.sentiment(message=message).sentiment # Add hard-coded business logic is_urgent = (sentiment < 3) or (category == "billing") return dspy.Prediction(category=category, sentiment=sentiment, urgent=is_urgent) ``` By using `dspy.ChainOfThought` instead of `dspy.Predict`, you automatically instruct the model to reason through the problem before providing the final answer, which is often more accurate for nuanced classification tasks. Section 3: Adapters and Token Efficiency While signatures define the intent, **Adapters** determine how that intent is formatted for the LLM. By default, DSPy uses a JSON adapter, but this can be inefficient for complex nested objects. Kevin Madura highlights that using alternative formats like BAML can improve performance by 5-10% because they are more intuitive for models to parse and use fewer tokens. ```python from dspy.adapters import ChatAdapter, JSONAdapter from baml_adapter import BAMLAdapter # Hypothetical specialized adapter Switching adapters is a one-line change that doesn't break your program logic with dspy.context(adapter=BAMLAdapter()): response = my_module(input_data=data) ``` Adapters live between the Signature and the LLM call, acting as the "translator" that turns your Python objects into the final string sent over the wire. Section 4: The Power of Optimizers The most distinctive feature of DSPy is the **Optimizer** (formerly called Teleprompters). Optimizers are algorithms that tune the prompts in your program to maximize a specific **Metric**. This is "AI building AI": the system tries different prompt variations and few-shot examples, measures them against your ground truth data, and keeps the version that performs best. The Optimization Flow 1. **Define a Dataset**: You need 10 to 100 examples of inputs and expected outputs. 2. **Define a Metric**: This can be a simple equality check or a "LLM-as-a-judge" metric that evaluates subjective quality. 3. **Run the Optimizer**: Algorithms like MIPRO (Multi-objective In-context Prompt Optimization) will iteratively refine your program. ```python from dspy.telepropmt import MIPRO Setup the optimizer optimizer = MIPRO(metric=my_accuracy_metric, num_candidates=10) Compile the program (this is where the 'training' happens) optimized_program = optimizer.compile(SupportAnalyzer(), trainset=my_dataset) Save the optimized state optimized_program.save("optimized_support_v1.json") ``` This compiled object contains the highly tuned prompts that the optimizer discovered. You can then load this program in production, ensuring that your small, cheap model (like GPT-4o mini) performs nearly as well as a larger, expensive model. Syntax Notes * **Dot Notation**: DSPy predictions return objects that allow for easy access via dot notation (e.g., `response.sentiment`). * **Context Managers**: Use `dspy.context` or `dspy.settings.configure` to switch models or adapters globally or within a specific block of code. This is invaluable for "model mixing" where you use a cheap model for classification and a powerful model for reasoning. * **Type Hinting**: Always use Python type hints in signatures (`text:str -> summary:str`). DSPy uses these to validate the LLM's response before it ever reaches your application logic. Practical Examples * **Document Routing**: A pipeline that takes a PDF, uses an image-capable model (Gemini 2.0 Flash) to classify the layout, and then routes it to a specialized summarizer module if it's a contract, or an extraction module if it's an SEC filing. * **Boundary Detection**: In legal tech, identifying where the "Main Agreement" ends and "Schedule A" begins. By passing page-level classifications into a DSPy module, the system can determine logical document boundaries with high precision. * **Cost Reduction**: Taking a complex reasoning task that currently requires GPT-4o and using DSPy optimizers to find a prompt strategy that allows Claude 3 Haiku to achieve the same accuracy at 1/10th the cost. Tips & Gotchas * **Caching**: DSPy caches LLM responses by default. If you change your code but the output doesn't change, check if you're hitting the cache. Changing a single space in a signature string will bust the cache. * **Field Naming**: The names of your input and output fields *are* prompts. If you name a field `output1`, the model will struggle. If you name it `summarized_legal_clause`, the model's performance will naturally improve. * **The Optimizer is Not Magic**: An optimizer cannot fix a fundamentally broken program logic. Build your program first, ensure it works on a handful of examples manually, and *then* use the optimizer to squeeze out the final 10-20% of performance. * **Observability**: Always use a tool like Phoenix or the `dspy.inspect_history(n=1)` command during development to see exactly what strings are being sent to the LLM. DSPy adds a lot of "boilerplate" to your prompts that you need to be aware of.

The Myth of the Unavoidable Bug Most users experience software as something that "just works" until it suddenly doesn't. For the person using a banking app or a camera, a bug is a fleeting frustration. For the developer, however, bugs are a source of constant atmospheric pressure—a reality of on-call rotations, pager alerts, and the relentless creep of technical debt. We have conditioned ourselves to believe that perfection is impossible, citing millions of lines of code, ambiguous specifications, and the sheer unpredictability of the physical world. Johann Schleier-Smith from Temporal Technologies challenges this defeatist status quo. He argues that the industry already knows how to build Zero-Bug Software. The methodologies have existed for decades, tucked away in the high-stakes corridors of aerospace and medical engineering. The primary barrier has never been a lack of knowledge; it has been the crushing weight of economics. High-assurance software traditionally costs upwards of $2,500 per line of code, a price point that renders it inaccessible for 99% of commercial applications. We are now entering an era where AI agents could bridge this 100x cost gap, making aerospace-grade reliability the default for every digital interaction. Lessons from the Flight Deck and Deep Space The Airbus A320 stands as a monument to what is possible when the industry rejects defect tolerance. Its control software, developed in the 1980s, has never been implicated in a serious flight incident. This wasn't achieved through luck, but through a rigorous adherence to N-version programming: separate teams using different processors (Intel x86 versus Motorola) and distinct operating systems to ensure that a single logic error couldn't bring down the aircraft. Similarly, NASA demonstrated near-perfection with the Space Shuttle program. Over its final versions, the software averaged only one error per 420,000 lines of code. This level of precision is roughly 1,000 times more reliable than typical commercial software. These systems prioritize static memory allocation, explicit error handling, and the total decoupling of verification teams from development teams. While critics argue that such processes stifle innovation, the data suggests that quality through process is the only proven path to absolute reliability. The Three Pillars of Manageable Complexity To understand how we move toward zero bugs, we must revisit the foundation of computer science. The first pillar is the high-level language. By moving away from assembly in the 1950s and 60s, we gained a 10x productivity boost by abstracting machine implementation details like registers and memory layout. This allows us to focus on "essential complexity"—the logic of the problem itself—rather than the quirks of the hardware. Edgar Dijkstra introduced the second pillar: structured programming. By eliminating the "go-to" statement and replacing it with sequences, selections, and iterations, developers gained the ability to use compositional reasoning. This means you can understand a block of code by looking at its immediate context rather than tracing a tangled web of jumps. Finally, David Parnas gave us modularity. Modularity allows for local reasoning, ensuring that as systems grow, the complexity scales linearly rather than exponentially. These three pillars are not just historical footnotes; they are the exact features that make code interpretable for Large Language Models (LLMs) today. Formal Methods and the Power of Proof While testing only proves the presence of bugs, formal methods can prove their absence. Languages like Daphne allow developers to write proofs directly alongside their code. When you run a verifier, it uses automated reasoning to ensure that every assertion holds true across all possible execution paths. We are seeing a renaissance in these techniques. The seL4 microkernel is a fully verified operating system used in security-critical applications. The CompCert compiler is a verified C compiler that guarantees the generated machine code exactly matches the source program’s intent. Even the Internet itself is increasingly protected by Project Everest, which provides verified cryptographic libraries. The speed and success rates of these verification tools have improved by orders of magnitude over the last 20 years, turning what was once a theoretical academic exercise into a commercially viable toolset. Engineering the Agentic Future The rise of Agentic Coding introduces a paradox. While LLMs are non-deterministic and prone to hallucinations, they possess a unique resilience: the ability to handle ambiguity and unanticipated inputs that would crash traditional rigid software. The key to "Software 3.0"—as Andrej Karpathy calls it—is applying old high-assurance processes to new AI workflows. Instead of asking an LLM to just "write code," we should be prompting it to conduct explicit risk analysis and write "safety cases" for its logic. We can emulate the Airbus model by using one foundation model (like GPT-4) to write the tests and another (Claude) to write the code. When agents are tasked with verifying their own work through formal methods, the cost of high-assurance code plummets. Schleier-Smith notes that while human-written high-assurance code costs $2,500 per line, agent-generated code can be produced for pennies. This 10,000x reduction in cost is the catalyst for the zero-bug vision. Once agents routinely produce software with fewer defects than humans, adoption will reach a point of absolute takeoff, fundamentally altering our expectations of what software can—and should—be.

The software development industry is currently navigating a chaotic transition into the AI age. We see a flood of new models from OpenAI, Anthropic, and Google, each claiming to be industry-leading. For developers, the challenge isn't just using these tools, but understanding which ones actually work. We have moved past the era of simple chat interfaces and entered a phase of "vibe coding"—a term coined by Andrej Karpathy that suggests we can build entire products by simply managing the "vibe" of the AI's output. While the hype is intoxicating, professional engineering requires moving beyond vibes and into structured, high-leverage workflows. Decoding the Benchmarks To choose the right tool, you must understand how these models are measured. We have transitioned away from the HumanEval era. While HumanEval was the gold standard in 2021, modern models score so high on its 164 Python tasks that it no longer differentiates quality. Today, we look to more rigorous tests like SWE-bench. This benchmark uses real-world bugs from production Python projects. When Claude 3.5 Sonnet hits a 73% success rate on these tasks, it isn't just completing a toy function; it is submitting functional patches for complex, multi-file issues. Another critical metric is the Aider Polyglot benchmark, which evaluates how well models handle localized edits across multiple languages like Go and Rust. This tracks efficiency and token cost, providing a practical view of which models are actually viable for daily production use. The Vibe Coding Paradox Andrej Karpathy sparked a firestorm with the concept of vibe coding—accepting all AI suggestions and letting the model drive the entire development process. This trend sits at the peak of inflated expectations on the Gartner Hype Cycle. History repeats itself here; the Agile Manifesto faced similar cynicism in 2001 when critics called it an attempt to undermine engineering discipline. The reality is that AI is a chainsaw. It is incredibly powerful but has jagged edges. If you operate it without a leash, you risk shipping vulnerabilities and "software burrows"—unstable patches held together by digital magic. The goal isn't to let the AI take the wheel entirely but to maintain human control over these high-powered agents. Shifting Mental Gears: Ask, Edit, and Agent Effective AI pair programming requires shifting between distinct modes. **Ask Mode** serves as your conversational debugger, possessing read-only access to answer architectural questions. **Edit Mode** is for precision surgery; the model sees specific files and generates diffs for localized refactors. **Agent Mode** is the most powerful, allowing the AI to search the repository, run terminal commands, and execute tests until a feature is complete. Using the wrong mode for a task leads to context window bloat and poor results. For instance, don't use Agent mode for a simple variable rename; use Edit mode to keep the model's focus narrow and surgical. Advanced Workflows for High-Performance Teams To truly integrate AI, you must codify your preferences. Use global and project-specific instruction files (like `.cursorrules`) to define your naming conventions and architectural patterns. This eliminates the need to constantly correct the AI on small stylistic choices. Furthermore, embrace **Multi-Agent Workflows**. Research shows that a "Reflection" pattern—where one model writes code and a second model reviews it—can boost accuracy by up to 20%. By supplying the reviewer's critique back to the writer, you create a self-correcting loop that catches bugs before they reach your local environment. This is the difference between "vibing" and professional engineering.