DSPy

Overview The landscape of Large Language Model (LLM) development is undergoing a fundamental shift away from "prompt engineering" toward a rigorous programming paradigm. DSPy represents this evolution, providing a declarative framework for building modular software where LLMs are treated as first-class citizens. Instead of manually tweaking strings to coax specific behaviors out of a model, developers define the **intent** of their program through typed interfaces and logical modules. Kevin Madura, a technical consultant at AlixPartners, argues that this transition is essential for enterprise-grade applications that require testability, robustness, and transferability across different models. This tutorial explores how to use DSPy to decompose complex business logic into maintainable Python code. We will examine the core primitives that allow you to separate the structure of your program from the implementation details of the underlying LLM. By the end of this guide, you will understand how to build a multi-stage pipeline that can classify, route, and process various document types using optimized prompting strategies that the system generates for you. Prerequisites To follow this tutorial, you should have a baseline understanding of the following concepts and tools: * **Python Programming**: Familiarity with classes, decorators, and asynchronous programming in Python. * **Pydantic**: Knowledge of Pydantic for data validation and settings management, as it underpins much of DSPy's type hinting. * **LLM Basics**: An understanding of how LLMs process tokens and the general concept of system prompts vs. user messages. * **Environment Setup**: A working Python environment with an API key for a provider like OpenAI, Anthropic, or Google Cloud (or an aggregator like OpenRouter). Key Libraries & Tools * **DSPy**: The core declarative framework used to structure and optimize LLM programs. * **LightLLM**: Used under the hood by DSPy to provide a unified interface for calling various model providers. * **Attachments**: A utility library that simplifies working with disparate file types (PDFs, images) and converting them into LLM-friendly formats. * **Phoenix**: An observability platform from Arize AI used for tracing and debugging LLM calls within the DSPy ecosystem. * **BAML**: A domain-specific language for extracting structured data from LLMs, which can be used as an adapter within DSPy for better token efficiency. Section 1: Signatures as Declarative Intent The heartbeat of any DSPy program is the **Signature**. A signature defines *what* a task should accomplish without specifying *how* it should be prompted. This is a critical distinction: you are defining the inputs and outputs, and DSPy handles the transformation into a prompt. Shorthand Signatures For simple tasks, you can use a shorthand string notation. This is ideal for rapid prototyping: ```python import dspy A simple sentiment classifier shorthand sentiment_predictor = dspy.Predict("text -> sentiment:int") response = sentiment_predictor(text="The service was absolute garbage.") print(response.sentiment) ``` In this example, `text -> sentiment:int` tells DSPy that the input field is named `text` and the output field is an integer named `sentiment`. Class-based Signatures For more complex enterprise logic, class-based signatures allow you to provide docstrings and field descriptions that the model uses to understand the context. These descriptions essentially function as "mini-prompts" embedded within your code structure. ```python class DocumentClassifier(dspy.Signature): """Classify the type of document based on visual and text content.""" document_images = dspy.InputField(desc="Images of the first few pages of the document") document_type = dspy.OutputField(desc="One of: SEC_FILING, PATENT, CONTRACT, OTHER") Usage classifier = dspy.Predict(DocumentClassifier) ``` Section 2: Building Logic with Modules **Modules** are the organizational units of DSPy, analogous to layers in a neural network. A module wraps one or more signatures and can include custom control flow, database calls, or other Python logic. Every module inherits from `dspy.Module` and implements an `__init__` method to define its components and a `forward` method for the execution logic. ```python class SupportAnalyzer(dspy.Module): def __init__(self): super().__init__() self.categorize = dspy.ChainOfThought("message -> category") self.sentiment = dspy.Predict("message -> sentiment:int") def forward(self, message): category = self.categorize(message=message).category sentiment = self.sentiment(message=message).sentiment # Add hard-coded business logic is_urgent = (sentiment < 3) or (category == "billing") return dspy.Prediction(category=category, sentiment=sentiment, urgent=is_urgent) ``` By using `dspy.ChainOfThought` instead of `dspy.Predict`, you automatically instruct the model to reason through the problem before providing the final answer, which is often more accurate for nuanced classification tasks. Section 3: Adapters and Token Efficiency While signatures define the intent, **Adapters** determine how that intent is formatted for the LLM. By default, DSPy uses a JSON adapter, but this can be inefficient for complex nested objects. Kevin Madura highlights that using alternative formats like BAML can improve performance by 5-10% because they are more intuitive for models to parse and use fewer tokens. ```python from dspy.adapters import ChatAdapter, JSONAdapter from baml_adapter import BAMLAdapter # Hypothetical specialized adapter Switching adapters is a one-line change that doesn't break your program logic with dspy.context(adapter=BAMLAdapter()): response = my_module(input_data=data) ``` Adapters live between the Signature and the LLM call, acting as the "translator" that turns your Python objects into the final string sent over the wire. Section 4: The Power of Optimizers The most distinctive feature of DSPy is the **Optimizer** (formerly called Teleprompters). Optimizers are algorithms that tune the prompts in your program to maximize a specific **Metric**. This is "AI building AI": the system tries different prompt variations and few-shot examples, measures them against your ground truth data, and keeps the version that performs best. The Optimization Flow 1. **Define a Dataset**: You need 10 to 100 examples of inputs and expected outputs. 2. **Define a Metric**: This can be a simple equality check or a "LLM-as-a-judge" metric that evaluates subjective quality. 3. **Run the Optimizer**: Algorithms like MIPRO (Multi-objective In-context Prompt Optimization) will iteratively refine your program. ```python from dspy.telepropmt import MIPRO Setup the optimizer optimizer = MIPRO(metric=my_accuracy_metric, num_candidates=10) Compile the program (this is where the 'training' happens) optimized_program = optimizer.compile(SupportAnalyzer(), trainset=my_dataset) Save the optimized state optimized_program.save("optimized_support_v1.json") ``` This compiled object contains the highly tuned prompts that the optimizer discovered. You can then load this program in production, ensuring that your small, cheap model (like GPT-4o mini) performs nearly as well as a larger, expensive model. Syntax Notes * **Dot Notation**: DSPy predictions return objects that allow for easy access via dot notation (e.g., `response.sentiment`). * **Context Managers**: Use `dspy.context` or `dspy.settings.configure` to switch models or adapters globally or within a specific block of code. This is invaluable for "model mixing" where you use a cheap model for classification and a powerful model for reasoning. * **Type Hinting**: Always use Python type hints in signatures (`text:str -> summary:str`). DSPy uses these to validate the LLM's response before it ever reaches your application logic. Practical Examples * **Document Routing**: A pipeline that takes a PDF, uses an image-capable model (Gemini 2.0 Flash) to classify the layout, and then routes it to a specialized summarizer module if it's a contract, or an extraction module if it's an SEC filing. * **Boundary Detection**: In legal tech, identifying where the "Main Agreement" ends and "Schedule A" begins. By passing page-level classifications into a DSPy module, the system can determine logical document boundaries with high precision. * **Cost Reduction**: Taking a complex reasoning task that currently requires GPT-4o and using DSPy optimizers to find a prompt strategy that allows Claude 3 Haiku to achieve the same accuracy at 1/10th the cost. Tips & Gotchas * **Caching**: DSPy caches LLM responses by default. If you change your code but the output doesn't change, check if you're hitting the cache. Changing a single space in a signature string will bust the cache. * **Field Naming**: The names of your input and output fields *are* prompts. If you name a field `output1`, the model will struggle. If you name it `summarized_legal_clause`, the model's performance will naturally improve. * **The Optimizer is Not Magic**: An optimizer cannot fix a fundamentally broken program logic. Build your program first, ensure it works on a handful of examples manually, and *then* use the optimizer to squeeze out the final 10-20% of performance. * **Observability**: Always use a tool like Phoenix or the `dspy.inspect_history(n=1)` command during development to see exactly what strings are being sent to the LLM. DSPy adds a lot of "boilerplate" to your prompts that you need to be aware of.