GPT-4o

The agentic revolution of the VS Code fork Cursor 3 represents a fundamental pivot in how we think about integrated development environments. It is no longer just a VS Code fork with a chat sidebar; it is evolving into a dedicated multi-agent environment. This shift mirrors the trajectory of tools like Conductor and Solarterm, placing the developer in the role of a high-level orchestrator rather than a line-by-line writer. The interface now allows for parallel workspaces where separate agents can tackle different tasks simultaneously, signaling a move toward "agent-first" development. Performance showdown across frontier models Testing Cursor 3 across different models reveals significant variance in both speed and capability. In a head-to-head comparison using a Laravel CRUD task, Composer 2 clocked in at a blistering 3 minutes and 21 seconds. While fast, it lacked the depth of GPT-4o (referred to as GPT-54 in the interface), which took nearly 9 minutes but implemented more nuanced features like post counts in category tables. Claude 3.5 Opus (Opus 4.6) lagged significantly in speed, though it delivered high-quality code. The takeaway is clear: speed often comes at the cost of architectural depth, and Composer 2 is built for velocity over complexity. Cloud agents and the infrastructure overhead One of the most ambitious features is the introduction of cloud agents. These allow you to run prompts in a remote virtual machine, theoretically freeing your local resources. However, the experience feels unpolished. During testing, the cloud environment lacked basic binaries like PHP, forcing the agent to spend valuable time and tokens installing dependencies and generating app keys. While it eventually succeeded in creating a pull request, the process felt slower and more cumbersome than local execution. Unless you are away from your main machine, the local agent remains the superior choice for efficiency. The steep cost of agentic orchestration Price remains the biggest hurdle for Cursor 3 adoption. Running a single multi-agent session for a simple CRUD project consumed approximately $5 worth of usage from a standard monthly plan. For context, a few hours of intensive agentic work could easily exhaust a user's monthly token quota. Cursor essentially acts as a middleman, paying API rates to providers like Anthropic and OpenAI, then passing those costs (with a premium) to the user. Compared to Claude Code or Codeium, which may offer different usage tiers, Cursor feels like a luxury tool that requires careful management of "max mode" to avoid a billing disaster. Final verdict on the agentic workspace Directionally, Cursor 3 is brilliant. It anticipates a future where we prompt, review, and merge rather than type. However, the current pricing model and the overhead of cloud environments make it a hard sell for the budget-conscious developer. If you value the ability to run three models against the same problem to find the best solution, the workflow is unmatched. For everyone else, it’s a glimpse into an expensive future that still needs a few more iterations to become a daily driver.

Overview Fixing a production bug involves more than just writing new code. The real challenge lies in **reproduction**. If you cannot replicate the failure, you cannot guarantee the fix works for the specific scenario reported by the user. By integrating Test-Driven Development (TDD) principles into AI agent workflows, we move from "guessing and checking" to verified engineering. This tutorial explores how to configure agents like Claude MD to follow a strict reproduce-first protocol. Prerequisites To follow this guide, you should understand: * **Basic PHP/Laravel**: The examples use the Laravel framework. * **Testing Fundamentals**: Familiarity with unit and integration tests. * **AI Agents**: Understanding how tools like Claude interact with local codebases. Key Libraries & Tools * **Claude MD / CodeX**: Developer-centric AI agents that can read, write, and execute terminal commands within your project. * **Pest PHP**: A graceful testing framework for Laravel used here to run regression tests. * **Claude 3.5 Sonnet / Opus**: The underlying LLMs that power the logic of the code exploration and fix generation. Code Walkthrough: The Fail-First Workflow Step 1: The Failing Test Instead of letting the AI jump straight to a fix, we instruct it to write a test that fails against the current buggy codebase. For a project bulk update missing a permission check, the test should attempt to update a project belonging to a different user. ```php // Example failing test generated by the agent it('prevents updating projects that do not belong to the user', function () { $user = User::factory()->create(); $otherUser = User::factory()->create(); $project = Project::factory()->create(['user_id' => $otherUser->id]); $response = $this->actingAs($user)->patch("/projects/{$project->id}", [ 'name' => 'Hacked Name' ]); $response->assertStatus(403); }); ``` Step 2: Verification of Failure The agent executes this test. Seeing the test return a `200 OK` or `500 Error` instead of the expected `403 Forbidden` confirms the bug is reproducible. Step 3: The Fix and Verification Once reproduced, the agent applies the fix—likely a simple ownership check in the controller—and reruns the same test. A passing result now provides a true regression suite. Syntax Notes When configuring Claude MD, your `guidelines.md` or prompt must be explicit. Use active instructions: "Investigate the codebase, then write a failing test first." Avoid vague requests like "Use sub-agents," as these often lead to complexity without clarity in the merge process. Tips & Gotchas * **Trust but Verify**: AI-generated tests can sometimes have logic errors. Always review the test assertions to ensure they match the real-world bug. * **Model Choice**: While GPT-4o and Opus handle test generation well, cheaper models like Sonnet may skip the testing phase unless your instructions are strictly enforced in the system prompt.

Overview of the Multi-Provider AI Integration Implementing AI features within a Laravel ecosystem often feels deceptively simple until you confront the realities of production-grade integration. In this tactical evaluation, a Filament-based CMS serves as the testing ground for the Laravel AI SDK, a tool designed to unify interactions across diverse Large Language Model (LLM) providers. The scenario involves four typical AI operations: title suggestion, tweet generation, full-text translation, and image creation. By stress-testing providers like OpenAI, Anthropic, Google, and DeepSeek, we move past theoretical capabilities to measure the cold, hard metrics of latency, cost-efficiency, and reliability. Key Strategic Decisions: Model Selection and Prompt Engineering A critical strategic move involves categorizing models by their "weight class." For lightweight tasks like title generation, utilizing expensive flagship models like Claude 3 Opus is a tactical error. The analysis reveals that cheaper models like Claude 3 Haiku or GPT-4o mini deliver comparable results for a fraction of the cost. A robust implementation strategy must also prioritize system prompt persistence. Storing these prompts in a database table rather than hard-coding them allows for real-time iteration and adjustments based on model-specific quirks, such as Gemini's tendency to ignore character limits in tweet generation. Performance Breakdown: Speed vs. Cost The data exposes a massive rift between provider promises and actual API performance. DeepSeek emerges as a dominant force in cost-efficiency, processing extensive text for less than a single cent. Conversely, Claude 3 Opus represents the premium ceiling, costing significantly more per prompt without a proportional increase in quality for simple CMS tasks. Latency is the hidden killer of user experience. While Groq delivers lightning-fast inferences, others like Gemini 1.5 Pro occasionally exceed 20 seconds for basic tasks. The most surprising finding remains the inconsistency of "mini" models; GPT-4o mini frequently lagged behind its larger sibling, GPT-4o, proving that smaller does not always mean faster in the world of cloud APIs. Critical Moments: Failures and Timeouts The translation and image generation tests served as the ultimate stress points. Translation tasks frequently triggered 60-second PHP timeouts, highlighting a desperate need for asynchronous processing. For instance, Gemini 1.5 Flash and Groq handled long-form translation with relative stability, but more complex models struggled to finish within the execution window. Image generation presented its own set of failures, often triggered by internal safety filters or "unknown finish reasons." These moments demonstrate that no provider is 100% reliable; a failure-tolerant architecture using try-catch blocks and human-readable error messages is non-negotiable. Future Implications: The Hybrid Model Approach The takeaway for developers is clear: do not marry a single provider. The Laravel AI SDK facilitates a hybrid strategy where DeepSeek handles high-volume translations, Groq generates rapid-fire titles, and OpenAI produces the most vibrant images. Moving forward, developers must implement queue-based architectures and WebSockets to manage long-running AI tasks, ensuring that the "magic" of AI doesn't break the fundamental responsiveness of the web application.

Overview: The Context Gap in AI Development AI agents have changed how we write code, but they often struggle with the nuances of specific frameworks. Standard models like Claude 3.5 Sonnet or GPT-4o possess vast general knowledge but lack the hyper-specific context of your local Laravel project. This lead to hallucinations, outdated syntax, or the AI suggesting patterns that conflict with your application's architecture. Laravel Boost solves this by acting as a bridge. It injects project-specific metadata, documentation, and "skills" directly into your AI agent's reasoning loop. Instead of manually feeding documentation to a chat window, Boost automates the context delivery. Version 2.0 introduces a major shift from a monolithic guideline approach to a modular, "skills-first" architecture. This reduces context bloat, saves on token costs, and makes the AI significantly more accurate by only providing the information it needs at that exact moment. Prerequisites To follow this guide and implement Boost 2.0, you should be comfortable with the following: * **PHP 8.2+:** Boost 2.0 has officially dropped support for PHP 8.1. * **Laravel 11 or 12:** Older versions like Laravel 10 are supported only by legacy versions of Boost (v1.x). * **Composer:** Basic knowledge of managing PHP dependencies. * **AI Coding Agents:** Familiarity with tools like Cursor, Claude Code, GitHub Copilot, or Juni. Key Libraries & Tools * **Laravel Boost:** The core CLI tool and package that manages AI context and skills. * **Laravel MCP:** A package for building Model Context Protocol servers, allowing AI agents to interact with your app's internal state (routes, database schemas, etc.). * **Remotion:** A React-based framework for programmatic video creation, often used as a demonstration of complex AI skill integration. * **Prism:** A Laravel package for working with LLMs, used to demonstrate how documentation can be bundled directly into vendor folders for AI consumption. Code Walkthrough: Installing and Configuring Boost 2.0 Setting up Boost 2.0 is a methodical process. It begins with the Laravel installer and moves into a randomized, aesthetically pleasing configuration CLI. 1. Installation First, ensure your Laravel installer is up to date to access the built-in Boost prompts during new project creation. If you are adding it to an existing project, use Composer: ```bash composer require laravel/boost --dev ``` 2. Initialization Run the install command to start the interactive configuration. ```bash php artisan boost:install ``` This command triggers a CLI interface featuring randomized gradients—a touch of "developer joy" added by Pushpak Chhajed. You will be prompted to select which features to configure: AI Guidelines, Agent Skills, or the MCP server. 3. Selecting Your AI Agent Boost 2.0 simplifies agent selection. Instead of choosing both an IDE and an agent, you now choose the specific agentic tool you use daily, such as Claude Code or Cursor. Boost will then automatically determine the correct file paths for these tools. 4. Automated Skill Syncing To ensure your AI context stays updated as your project evolves, add the update command to your `composer.json` file: ```json "scripts": { "post-update-cmd": [ "@php artisan boost:update" ] } ``` This ensures that every time you update your dependencies, Boost re-scans your `composer.json` and syncs the relevant skills for packages like Inertia, Tailwind CSS, or Livewire. Deep Dive into Skills vs. Guidelines Understanding the distinction between these two features is critical for a clean development workflow. Guidelines: The Global Rules Guidelines are persistent. They contain high-level rules that the AI should *always* know. For example, if you always use Pest for testing or strictly follow an Action-based architecture, these belong in your guidelines. However, shoving every package's documentation into a guideline leads to "context fatigue," where the AI becomes overwhelmed and starts to hallucinate. Skills: The On-Demand Context Skills are modular Markdown files. They aren't loaded into the AI's memory until they are needed. Each skill has a name and a description in its front matter. When you ask the AI to "build a new UI component with Tailwind," the agent sees the keyword "Tailwind," looks at its available skills, and activates the Tailwind CSS skill. This keeps the prompt lean and the output precise. Syntax Notes: Custom Skill Creation Creating a custom skill allows you to automate highly specific tasks, like generating pull request descriptions or adhering to internal API versioning standards. Skills rely on a specific Markdown front matter format. ```markdown --- name: my-custom-skill description: Use this skill when generating API endpoints or PR descriptions. --- My Custom Skill Rules - Always use the `App\Actions` namespace for business logic. - Ensure all API responses are wrapped in a standard `JsonResource`. - Pull Request descriptions must include a 'Breaking Changes' section. ``` When you save this in a local `.boost/skills` directory and run `php artisan boost:update`, Boost replicates this file into the hidden configuration folders of your chosen AI agents (e.g., `.cursor/rules` or `.claudecode/skills`). Practical Examples Automating Pull Requests You can create a skill that teaches an agent how to use the GitHub CLI. By invoking the skill with a slash command (e.g., `/create-pr`), the AI can analyze your staged changes, write a formatted description, and execute the CLI command to open the PR. Package-Specific Intelligence If you build a project using Filament, you don't want the AI thinking about Filament when you are just debugging a console command. By using a Filament skill, the AI only accesses those specific layout and component rules when you are actively working on the admin panel. Tips & Gotchas * **Git Management:** Never commit the auto-generated agent folders (like `.cursor/rules`) to your repository. These are local mirrors. Only commit the `.boost` folder and your `boost.json` file. This allows your teammates to run `boost:install` and get the exact same AI behavior on their machines. * **Hallucination Prevention:** If your AI starts ignoring your project structure, check your guideline length. If it exceeds 500 lines, move package-specific rules into individual skills. * **Legacy Projects:** Do not attempt to use Boost 2.0 on Laravel 10 projects. The dependency tree for the new MCP features and skills requires the modern internals found in Laravel 11 and up. * **Manual Invocation:** If an agent fails to auto-detect a skill, you can usually force it by using a slash command in the chat interface. Most modern agents support `/` to list and select active skills.

The Digital Renaissance of Open Source For years, a silent frustration plagued the technological world: the recurring disappointment of Chinese open-source models that shimmered on benchmarks but crumbled under the weight of real-world complexity. We call this phenomenon **benchmaxing**. It involves optimizing models specifically for testing datasets while ignoring the messy, organic logic required for human interaction. Kimi K2.5, the latest release from Moonshot AI, suggests we have reached a turning point where the artifact finally matches the promise. The Agent Swarm Architecture One cannot discuss Kimi K2.5 without examining its most provocative feature: the **Agent Swarm**. While traditional Large Language Models (LLMs) operate as a single, linear intelligence, this model can deploy up to 100 sub-agents in parallel. This decentralized approach mimics a workshop of specialized artisans rather than a lone scholar. This parallelization results in a 4.5x speed increase for complex tool calls, allowing the system to verify its own logic across multiple threads simultaneously. It is a structural evolution that reflects the complex, multi-layered societies of our own history. Synthesis of Vision and Code The most grueling trial for any modern model remains its ability to translate visual stimuli into functional logic. In tests involving a high-fidelity website recording, Kimi K2.5 attempted to recreate a complex front-end experience from video alone. While it missed the subtle 'smoke' cursor effects, it successfully replicated the core layout, interactive 'eye' elements, and brand essence. This capability extends beyond mere imitation; it suggests an internal understanding of how visual components map to underlying structural code. In single-shot coding tests, the model even constructed a functional 'Melvore Idol' style game—complete with inventory systems and experience tracking—from a single prompt. Analysis of the Global Hierarchy When we look at the market share by token usage, Google and Anthropic still hold the high ground. However, the emotional intelligence scores tell a different story. Kimi K2.5 recently seized the number one spot on the EQ Bench, surpassing GPT-4o and Gemini 1.5 Pro. It indicates that the model excels at creative writing and abstract nuances—areas where open-source models historically struggled. While it remains a newcomer in token market share, its performance suggests a looming disruption to the established Western dominance. Final Verdict Kimi K2.5 is a rare specimen that justifies the surrounding fervor. Its combination of swarm agentics and vision-to-code synthesis makes it a formidable tool for developers and creative thinkers alike. While the gap between high-res reality and model output still exists, the distance has closed significantly. It is no longer a matter of if open-source will catch up, but rather when the established giants will have to defend their territory.

Overview The landscape of Large Language Model (LLM) development is undergoing a fundamental shift away from "prompt engineering" toward a rigorous programming paradigm. DSPy represents this evolution, providing a declarative framework for building modular software where LLMs are treated as first-class citizens. Instead of manually tweaking strings to coax specific behaviors out of a model, developers define the **intent** of their program through typed interfaces and logical modules. Kevin Madura, a technical consultant at AlixPartners, argues that this transition is essential for enterprise-grade applications that require testability, robustness, and transferability across different models. This tutorial explores how to use DSPy to decompose complex business logic into maintainable Python code. We will examine the core primitives that allow you to separate the structure of your program from the implementation details of the underlying LLM. By the end of this guide, you will understand how to build a multi-stage pipeline that can classify, route, and process various document types using optimized prompting strategies that the system generates for you. Prerequisites To follow this tutorial, you should have a baseline understanding of the following concepts and tools: * **Python Programming**: Familiarity with classes, decorators, and asynchronous programming in Python. * **Pydantic**: Knowledge of Pydantic for data validation and settings management, as it underpins much of DSPy's type hinting. * **LLM Basics**: An understanding of how LLMs process tokens and the general concept of system prompts vs. user messages. * **Environment Setup**: A working Python environment with an API key for a provider like OpenAI, Anthropic, or Google Cloud (or an aggregator like OpenRouter). Key Libraries & Tools * **DSPy**: The core declarative framework used to structure and optimize LLM programs. * **LightLLM**: Used under the hood by DSPy to provide a unified interface for calling various model providers. * **Attachments**: A utility library that simplifies working with disparate file types (PDFs, images) and converting them into LLM-friendly formats. * **Phoenix**: An observability platform from Arize AI used for tracing and debugging LLM calls within the DSPy ecosystem. * **BAML**: A domain-specific language for extracting structured data from LLMs, which can be used as an adapter within DSPy for better token efficiency. Section 1: Signatures as Declarative Intent The heartbeat of any DSPy program is the **Signature**. A signature defines *what* a task should accomplish without specifying *how* it should be prompted. This is a critical distinction: you are defining the inputs and outputs, and DSPy handles the transformation into a prompt. Shorthand Signatures For simple tasks, you can use a shorthand string notation. This is ideal for rapid prototyping: ```python import dspy A simple sentiment classifier shorthand sentiment_predictor = dspy.Predict("text -> sentiment:int") response = sentiment_predictor(text="The service was absolute garbage.") print(response.sentiment) ``` In this example, `text -> sentiment:int` tells DSPy that the input field is named `text` and the output field is an integer named `sentiment`. Class-based Signatures For more complex enterprise logic, class-based signatures allow you to provide docstrings and field descriptions that the model uses to understand the context. These descriptions essentially function as "mini-prompts" embedded within your code structure. ```python class DocumentClassifier(dspy.Signature): """Classify the type of document based on visual and text content.""" document_images = dspy.InputField(desc="Images of the first few pages of the document") document_type = dspy.OutputField(desc="One of: SEC_FILING, PATENT, CONTRACT, OTHER") Usage classifier = dspy.Predict(DocumentClassifier) ``` Section 2: Building Logic with Modules **Modules** are the organizational units of DSPy, analogous to layers in a neural network. A module wraps one or more signatures and can include custom control flow, database calls, or other Python logic. Every module inherits from `dspy.Module` and implements an `__init__` method to define its components and a `forward` method for the execution logic. ```python class SupportAnalyzer(dspy.Module): def __init__(self): super().__init__() self.categorize = dspy.ChainOfThought("message -> category") self.sentiment = dspy.Predict("message -> sentiment:int") def forward(self, message): category = self.categorize(message=message).category sentiment = self.sentiment(message=message).sentiment # Add hard-coded business logic is_urgent = (sentiment < 3) or (category == "billing") return dspy.Prediction(category=category, sentiment=sentiment, urgent=is_urgent) ``` By using `dspy.ChainOfThought` instead of `dspy.Predict`, you automatically instruct the model to reason through the problem before providing the final answer, which is often more accurate for nuanced classification tasks. Section 3: Adapters and Token Efficiency While signatures define the intent, **Adapters** determine how that intent is formatted for the LLM. By default, DSPy uses a JSON adapter, but this can be inefficient for complex nested objects. Kevin Madura highlights that using alternative formats like BAML can improve performance by 5-10% because they are more intuitive for models to parse and use fewer tokens. ```python from dspy.adapters import ChatAdapter, JSONAdapter from baml_adapter import BAMLAdapter # Hypothetical specialized adapter Switching adapters is a one-line change that doesn't break your program logic with dspy.context(adapter=BAMLAdapter()): response = my_module(input_data=data) ``` Adapters live between the Signature and the LLM call, acting as the "translator" that turns your Python objects into the final string sent over the wire. Section 4: The Power of Optimizers The most distinctive feature of DSPy is the **Optimizer** (formerly called Teleprompters). Optimizers are algorithms that tune the prompts in your program to maximize a specific **Metric**. This is "AI building AI": the system tries different prompt variations and few-shot examples, measures them against your ground truth data, and keeps the version that performs best. The Optimization Flow 1. **Define a Dataset**: You need 10 to 100 examples of inputs and expected outputs. 2. **Define a Metric**: This can be a simple equality check or a "LLM-as-a-judge" metric that evaluates subjective quality. 3. **Run the Optimizer**: Algorithms like MIPRO (Multi-objective In-context Prompt Optimization) will iteratively refine your program. ```python from dspy.telepropmt import MIPRO Setup the optimizer optimizer = MIPRO(metric=my_accuracy_metric, num_candidates=10) Compile the program (this is where the 'training' happens) optimized_program = optimizer.compile(SupportAnalyzer(), trainset=my_dataset) Save the optimized state optimized_program.save("optimized_support_v1.json") ``` This compiled object contains the highly tuned prompts that the optimizer discovered. You can then load this program in production, ensuring that your small, cheap model (like GPT-4o mini) performs nearly as well as a larger, expensive model. Syntax Notes * **Dot Notation**: DSPy predictions return objects that allow for easy access via dot notation (e.g., `response.sentiment`). * **Context Managers**: Use `dspy.context` or `dspy.settings.configure` to switch models or adapters globally or within a specific block of code. This is invaluable for "model mixing" where you use a cheap model for classification and a powerful model for reasoning. * **Type Hinting**: Always use Python type hints in signatures (`text:str -> summary:str`). DSPy uses these to validate the LLM's response before it ever reaches your application logic. Practical Examples * **Document Routing**: A pipeline that takes a PDF, uses an image-capable model (Gemini 2.0 Flash) to classify the layout, and then routes it to a specialized summarizer module if it's a contract, or an extraction module if it's an SEC filing. * **Boundary Detection**: In legal tech, identifying where the "Main Agreement" ends and "Schedule A" begins. By passing page-level classifications into a DSPy module, the system can determine logical document boundaries with high precision. * **Cost Reduction**: Taking a complex reasoning task that currently requires GPT-4o and using DSPy optimizers to find a prompt strategy that allows Claude 3 Haiku to achieve the same accuracy at 1/10th the cost. Tips & Gotchas * **Caching**: DSPy caches LLM responses by default. If you change your code but the output doesn't change, check if you're hitting the cache. Changing a single space in a signature string will bust the cache. * **Field Naming**: The names of your input and output fields *are* prompts. If you name a field `output1`, the model will struggle. If you name it `summarized_legal_clause`, the model's performance will naturally improve. * **The Optimizer is Not Magic**: An optimizer cannot fix a fundamentally broken program logic. Build your program first, ensure it works on a handful of examples manually, and *then* use the optimizer to squeeze out the final 10-20% of performance. * **Observability**: Always use a tool like Phoenix or the `dspy.inspect_history(n=1)` command during development to see exactly what strings are being sent to the LLM. DSPy adds a lot of "boilerplate" to your prompts that you need to be aware of.

The Shift from Static Prompts to Dynamic Learning Software development is hitting a wall with Large Language Model (LLM) agents. We have built systems that work 80% of the time, but the remaining 20%—the "reliability gap"—remains stubbornly open. Traditionally, we have tried to close this gap by manually tweaking prompts, a process that is both unscalable and fragile. SallyAnn DeLucia and Fuad Ali from Arize AI argue that the industry needs to move away from static instructions entirely. Instead, developers should implement **prompt learning**, a technique that borrows principles from Reinforcement Learning to create a self-correcting optimization loop. Unlike traditional prompt engineering, where a human tries to guess what words might steer the model better, prompt learning treats the prompt as a set of weights that can be updated based on structured feedback. The core philosophy is that the most valuable data in your system isn't just the final output; it is the **English feedback** explaining *why* an output failed. By capturing human or LLM-based explanations of failures and feeding them back into an optimizer, you can achieve performance gains—like a 15% improvement in coding accuracy—without touching the underlying model architecture or training data. Prerequisites and the Optimization Stack To build a prompt learning loop, you need a baseline understanding of Python and Jupyter Notebooks. Conceptually, you should be familiar with evaluation frameworks and the idea of "LLM-as-a-judge." Key Libraries & Tools * **Arize Phoenix**: An open-source observability library used for tracing and evaluating LLM applications. * **OpenAI SDK**: Used here for both the core agent logic and the evaluators (specifically GPT-4o or newer models). * **Nest-asyncio**: A utility to allow nested asynchronous loops in Jupyter, which is critical for running parallel evaluations quickly. * **Pandas**: Necessary for managing the training and testing datasets that drive the optimization process. Architecting the Multi-Step Optimization Loop Setting up the environment requires specific attention to library versions. A common pitfall in these rapidly evolving ecosystems is version mismatch. For this tutorial, ensure you are using `arize-phoenix >= 2.2.0` to avoid package conflicts during evaluation. ```python import phoenix as px import nest_asyncio Patch for Jupyter environments to handle async calls nest_asyncio.apply() Configuration parameters NUM_SAMPLES = 50 TRAIN_SPLIT = 0.8 OPTIMIZATION_LOOPS = 5 ``` The loop consists of three logical stages: **Generation**, **Evaluation**, and **Refinement**. You start by splitting your dataset into a training set (used to generate the new prompt) and a test set (used to verify that the new prompt actually performs better). Building Custom Evaluators as High-Fidelity Signals A prompt learning loop is only as strong as its evaluators. If your evaluator provides a simple "Incorrect" label without context, the optimizer has no idea how to fix the instruction. You must initialize evaluators that provide **detailed explanations**. ```python Initializing the Classification Evaluator evaluate_output = px.evals.OpenAIModel( model="gpt-4o", template=EVAL_TEMPLATE, # A template defined in external files choices=["correct", "incorrect"] ) ``` In this workshop, SallyAnn DeLucia highlights the "Rule Checker"—a specialized evaluator that performs a granular, rule-by-rule analysis of the output. This creates a high-dimensional feedback signal. Instead of telling the optimizer "this failed," it says "this failed because it didn't adhere to the JSON schema in rule #3." This level of specificity is what allows the Prompt Learning SDK to rewrite the system prompt effectively. Syntax Notes and Implementation Details When writing the optimization logic, pay attention to the **response format** and **temperature**. For consistent results during an automated optimization loop, setting `temperature=0` is standard practice. ```python async def generate_output(data, system_prompt): response = await client.chat.completions.create( model="gpt-4o", messages=[ {"role": "system", "content": system_prompt}, {"role": "user", "content": data} ], response_format={"type": "json_object"}, temperature=0 ) return response.choices[0].message.content ``` The `response_format` parameter is a critical language feature in the OpenAI API that ensures the model outputs valid JSON. This is vital when the task involves web page creation or structured data, as it prevents the optimizer from getting distracted by formatting errors and allows it to focus on logic and content. Practical Case Study: The 15% Performance Jump To prove the efficacy of this method, Arize AI applied prompt learning to OpenDevin, an open-source coding agent. The original system prompt was remarkably simple, lacking specific rules for error handling or test requirements. By running this exact optimization loop, the system generated a new prompt that included a robust "Rules" section. This optimized prompt improved the agent's performance on the SWE-bench benchmark by 15%. Most importantly, the optimized agent (using GPT-4o) approached the performance of much more expensive models like Claude 3.5 Sonnet while costing two-thirds less. This demonstrates that "expertise" can be engineered into a prompt through data-driven iterations, often negating the need for expensive fine-tuning. Tips and Debugging Your Loop 1. **Avoid Over-Optimization**: There is a temptation to run 20 or 30 loops. However, Fuad Ali notes that significant gains usually occur within the first 3-5 loops. Beyond that, you risk overfitting to the specific quirks of your training set. 2. **Optimize the Evaluator First**: If your prompt learning loop isn't working, the problem is likely your evaluator. You should optimize the evaluator's prompt with the same rigor as your agent's prompt. 3. **Use Logprobs for Confidence**: If you aren't sure if the model's "Incorrect" label is reliable, look at the logprobs (logarithmic probabilities) of the token. Low confidence in the evaluator's label should trigger a human review. 4. **Handling Multi-Agent Systems**: While the current SDK focuses on independent tasks, you can optimize multi-agent systems by treating each agent's hand-off as a discrete step for prompt learning. By treating prompts as software that requires a CI/CD-like iteration cycle, developers can finally build agents that aren't just "cool prototypes" but reliable production tools.