TJ Miller

Overview: The Unified AI Strategy for Laravel Building AI features often feels like a fragmented journey. Developers usually jump between specialized APIs for text, separate services for images, and complex libraries for audio transcription. The Laravel AI SDK changes this by providing a unified, first-party toolkit that handles the heavy lifting of AI integration. It treats AI as a core application concern, much like how Laravel handles databases or queues. By abstracting the differences between providers like OpenAI, Anthropic, and Gemini, it allows you to write cleaner, more maintainable code that isn't locked into a single vendor's API. Taylor Otwell designed the SDK to feel "Laravel-esque." This means leaning into conventions like class-based agents, fluent API chains, and deep integration with the existing Laravel ecosystem. Whether you need to summarize an issue in a project management tool, generate realistic speech via ElevenLabs, or perform semantic search on a mountain of PDFs, the SDK provides the scaffolding to do it efficiently. It moves AI from being an experimental add-on to a standard part of the modern developer's workflow. Prerequisites and Environment Setup Before you begin building, ensure you have a standard Laravel environment ready. You should be comfortable with PHP, Composer, and basic Laravel concepts like controllers and service providers. You will also need API keys from at least one AI provider. While the SDK supports local models via Ollama, production applications typically require keys for OpenAI or Anthropic. To get started, install the package via Composer: ```bash composer require laravel/ai ``` After installation, publish the configuration and migrations: ```bash php artisan vendor:publish --tag="ai-config" php artisan vendor:publish --tag="ai-migrations" php artisan migrate ``` The configuration file (`config/ai.php`) allows you to define your default providers. You can set different defaults for different modalities—for instance, using Claude for text and DALL-E for images. This flexibility is a core strength of the SDK. Key Libraries & Tools * **Laravel AI SDK**: The primary toolkit for interacting with LLMs, image generators, and audio services. * **Prism**: A community package by TJ Miller that serves as the query builder layer for the SDK's text generation. * **ElevenLabs**: Integrated for high-quality text-to-speech capabilities. * **Ollama**: Enables running local models for development and testing without incurring API costs. * **Laravel Boost**: A local MCP server that provides AI agents with context about your specific Laravel codebase. * **PostgreSQL with PGVector**: Used for storing and searching vector embeddings locally. Code Walkthrough: Implementing Agents and Tools 1. Creating an Agent The Agent class is the heart of the SDK. It encapsulates the identity of your AI. Instead of passing long strings of instructions in every controller, you define them once in a reusable class. You can generate one using the Artisan command: ```bash php artisan make:agent SalesCoachAgent ``` In the generated class, you define the system prompt and the models to use. The `instructions` method is where you set the "personality" and guardrails for the agent. ```php namespace App\Agents; use Laravel\AI\Agent; class SalesCoachAgent extends Agent { public function instructions(): string { return "You are an expert sales coach. Analyze the provided transcript and offer three actionable improvements."; } } ``` 2. Using Structured Output One of the most powerful features is getting the AI to return data in a specific format rather than a messy string. The SDK uses a JSON Schema builder to ensure the model follows your rules. This makes it possible to save AI responses directly into your database without fragile regex parsing. ```php use App\Agents\SalesCoachAgent; use Laravel\AI\Schema; $agent = new SalesCoachAgent(); $response = $agent->predict( prompt: "Analyze the call from yesterday.", schema: Schema::object([ 'sentiment' => Schema::string()->description('Overall tone of the customer'), 'score' => Schema::integer()->description('Score from 1-10'), 'follow_up_needed' => Schema::boolean(), ]) ); // Access data directly as an array echo $response['sentiment']; ``` 3. Integrating Tools (Function Calling) Tools allow your AI to actually *do* things. You can give an agent the ability to search the web, fetch a URL, or even query your own database. The SDK comes with several provider tools built-in, but you can also write your own custom tools by extending the `Tool` class and implementing a `handle` method. ```php use Laravel\AI\Tools\WebSearch; class ResearcherAgent extends Agent { public function tools(): array { return [ new WebSearch(), ]; } } ``` When you prompt this agent, it will realize it needs more info, call the `WebSearch` tool, and then use the results to finish its answer. This turns a static LLM into a dynamic assistant. Syntax Notes: Attributes and Traits The Laravel AI SDK makes heavy use of PHP attributes to simplify configuration. These attributes allow you to stay updated with the latest model advancements without changing your code logic. * **`#[UseCheapestModel]`**: Instructs the SDK to use the most cost-effective model for a specific provider (e.g., GPT-4o-mini or Claude Haiku). This is perfect for simple tasks like summarization. * **`#[UseSmartestModel]`**: Forces the use of the flagship model (e.g., Claude 3.7 Sonnet) for tasks requiring high reasoning capabilities. * **`RemembersConversations` trait**: Adding this to your agent automatically manages database storage for chat history, ensuring the AI remembers previous messages without you manually passing a growing array of context. Practical Examples: The 'Larvis' Workflow A practical application of this tech is building a voice-enabled assistant like "Larvis." The workflow demonstrates the multi-modal nature of the SDK: 1. **Transcription**: The user uploads an audio file of their question. The SDK uses a `transcribe` method (typically via Whisper) to convert audio to text. 2. **Context Retrieval**: The agent fetches relevant local documents (like Markdown files) and injects them into the prompt to provide specific knowledge the LLM wasn't trained on. 3. **Inference**: The agent generates a text response based on the transcription and the local documents. 4. **Speech Synthesis**: The text response is passed to the `audio` method, using ElevenLabs to generate a high-quality voice response that is sent back to the user. This entire pipeline, which would previously take dozens of different library integrations, can now be handled in a single Laravel controller using less than 50 lines of code. Tips & Gotchas * **Context Bloat**: Be careful not to attach too many tools or files to every request. Every tool definition and message in a conversation history consumes tokens, which increases latency and cost. Use the `RemembersConversations` settings to prune old messages. * **Failover Logic**: In production, always define a fallback provider. If OpenAI is experiencing downtime or you hit a rate limit, the SDK can automatically switch to Anthropic to keep your app running. * **Local Development**: Use Ollama for your daily coding to save money. You can switch your local `.env` to point the AI provider to `http://localhost:11434` to test your logic for free. * **Async Processing**: For long-running tasks like transcribing a massive video file or generating a complex image, use the `queue` method. This offloads the work to your Laravel worker and prevents your web request from timing out.