Mixtral

Overview Most recommendation systems rely on disjoint, task-specific models that require separate training and maintenance. LinkedIn engineers Hamed Firooz and Maziar Sanjabi changed this by building a unified foundation model capable of handling multiple personalization tasks simultaneously. By treating recommendation as a language problem, they achieved zero-shot capabilities for new features and improved performance for "cold start" users who have minimal interaction history. Prerequisites To follow this workflow, you should be comfortable with basic **Machine Learning (ML)** concepts, specifically how **Transformers** use attention mechanisms. Familiarity with model training phases like fine-tuning and distillation is essential, as is a basic understanding of hardware constraints in production environments. Key Libraries & Tools * **SGlang & vLLM**: Used for developing custom kernels to handle specialized attention masks. * **PyTorch Lightning**: Provided the backbone for model training and automation. * **Mixtral**: Served as the base architecture for scaling experiments up to 8x22B parameters. * **FP8/FP32 Precision**: Formats used to balance computational speed with numerical stability. Code Walkthrough: The Promptification Logic The core of the LinkedIn approach is transforming raw user data into a structured prompt that an LLM can reason about. This process, termed "promptification," involves three distinct blocks of information. ```python def create_rec_prompt(member_profile, history, candidate_item): instruction = "Determine if this member will engage with the following item." # Profile context helps with cold-start users profile_context = f"Member Title: {member_profile['job_title']}, Skills: {member_profile['skills']}" # Historical interactions provide the pattern history_block = "\n".join([f"Action: {h['action']} on Item: {h['item_name']}" for h in history]) # The prediction target query = f"Target Item: {candidate_item['title']}. Prediction:" return f"{instruction}\n{profile_context}\n{history_block}\n{query}" ``` To keep performance high, the team implemented a **sparsified attention mask**. Instead of every candidate item attending to every other item (which is $O(N^2)$), they used a mask where items only attend to the historical user profile. This prevents unnecessary computation between 500+ candidate items in a single query. Syntax and Optimization Notes The team discovered that **calibration** is the most sensitive part of ranking. While they utilized **FP8** for internal layers to save memory, they kept the **LM Head** in **FP32**. Lower precision in the final layer caused "probability collapse," making it impossible for the system to distinguish between a 0.75 and a 0.80 relevancy score. This distinction is critical for sorting a list of jobs or feed posts effectively. Practical Examples LinkedIn uses this unified model to solve several "out-of-domain" tasks where no specific training data exists for a new surface. For example, if a new "Niche Interests" feed is launched, the model can recommend content immediately by following the provided instructions rather than waiting months for enough click data to train a traditional Gradient Boosted Decision Tree (GBDT) model. Tips & Gotchas * **Step-by-Step Distillation**: Never jump directly from a 150B parameter model to a 1B model. The LinkedIn team found that distilling from 150B to 8B, then 8B to 3B, preserved reasoning capabilities that were lost in a single-step jump. * **The Context Length Trap**: While more history generally helps, performance often drops after a certain context length. This isn't because the data is bad, but because current open-source models often fail to generalize to extremely long contexts during inference. * **Automation is Non-Negotiable**: To optimize for production, the team automated everything from quantization parameters to results logging. This allowed them to iterate through 7x latency reductions without manual intervention for every experiment.