LinkedIn cuts LLM latency by 7x to power real-time recommendations

Prerequisites

To follow this workflow, you should be comfortable with basic Machine Learning (ML) concepts, specifically how Transformers use attention mechanisms. Familiarity with model training phases like fine-tuning and distillation is essential, as is a basic understanding of hardware constraints in production environments.

Key Libraries & Tools

• SGlang & vLLM: Used for developing custom kernels to handle specialized attention masks.
• PyTorch Lightning: Provided the backbone for model training and automation.
• Mixtral: Served as the base architecture for scaling experiments up to 8x22B parameters.
• FP8/FP32 Precision: Formats used to balance computational speed with numerical stability.

Code Walkthrough: The Promptification Logic

The core of the LinkedIn approach is transforming raw user data into a structured prompt that an LLM can reason about. This process, termed "promptification," involves three distinct blocks of information.

def create_rec_prompt(member_profile, history, candidate_item):
    instruction = "Determine if this member will engage with the following item."
    
    # Profile context helps with cold-start users
    profile_context = f"Member Title: {member_profile['job_title']}, Skills: {member_profile['skills']}"
    
    # Historical interactions provide the pattern
    history_block = "\n".join([f"Action: {h['action']} on Item: {h['item_name']}" for h in history])
    
    # The prediction target
    query = f"Target Item: {candidate_item['title']}. Prediction:"
    
    return f"{instruction}\n{profile_context}\n{history_block}\n{query}"

To keep performance high, the team implemented a sparsified attention mask. Instead of every candidate item attending to every other item (which is $O(N^2)$), they used a mask where items only attend to the historical user profile. This prevents unnecessary computation between 500+ candidate items in a single query.

Syntax and Optimization Notes

The team discovered that calibration is the most sensitive part of ranking. While they utilized FP8 for internal layers to save memory, they kept the LM Head in FP32. Lower precision in the final layer caused "probability collapse," making it impossible for the system to distinguish between a 0.75 and a 0.80 relevancy score. This distinction is critical for sorting a list of jobs or feed posts effectively.

Practical Examples

LinkedIn uses this unified model to solve several "out-of-domain" tasks where no specific training data exists for a new surface. For example, if a new "Niche Interests" feed is launched, the model can recommend content immediately by following the provided instructions rather than waiting months for enough click data to train a traditional Gradient Boosted Decision Tree (GBDT) model.

Tips & Gotchas

• Step-by-Step Distillation: Never jump directly from a 150B parameter model to a 1B model. The LinkedIn team found that distilling from 150B to 8B, then 8B to 3B, preserved reasoning capabilities that were lost in a single-step jump.
• The Context Length Trap: While more history generally helps, performance often drops after a certain context length. This isn't because the data is bad, but because current open-source models often fail to generalize to extremely long contexts during inference.
• Automation is Non-Negotiable: To optimize for production, the team automated everything from quantization parameters to results logging. This allowed them to iterate through 7x latency reductions without manual intervention for every experiment.