Llama 8B matches 70B performance with NVIDIA data flywheels

Most developers assume that building a more capable AI agent requires upgrading to the next massive Large Language Model. However, Sylendran Arunagiri from NVIDIA argues that high-performance agents actually depend on data flywheels—continuous loops that process data, customize models, and evaluate performance in production. Instead of chasing parameter counts, a data flywheel focuses on curating ground truth from user feedback and inference logs to refine smaller, more efficient models.

Internal success with NV Info Agent

NVIDIA applied this concept to its NV Info Agent, an internal tool designed to help employees navigate HR benefits, IT help, and financial data. The architecture uses a router agent to direct queries to specific domain experts. Initially, a Llama 3.1 70B model achieved a 96% baseline accuracy for routing, while the smaller Llama 3.1 8B variant struggled at a dismal 14%.

Rather than abandoning the small model, the team used the NVIDIA NeMo microservices suite to build a flywheel. By curating just 685 data points from employee feedback and using LLM-as-a-judge techniques via NeMo Evaluator, they fine-tuned the 8B model. The result was transformative: the 8B model reached parity with the 70B model's 96% accuracy but delivered a 70% reduction in latency and a massive 98% saving in inference costs. Even a tiny 1B model reached 94% accuracy, proving that data quality beats model size.

Building the loop with NeMo microservices

Constructing these flywheels is now a modular process thanks to NVIDIA NeMo microservices. Tools like NeMo Curator handle high-quality data sets, while NeMo Customizer implements techniques like LoRA and SFT. NeMo Guardrails ensures safety, and NeMo Retriever manages RAG pipelines. These components function like Lego pieces, allowing developers to deploy optimized NVIDIA NIM inference microservices anywhere from the cloud to the edge.

A framework for agentic evolution

Effective flywheels follow a four-step cycle: monitor, analyze, plan, and execute. It begins with collecting explicit and implicit user signals to identify model drift. Developers must then attribute failures—determining if an error stemmed from incorrect routing or poor retrieval. This analysis feeds the planning stage, where synthetic data is generated for fine-tuning. Finally, execution involves setting a regular cadence for retraining, ensuring the agent remains relevant as business data evolves. This transition from static deployment to a living, learning system is the key to sustainable GenAI operations.