AI Engineer World's Fair 2025
Effective AI Agents Need Data Flywheels, Not The Next Biggest LLM – Sylendran Arunagiri, NVIDIA
Loading slides…
Read the talk
Effective AI Agents Need Data Flywheels, Not The Next Biggest LLM
NVIDIA’s employee-support router shows how production feedback, expert review and task-specific fine-tuning can make smaller models viable replacements for larger ones.
From a talk by Sylendran Arunagiri
Before you start: Familiarity with LLM inference, fine-tuning and retrieval-augmented generation will help; no NVIDIA platform experience is required.
Agents need a learning loop
How do you keep an AI agent relevant and helpful after deployment? Its model must continue serving a task while the knowledge, users and expectations around that task change. Sylendran Arunagiri, from NVIDIA’s Generative AI Platforms team, approaches this problem through data flywheels: repeated cycles of learning from operational data, illustrated through NVIDIA’s internal employee-support agent.
Customer-service agents, software-security agents and research agents have different jobs, but share a basic cycle: perceive, reason and act. They inspect data, form a plan and use tools, functions or external systems to carry it out. Feedback completes that cycle. Capturing whether the result helped the user gives the system evidence it can use to refine its behavior as preferences and needs evolve.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Changing data and rising costs
Enterprise knowledge changes rapidly as new business intelligence enters the system. Customer needs and user preferences change alongside it. Meanwhile, serving larger models becomes more expensive as usage grows. An agent that works at launch therefore faces both a relevance problem and an operating-cost problem.
A data flywheel starts with enterprise data and repeatedly processes and curates it, customizes models, evaluates their behavior and applies guardrails for safer interactions. Retrieval-augmented generation, or RAG, supplies relevant enterprise knowledge when answering a query. Production inference data, business intelligence and user feedback then feed the next round of ground-truth curation.
The objective is to find smaller models that meet the task’s accuracy requirement while improving latency, inference speed and total cost of ownership. That requires evaluating both existing models and newly available candidates against the work the agent actually performs. Model size is a candidate-selection variable; task accuracy is the acceptance criterion.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Map services to the loop
NeMo Microservices provides components for the different stages of this process. The division of responsibilities matters because improving an agent involves more than training its model.
| Service | Responsibility in the flywheel |
|---|---|
| NeMo Curator | Prepare high-quality training data, including multimodal data |
| NeMo Customizer | Customize models with LoRA, P-tuning and full supervised fine-tuning |
| NeMo Evaluator | Run academic or institutional benchmarks and LLM-as-a-judge evaluations |
| NeMo Guardrails | Constrain interactions for privacy, security and safety |
| NeMo Retriever | Build retrieval-augmented generation pipelines |
These are the capabilities presented in the recording; the customization techniques describe that historical offering, not a guarantee of identical support in every current release.
The services expose API endpoints, allowing customization, evaluation and guardrailing to be coordinated through API calls. Arunagiri describes deployment across on-premises infrastructure, cloud, data centers and edge environments, with NVIDIA enterprise support. That platform-level flexibility should not be read as a claim that every component or model fits arbitrary edge hardware.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From production interaction to model promotion
Consider a customer-service agent assembled from these components. A user interacts with its frontend; guardrails protect the interaction; a backend model served through NVIDIA NIM provides optimized inference. Alongside that serving path, a separate improvement loop gathers the data needed to decide which model should power the agent.
The improvement loop follows a concrete sequence:
- Curate operational data and store it in NeMo Data Store.
- Use Customizer and Evaluator to repeat training and evaluation across candidate models.
- Check whether a candidate meets the target accuracy.
- Have an IT administrator or AI engineer promote the accepted model to serve the agent through NIM.
The promotion gate connects experimentation to deployment. Training a candidate does not itself replace the production model.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
NVinfo’s routing problem
NVIDIA’s internal NVinfo agent gives employees access to enterprise knowledge across HR benefits, financial earnings, IT help and product documentation. Its architecture separates those domains into expert agents rather than asking one undifferentiated agent to handle every query.
An employee query passes through safety and security guardrails before reaching an LLM-powered router. Domain experts have RAG pipelines that retrieve the information needed to answer questions in their respective areas. Production inference logs and user feedback feed a flywheel, while subject-matter experts and human review help curate ground truth. Customizer and Evaluator compare models that could be promoted as the router’s NIM.
The red boxes in the architecture identify places where the multi-model system can fail. Routing is one such point: the router must infer the query’s intent and context, select an expert, and hand the query to that expert’s retrieval pipeline. The optimization question is narrow and measurable: which expert should receive this query, and can a faster, cheaper model make that decision accurately?
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The baseline gap is a starting point
The untuned Llama 3.1 70B and Llama 3.3 70B variants reportedly achieved 96% routing accuracy, while Llama 3.1 8B achieved below 14%. These measurements concern selecting the correct expert agent, not the quality of the final answer. The accompanying slide also cautions that the prompt optimized for the 70B model may not suit the 8B model.
Such a large baseline gap can make the smaller model look unviable before any customization begins. But this comparison only establishes how the pretrained models perform under the existing setup. It does not establish how well a smaller model can learn the specific routing task from curated examples.
The team ran the 70B model and circulated a feedback form asking NVIDIA employees to submit queries and report whether the responses were useful. They collected 1,224 responses: 729 satisfactory and 495 unsatisfactory. Response satisfaction and routing accuracy are different measurements: an unhappy user identifies a problem worth investigating, but does not identify which component caused it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Attribute failures before creating training data
NeMo Evaluator used an LLM judge to investigate the 495 unsatisfactory responses and flagged 140 as incorrect routing. Subject-matter experts and manual analysis confirmed 32 genuine routing failures. This second review separates a proposed failure attribution from a training label the team is prepared to trust. Treating every unsatisfactory answer—or every judge verdict—as a router error would train against the wrong target.
Arunagiri reports a ground-truth dataset of 685 examples, split 60% for training or fine-tuning and 40% for testing and evaluation. The recording does not spell out the full accounting between the feedback categories and that final dataset. The central mechanism is nevertheless clear: feedback triggers investigation, reviewed attribution produces task-specific ground truth, and the train/test split supports both customization and measurement.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
What smaller models achieved
The reported baseline combined 96% routing accuracy for the 70B model with a 0.26-second time to first token. The untuned 8B model had approximately 14% accuracy but almost 70% lower latency. The smaller model therefore offered a useful speed advantage before it offered acceptable task performance.
The team then fine-tuned smaller variants, including 8B and 3B models. The resulting comparison changed the deployment choices:
| Router model | Reported routing accuracy |
|---|---|
| Untuned 70B | 96% |
| Fine-tuned 8B | Matched the 70B model’s 96% |
| 1B after customization | 94% |
The fine-tuned 8B model closed the baseline accuracy gap. The 1B result introduced another choice: whether the application could accept a two-percentage-point accuracy reduction in exchange for lower resource requirements. That decision depends on how consequential a routing failure is for the use case.
For a proposed 1B deployment in place of 70B, Arunagiri claims 98% lower inference cost, a 70-fold model-size reduction and 70-fold lower latency. The recording does not specify hardware, serving configuration or the cost methodology, so the cost and latency figures should remain attributed claims rather than portable performance expectations. A parameter-count reduction alone does not establish either saving.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keep evaluating replacements
The useful outcome is not a permanent decision to use one particular small model. It is a repeatable process for finding replacements as production knowledge accumulates and new models become available. Periodic evaluation and fine-tuning can surface smaller candidates that meet the needs of a workflow currently served by a larger model.
Ongoing production logs and knowledge provide fresh material for that process. NeMo Microservices supplies the curation, customization and evaluation components; NIM supplies the serving layer for accepted models. Keeping those paths connected turns a one-time optimization experiment into an operating capability.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Monitor, analyze, plan and execute
The closing framework turns the case study into four recurring operational responsibilities:
- Monitor. Design intuitive ways to collect explicit and implicit feedback while respecting privacy requirements. Use those signals to detect inaccuracies and model drift.
- Analyze. Investigate why the agent behaved as it did. Classify errors, attribute failures to the responsible component and curate ground truth from that analysis.
- Plan. Identify candidate models, consider synthetic datasets, run experiments and fine-tune. Evaluate resource use and cost alongside task performance before choosing a change.
- Execute. Put the change into operation and establish a regular cadence for tracking accuracy, latency, performance and production logs.
Execution includes managing the end-to-end GenAI Ops pipeline after the training cycle finishes. Fresh production behavior becomes the evidence for the next cycle, allowing the agent’s models to keep adapting to the work they must perform.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Version 25.4.0 documentation for preparing datasets, submitting customization jobs, evaluating models, and deploying through NIM.
Further reading
The speaker's companion explanation of NeMo services, NVInfo's router, and the smaller-model optimization results.
Historical introduction to Customizer's parameter-efficient tuning techniques and deployment architecture.
A separate customer-support tool-calling demonstration explaining log ingestion, teacher-based training, evaluation, and model promotion.
Updates since the talk
Later NVInfo research detailing feedback analysis, dataset curation, routing experiments, and the MAPE improvement loop.
Reference implementation and architecture documentation. Deprecated in April 2026 and no longer recommended for new production deployments.
Full transcript
Loading transcript…