AI Engineer World's Fair 2025

Effective AI Agents Need Data Flywheels, Not The Next Biggest LLM – Sylendran Arunagiri, NVIDIA

Loading slides…

Read the talk

Effective AI Agents Need Data Flywheels, Not The Next Biggest LLM

NVIDIA’s employee-support router shows how production feedback, expert review and task-specific fine-tuning can make smaller models viable replacements for larger ones.

From a talk by Sylendran Arunagiri

Before you start: Familiarity with LLM inference, fine-tuning and retrieval-augmented generation will help; no NVIDIA platform experience is required.

Changing data and rising costs

Enterprise knowledge changes rapidly as new business intelligence enters the system. Customer needs and user preferences change alongside it. Meanwhile, serving larger models becomes more expensive as usage grows. An agent that works at launch therefore faces both a relevance problem and an operating-cost problem.

Slide lists rapidly aging data, evolving customer needs, growing inference costs, and rising accuracy targets beneath a call for continuous updates.Open full source frame
Four challenges of operating AI agents at scale.

A data flywheel starts with enterprise data and repeatedly processes and curates it, customizes models, evaluates their behavior and applies guardrails for safer interactions. Retrieval-augmented generation, or RAG, supplies relevant enterprise knowledge when answering a query. Production inference data, business intelligence and user feedback then feed the next round of ground-truth curation.

The objective is to find smaller models that meet the task’s accuracy requirement while improving latency, inference speed and total cost of ownership. That requires evaluating both existing models and newly available candidates against the work the agent actually performs. Model size is a candidate-selection variable; task accuracy is the acceptance criterion.

1:542:15
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:54 · section reference included

Map services to the loop

NeMo Microservices provides components for the different stages of this process. The division of responsibilities matters because improving an agent involves more than training its model.

ServiceResponsibility in the flywheel
NeMo CuratorPrepare high-quality training data, including multimodal data
NeMo CustomizerCustomize models with LoRA, P-tuning and full supervised fine-tuning
NeMo EvaluatorRun academic or institutional benchmarks and LLM-as-a-judge evaluations
NeMo GuardrailsConstrain interactions for privacy, security and safety
NeMo RetrieverBuild retrieval-augmented generation pipelines

These are the capabilities presented in the recording; the customization techniques describe that historical offering, not a guarantee of identical support in every current release.

The services expose API endpoints, allowing customization, evaluation and guardrailing to be coordinated through API calls. Arunagiri describes deployment across on-premises infrastructure, cloud, data centers and edge environments, with NVIDIA enterprise support. That platform-level flexibility should not be read as a claim that every component or model fits arbitrary edge hardware.

Circular diagram connects enterprise data, NeMo Curator, Customizer, Evaluator, Guardrails, and Retriever around NVIDIA NIM. Bottom labels include Enterprise-Grade and Run Anywhere.Open full source frame
NeMo microservices arranged around NVIDIA NIM in a data flywheel.
3:464:05
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:46 · section reference included

From production interaction to model promotion

Consider a customer-service agent assembled from these components. A user interacts with its frontend; guardrails protect the interaction; a backend model served through NVIDIA NIM provides optimized inference. Alongside that serving path, a separate improvement loop gathers the data needed to decide which model should power the agent.

The improvement loop follows a concrete sequence:

  1. Curate operational data and store it in NeMo Data Store.
  2. Use Customizer and Evaluator to repeat training and evaluation across candidate models.
  3. Check whether a candidate meets the target accuracy.
  4. Have an IT administrator or AI engineer promote the accepted model to serve the agent through NIM.

The promotion gate connects experimentation to deployment. Training a candidate does not itself replace the production model.

5:275:42
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:27 · section reference included

NVinfo’s routing problem

NVIDIA’s internal NVinfo agent gives employees access to enterprise knowledge across HR benefits, financial earnings, IT help and product documentation. Its architecture separates those domains into expert agents rather than asking one undifferentiated agent to handle every query.

An employee query passes through safety and security guardrails before reaching an LLM-powered router. Domain experts have RAG pipelines that retrieve the information needed to answer questions in their respective areas. Production inference logs and user feedback feed a flywheel, while subject-matter experts and human review help curate ground truth. Customizer and Evaluator compare models that could be promoted as the router’s NIM.

The red boxes in the architecture identify places where the multi-model system can fail. Routing is one such point: the router must infer the query’s intent and context, select an expert, and hand the query to that expert’s retrieval pipeline. The optimization question is narrow and measurable: which expert should receive this query, and can a faster, cheaper model make that decision accurately?

Architecture diagram connects a user through guardrails and a router to domain experts, retrieval, reranking, answer generation, and citations. Red boxes identify errors, while user feedback and SME input feed a NeMo data flywheel.Open full source frame
NVInfoBot’s routing, retrieval, and feedback architecture.
6:376:58
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:37 · section reference included

The baseline gap is a starting point

The untuned Llama 3.1 70B and Llama 3.3 70B variants reportedly achieved 96% routing accuracy, while Llama 3.1 8B achieved below 14%. These measurements concern selecting the correct expert agent, not the quality of the final answer. The accompanying slide also cautions that the prompt optimized for the 70B model may not suit the 8B model.

Bar chart shows Llama 3.1 70B and 3.3 70B at 96.0, Llama 3.1 8B at 13.6, Llama 3.2 3B at 53.1, and Llama 3.2 1B at 13.6. A callout warns that the prompt optimized for 70B may not suit 8B.Open full source frame
Baseline model results using the existing prompt without fine-tuning.

Such a large baseline gap can make the smaller model look unviable before any customization begins. But this comparison only establishes how the pretrained models perform under the existing setup. It does not establish how well a smaller model can learn the specific routing task from curated examples.

The team ran the 70B model and circulated a feedback form asking NVIDIA employees to submit queries and report whether the responses were useful. They collected 1,224 responses: 729 satisfactory and 495 unsatisfactory. Response satisfaction and routing accuracy are different measurements: an unhappy user identifies a problem worth investigating, but does not identify which component caused it.

9:5310:15
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:53 · section reference included

Attribute failures before creating training data

NeMo Evaluator used an LLM judge to investigate the 495 unsatisfactory responses and flagged 140 as incorrect routing. Subject-matter experts and manual analysis confirmed 32 genuine routing failures. This second review separates a proposed failure attribution from a training label the team is prepared to trust. Treating every unsatisfactory answer—or every judge verdict—as a router error would train against the wrong target.

Arunagiri reports a ground-truth dataset of 685 examples, split 60% for training or fine-tuning and 40% for testing and evaluation. The recording does not spell out the full accounting between the feedback categories and that final dataset. The central mechanism is nevertheless clear: feedback triggers investigation, reviewed attribution produces task-specific ground truth, and the train/test split supports both customization and measurement.

11:4211:58
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:42 · section reference included

What smaller models achieved

The reported baseline combined 96% routing accuracy for the 70B model with a 0.26-second time to first token. The untuned 8B model had approximately 14% accuracy but almost 70% lower latency. The smaller model therefore offered a useful speed advantage before it offered acceptable task performance.

The team then fine-tuned smaller variants, including 8B and 3B models. The resulting comparison changed the deployment choices:

Router modelReported routing accuracy
Untuned 70B96%
Fine-tuned 8BMatched the 70B model’s 96%
1B after customization94%

The fine-tuned 8B model closed the baseline accuracy gap. The 1B result introduced another choice: whether the application could accept a two-percentage-point accuracy reduction in exchange for lower resource requirements. That decision depends on how consequential a routing failure is for the use case.

For a proposed 1B deployment in place of 70B, Arunagiri claims 98% lower inference cost, a 70-fold model-size reduction and 70-fold lower latency. The recording does not specify hardware, serving configuration or the cost methodology, so the cost and latency figures should remain attributed claims rather than portable performance expectations. A parameter-count reduction alone does not establish either saving.

12:3612:51
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

12:36 · section reference included

Keep evaluating replacements

The useful outcome is not a permanent decision to use one particular small model. It is a repeatable process for finding replacements as production knowledge accumulates and new models become available. Periodic evaluation and fine-tuning can surface smaller candidates that meet the needs of a workflow currently served by a larger model.

Ongoing production logs and knowledge provide fresh material for that process. NeMo Microservices supplies the curation, customization and evaluation components; NIM supplies the serving layer for accepted models. Keeping those paths connected turns a one-time optimization experiment into an operating capability.

13:5814:20
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:58 · section reference included

Monitor, analyze, plan and execute

The closing framework turns the case study into four recurring operational responsibilities:

  1. Monitor. Design intuitive ways to collect explicit and implicit feedback while respecting privacy requirements. Use those signals to detect inaccuracies and model drift.
  2. Analyze. Investigate why the agent behaved as it did. Classify errors, attribute failures to the responsible component and curate ground truth from that analysis.
  3. Plan. Identify candidate models, consider synthetic datasets, run experiments and fine-tune. Evaluate resource use and cost alongside task performance before choosing a change.
  4. Execute. Put the change into operation and establish a regular cadence for tracking accuracy, latency, performance and production logs.

Execution includes managing the end-to-end GenAI Ops pipeline after the training cycle finishes. Fresh production behavior becomes the evidence for the next cycle, allowing the agent’s models to keep adapting to the work they must perform.

14:4815:13
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

14:48 · section reference included

Resources

From the talk

Updates since the talk

Transcript download
Full transcript

Loading transcript…