AI Engineer Code 2025
Stop Fine-Tuning to Fix Retrieval Problems — Anant Srivastava
Loading slides…
Read the talk
Stop Fine-Tuning to Fix Retrieval Problems
Anant Srivastava explains how to assign behavior to prompts, changing knowledge to memory, and stable task reflexes to weights—and how to move information among them as an agent learns from its work.
From a talk by Anant Srivastava
At a glance
Ideas worth remembering
Prompt edits, indexed documents and training samples each decide where behavior or knowledge lives. Review their combined effects as architecture.
Use authored prompts for small, stable behavior; use memory for changing, large, citable or access-controlled knowledge.
When an assistant lacks the right document chunks, fix retrieval. Fine-tuning on runbooks can encode stale facts while preserving the original retrieval failure.
Fine-tune stable task reflexes when human judgments have converged and capability or cost justifies the change. Keep humans at contested edges and monitor drift.
Learning a recurring format changes future retrieval needs: format examples may become unnecessary even while factual knowledge remains in memory.
Every prompt edit decides where knowledge lives
Enterprise knowledge lives in files, databases and APIs. A model consumes tokens and produces tokens; the system around it decides which knowledge reaches inference and how. Anant Srivastava opens with that architectural question: should something arrive through authored instructions, through retrieval and memory, or through a change to model weights?
Teams often answer by escalation. An answer is wrong, so someone edits the prompt. If that fails, someone investigates retrieval. If answers remain wrong, fine-tuning becomes the next proposed fix. This treats three tools as successive levels of intervention, even though each changes a different part of the system.
A quieter mechanism produces the same architectural drift during ordinary product work:
- Prompt edits decide which behavior lives in instructions.
- Indexed documents decide which knowledge lives in retrievable memory.
- Training samples decide what the model absorbs into its weights.
These decisions happen even when nobody calls them architecture. Their effects accumulate across teams.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The catalog changes, but the old product names survive
The support assistant example follows several individually plausible changes. After launch, a product manager adjusts its tone through the prompt. The support team adds refund-policy documents to retrieval. When answers still disappoint, an engineer inserts the product catalog into the prompt. Later, the machine-learning team fine-tunes the model on support tickets covering up to six months of history.
Then a new catalog launches. Updating the catalog in the prompt looks like the obvious maintenance step, yet the assistant continues returning product names that no longer exist. Srivastava attributes this failure to the historical tickets: fine-tuning absorbed product information alongside the support behavior those tickets demonstrated. Replacing the prompt’s catalog changes one source of information while leaving the learned associations in the weights.
Where did the obsolete names survive the update? The diagram follows the two paths by which catalog knowledge entered the assistant. The prompt path receives a replacement; the training path retains historical product information. That split explains why updating the visible catalog can leave the underlying failure intact.
Six months of normal product work created a system that nobody owned as a whole. Everyone owned a piece. Srivastava includes himself among engineers who have made these decisions without enough thought; the useful correction is to ask where each piece belongs before adding it.
Two catalog paths, only one updated
Read the diagram as text
- New product catalog. prompt-path — Launched after the assistant had accumulated several changes.
- Updated prompt. prompt-path — The engineer replaces the catalog supplied in instructions.
- Historical support tickets. training-path — Up to six months of tickets contain product information.
- Fine-tuned weights. training-path — Historical catalog information enters the model through training.
- Obsolete product names. The assistant still returns names that do not exist in the new catalog.
- New product catalog → Updated prompt: Replace catalog in prompt.
- Historical support tickets → Fine-tuned weights: Fine-tune on historical tickets.
- Updated prompt → Obsolete product names: Update does not eliminate the failure.
- Fine-tuned weights → Obsolete product names: Learned product information persists.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Prompt: small, stable instructions about behavior
The prompt’s job is to specify tone, persona and behavior through instructions that are small, stable and editable. For the SaaS support agent, that means maintaining a professional tone, offering human escalation after three failures, and giving concrete next steps. Those requirements remain useful across users and queries.
A product catalog has different maintenance and inference costs. Carrying it in the prompt supplies factual context even when much of it is irrelevant to the question, and the additional tokens cost money. Srivastava also warns about the “lost in the middle” problem with long context. His diagnostic is whether the material is small, stable and concerned with how to behave.
Here, prompt includes authored instructions wherever they originate. A system prompt and instructions loaded from Markdown files serve the same architectural role. The important distinction is what they tell the model to do, rather than which file or message holds them.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Memory: retrieve current knowledge with the right scope
Memory grounds an agent in knowledge that is current, large or citable. Current knowledge changes faster than it is practical to fine-tune. Large collections exceed what belongs in a standing prompt. Citable knowledge must remain connected to a source that the production system can identify.
Memory covers two arrangements in this talk. An agent can acquire information about the person interacting with it. An external application can also maintain enterprise knowledge—such as refund policies—and retrieve it through retrieval-augmented generation, or RAG. The agent need not manage the underlying collection for that collection to serve as memory.
Retrieval supplies material to reason over; the model still needs the ability to use it. Srivastava’s warning is concrete: if a model cannot reason across two, three or five documents, supplying fifty does not solve that capability problem. More context can add confusion when the missing ingredient is reasoning.
Access control adds another reason to keep knowledge in memory. If one user must not see another person’s information, retrieval must control which information reaches that user’s context. The diagnostic therefore includes scope: does this knowledge belong to a particular user, repository or other permitted collection?
An organizational code assistant makes these requirements practical. Repositories change frequently, so fine-tuning their contents creates a maintenance problem. Inserting the entire codebase into a standing prompt also carries irrelevant material into inference. Relevant code can enter the context window when a task needs it; the collection itself belongs in retrievable memory.
Two retrieval mechanisms keep that collection useful:
- Code-aware chunking. Chunk code with its structure in mind, potentially using an abstract syntax tree (AST), rather than treating it only as undifferentiated text.
- Metadata and filtering. Attach repository identity and relevant user-permission information to chunks. Keeping that metadata alongside the chunk makes it available when selecting the material for a request.
The aim is to return the code relevant to the task and its scope. Returning many loosely selected chunks produces what Srivastava calls “RAG mush”: the database supplies volume, and the model inherits the confusion.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Weights: avoid freezing knowledge that still changes
Rate of change is the starting diagnostic for weights. Fine-tuning can encode a stable pattern, but stability alone does not justify training. There must be a strong reason to make the change. Encoding a pattern too early risks freezing a decision boundary while the task is still evolving.
The subtle mistake is an internal document assistant that answers poorly, prompting the team to fine-tune on its documentation, runbooks or procedures to teach the domain. In Srivastava’s example, the missing ingredient is the right document chunks. Training on those documents gives the model an older version of the facts while leaving the retrieval defect unresolved.
The repair follows the failure’s cause: improve retrieval so the assistant receives the relevant current procedure. Runbooks belong in external memory, where the system can update and retrieve them. A wrong answer alone does not diagnose a need for fine-tuning.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Fine-tune the stable center, retain humans at the contested edge
Content moderation and claims processing offer a different path. Start with a capable model making recommendations and humans correcting them. Early corrections may be inconsistent. Over time, repeated decisions can reveal a stable center of the task and a contested edge where human judgment remains necessary.
As human override rates settle, that stable center becomes a candidate for fine-tuning. Convergence is something to observe, rather than assume every task will reach. The resulting model still needs monitoring for drift: if the previously settled pattern changes, its learned response can become inappropriate. Human involvement remains at the contested edge.
Medical claims coding sharpens the distinction between facts and reflexes. Doctors’ notes are mapped to ICD-10 codes for uses such as insurance. Srivastava cites seventy thousand codes, yet recommends against fine-tuning simply to memorize the collection. Codes can change, and covering the collection through training would require substantial training data.
The training target is the repeatable task behavior: understanding the input format and making the appropriate selection across formats. The code collection remains knowledge to consult. This is the practical force of “reflexes, not facts”: fine-tuning should teach a settled operation the model must perform repeatedly.
After stability and human convergence, there are two reasons to consider the expense of fine-tuning:
- Capability. The model needs to acquire a task behavior it cannot perform adequately.
- Cost. A repeated, well-understood pattern may let a smaller fine-tuned model handle volume more cheaply.
Srivastava expects cost to be the more common motivation when frontier models already have sufficient capability. He gives no universal savings figure; the case depends on the pattern and workload.
The decision lens is compact: prompt for behavior, memory for facts and knowledge, weights for learned reasoning or task reflexes. It is a heuristic with exceptions. Its value is that each proposed change must answer a question about the job it will perform, rather than merely occupy the next rung after an unsuccessful fix.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The harness moves information as the agent learns
Assigning information to a place is only the beginning. A useful architecture lets information circulate. Work performed in the context window generates signals, some of which become lasting memory. A later session retrieves relevant memory into context. The harness—the surrounding system that stores information and manages these flows—makes experience available beyond a single interaction.
A further flow turns stable experience into training material. Over time, retrieval patterns and recurring formats reveal things the model could learn reflexively. Those selected patterns move from memory into fine-tuning. This requires the earlier diagnostic: a recurring fact does not become a good training target merely because it appears often.
What changes after a format becomes a learned reflex? The cycle below separates the flow of information into context from the effect of fine-tuning on future retrieval. The return arrow from weights to memory means retrieval needs change; it does not mean the system copies the model’s weights into a database.
Return to medical coding. Before fine-tuning, the system may retrieve examples to show the model the notes-and-code format on each request. Once the model learns that format reflexively, those particular examples need not be supplied every time. The system still needs the relevant factual knowledge, but the material required to teach the format at inference has changed.
That is how an agent can improve through doing its job: experience becomes memory, settled patterns become reflexes, and those reflexes alter what future work needs to retrieve. Srivastava closes by calling the model the easy part. The substantial architectural work is the harness that puts information in the right place and manages its movement over time.
Experience changes both the model and what it retrieves
Read the diagram as text
- Prompt and context window. Instructions and retrieved information support the current session.
- Memory. Retain useful signals and supply knowledge to later sessions.
- Fine-tuned weights. Learn stable formats and recurring task reflexes.
- Prompt and context window → Memory: Keep useful signals from work.
- Memory → Prompt and context window: Retrieve information for a new session.
- Memory → Fine-tuned weights: Fine-tune selected stable patterns.
- Fine-tuned weights → Memory: Change which examples need retrieval.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
Related talks
- Beating RL With Reflection: GEPA and Optimize Anything
Explores how execution traces and human feedback can improve prompts, agent programs and repository skills, giving a complementary mechanism for turning work experience into reusable behavior.
Full transcript
Loading transcript…