AI Engineer World's Fair 2026

User Signal Die at the Retrieval Boundary

Loading slides…

Read the talk

User Signal Die at the Retrieval Boundary

An agent can reason, call tools, and finish a task without learning anything from the result. Outcome-aware retrieval connects reviewed runs to the context and skills used next time.

From a talk by Sonam Pankaj

Before you start: Basic familiarity with LLM tool calls, retrieval-augmented generation, and SQL queries will help you follow the examples.

What changes after an agent fails?

An agent finishes a task, and an evaluator marks the result wrong. What makes the next attempt any different? An LLM can have agency to reason, invoke tools, interact with the world, and retrieve memory without having a mechanism for learning from what worked or failed. That missing loop is the starting problem for Sonam Pankaj, who introduces herself as CEO and co-founder of Starlight Search.

Consider the ReAct execution pattern: a user supplies a prompt; the agent reasons, calls tools, retrieves information or searches, and repeats until the task is complete. The loop explains how an agent acts. It does not, by itself, explain how the outcome of one run changes the next run. Completing an execution loop is different from closing a learning loop.

Pankaj diagnoses repeated failures as partly a retrieval problem: the system keeps selecting context in the same way, even after that context has contributed to unsuccessful runs. She cites an 85% AI-project production failure rate, attributed to Gartner and a McKinsey 2025 report, and a 73% share of RAG pipeline failures attributed to retrieval rather than generation. Those figures lack verified study populations and measurement conditions here; they frame the motivation, rather than establish how often this mechanism causes failure. Context stuffing is the related problem: adding more retrieved material does not ensure that it helps the task.

The criticism echoes Ram Sriharsha’s post on broken agent memory: optimizing lookup speed and cost can make incorrect answers cheaper and faster without teaching retrieval which memories were useful. Retrieval needs a signal about outcomes, not only an efficient way to find similar text.

0:010:25
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:01 · section reference included

Getting evaluation signals out of the dashboard

The third problem is that agents are not outcome-informed. Observability can capture every tool call, LLM completion, and exception, yet those records need not influence subsequent execution. Together, static retrieval, context stuffing, and missing outcome feedback describe a system that can record its mistakes without adapting to them.

Slide with three bullets: Retrieval is Static; Context Stuffing; Agents are not outcome-informed.Open full source frame
Three problems: static retrieval, context stuffing, and agents that are not outcome-informed.

An evaluation suite adds a judgment—pass or fail—but that judgment may never reach the agent’s context, its SKILL.md files, or its next action. The missing layer consumes both traces and evaluations and converts them into retrieval guidance for future runs. Traces explain what happened; evaluations establish whether the result was acceptable; retrieval guidance makes that information available when a relevant task returns.

Without that connection, improvement becomes a manual engineering cycle: inspect the runs, rewrite prompts, redeploy, consider a more expensive model, restructure tools or the harness, or fine-tune a custom model. These interventions can change behavior, but an engineer must repeatedly translate observed failure into a system change.

2:162:37
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:16 · section reference included

Remembering preferences versus learning usefulness

Memory already sounds like the solution, but what the system remembers matters. User profiles, conversation history, extracted facts, and long-lived preferences support personalization. Remembering how someone likes to be addressed improves a chat experience; it does not necessarily teach an agent how to complete a production task. Pankaj names LangChain and Mem0 in this comparison, contrasting preference-oriented memory with the outcome-learning behavior she wants. This is a distinction between memory designs, rather than a complete assessment of those projects’ capabilities.

Retrieval signalWhat it helps select
Embedding similarityMemories related to the current task
Similarity plus outcome utilityRelated memories that have helped execution

The proposed utility score reweights similarity according to how useful a memory has been for an agent’s task. Past traces and outcomes become part of selection, so two relevant memories need not receive equal priority if their execution histories differ.

3:363:47
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:36 · section reference included

Retrieve a lesson that changes the next action

Pankaj calls the proposed runtime layer Agent AgX, expanded as agents with runtime experience. Its intended role is to improve production agents through experience without pre-training, fine-tuning, or manual prompt engineering. She contrasts this with the compile-time approach she associates with DSPy: baking lessons into a prompt, versus adapting context while the agent executes tasks.

The retrieval decision combines semantic relevance to the current task with evidence that a memory historically helped or hurt execution. An evaluation outcome becomes a first-class reranking signal. It is no longer only feedback for someone reading a dashboard; it influences which guidance reaches the agent. The talk describes this weighting conceptually, without specifying an exact utility equation.

A refund request makes the distinction concrete. A fact-oriented memory might say that a user prefers a dark theme or a shorter name. A reasoning memory instead says to check settlement before issuing a refund, so the customer is not refunded twice. The useful memory carries a decision rule and its reason: inspect the relevant transaction state before taking the consequential action.

That lesson should enter context when the task calls for it. Reranking by usefulness and updating context according to the task are intended to avoid context stuffing: the goal is not to supply every remembered fact, but to supply the reasoning that can improve the current decision.

Two panels compare facts and reasoning. Facts list static information without context or history; reasoning says to check settlement before refund and describes usefulness ranking and task-based context.Open full source frame
Memory as reasoning contrasts static user preferences with actionable refund guidance.
4:475:01
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:47 · section reference included

From individual memories to reusable skills

Pankaj introduces Reflect as the memory system evaluated on what she calls τ-bench, describing the task as following policy correctly. Pankaj reports τ-bench performance rising from 66% to 76% with Reflect without baked-in skills, and to 80% with skills. The recording does not establish the exact benchmark variant, scoring protocol, model configuration, or memory exposure, so the percentages are reported results rather than a reproducible comparison.

Skills provide a second destination for accumulated experience. After enough memories exist—Pankaj gives roughly ten as an example—the system can consolidate their reasoning into a skill that the agent consults. Instead of retrieving every individual lesson indefinitely, the agent can draw on a maintained piece of task guidance.

Consider a product SQL agent whose system prompt still mentions a column that is no longer available. That stale instruction can survive even as the agent repeatedly encounters the changed database. An updated skill can tell the agent not to use the obsolete column, and subsequent calls to the skill can expose the corrected guidance. This places some operational knowledge in an updateable artifact rather than leaving it fixed in the system prompt.

6:427:05
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:42 · section reference included

Reported gains and the limits of feedback

Pankaj says similar behavior appears with GPT-5.4, then broadens the discussion to agentic evaluation: reasoning, planning, and tool use over extended workflows, rather than only static question answering. She presents the following Humanity’s Last Exam comparison.

ConfigurationSpeaker-reported score
Baseline35.7%
With RAG47.5%
With an unspecified memory system58.2%
With Reflect Memory System61.3%

The model, tool access, evaluation subset, and memory protocol for these scores are not established in the recording. Pankaj also cites similar trends on BigCodeBench and a benchmark named LifeLongTV in the transcript, without giving numerical results for them.

Outcome-based retrieval introduces its own dependencies:

  • Cold start: Retrieval begins as pure semantic search until enough reviews accumulate.
  • Utility drift: Usefulness can drift, and similar memories can recur in retrieval.
  • Review quality: Noisy outcome labels make the utility signal noisy.
  • Reranking control: A hyperparameter, λ, is associated with credit and reranking.

Pankaj says Reflect reduces most of these problems, but gives no detailed mitigation procedure and identifies cold start as the remaining constraint. The system cannot use a history of reviewed outcomes before that history exists.

7:588:16
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

7:58 · section reference included

A product search with no memories

The live demonstration uses an agent that searches a product catalog stored in a SQL database. Pankaj asks it to find a gaming mouse. The run retrieves zero memories and responds that it cannot find one. She opens the dashboard to inspect what happened. This is the cold-start situation described earlier: there is no retrieved experience to guide a different response when the search fails.

During the dashboard transition, Pankaj briefly returns to evaluation, naming AgentBench and again describing reasoning and planning across tool-using workflows. The demonstration then returns to the unsuccessful catalog search, where the next step is to supply a review of the run.

9:449:57
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:44 · section reference included

A failed search becomes guidance to broaden the query

The feedback changes what counts as an acceptable result. A wireless mouse exists in the database, so Pankaj marks the run as failed and supplies the instruction “Give any relevant.” The review is attached to the input, response, trajectory, and tool call; the initial product-search result is empty. The lesson is therefore more specific than a negative rating: when an exact product search fails, a relevant alternative may satisfy the request.

A later run finds a Wireless Mouse. The displayed trace contains the task “find me a gaming mouse” and a product response, with review status still pending. In the narration, however, Pankaj describes the later input as “mouse.” The visible result shows a relevant product being returned after feedback, but the query discrepancy and broadened acceptance criterion prevent treating this as a controlled, identical-query comparison.

Reflect trace detail shows the task “find me a gaming mouse” and a response listing a Wireless Mouse with product details. Review status is pending.Open full source frame
The later product lookup returns a Wireless Mouse.

Pankaj emphasizes the tool trajectory: the earlier run made a product-search call and received an empty result; the later run still uses the product-search tool but obtains a relevant answer. The tool itself need not be replaced for retrieved guidance to change how the agent uses it.

For a catalog represented by a products table with a name column, the SQL distinction can be expressed as a specific search followed, if empty, by a broader one:

sql

SELECT name
FROM products
WHERE LOWER(name) LIKE '%gaming mouse%';

-- If the first search is empty and alternatives are acceptable:
SELECT name
FROM products
WHERE LOWER(name) LIKE '%mouse%';

This illustrates the search lesson, rather than reproducing the dashboard’s exact SQL. A name such as Wireless Mouse matches the broader query. Returning it as a related product preserves the distinction between finding an alternative and establishing that it meets the original gaming requirement.

11:5612:07
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:56 · section reference included

Keep the lesson, then maintain the skill

The memory card makes the resulting guidance visible: it records the failed gaming-mouse search, identifies the mistake, describes a corrective action, associates the lesson with the search tool, and recommends broadening a query when no results appear. This is the reasoning-memory pattern from the refund example, now attached to a concrete tool failure. Memories are retrieved according to utility, and that utility is reranked as evidence of usefulness accumulates.

Reflect memory card for “find me a gaming mouse” shows a failure summary, key mistake, correct action, applicable search tool, and guidance to broaden a query when no results appear.Open full source frame
A reviewed failure becomes guidance to broaden unsuccessful product searches.

The final step is to consolidate reviewed experience into a skill. Pankaj describes memory scores changing with reviews, then gives about five reviews as an example of when findings could be baked into a skill. Earlier she used about ten memories; these are illustrative counts, not an established fixed threshold. The later Project Skill view still shows an unmet review threshold, so the demonstration explains the path to publication without showing a completed skill update.

The durable change is guidance the agent can consult without rewriting its system prompt. Individual reviewed runs produce memories; usefulness affects which memories reach future tasks; accumulated lessons can become maintained skills. That is how an outcome can cross the retrieval boundary and influence what the agent does next.

14:0214:26
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

14:02 · section reference included

Resources

From the talk

Updates since the talk

  • Current documentation explains how traces and reviews create memories, update utility, and influence retrieval, with a worked refund example.

Transcript download
Full transcript

Loading transcript…