# User Signal Die at the Retrieval Boundary

Sonam Pankaj

AI Engineer World's Fair 2026

[Talk page](https://ai.engineer/talks/Jx4ZFEAq6bY-user-signal-die-at-retrieval-boundary) · [Video](https://www.youtube.com/watch?v=Jx4ZFEAq6bY)

## User Signal Die at the Retrieval Boundary

An agent can reason, call tools, and finish a task without learning anything from the result. Outcome-aware retrieval connects reviewed runs to the context and skills used next time.

Basic familiarity with LLM tool calls, retrieval-augmented generation, and SQL queries will help you follow the examples.

### What changes after an agent fails?

[0:01](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=1s)

An agent finishes a task, and an evaluator marks the result wrong. What makes the next attempt any different? An LLM can have agency to reason, invoke tools, interact with the world, and retrieve memory without having a mechanism for learning from what worked or failed. That missing loop is the starting problem for Sonam Pankaj, who introduces herself as CEO and co-founder of Starlight Search.

Consider the [ReAct](https://arxiv.org/abs/2210.03629) execution pattern: a user supplies a prompt; the agent reasons, calls tools, retrieves information or searches, and repeats until the task is complete. The loop explains how an agent acts. It does not, by itself, explain how the outcome of one run changes the next run. **Completing an execution loop is different from closing a learning loop.**

Pankaj diagnoses repeated failures as partly a retrieval problem: the system keeps selecting context in the same way, even after that context has contributed to unsuccessful runs. She cites an 85% AI-project production failure rate, attributed to Gartner and a McKinsey 2025 report, and a 73% share of RAG pipeline failures attributed to retrieval rather than generation. Those figures lack verified study populations and measurement conditions here; they frame the motivation, rather than establish how often this mechanism causes failure. Context stuffing is the related problem: adding more retrieved material does not ensure that it helps the task.

The criticism echoes [Ram Sriharsha’s post on broken agent memory](https://www.linkedin.com/posts/harsha340_your-agents-memory-is-broken-heres-why-activity-7425652489246527488-IUDN): optimizing lookup speed and cost can make incorrect answers cheaper and faster without teaching retrieval which memories were useful. Retrieval needs a signal about outcomes, not only an efficient way to find similar text.

### Getting evaluation signals out of the dashboard

[2:16](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=136s)

The third problem is that agents are not **outcome-informed**. Observability can capture every tool call, LLM completion, and exception, yet those records need not influence subsequent execution. Together, static retrieval, context stuffing, and missing outcome feedback describe a system that can record its mistakes without adapting to them.

![Slide with three bullets: Retrieval is Static; Context Stuffing; Agents are not outcome-informed.](https://ai.engineer/api/talk-essay-media/f1eb1abd33818c195f21cb67d080983e13e92a47b01734efbe04dd4dfa40698c)

Three problems: static retrieval, context stuffing, and agents that are not outcome-informed.

An evaluation suite adds a judgment—pass or fail—but that judgment may never reach the agent’s context, its `SKILL.md` files, or its next action. The missing layer consumes both traces and evaluations and converts them into **retrieval guidance for future runs**. Traces explain what happened; evaluations establish whether the result was acceptable; retrieval guidance makes that information available when a relevant task returns.

Without that connection, improvement becomes a manual engineering cycle: inspect the runs, rewrite prompts, redeploy, consider a more expensive model, restructure tools or the harness, or fine-tune a custom model. These interventions can change behavior, but an engineer must repeatedly translate observed failure into a system change.

### Remembering preferences versus learning usefulness

[3:36](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=216s)

Memory already sounds like the solution, but what the system remembers matters. User profiles, conversation history, extracted facts, and long-lived preferences support personalization. Remembering how someone likes to be addressed improves a chat experience; it does not necessarily teach an agent how to complete a production task. Pankaj names LangChain and Mem0 in this comparison, contrasting preference-oriented memory with the outcome-learning behavior she wants. This is a distinction between memory designs, rather than a complete assessment of those projects’ capabilities.

| Retrieval signal | What it helps select |
| --- | --- |
| Embedding similarity | Memories related to the current task |
| Similarity plus outcome utility | Related memories that have helped execution |

The proposed **utility score** reweights similarity according to how useful a memory has been for an agent’s task. Past traces and outcomes become part of selection, so two relevant memories need not receive equal priority if their execution histories differ.

### Retrieve a lesson that changes the next action

[4:47](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=287s)

Pankaj calls the proposed runtime layer Agent AgX, expanded as agents with runtime experience. Its intended role is to improve production agents through experience without pre-training, fine-tuning, or manual prompt engineering. She contrasts this with the compile-time approach she associates with DSPy: baking lessons into a prompt, versus adapting context while the agent executes tasks.

The retrieval decision combines semantic relevance to the current task with evidence that a memory historically helped or hurt execution. **An evaluation outcome becomes a first-class reranking signal.** It is no longer only feedback for someone reading a dashboard; it influences which guidance reaches the agent. The talk describes this weighting conceptually, without specifying an exact utility equation.

A refund request makes the distinction concrete. A fact-oriented memory might say that a user prefers a dark theme or a shorter name. A reasoning memory instead says to check settlement before issuing a refund, so the customer is not refunded twice. The useful memory carries a decision rule and its reason: inspect the relevant transaction state before taking the consequential action.

That lesson should enter context when the task calls for it. Reranking by usefulness and updating context according to the task are intended to avoid context stuffing: the goal is not to supply every remembered fact, but to supply the reasoning that can improve the current decision.

![Two panels compare facts and reasoning. Facts list static information without context or history; reasoning says to check settlement before refund and describes usefulness ranking and task-based context.](https://ai.engineer/api/talk-essay-media/8928137ab198f2ed1555d93be6e797aacd7f4ac6df7e4aa4a6ce283e4b7b86fa)

Memory as reasoning contrasts static user preferences with actionable refund guidance.

### From individual memories to reusable skills

[6:42](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=402s)

Pankaj introduces Reflect as the memory system evaluated on what she calls τ-bench, describing the task as following policy correctly. Pankaj reports τ-bench performance rising from 66% to 76% with Reflect without baked-in skills, and to 80% with skills. The recording does not establish the exact benchmark variant, scoring protocol, model configuration, or memory exposure, so the percentages are reported results rather than a reproducible comparison.

Skills provide a second destination for accumulated experience. After enough memories exist—Pankaj gives roughly ten as an example—the system can consolidate their reasoning into a skill that the agent consults. Instead of retrieving every individual lesson indefinitely, the agent can draw on a maintained piece of task guidance.

Consider a product SQL agent whose system prompt still mentions a column that is no longer available. That stale instruction can survive even as the agent repeatedly encounters the changed database. An updated skill can tell the agent not to use the obsolete column, and subsequent calls to the skill can expose the corrected guidance. This places some operational knowledge in an updateable artifact rather than leaving it fixed in the system prompt.

### Reported gains and the limits of feedback

[7:58](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=478s)

Pankaj says similar behavior appears with **GPT-5.4**, then broadens the discussion to agentic evaluation: reasoning, planning, and tool use over extended workflows, rather than only static question answering. She presents the following [Humanity’s Last Exam](https://arxiv.org/abs/2501.14249) comparison.

| Configuration | Speaker-reported score |
| --- | ---: |
| Baseline | 35.7% |
| With RAG | 47.5% |
| With an unspecified memory system | 58.2% |
| With Reflect Memory System | 61.3% |

The model, tool access, evaluation subset, and memory protocol for these scores are not established in the recording. Pankaj also cites similar trends on BigCodeBench and a benchmark named LifeLongTV in the transcript, without giving numerical results for them.

Outcome-based retrieval introduces its own dependencies:

- **Cold start:** Retrieval begins as pure semantic search until enough reviews accumulate.
- **Utility drift:** Usefulness can drift, and similar memories can recur in retrieval.
- **Review quality:** Noisy outcome labels make the utility signal noisy.
- **Reranking control:** A hyperparameter, λ, is associated with credit and reranking.

Pankaj says Reflect reduces most of these problems, but gives no detailed mitigation procedure and identifies cold start as the remaining constraint. The system cannot use a history of reviewed outcomes before that history exists.

### A product search with no memories

[9:44](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=584s)

The live demonstration uses an agent that searches a product catalog stored in a SQL database. Pankaj asks it to find a gaming mouse. The run retrieves zero memories and responds that it cannot find one. She opens the dashboard to inspect what happened. This is the cold-start situation described earlier: there is no retrieved experience to guide a different response when the search fails.

During the dashboard transition, Pankaj briefly returns to evaluation, naming AgentBench and again describing reasoning and planning across tool-using workflows. The demonstration then returns to the unsuccessful catalog search, where the next step is to supply a review of the run.

### A failed search becomes guidance to broaden the query

[11:56](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=716s)

The feedback changes what counts as an acceptable result. A wireless mouse exists in the database, so Pankaj marks the run as failed and supplies the instruction “Give any relevant.” The review is attached to the input, response, trajectory, and tool call; the initial product-search result is empty. The lesson is therefore more specific than a negative rating: when an exact product search fails, a relevant alternative may satisfy the request.

A later run finds a Wireless Mouse. The displayed trace contains the task “find me a gaming mouse” and a product response, with review status still pending. In the narration, however, Pankaj describes the later input as “mouse.” The visible result shows a relevant product being returned after feedback, but the query discrepancy and broadened acceptance criterion prevent treating this as a controlled, identical-query comparison.

![Reflect trace detail shows the task “find me a gaming mouse” and a response listing a Wireless Mouse with product details. Review status is pending.](https://ai.engineer/api/talk-essay-media/3ad3bdf567491367a83c1fb6efe15d7c810af42e7b7f09db36bd2615e611bae2)

The later product lookup returns a Wireless Mouse.

Pankaj emphasizes the tool trajectory: the earlier run made a product-search call and received an empty result; the later run still uses the product-search tool but obtains a relevant answer. The tool itself need not be replaced for retrieved guidance to change how the agent uses it.

For a catalog represented by a `products` table with a `name` column, the SQL distinction can be expressed as a specific search followed, if empty, by a broader one:

```sql
SELECT name
FROM products
WHERE LOWER(name) LIKE '%gaming mouse%';

-- If the first search is empty and alternatives are acceptable:
SELECT name
FROM products
WHERE LOWER(name) LIKE '%mouse%';
```

This illustrates the search lesson, rather than reproducing the dashboard’s exact SQL. A name such as `Wireless Mouse` matches the broader query. Returning it as a related product preserves the distinction between finding an alternative and establishing that it meets the original gaming requirement.

### Keep the lesson, then maintain the skill

[14:02](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=842s)

The memory card makes the resulting guidance visible: it records the failed gaming-mouse search, identifies the mistake, describes a corrective action, associates the lesson with the search tool, and recommends broadening a query when no results appear. This is the reasoning-memory pattern from the refund example, now attached to a concrete tool failure. Memories are retrieved according to utility, and that utility is reranked as evidence of usefulness accumulates.

![Reflect memory card for “find me a gaming mouse” shows a failure summary, key mistake, correct action, applicable search tool, and guidance to broaden a query when no results appear.](https://ai.engineer/api/talk-essay-media/4bad3d82341990ff2228a65d005c1055b98d0b24e0765b008e1fe311a814a235)

A reviewed failure becomes guidance to broaden unsuccessful product searches.

The final step is to consolidate reviewed experience into a skill. Pankaj describes memory scores changing with reviews, then gives about five reviews as an example of when findings could be baked into a skill. Earlier she used about ten memories; these are illustrative counts, not an established fixed threshold. The later Project Skill view still shows an unmet review threshold, so the demonstration explains the path to publication without showing a completed skill update.

**The durable change is guidance the agent can consult without rewriting its system prompt.** Individual reviewed runs produce memories; usefulness affects which memories reach future tasks; accumulated lessons can become maintained skills. That is how an outcome can cross the retrieval boundary and influence what the agent does next.

## Resources

- [Reflect documentation](https://docs.starlight-search.com/) — Current documentation explains how traces and reviews create memories, update utility, and influence retrieval, with a worked refund example.
- [Reflect integration skills](https://github.com/StarlightSearch/reflect-skills) — An installable coding-agent skill for adding Reflect memory to Python agents across several frameworks.
- [Your Agent’s Memory Is Broken. Here’s Why](https://www.linkedin.com/posts/harsha340_your-agents-memory-is-broken-heres-why-activity-7425652489246527488-IUDN) — Ram Sriharsha's post argues that agent retrieval needs to learn, beyond improving lookup speed and cost.
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://arxiv.org/abs/2210.03629) — The original paper introducing interleaved reasoning and actions for language-model agents.
- [Humanity's Last Exam](https://arxiv.org/abs/2501.14249) — The original benchmark paper; its arXiv record includes subsequent revisions.

## Slides

### What is an agent?

A ReAct agent loops between an LLM and execution, calling tools, retrieval or web search until the task is complete. The speaker points out that this basic architecture lacks a way to learn from outcomes.

[0:57](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=57s) [An agent definition above a diagram showing a user, an LLM, execution, tools and retrieval, and a feedback loop.](https://assets.ai.engineer/sites/aiecode2025/sha256/4fa4d940c1a5fc7155aa46a856504480b7b7c5dda70ecc36a017a5eca59a765e.webp)

What is an agent?

An agent is an LLM with agency: it reasons, invokes TOOLS to interact with the world, RETRIEVES from memory, and loops until the TASK is complete.

React Agent
LLM → Execute → Tools/ Retrieval/ websearch

### Agents keep failing at the same tasks

The speaker identifies learning from outcomes as missing from the basic ReAct architecture. The slide cites 85% of AI projects failing in production and fewer than 20% of AI pilots scaling to production within 18 months.

[1:22](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=82s) [A slide states that agents keep failing at the same tasks and cites Gartner and McKinsey statistics about AI projects and pilots.](https://assets.ai.engineer/sites/aiecode2025/sha256/b3887105bcc2e8c4d72dc0714234f77b4e0c9642d320400f8914d9d72420d2f1.webp)

Agents keep failing at the same tasks.

Gartner's 2025 AI deployment survey found that 85% of AI projects fail in production. McKinsey's 2025 State of AI report found that fewer than 20% of AI pilots scale to production within 18 months.

### Retrieval problems

The speaker points to static retrieval and context stuffing as problems, arguing that agents need to learn from outcomes instead of repeating the same failures.

[1:32](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=92s) [Two bullets read: retrieval is static and context stuffing.](https://assets.ai.engineer/sites/aiecode2025/sha256/ba947e744a4abe67a4382db368b4038101926575605f5c0ab9c98b04c273f4a3.webp)

➤ Retrieval is Static
➤ Context Stuffing

### Making retrieval learn

Ram Sriharsha argues that the industry made wrong answers faster and cheaper while forgetting to make retrieval learn. He also calls context stuffing a money pit.

[2:08](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=128s) [A post argues that agent memory is costly and retrieval should learn from outcomes.](https://assets.ai.engineer/sites/aiecode2025/sha256/a7aa12404bda5201cfa59fa71c83c7a9208e22d86ad9adb81bbfa4bb6da83ef6.webp)

Ram Sriharsha · 1st
Ex-CTO @ Pinecone | Researching what happens when you stop scaling and [illegible]
2mo

If you’re building AI agents today, I have a confession: your agent’s memory is probably broken and you are probably paying too much.

Why is the world’s most advanced coding agent (Anthropic’s Claude Code) using grep, a tool from 1973 instead of vector search?

Because we’ve been optimizing for the wrong things. We made the wrong answers faster and cheaper, but we forgot to make retrieval learn.

I wrote about why the industry overcorrected, why “context stuffing” is a money pit, and what actually comes next for agentic memory.

Interface text:
agentRTX

### Problems with agent memory

The speaker identifies static retrieval and context stuffing as agent-memory problems, and says agents lack a link from evaluation outcomes to action.

[2:14](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=134s) [Three points: static retrieval, context stuffing, and agents that are not outcome-informed.](https://assets.ai.engineer/sites/aiecode2025/sha256/a9663249ece00f9227a976e877a76b0c9c57396107a4cfecfd06c694afe654e5.webp)

• Retrieval is Static
• Context Stuffing
• Agents are not outcome-informed.

### The gap

Eval results stay in dashboards, leaving agents without reasons why past runs passed or failed. The proposed layer turns traces and eval outcomes into retrievable guidance for future runs.

[2:56](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=176s) [The slide says eval signals remain in dashboards and proposes converting traces and eval outcomes into retrievable guidance for future runs.](https://assets.ai.engineer/sites/aiecode2025/sha256/ef3dfb88b00b6c77c8dc43266bae8cecc4c2818964fb411ed9a5910a09fa5dcb.webp)

The GAP

It has no access to why yesterday's runs passed or failed. The eval signal dies in a dashboard.

This is the missing layer: a system that consumes traces, absorbs eval outcomes, and converts both into retrievable guidance for future runs.

### The manual improvement tax

The speaker describes a manual improvement tax: engineers review evals and observability, then change prompts, models, the tool-call harness, or fine-tuning.

[3:19](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=199s) [Manual improvement steps include rewriting prompts, upgrading models, restructuring the tool-call harness, and fine-tuning custom models.](https://assets.ai.engineer/sites/aiecode2025/sha256/33d5b18a39350f10de68a0740b4273fdebf23fa6db5f93f8b77ab46e3ec71b0e.webp)

The Manual Improvement TAX

Rewrite Prompts and redeploy
Upgrade to expensive models
Restructure tool-call harness
Fine-tune custom models

### Why are current memories failing?

The speaker argues that memory has not solved the manual improvement burden, then describes current systems as storing user preferences, profiles and conversation history.

[3:34](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=214s) [A slide asks why current memories are failing.](https://assets.ai.engineer/sites/aiecode2025/sha256/287e7b6bcf984c4eb60de8e9b0fd1c5ad7e35b64ca431b23c9ec0b17fe57212d.webp)

Why are current memories failing?

### Utility Score

The utility score weights semantic similarity to the current task by whether memories helped or hurt past outcomes, making evaluation results a signal in retrieval ranking.

[5:12](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=312s) [A slide explains retrieval using semantic similarity weighted by whether memories historically helped or hurt.](https://assets.ai.engineer/sites/aiecode2025/sha256/d5dbcbb3fa154b8456ea369e09e1f7457b4a4690d0cda5d8f3fb2143d499bf6e.webp)

Introduces Utility Score.

You do not retrieve by keyword.
You retrieve by semantic similarity to the current task, weighted by whether those [illegible]mories have historically helped or hurt. The eval outcome becomes a first-class signal in the retrieval ranking, not just a post-mortem footnote.

### Memory as Reasoning

The slide contrasts static preferences with reasoning reranked for usefulness. In the refund example, a bot can check settlement first to avoid paying twice; context is updated using task history.

[5:38](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=338s) [A diagram contrasts static user preference facts with task-specific reasoning that is reranked for usefulness and informed by history.](https://assets.ai.engineer/sites/aiecode2025/sha256/5b4dda98582641f8c1ed16716f777aa5fbb28014d007a1f0dd3537dc8649b014.webp)

Memory as Reasoning

facts
User preferences

Static,
No Context
No history

Reasoning
“Check settlement before Refund”

Reranked based on usefulness
Context is updated based on task
learned from history

### Limitations

During cold start, search is purely semantic until enough reviews accumulate. Noisy review labels can make utility noisy, and lambda is a hyperparameter associated with credit and re-ranking.

[8:49](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=529s) [The slide lists cold start, utility drift, review quality, and lambda.](https://assets.ai.engineer/sites/aiecode2025/sha256/388f0c16d29c706d04cbb711a1de3954583d98b2c2e660dab9a9bca0306cdcee.webp)

Limitations

Cold Start
Utility Drift
Review quality
Lambda

### Product search and human review demo

The presenter asks the SQL product agent for a gaming mouse; it finds none. The dashboard then shows the request as a pending trace alongside earlier searches and their review statuses.

[10:13](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=613s) [A terminal shows a product search prompt beneath code describing a deferred human review.](https://assets.ai.engineer/sites/aiecode2025/sha256/7bc9b7dbd595997521e37f8909caff5b23e351f71200b98ee3ceadffe163a0ed.webp)

    initialize_database,
    list_products,
    search_products_by_name,
)

# Integration choices captured from the user:
# - Framework: OpenAI Agents SDK (same as demo.py).
# - Agent shape: product-sql-agent. Tools query a local SQLite database.
# - Review source: HUMAN (deferred). No inline judge call. result=None on
#   set_output; reviewer grades the trace later via dashboard or
#   review_trace.
# - Pass/fail definition is left to the human grader; document yours in
#   your team's review rubric, not in this file.

Prompt (e.g. 'Get product 2' or 'price of "Wireless Mouse"'): Find me a gaming mouse
REFLECT_PROJECT_ID not set; using default 'reflect-demo'. Set it explicitly for multi-tenant use.

Interface text:
demo_human_review.py
Problems  Output  Debug Console  Terminal  Ports  Postman Console
[illegible]$ python demo_human_review.py

[10:40](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=640s) [A dashboard lists agent traces and their review statuses, including a gaming mouse request.](https://assets.ai.engineer/sites/aiecode2025/sha256/d97d5471bdc7ac0f7aad7677b12d76a0d56dd327082c612505f35ffa2fa9dfdb.webp)

Traces - reflect-demo
Runs, outputs, and review status.

TIMESTAMP  NAME  STATUS  OUTPUT
[illegible]  Find me a gaming mouse  PENDING  I couldn't find any gaming mouse in the product catalog. If you have a specific produc...
[illegible]  what are the composites used to design flight parts?  PENDING  I searched for composites named "epoxy" and "[illegible]" which are often used for designing...
[illegible]  What are the composites I can use for designing flight parts?  FAIL  There are no products specifically listed as composites for designing flight parts in the...
[illegible]  SPSG  PENDING  Could you please specify what product information you are looking for? You can...
[illegible]  get product 1  PENDING  Product 1 details: Name: Mechanical Keyboard Description: Compact 75% keyboard with...

Showing 1-5 of 5

Interface text:
reflect
PROJECT
reflect-demo
Overview
Projects
Traces
Reviews
Memories
API Keys
Billing & Usage
Select page
No traces selected
All
10 / page
Open
Prev  Page 1 / 1  Next

### Reviewing a trace

The trace pairs a gaming mouse request with the agent’s unsuccessful catalog search and offers pass/fail review controls.

[10:44](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=644s) [An open trace shows a gaming mouse request, the agent response and pass or fail controls.](https://assets.ai.engineer/sites/aiecode2025/sha256/55f0eb0c8d2567b684182095fe10337505dd987d58f00c6bb404f6224b0ceaeb.webp)

TRACE  PENDING
Find me a gaming mouse
Model: gpt-4.1-mini
Prev  Next

Mark pass  Mark fail

INPUT
Task
Find me a gaming mouse

OUTPUT
Response
I couldn't find any gaming mouse in the product catalog. If you have a specific product name or ID in mind, please let me know!

Trajectory
Messages and tool calls.
Show

Interface text:
reflect
PROJECT
reflect-demo
Overview
Projects
Traces
Reviews
Memories
API Keys
Billing & Usage

Traces - reflect-demo
Runs, outputs, and review status.
Select page
No traces selected
TIMESTAMP  NAME
Find me a gaming mouse
[illegible]
get product 1
Showing 1-5 of 5

### AgentBench comparison

AgentRTX leads both displayed comparisons, scoring 61.3% in one and 98 in the other. The slide leaves the two comparisons unnamed.

[11:07](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=667s) [An AgentBench slide compares AgentRTX with other approaches using two sets of scores.](https://assets.ai.engineer/sites/aiecode2025/sha256/089e3b96641fd3fdca901309f7ff630771c6fd669343b2515422e73f13c71db5.webp)

4. Agentbench

tests a model's ability to reason, plan, and use tools over extended, multi-step workflows, rather than just measuring static question-answering accuracy

AgentRTX 61.3
Mem0 58
RAG 47
Baseline 35.7

AgentRTX 98
DSPy 82
Baseline 57

Interface text:
File
Resize
Editing
Ask Canva
Edit
5.0s
Animate
Position
Present
Design
Elements
Text
Brand
Uploads
Tools
Projects
Apps

### Reviewing a failed mouse search

The presenter marks the failed gaming mouse search and submits feedback to “Give any relevant product.” She says a wireless mouse is in the database.

[12:02](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=722s) [A review dashboard shows a failed gaming mouse search and feedback requesting a relevant product.](https://assets.ai.engineer/sites/aiecode2025/sha256/c70d739fb56efbdfab1d6d99a9337b42899332b20bbad0d97337a097b899078a.webp)

TRACE  FAIL
Find me a gaming mouse
Model: gpt-4.1-mini

Mark pass  Mark fail
Give any relevant product.
Submitting...  Cancel

INPUT
Task
Find me a gaming mouse

OUTPUT
Response
I couldn't find any gaming mouse in the product catalog. If you have a specific product name or ID in mind, please let me know!

Trajectory
Messages and tool calls

Interface text:
reflect
PROJECT
reflect-demo
Overview
Projects
Traces
Reviews
Memories
API Keys
Billing & Usage
Traces - reflect-demo
Runs, outputs, and review status
Select page
No traces selected
TIMESTAMP  NAME
Find me a gaming mouse
[illegible]
Showing 5 of 5

### Inspecting the failed search trajectory

The trace shows search_products returning an empty product list. The assistant then says it could not find a gaming mouse in the catalog.

[12:27](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=747s) [The review dashboard shows a product search tool call, an empty product list, and the resulting response.](https://assets.ai.engineer/sites/aiecode2025/sha256/4cd4062715e6c4fe0de9f9fada3eca6fc944edfcd3f3ce84da3c5522ed5aadc4.webp)

TRACE  FAIL
Find me a gaming mouse
Model: gpt-4.1-mini
Prev  Next

Assistant
MESSAGE 2

TOOL CALLS
search_products
query
limit

Tool
MESSAGE 3
{"products": []}

Assistant
MESSAGE 4
I couldn't find any gaming mouse in the product catalog. If you have a specific product name or ID in mind, please let me know!

Interface text:
reflect
PROJECT
reflect-demo
Overview
Projects
Traces
Reviews
Memories
API Keys
Billing & Usage
Traces · reflect-demo
Runs, outputs, and review status.
Select page
No traces selected
TIMESTAMP
NAME
Find me a gaming mouse
[illegible]

### Reviewing the product response

For a gaming mouse request, the trace returns a wireless mouse and offers to search again if it doesn’t fit. The speaker contrasts this with an earlier search that returned no products.

[13:16](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=796s) [A trace panel shows a gaming mouse request and a response describing a wireless mouse.](https://assets.ai.engineer/sites/aiecode2025/sha256/cd851d0a8211a18fd50792d1ad01bc9f272d5503c90143ce51ae3ef5a8122571.webp)

find me a gaming mouse
Model: gpt-4.1-mini

INPUT
Task
find me a gaming mouse

OUTPUT
Response
I found a product related to a mouse:
- Name: Wireless Mouse
- Description: Ergonomic mouse with silent click buttons.
- Price: $39.50
- Stock available: 85 units
- Review score: 4.5

If you are specifically looking for a gaming mouse and this doesn't fit your need, please let me know, and I can try searching with other terms or explore further options.

Trajectory
Messages and tool calls.

Interface text:
reflect
PROJECT
reflect-demo
Overview
Projects
Traces
Reviews
Memories
API Keys
Billing & Usage
Traces · reflect-demo
Runs, outputs, and review status.
Select page
No traces selected
TIMESTAMP
NAME
[illegible]
TRACE
PENDING
Prev
Next
Mark pass
Mark fail
Hide

### Failed product search trajectory

The earlier trace calls search_products once, receives an empty products list, and tells the user it could not find a gaming mouse.

[13:40](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=820s) [A failed trace shows a product search returning an empty list and an assistant response saying no gaming mouse was found.](https://assets.ai.engineer/sites/aiecode2025/sha256/27830c8570a3a8c8a2b912a3af9cb843c500b5545cfb32b21c4d900145eb643a.webp)

TRACE  FAIL
find me a gaming mouse
Model: gpt-4.1-mini
Prev  Next

Assistant
MESSAGE 2
TOOL CALLS
search_products
query
limit

Tool
MESSAGE 3
{"products": []}

Assistant
MESSAGE 4
I couldn't find any product listed as a "gaming mouse" in the current catalogue. If you want, you can specify another product or provide more details.

Interface text:
reflect
PROJECT
reflect-demo
Overview
Projects
Traces
Reviews
Memories
API Keys
Billing & Usage
Traces - reflect-demo
Runs, outputs, and review status.
Select page
No traces selected
TIMESTAMP
NAME
find me a gaming mouse
[illegible]
Showing 1-7 of 7

### Product search returns a result

The search_products call returns a Wireless Mouse record. The speaker says the changed production trajectory receives an answer fed in as feedback.

[13:49](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=829s) [A trace shows a product search tool call and a returned wireless mouse record.](https://assets.ai.engineer/sites/aiecode2025/sha256/16df2a557df575b9a22e64c87a3ea1a6ec2c7c0136daf4555c26b19a04126d68.webp)

TRACE  PENDING
find me a gaming mouse
Model: gpt-4.1-mini
Prev  Next

Assistant
MESSAGE 2
TOOL CALLS
search_products
query
limit

Tool
MESSAGE 3
{"products": [{"id": 2, "name": "Wireless Mouse", "description": "Ergonomic mouse with silent click buttons.", "price": 39.5, "stock": 85, "review": 4.3}]}

Assistant
MESSAGE 4
I found a product related to a mouse:

Interface text:
reflect
PROJECT
reflect-demo
Overview
Projects
Traces
Reviews
Memories
API Keys
Billing & Usage
Traces - reflect-demo
Runs, outputs, and review status.

### Response after the product search

The search returns a wireless mouse, and the assistant offers to search again if it does not meet the request for a gaming mouse.

[13:56](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=836s) [A trace shows the returned wireless mouse record followed by the assistant's product response.](https://assets.ai.engineer/sites/aiecode2025/sha256/1bf8e410c9afd4706aa31e51888e593e86661a02f7e61622a97ed8821aaf8729.webp)

find me a gaming mouse
Model: gpt-4.1-mini

Tool
MESSAGE 3
{"products": [{"id": 2, "name": "Wireless Mouse", "description": "Ergonomic mouse with silent click buttons.", "price": 39.5, "stock": 85, "review": 4.3}]}

Assistant
MESSAGE 4
I found a product related to a mouse:
• Name: Wireless Mouse
• Description: Ergonomic mouse with silent click buttons.
• Price: $39.50
• Stock available: 85 units
• Review score: 4.3

If you are specifically looking for a gaming mouse and this doesn't fit your need, please let me know, and I can try searching with other terms or explore further options.

Interface text:
reflect
PROJECT
reflect-demo
Overview
Projects
Traces
Reviews
Memories
API Keys
Billing & Usage
Traces - reflect-demo
Runs, outputs, and review status.
TRACE
PENDING
Prev
Next
limit

## Full transcript

[0:01](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=1s) Hey, everyone. I'm Sonam. I'm the CEO and co-founder of Starlight Search, and today my talk is, user signals die at retrieval boundary. So we'll look into, uh, what are agents, essentially why agent fails, what is, uh, the cause of fails in retrieval particularly, and how to make actually signals cross the retrieval boundary, and how to make your

[0:25](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=25s) agent basically outcome-aware. So, uh, let's get started. So what is an agent? An agent is an LLM that has agency to reason, invoke tools, uh, interact with the real world, retrieve, uh, the memory to complete the task.

[0:43](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=43s) One major loop here is missing is, uh, learning. It should also learn from what worked and what it didn't work. Suppose, uh, if I have to, uh,

[0:54](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=54s) explain what is agent, I can explain with ReAct agent. So if I have to explain, uh, what agent is, I'll explain it with ReAct agent. So basically, user prompt the agent, uh, uh, execute it in a loop, uh, call tool, retrieval, search, and then pause when the task is complete.

[1:14](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=74s) This is very basic ReAct architecture. One thing that is missing is how to make agent learn from the outcome.

[1:24](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=84s) So agent keeps failing at the same task. Gartner reports that eighty-five percent of AI projects fail in production, and so does McKinsey's twenty-twenty-five report. The problem came out to be most of the time is that retrieval is static.

[1:40](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=100s) Seventy-three percent of, uh, RAG pipeline fails because of retrieval, not generation, and context stuffing. So a recent, uh, uh, post from Ram Sriharsha, the ex-CTO of Pinecone said, uh, "We have been optimizing for the wrong thing.

[1:59](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=119s) You are paying a lot for your agent's memory. It is probably broken, and we have been optimizing for the wrong things. We made wrong answers appear faster and cheaper that we forgot to make retrieval learn."

[2:12](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=132s) So why this is-- does, does, does it matter?

[2:16](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=136s) Again, uh, the third problem is agents are not outcome-informed. So there's a missing layer between evals and action. Your observability has all the traces, uh, all the stack that capture-- eh-uh, observability is the stack that capture every tool call, every LLM completion, every exceptions.

[2:37](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=157s) Your eval suite judges whether the final output was correct or wrong, basically pass or fail. But these evals are not reflected in agent, uh, context, SKILL.md files, or agent action in any ways.

[2:57](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=177s) Right? So the agent doesn't have any access to why yesterday's runs, uh, runs passed or failed. Eval signal dies in the dashboard. This is a missing layer, a system that consume traces, absorb eval, and convert both into retrieval guidance for future runs.

[3:17](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=197s) So there's a manual improvement tax. An engineer actually has to sit and see if the eval and observability perform well, rewrite the prompt, redeploy it, upgrade-- either upgrade to expensive model, restructure tool or harness, or fine-tune the custom models.

[3:36](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=216s) Why are current memories failing? Why-- Memories was designed to actually address this, but, uh, uh, it's not.

[3:47](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=227s) So let's see, uh, eh, what we have in-- as a current system, uh, and current memory is that they basically store user preferences, profile, conversational history, or long-lived personal-- personalization.

[4:05](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=245s) So chat experience is not self-improving learning systems for production. If you see, uh, the already existing approach in the, uh, market, there's, uh, uh, LangChain, uh, there's Mem0, which does extracted fact, preferences.

[4:21](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=261s) A user's retrieval signal is, uh, embedding similarity. Does it learn from outcome? No. Uh, so we have come up with something called utility score, which is a similarity we-weighted by how useful it is for the agent to execute the task.

[4:41](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=281s) It has actually the history of past traces and past outcomes.

[4:47](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=287s) So we came up with Agent AgX, and that is agents with runtime experience. It's a runtime, uh, layer that let production agent improve from experiences without pre-training, fine-tuning, or manual prompt engineering.

[5:01](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=301s) Ha-- It's a bit different from compile time like DSPy because, uh, you bake in all the lessons in the prompt. Here, it's actually improving, uh, while it is executing the task.

[5:14](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=314s) So let's, uh, again introduce this, uh, utility score. So you do not retrieve by keyword, you retrieve by semantic similarity to the current task, weighted by whether those memories have historically helped or hurt the execution or the outcome.

[5:32](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=332s) The eval outcome becomes a first-class signal in the retrieval re-ranking and not just for feedback.

[5:41](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=341s) One of the key things is it treat memory as reasoning, not as facts, statics fact with no context and no history, but reasoning. Like suppose, uh, if, uh, if there's a-- if there is a customer support bot looking for refund, it will not only say, "Hey, user prefers," uh, uh, uh-

[6:06](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=366s) Dark theme or user prefers, uh, be-- to be called by a shorter name. It's actually reason about a query, like, uh, if some-someone asks for refund, you should check the settlement, uh, before refunding it so that the, the, the customer doesn't get paid, uh, refund twice.

[6:26](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=386s) So re-rank based on usefulness. Context is updated based on tasks. So this is a very big thing, uh, because most of the agents fail with context stuffing, and this has been brought up in the past and learned from history and reasoning, right?

[6:42](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=402s) Talking about benchmarks. So, uh, we have benchmarked, uh, our memory system Reflect with-- on τ-bench, which essentially measure if agents have followed a policy well or not. So we have seen, uh, the, uh, uh, the performance improve from sixty-six to seventy-six percent without baking in, uh, skills.

[7:05](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=425s) And with skills, uh, Reflect performs at eighty. So, uh, once there are, uh, enough memory, like ten, uh, memories, uh, what we do is we bake in the, uh, reasoning and the understanding into skills so that your o-agent always remains updated.

[7:24](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=444s) What happens most of the time, we have seen, suppose you have a product SQL agent, and there's a system-- there's a column in system prompt. Even though that column is no useful anymore, it remains as the system prompt.

[7:36](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=456s) So there's no system right now that can update that, "Hey, there's no column, uh, right now called this, so maybe probably you shouldn't entertain it in the future." And this is possible with skills that, uh, because agents always uses, uh, calls that skill, uh, updated skill all the time.

[7:58](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=478s) And the similar behavior has been s- uh, seen in GPT-5.4. Uh, we have also benchmarked it on agentic tasks, which essentially test a model's ability to reason, plan, and use tools over extended multi-step workflows rather than measuring a static Q&A.

[8:16](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=496s) So you can see here, uh, with the, suppose the Humanity's Last Exam, uh, with RAG, you get forty-seven point five. If it is starting from the baseline thirty-five point seven, uh, with the, uh, the memory system, it gets to fifty-eight point two, but with the Reflect Memory System, uh, it gets to sixty-one point three percent.

[8:38](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=518s) So this k- uh, this kind of trend is shown in another, uh, agentic benchmarks, uh, as well, like, uh, BigCodeBench, LifeLongTV, et cetera. So of course, there are limitations to this, uh, uh, to this approach.

[8:56](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=536s) First of all, there's a cold start. So in the beginning, it's pure semantic search until, uh, enough reviews have been accumulated. Um, there's a utility l-drift. Maybe sometimes, uh, similar memories could come up.

[9:12](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=552s) There are a lot of problems that could come at scale with this experience, but we have combat most of them. Uh, there's a u-review quality, so noisy labels can make the utility noisy as well.

[9:25](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=565s) And there is a hyperparameter called lambda that is associated with credit and re-ranking. So we have bu-built Reflect in such a way that most of these problems and most of these, uh, limits are now reduced, uh, except for cold start, which we cannot do much about it.

[9:44](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=584s) Uh, let's now get into the demo. So let's check this demo.

[9:57](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=597s) Um, so it's a-- basically a product SQL demo. Uh, I'll give-- ask it to search some product in a SQL database, and let's see if it is able to find it out.

[10:25](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=625s) So I gave it, "Find me a gaming mouse."

[10:30](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=630s) Zero memories retrieved. I couldn't find a gaming mouse. Uh, okay. So maybe let's just go and see what's happening in the dashboard.

[10:43](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=643s) Okay, it came out that... So coming on another benchmark that with the AgentBench, which ess-essentially measures if agent has actually done the right reasoning, planning, and, uh, have followed.

[11:09](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=669s) Which essentially test a model's ability to reason, plan, and use tools over extended multi-step workflows rather than measuring a static Q&A. So you can see here, uh, with the, suppose the Humanity's Last Exam, uh, with RAG, you get forty-seven point five.

[11:28](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=688s) If it is starting from the baseline thirty-five point seven, uh, with the, uh, the memory system, it gets to fifty-eight point two, but with the Reflect Memory System, uh, it gets to sixty-one point three percent.

[11:41](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=701s) So this k- uh, this kind of trend is shown in another, uh, agentic benchmarks, uh, as well, like, uh, BigCodeBench, LifeLongTV, et cetera.

[11:56](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=716s) Wait, let's check. Uh, it couldn't find-- I couldn't find any gaming mouse in the product catalog. So suppose I want to mark it fail and s-tell it

[12:07](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=727s) Give any relevant. Because there is, uh, there is a wireless, uh, mouse in the database, and I have submitted the failure. Uh, this was the input, this was the response.

[12:25](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=745s) I couldn't find any, and this was a trajectory and tool call it took. So you can see the product is empty

[12:33](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=753s) with this product or with this query. Let's, uh, let's check again what happens now.

[12:48](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=768s) Let's see now, now what happens. ID, uh...

[13:05](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=785s) So it searched a wireless mouse. Uh, let's see what happens in the dashboard now.

[13:16](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=796s) Finally, I gave it mouse was the input. The response was, "I found a product related to mouse," as we asked it to find any kind of relevant. Let-- But the most important thing is how the tool call evolved.

[13:30](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=810s) You can see that previously, uh, the tool calls, uh, or the trajectory, uh, used to look just with one search, that is search products, uh, it called, uh, and the product was empty.

[13:45](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=825s) Now, the trajectory has changed in production, and, uh, it found something in the product, and it- it's still calling search product tool call, but it is getting some answer that we fed in as a feedback.

[14:02](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=842s) So, uh, that's the demo, and the most important thing that is happening over here is it's forming memories, which is retrieved based on the utility score. That is the score which basically keeps improving-- uh, keeps re-ranking itself on the basis of how useful that memory was.

[14:26](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=866s) So, um... You can see from these traces, past traces which memory. Uh, so this memory scores keep changing, and after a certain while, after suppose a five reviews need, you can bake in these findings or these, uh, new

[14:50](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=890s) updates in a skill without, uh, so without changing any system prompt, you can actually update certain things that agent, uh, uh, draws, uh, a lot from.

[15:05](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=905s) So, uh, uh, you can update the skill, which is very cool. So, uh, I hope everything was...

[15:15](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=915s) I hope you get to try, uh, this new, uh, feature, this new, uh, runtime experience that we have built.

[15:26](https://www.youtube.com/watch?v=Jx4ZFEAq6bY&t=926s) If you want more details, you can visit our website, and, uh, you can also contact me if at [REDACTED:email_address] or visit me. Thank you.
