# An Agent That Fixes Its Own Mistakes - Saloni Gandhi & Ziwei Guo, LinkedIn

Ziwei Guo, LinkedIn · Saloni Gandhi, LinkedIn

[Talk page](https://ai.engineer/talks/yH7_SIIgI-Y-agent-that-fixes-its-own-mistakes) · [Video](https://www.youtube.com/watch?v=yH7_SIIgI-Y)

## An Agent That Fixes Its Own Mistakes

LinkedIn’s first GEPA-optimized prompt for its user-understanding agent reached production after measurement and human review. A separate review of the evaluators exposed a subtler error. The system could blame the scoring rubric when the judge had misapplied it.

### Which part of the agent should a production failure change?

[0:01](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1s)

A production trace records what LinkedIn’s career agent did on a real request. When something goes wrong, the engineers maintaining the agent still have to decide what to change. Ziwei Guo and Saloni Gandhi built a system around one part of the career experience, the agent that infers a member’s goals and preferences. Their starting point was an evaluator that could inspect the traces and identify failures that mattered to members.

The team calls the process recursive self-improvement, or RSI. The agent logs its output and the steps that produced it. An improvement engine uses those records to evaluate the output, investigate the cause of a failure and propose an edit. Once a validated edit enters production, the revised agent produces the traces for a later round.

Several parts of the agent harness, the software around the base model, can influence an answer. Planning logic chooses an execution path. Context and memory preserve session state. Tool interfaces give the agent access to APIs and datasets, and storage retains outputs and generation logs. The team chose prompt edits as the simplest changes to propose and test while keeping the base model fixed.

Guo describes a progression in what the optimizer might eventually be allowed to edit. It could change how the agent’s context is structured, design the workflow, then modify the harness code that runs it. The most ambitious step would let the optimizer revise its own code. The production work in this talk tackles the first step, changing the instructions.

### How an edit earns its place in the next run

[5:09](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=309s)

One iteration has seven steps. The first two produce evidence. Run the agent on a query, keeping its output and execution trace. Then have a judge evaluate the run and explain where the output succeeded or fell short.

Steps three and four turn a judgment into a proposed edit. A reflection agent investigates what could account for the score. Possibilities include a model error, a prompt that left something out, a tool problem or a mistake in evaluation. The diagnosis guides the proposal. This matters even when the only edit the optimizer can make is to a prompt. A low score by itself gives it little reason to change a particular instruction.

The last three steps test the proposal. The system evaluates candidates using the same judge and rubric, incorporates a promising edit into the prompt, then compares the revised agent with the previous version on a held-out set. That final comparison asks whether the change also helps on examples outside those used to choose it. A validated update becomes the baseline for the next iteration.

### From a member ID to an inferred job preference

[9:20](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=560s)

The user-understanding agent takes a member ID. It fetches profile data, search history, location and preferences, then synthesizes those signals into fields describing the member. One field records which job titles the person appears to prefer. Entries might include *network manager* or *account manager*. What would let a reviewer check that inference?

Alongside the titles, the agent supplies its confidence, citations to member evidence and an explanation. If *network manager* appears frequently in saved searches, a citation can point to the activity behind that inference. The explanation can distinguish the roles of two sources of evidence. A preference the member stated can establish the primary title, while saved searches can support a related title the member might also consider.

The first-level judge, L1, assesses the output across 15 task dimensions using product-defined rubrics. One member’s output received an overall score of 2.31. Underneath it, fields such as goal summary and preferred job title each get a score and notes explaining the verdict. On this scale, 1 means poor, 2 fair and 3 very good. The field-level notes give the next stage a specific claim to investigate when an otherwise reasonable output scores poorly.

### Counting recurring problems across different members

[13:57](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=837s)

Reflection starts by grouping failing rows from members’ traces according to the dimension that scored poorly. This first grouping just uses the existing scores. Within each group, a second-level judge, L2, examines each row and assigns a free-text diagnosis and a cause tag. A shared symptom, such as a low score for preferred titles, can have different causes across members.

Free text creates a counting problem. One diagnosis says the member’s profile data is missing. Another says the profile gives an incomplete picture of the member’s browsing history. In Guo’s example, both identify a problem at the same level of detail for the team’s purposes. The available profile evidence cannot support the agent’s conclusion. Merging these descriptions under a common issue label makes their combined frequency visible. Ranking the labels then requires no further model judgment.

Who uses the diagnoses after a run? Engineers get a scorecard showing which problems recur most often. L2 gets something different on a later run. The system saves prior diagnoses and supplies them as context when L2 investigates fresh failures.

Where a row’s diagnosis goes
The scorecard counts issues across members in this run. The return path carries prior diagnoses into a later run, when L2 has new rows to examine.
- Failing rows — Group traces by the dimension that scored poorly.
- L2 diagnosis — Assign each row a cause tag and a free-text explanation.
- Common issue labels — Merge descriptions of the same underlying problem.
- Scorecard for engineers — Rank recurring issues by frequency.
- Saved root causes — Keep diagnoses available for future investigations.
- Failing rows → L2 diagnosis (Pass each row for diagnosis)
- L2 diagnosis → Common issue labels (Collect near-duplicate diagnoses)
- Common issue labels → Scorecard for engineers (Count occurrences of each issue)
- Common issue labels → Saved root causes (Retain the identified causes)
- Saved root causes → L2 diagnosis (Supply prior diagnoses on a later run)

### The first GEPA-optimized prompt reaches production

[18:37](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1117s)

GEPA uses the judge’s evidence and the reflection results to propose changes to the agent’s instructions. It works over mini-batches of sampled traces. Candidate prompts advance to validation, where the team can compare how well they address the observed problems. A prompt that survives the held-out comparison can replace the previous production baseline.

The team used human annotations to improve both L1 and L2 and studied variation across agent runs and model judgments. Meanwhile, engineers packaged the seven steps into a workflow that could suggest edits weekly, every two weeks or monthly, depending on need. Scheduling proposals left the release decision with people. The first GEPA-optimized pull request shipped after quantitative measurement and human review.

Gandhi describes a second outcome from comparing the evaluators’ decisions with human reviews. Disagreements exposed recurring faults in both the scoring rules and the system that explained failures. That raised two separate questions. Could L1 recognize a poor answer? And when L1 flagged one, could L2 identify what had gone wrong?

![Two panels report that the first RSI-generated changes reached production after human review and quantitative measurement, and that human annotation helped improve the evaluation judge and rubric and reduce evaluation variance.](https://ai.engineer/api/talk-essay-media/a57e018dfd148b6cdba801daaa51a836f0436defdccddcf56e57d519f37c9ef3)

The first production changes were paired with a study of how the evaluator’s judgments compared with human review.

### Human reviewers agreed most often with a score of 3

[25:12](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1512s)

Where was L1 reliable enough to guide an edit? The team analyzed about 180–200 human reviews, grouping them first by the score L1 had assigned. Full agreement was roughly 55% for outputs scored 1, 57% for those scored 2 and 80% for those scored 3. The judge recognized strong work more reliably than it distinguished degrees of weak work. Those low scores were precisely the ones that sent examples into reflection. [Score agreement in the recording, 25:31](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1531s)

Grouping the reviews by field reveals a different source of disagreement. Preferred job titles had among the weakest agreement. For domain or area of expertise, agreement was about 47%, and for work type it was about 50%. The two breakdowns point to different questions for the rubric. What distinguishes a 1 from a 2? What evidence should count for this particular field?

The annotations put concrete cases behind those figures.

- **Preferred titles.** L1 could accept a plausible title even when the agent had overlooked an important signal of the member’s intent. A reviewer checking the input could spot a missing preference that a check of the returned titles alone would miss.
- **Fields of expertise.** In the team’s example, *software engineering* is a domain, *machine learning engineer* is a role and *machine learning* is a skill. The rubric blurred those categories. It also gave little guidance on how much credit to award when an answer mixed correct and incorrect entries.
- **Formatting and inference.** The judge sometimes penalized harmless differences in how a value was written. A harder disagreement arose when the agent inferred a preference from current employment and the member had never stated that preference explicitly. Reviewers needed a rule for when that inference was justified.

Of the 37 judgments reviewers identified as too harsh, 31 involved an output L1 had scored 1. That concentration made partial credit a practical priority. A rule for recognizing the correct parts of a mixed answer could keep that answer from being treated as a complete failure.

### When the scoring rule is wrong, and when the judge is wrong

[28:31](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1711s)

Take the work-type value *full-time*. Capitalization alone could cost the agent credit. The revised rubric accepted formatting variants with the same meaning. The team also added guidance for normalizing job titles, handling concurrent employment and assessing inferences about seniority and geography. It revised the scoring rules to distinguish a poor answer from a partially correct one.

After the rubric changes, reported L1 agreement with human reviewers rose from about 57% to 64%. Mean error also fell, though the scale of that measure was unspecified. The agreement gain measures a change in the evaluator. Engineers had made the product’s definition of a correct judgment more explicit.

Reviewing L2 exposed a different problem. Human reviewers agreed more often when L2 attributed a failure to the underlying model. Agreement was much lower when it blamed the rubric. The original list of cause tags gave L2 no way to say that L1 had made a mistake. Faced with a reasonable rule that L1 had applied incorrectly, L2 sometimes labeled the rule itself defective.

L2 also sometimes repeated L1’s rationale without checking it against the agent’s output. A single mistaken judgment could then look like a finding that two evaluators agreed on. If GEPA acted on that diagnosis, it could propose an edit to a component that had behaved correctly. Improving L2’s ability to identify the cause became a priority in its own right.

For decisions about individual edits, the team used majority voting to reduce the influence of any one model judgment and held-out evaluations to check whether gains carried over. Gandhi also reports an improvement involving the threshold for adopting an edit and the reported confidence interval, without specifying a value or direction of change for either measure.

![Bullets describe missed job-title preferences, inconsistent partial credit for expertise, formatting penalties and disputes over inferred preferences. Of 37 judgments reviewers marked too harsh, 31 received score 1. The slide calls for clearer rules for partial credit and inferred evidence.](https://ai.engineer/api/talk-essay-media/86dd45128618494ec7ee6fe23d80579a98e8c9a69cc8ed7118f55d2f0272e5a9)

The review breaks disagreements down by score and failure mode, including overly harsh judgments and gaps in the rubric.

### What another agent team would need

[33:03](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1983s)

At the scale of a team, adoption poses a different question. What fraction of its prompt changes can RSI reliably generate and validate? LinkedIn also wants to measure downstream response quality and the time engineers save maintaining an agent. These are proposed measures for broader use. A count of edits would rise even if the quality of the answers stayed flat.

Many teams already have an agent that answers questions, summarizes material or completes a workflow. Their ability to evaluate it varies. Some have bespoke tests, others inspect outputs by hand, and relatively few have a reliable evaluation they can run repeatedly. That gap limits what a shared optimizer could do for them. To propose a targeted fix, it needs a diagnosis. To investigate the cause, the diagnostic system needs a dependable way to identify a failure in the first place.

LinkedIn’s proposed shared infrastructure would give teams five capabilities.

- **Evaluation.** Reuse evaluation frameworks and metrics, with judges that teams can calibrate against human feedback.
- **Feedback collection.** Collect execution traces continuously so recurring failures can be detected and investigated.
- **Optimization.** Use reflection and prompt optimization to generate and test candidate changes in a repeatable way.
- **Validation.** Compare candidates on held-out examples and run regression tests for behavior an edit might damage.
- **Self-service adoption.** Package the components so a team with an existing agent can use them without building the whole pipeline.

Gandhi frames the longer-term goal in terms of a slope. The target agent infers member preferences. The improvement engine finds failures, investigates them and tests changes. If agent quality is a function of time, an individual release can raise quality at one point on the curve. The ambition for the engine is to increase the rate of improvement across releases. For that slope to mean anything, the team has to be able to trust that a measured gain reflects a better answer for the member.

## Key takeaways

- Checking the titles an agent returned tests plausibility. Checking those titles against the member’s intent signals can also reveal a preference the agent omitted.
- A diagnostic judge needs a way to identify an incorrect score and evidence against which to check it. LinkedIn’s reflection agent sometimes blamed a sound rubric or repeated the first judge’s error.
- LinkedIn compared proposed prompt edits under a common rubric, tested the selected change on held-out examples and kept people involved in the production decision. The first GEPA-optimized pull request passed measurement and human review before release.

## Slides

### Outcomes and Learnings

Saloni introduces a lesson from the team’s work on RSI reliability. The quality of the evaluator mattered as much as the quality of the optimization system.

[23:26](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1406s) [The words “Outcomes and Learnings” appear on two lines in large dark type at the upper left of a white slide.](https://assets.ai.engineer/sites/aiecode2025/sha256/2f46cd3d927d2e908eb9617aa1bbdc3f4f24f9cc17d0681a190a2cb2cdd61ce6.webp)

Outcomes and
Learnings

in
PROPRIETARY + CONFIDENTIAL
12

### RSI production deployment and evaluation quality

LinkedIn shipped its first RSI-generated changes after human review and quantitative measurement. The team studied where its judge disagreed with reviewers, refined the rubric and reduced evaluation variance.

[23:56](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1436s) [Two panels report that the first RSI-generated changes reached production after human review and quantitative measurement, and that human annotation helped improve the evaluation judge and rubric and reduce evaluation variance.](https://assets.ai.engineer/sites/aiecode2025/sha256/a57e018dfd148b6cdba801daaa51a836f0436defdccddcf56e57d519f37c9ef3.webp)

1
First RSI system changes landed in production 🚀🚀🔥
Successfully deployed initial system updates generated by RSI after rigorous human review and quantitative measurement.

2
Eval Judge and Rubric quality
• Dedicated substantial effort to human-annotating the MU eval judge
• Improved overall judge quality and significantly reduced evaluation variance
• Enhanced rubric robustness for more reliable automated evaluation

in
PROPRIETARY + CONFIDENTIAL
13

### MU Eval L1 judge findings and rubric recalibration

Of the 37 judgments reviewers flagged as too harsh, 31 gave a score of 1. The team proposed clearer partial-credit rules so the judge would reserve score 1 for substantially incorrect or unusable answers.

[26:38](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1598s) [Bullets describe missed job-title preferences, inconsistent partial credit for expertise, formatting penalties and disputes over inferred preferences. Of 37 judgments reviewers marked too harsh, 31 received score 1. The slide calls for clearer rules for partial credit and inferred evidence.](https://assets.ai.engineer/sites/aiecode2025/sha256/86dd45128618494ec7ee6fe23d80579a98e8c9a69cc8ed7118f55d2f0272e5a9.webp)

Human Review - MU Eval (L1) Judge findings

Summary of Key Patterns

• Preferred Job titles: The judge often checks whether returned titles are plausible but misses expected titles or intent signals.
• Primary field of Expertise: The distinction between a domain, function, and skill is unclear, and partial credit for mixed-quality lists is inconsistent in the rubric.
• Formatting & Inferences: The judge sometimes penalizes formatting differences too severely or inconsistently infers target preferences from current employment.
• Score Alignment & Rubric Recalibration: Scores 1 and 2 have similar full-judgment alignment, 55.7% and 57.5%, while score 3 reaches 80.4%. 31 of the 37 too-harsh annotations occur on score 1 decisions.
  ○ This suggests we calibrate the boundary between scores 1 and 2, with clearer partial-credit rules through rubric changes so that score 1 is reserved for outputs that are substantially incorrect or unusable.
• Recurring Disputes:
  ○ Key Areas: Employment status, Geo, expertise & rank: Human reviewers frequently accept reasonable inferences, whereas the L1 rubric strictly required explicit evidence.
  ○ Gap: Need aligned guidelines on implicit vs. explicit candidate signal handling.

in
PROPRIETARY + CONFIDENTIAL
15

### Human Review – RSI Reflection Findings

Reviewers agreed in 70% of the 72 cases where the reflection agent blamed the model, but only 27% of the 22 where it blamed the rubric. The agent had no way to flag an error by the L1 judge, so it sometimes blamed the rubric instead.

[29:50](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1790s) [Four charts compare L2 reflections with human judgments by assigned cause, dimension and stated confidence. Reviewers agreed with 70% of model attributions, 50% of context attributions and 27% of rubric attributions. Agreement was 68% in the middle confidence band and 64% in the highest.](https://assets.ai.engineer/sites/aiecode2025/sha256/f2975d1dbebdce1a1e1a4d187073a16974d6fdb85a6225341f183a65fdad0d98.webp)

Human Review - RSI Reflection findings

Score alignment pattern
• Humans agree mostly with model issues
• low agreement on other causes – L2 could cheat → blame “model”
• Confidence scores did not align with performance

Dimension alignment pattern
• Lowest: employment type 33%, workplace type 44%, career stage 50%

Why humans disagreed
• Wrong root cause 18: no way to blame L1 judge→ fell back to "rubric"
• Nothing failed 15: 13 were L1 too harsh
• L2 reused L1's note; L2 didn’t put validation efforts

Human agrees, by cause L2 named
Y-axis: %
Y-axis ticks: 0, 50, 100
model (n=72): 70%
context (n=4): 50%
rubric (n=22): 27%

Cause: L2 vs human (n=35)
Rows: L2 said
Columns: Human said
         model  rubric  other
model       13       7      3
rubric       8       0      2
context      0       1      1

Human agrees, by dimension
X-axis ticks: 0, 20, 40, 60, 80, 100
Job nature: 33%
Work environment: 44%
Employment status: 50%
Career level: 50%
Desired locations: 57%
Field knowledge: 62%
Experience level: 62%
Preferred domains: 71%
Career goals: 80%
Sector type: 83%

Human agrees, by confidence L2 stated
Y-axis: %
Y-axis ticks: 0, 50, 100
<0.85 (n=16): 31%
0.85-0.95 (n=28): 68%
≥0.95 (n=54): 64%

in
PROPRIETARY + CONFIDENTIAL
17

### Quality Improvements

The team wanted the L1 judge to score more like a human reviewer and the L2 reflection to identify what went wrong. A mistaken diagnosis could send a prompt change to the wrong part of the system.

[31:26](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1886s) [Section divider headed “Quality Improvements” with the subtitle “RSI reflections and Judge.”](https://assets.ai.engineer/sites/aiecode2025/sha256/b4c9f79d9a0957a23e10aee75c6919a68a316b4d47b2fd45dcf5a492cb2daebd.webp)

Quality
Improvements

RSI reflections and Judge

in
PROPRIETARY + CONFIDENTIAL
18

### RSI Reflection Quality Improvements

Once L2 could flag an L1 judging error, it blamed the rubric in 8 of 300 verdicts, down from 37 of 300. Human agreement with those rubric diagnoses rose from 22% to 62%. Across all verdicts, agreement rose from 77% to 80%.

[31:30](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1890s) [A table lists changes to the L1 and L2 judges. The chart shows overall human agreement rising from 77% to 80%. For verdicts that attributed the issue to the rubric, agreement rose from 22% of 37 cases to 62% of 8 cases.](https://assets.ai.engineer/sites/aiecode2025/sha256/ab4ae22a4107af0fb593e84dbb9ec7c90ba33852c939f3723fd7a8991f9536a1.webp)

RSI Reflection Quality Improvements

Issue | Fix | Before → After
No "L1 judge was wrong" cause | Added "L1 judge wrong" cause | rubric blames 37 → 8
L2 reused L1's note as evidence | Reads agent output + reasoning | repeats agree 80 → 88%
Confidence float uninformative | Low / medium / high bands | shipped
L1 too harsh (37 of 180 reviews) | Production rubric + formatting rules | matches human 24 → 42%
No "nothing failed" verdict (15) | Open: deeper analysis ongoing | next

Human agrees, by cause L2 named · before vs after
Y-axis: Human agrees (%)
Y-axis ticks: 0, 25, 50, 75, 100
Legend: Before, After
All verdicts: Before 77% (n=300), After 80% (n=300)
"model": Before 87% (n=253), After 83% (n=275)
"rubric": Before 22% (n=37), After 62% (n=8)
"l1_judge" (new): After 33% (n=15)

in
PROPRIETARY + CONFIDENTIAL
19

### Acceleration Plan

The speakers propose standardizing evaluation, reflection and optimization so other agent teams could reuse the self-improvement loop developed for user understanding.

[33:48](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2028s) [White section divider with the large heading “Acceleration Plan” in the upper left.](https://assets.ai.engineer/sites/aiecode2025/sha256/3106134f99dfcef742f10a9cd61b617bd6d300957459ade9f93bef3d279b304b.webp)

Acceleration Plan

in
PROPRIETARY + CONFIDENTIAL
21

### End-to-end Evals Current State

Some teams evaluate agents with fragile custom workflows or one-off checks. Reliable evaluations help agents diagnose failures. Most agents lack both that reflection step and the recursive self-improvement loop that builds on it.

[34:15](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2055s) [Four layers run from Base Agent at the bottom through Agentic Evals and Agentic Reflection to RSI Loop at the top. Annotations describe bespoke or one-off evaluations and say most agents lack the upper two layers.](https://assets.ai.engineer/sites/aiecode2025/sha256/0b656b0e4450919e206f5bf62d8136717a171269fcb2633cb3b8c54a4f9ec2bf.webp)

End-to-end Evals Current State
Leveling up Evals is prerequisite to RSI

RSI Loop
Does not exists for most agents

Agentic Reflection
Does not exists for most agents

Agentic Evals
Some have it as a bespoke unstable workflow. Some just do one-off analysis.
Almost every team is asking for a platform

Base Agent

in
PROPRIETARY + CONFIDENTIAL
22

### Conclusion: Accelerating the Rate of AI Improvement

LinkedIn’s improvement engine diagnoses failures in its user understanding agent, proposes changes, and tests whether they help. The goal is a steeper performance curve as the agent improves faster over time.

[35:36](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2136s) [Four bullets describe the target agent, the improvement engine, repeated evaluation and validation, and the goal of faster improvement. A formula expresses that goal as increasing the rate of change in agent performance over time.](https://assets.ai.engineer/sites/aiecode2025/sha256/041c094424aad90de37ea6ee3b225688f48b7d8894f00c46a34df23959ee0ae2.webp)

Conclusion: Accelerating the Rate of AI Improvement

Recursive Self-Improvement isn't just about building better AI agents. It's about building systems that make AI agents better, faster.

• Target Agent: The AI system whose performance we're continuously improving.
• Improvement Engine: The feedback-driven system that evaluates, diagnoses failures, and proposes targeted changes.
• Closed-Loop Learning: Repeated cycles of evaluation, reflection, optimization, and validation drive measurable improvements.
• The Bigger Goal: Move beyond improving performance at a single point in time to accelerating the rate of improvement itself.

We're not just optimizing agent performance. We're optimizing how quickly agent performance improves.

RSI Goal: Increase d(Agent Performance)/dt

[36:57](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2217s) [Four bullets describe the target agent, the improvement engine, closed-loop learning, and the goal of faster improvement. An equation expresses that goal as increasing the rate of change in agent performance over time.](https://assets.ai.engineer/sites/aiecode2025/sha256/8b5853f84c97542ff6a105ad1797048341c79b6d523c10f5b433dbf5efc1a421.webp)

Conclusion: Accelerating the Rate of AI Improvement

Recursive Self-Improvement isn't just about building better AI agents. It's about building systems that make AI agents better, faster.

• Target Agent: The AI system whose performance we're continuously improving.
• Improvement Engine: The feedback-driven system that evaluates, diagnoses failures, and proposes targeted changes.
• Closed-Loop Learning: Repeated cycles of evaluation, reflection, optimization, and validation drive measurable improvements.
• The Bigger Goal: Move beyond improving performance at a single point in time to accelerating the rate of improvement itself.

We're not just optimizing agent performance. We're optimizing how quickly agent performance improves.

RSI Goal: Increase d(Agent Performance)/dt

### Thank you.

As the thank-you slide appears, the speaker closes with a question for recursive self-improvement: how can the team trust that its changes make the system better?

[38:26](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2306s) [Large white text reads “Thank you.” on a navy background with a blue border and a LinkedIn logo in the lower right.](https://assets.ai.engineer/sites/aiecode2025/sha256/33ff996b207fc4c3d4e89c8cdce38f90ab26c0598c4b2be7d6cb98de1244c9a6.webp)

Thank you.

in

## Full transcript

[0:01](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1s) Uh hello everyone. Uh so my name is uh

[0:04](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=4s) Zui and today uh together with uh Saloni

[0:07](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=7s) so we are going to present the recursive

[0:09](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=9s) self-improvement uh project that we are

[0:12](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=12s) doing at linking and for this RSI work

[0:15](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=15s) we focus on the user understanding and

[0:19](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=19s) yeah super excited to share the uh RSI

[0:22](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=22s) work we did at linking. So first uh just

[0:25](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=25s) a high level or overview on what is a

[0:27](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=27s) recursive self-improvement.

[0:30](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=30s) So the recursive self-improvement is a

[0:32](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=32s) build around some uh AI system and uh

[0:34](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=34s) this is the purpose of doing this is we

[0:37](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=37s) want to get a system that

[0:39](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=39s) self-improvements uh self-improves. So

[0:42](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=42s) it can learns from issues in production

[0:44](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=44s) trace and that work. So the AI system

[0:47](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=47s) can be some uh agent and I linking. So

[0:50](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=50s) one example is uh our career uh agent to

[0:54](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=54s) help uh our uh linking users members to

[0:57](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=57s) achieve their career goals and from the

[1:00](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=60s) agent so uh and there's uh many daily

[1:03](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=63s) development work that we discover the

[1:06](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=66s) issues we want to address and also we

[1:09](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=69s) will get the production traces and that

[1:12](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=72s) we can uh so they like we can log the

[1:14](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=74s) issues we found during the production

[1:16](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=76s) traffic and once we have this

[1:19](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=79s) informations So we will have some

[1:21](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=81s) evaluation engine for example the ar

[1:23](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=83s) judge and to uh check our uh traces and

[1:27](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=87s) to identify the issues and gives

[1:29](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=89s) feedbacks and based on the feedbacks and

[1:32](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=92s) we will uh use those feedback to keep uh

[1:35](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=95s) continuously uh iterate our system. So

[1:38](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=98s) we build this kind of like a loop to

[1:41](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=101s) keep uh improve our um AI system. And uh

[1:45](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=105s) one key thing we want to uh call out is

[1:47](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=107s) uh during this uh recursive self

[1:49](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=109s) improvement process the evaluation is

[1:52](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=112s) very critical and it must be objective

[1:55](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=115s) and well aligned with the user feedback

[1:57](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=117s) because this is a key to uh like

[1:59](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=119s) identify where the bottleneck is and

[2:01](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=121s) guide the next step for the uh system AI

[2:04](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=124s) system improvement.

[2:06](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=126s) And uh next we want to uh uh like

[2:09](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=129s) describe like show uh what kind of like

[2:12](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=132s) uh which component of the AI system we

[2:14](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=134s) can uh optimize. Uh so here take the

[2:18](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=138s) harness uh as as an example or like as

[2:21](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=141s) orchestrator. So there are different

[2:23](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=143s) components we can uh use we can uh

[2:26](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=146s) optimize by using the RSI. So for

[2:29](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=149s) example the first is a planning and

[2:31](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=151s) logics which is used to orchestrator

[2:34](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=154s) orchestrates execution and make a

[2:36](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=156s) decision uh past and the second is a

[2:38](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=158s) context and the memory where we preserve

[2:41](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=161s) and manage the session state and the

[2:44](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=164s) third one is a two interface that

[2:46](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=166s) interacts with external and internal

[2:48](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=168s) APIs and data sets. The last uh last but

[2:51](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=171s) not least is artifact storage uh that

[2:54](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=174s) where we stores outputs and the

[2:56](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=176s) generation logs and so they like uh and

[2:59](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=179s) we are on top of those we definitely

[3:01](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=181s) have some like evaluation uh logic and

[3:04](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=184s) uh so that to help us to evaluate uh

[3:06](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=186s) each um component and help us to keep uh

[3:09](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=189s) improve the uh AI system and we want to

[3:13](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=193s) uh like uh uh and another thing to uh

[3:16](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=196s) keep in mind that all those uh like

[3:18](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=198s) harness is a powered by the base uh iron

[3:22](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=202s) model but for the RSI uh so it's

[3:26](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=206s) currently the optimize of this base

[3:29](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=209s) model uh itself is not in uh scope but

[3:32](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=212s) we focus on the uh harness optimization

[3:36](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=216s) uh first

[3:38](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=218s) and uh here we want to uh like show uh

[3:41](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=221s) like this is the ladders that we can

[3:43](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=223s) climb to optimize the uh AI system by

[3:46](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=226s) using the uh RSI. So uh at the beginning

[3:50](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=230s) so typically so this is a start from

[3:52](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=232s) simple uh process at the beginning we

[3:54](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=234s) can focus on the prompts uh updates

[3:57](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=237s) because this is a most uh this is the

[3:59](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=239s) easiest and the most straightforward way

[4:02](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=242s) to uh keep iterate our AI system and

[4:06](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=246s) after that uh we can uh continue to

[4:08](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=248s) optimize the structured context of the

[4:11](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=251s) AI system and uh uh the and the for the

[4:15](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=255s) this is the L1 uh ladder and for the L2

[4:18](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=258s) tool ladder and which is after the

[4:20](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=260s) structure context optimization. We can

[4:23](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=263s) uh expand our scope to design the auto

[4:26](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=266s) automatically design the entire uh

[4:28](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=268s) workflow and uh uh so this is kind of

[4:31](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=271s) like uh like a meta information uh

[4:34](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=274s) design and uh after that and we can

[4:37](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=277s) automize the harness code uh itself and

[4:40](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=280s) as at the end we can uh go very

[4:43](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=283s) compactly to optimize the optimizer code

[4:46](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=286s) uh itself. So this is kind of the

[4:48](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=288s) different optimization ladders uh with

[4:51](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=291s) increasing complexity and uh uh for our

[4:55](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=295s) project. So currently uh our active

[4:57](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=297s) focus is a prompt uh optimization but uh

[5:01](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=301s) uh definitely like moving forward we are

[5:03](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=303s) extend our scope to uh more and more

[5:06](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=306s) complicated uh optimization uh in future

[5:09](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=309s) and uh here we want to show our uh uh

[5:13](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=313s) RSI uh strategy. So we break down our uh

[5:17](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=317s) RSI uh uh process uh into seven uh

[5:20](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=320s) steps. So the first step is to uh run

[5:23](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=323s) the agent and uh generate some uh

[5:26](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=326s) samples. So basically like for our

[5:29](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=329s) career agent uh that we are optimizing

[5:31](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=331s) for. So given a query we'll generate

[5:34](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=334s) some uh output and we will lock those uh

[5:37](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=337s) process in the uh traces for example the

[5:40](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=340s) last miss traces and uh once so this is

[5:43](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=343s) basically just the uh like the uh kind

[5:45](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=345s) of the agent uh trajectories and the

[5:48](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=348s) traces that we can uh uh use to uh

[5:51](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=351s) optimize and once we got those

[5:53](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=353s) information the next step step two is we

[5:56](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=356s) uh runs through the evation uh engine

[5:59](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=359s) for example the judge and to uh get some

[6:02](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=362s) judge uh output to tell us uh where we

[6:06](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=366s) are uh good at and we are where we are

[6:08](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=368s) not good at. So to get the uh uh future

[6:11](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=371s) iteration and then uh once we get the uh

[6:15](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=375s) um judge output then we go uh we we are

[6:19](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=379s) going to the RSI components and uh

[6:22](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=382s) starting from the uh step three. So this

[6:24](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=384s) is a kind of the RSI in the inside the

[6:26](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=386s) RSI loop and the first step is we will

[6:30](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=390s) uh we will do the agent reflection uh

[6:33](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=393s) sometime we also call it a root causing

[6:35](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=395s) so they like based on the judge uh

[6:37](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=397s) output we want to understand what is a

[6:39](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=399s) root cause uh that we don't have a good

[6:43](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=403s) uh output for example is this is a agent

[6:45](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=405s) issue a model issue u prompt issue or

[6:49](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=409s) like a two uh two two issue or some

[6:51](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=411s) other issues or sometimes it's Even it

[6:54](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=414s) can even be a judge issue or rubric

[6:56](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=416s) issue. And uh once we got the root cause

[6:59](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=419s) and uh uh we will uh the next step is to

[7:02](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=422s) uh based on the root cause uh we

[7:05](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=425s) identified we want uh we can generate

[7:07](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=427s) some uh addits proposal. So uh currently

[7:11](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=431s) the addits proposal are just uh prompt

[7:14](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=434s) but as we are climbing the climbing the

[7:16](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=436s) ladder uh for the SI loop. So in future

[7:19](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=439s) this uh additis can also be some like a

[7:22](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=442s) new design of the workflow the harness

[7:24](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=444s) code or even the optimizer optimizer

[7:27](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=447s) code and uh uh so yeah that's uh step

[7:31](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=451s) four to generate the edits proposal and

[7:33](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=453s) the next step is to uh validate our

[7:37](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=457s) edits because uh some edits can be uh

[7:40](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=460s) good some can be bad uh so we will use

[7:42](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=462s) the same uh evaluation engine for

[7:44](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=464s) example the same judge with the same

[7:47](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=467s) rubrics to uh to uh evaluate uh our

[7:50](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=470s) addit validation and uh uh once we found

[7:54](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=474s) some like a good candidates and we will

[7:56](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=476s) uh naturally the next step is to uh

[7:59](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=479s) evolve our uh prompt to adapt the uh the

[8:03](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=483s) uh the proposed edits uh we have and uh

[8:07](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=487s) uh once we uh once the prompt is uh

[8:09](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=489s) updated. So the very last step which is

[8:12](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=492s) a step seven here is we run the

[8:14](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=494s) evaluation uh again on a hold up side

[8:17](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=497s) and to confirm uh our uh this kind of

[8:20](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=500s) the uh new system is indeed performs

[8:23](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=503s) better uh than the old system after the

[8:26](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=506s) SI optimization and uh uh this will and

[8:30](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=510s) also like uh after this hold out

[8:33](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=513s) validation uh like we will make the uh

[8:36](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=516s) prompt updates and this will become our

[8:39](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=519s) uh new baseline and uh and we are uh in

[8:43](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=523s) future we will like continue to do this

[8:46](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=526s) uh RSI uh iteration to keep improving uh

[8:49](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=529s) our system uh with some uh cadence.

[8:53](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=533s) Uh okay. Yeah. Next uh like uh we are we

[8:55](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=535s) can uh like dig deeper into uh each step

[8:59](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=539s) so that we can get uh so so to help you

[9:02](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=542s) to understand more details on what we

[9:04](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=544s) are uh doing here. for example, what is

[9:05](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=545s) the agent we are optimizing and what is

[9:08](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=548s) like a steps uh like a detail steps uh

[9:11](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=551s) in each uh this high level steps to uh

[9:14](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=554s) help us uh to help you to understand

[9:16](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=556s) more of of our uh SI work here. So the

[9:20](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=560s) first step I I want to show uh so this

[9:23](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=563s) is the first uh the step we generated

[9:25](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=565s) the output and the generated trace and

[9:27](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=567s) uh so first we want to I want to uh help

[9:30](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=570s) you to understand the agent uh we are

[9:32](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=572s) optimizing here. So this is a uh user uh

[9:35](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=575s) understanding uh agent uh at linking. So

[9:39](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=579s) uh so like you can see we uh in this

[9:41](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=581s) slide we are showing the input and the

[9:43](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=583s) output of this uh agent. So the input is

[9:46](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=586s) just a simply member ID also it's just a

[9:50](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=590s) user ID and uh uh by uh given user ID we

[9:54](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=594s) can fetch uh some information uh of this

[9:57](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=597s) user for example this user search

[9:59](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=599s) history uh profile data uh and also some

[10:02](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=602s) like a location data and the preference

[10:04](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=604s) data those kind of things and we have

[10:07](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=607s) this user understanding agent to uh like

[10:09](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=609s) take those users uh history or behavior

[10:12](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=612s) data profile data as a input and uh do

[10:15](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=615s) the uh do a lot like this kind of um uh

[10:18](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=618s) agentic uh synthesize steps and we

[10:21](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=621s) generate the uh output. So for this

[10:24](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=624s) output we call it is a uh memor sending

[10:26](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=626s) or the user understanding uh output. So

[10:29](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=629s) here is a high level uh structure of the

[10:32](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=632s) uh this output is so given a member. So

[10:35](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=635s) we will generate some uh fields to

[10:38](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=638s) describe uh this uh member based on our

[10:41](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=641s) agent understanding. For example, what

[10:43](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=643s) is the preferred uh job title uh of this

[10:46](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=646s) member? This is just one uh field of

[10:48](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=648s) this uh this user understanding uh

[10:50](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=650s) output. So for this one, this is just a

[10:54](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=654s) example of the preferred job titles. So

[10:57](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=657s) it can be some like a network manager,

[10:59](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=659s) account manager those kind of things and

[11:01](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=661s) we'll give a confidence on like how

[11:03](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=663s) confidence this uh agent think uh like

[11:07](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=667s) this is indeed the preferred job titles

[11:10](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=670s) for this user and uh also uh we'll get

[11:13](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=673s) uh generate some uh citation data. So

[11:16](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=676s) they like this is the uh like why the

[11:18](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=678s) this is where we can tell why the agent

[11:21](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=681s) think those uh titles is a preferred job

[11:24](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=684s) title for this particular uh member. Uh

[11:27](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=687s) for example it could be uh the reason

[11:29](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=689s) could be like uh in the users uh save

[11:32](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=692s) the search history like those job titles

[11:35](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=695s) uh appears very frequently. So the agent

[11:38](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=698s) think oh this is a strong evidence uh

[11:40](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=700s) that to support that the preferred job

[11:42](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=702s) title is a network uh manager uh for

[11:44](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=704s) this uh for this uh user and uh uh as a

[11:49](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=709s) uh and uh finally this agent can also

[11:52](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=712s) output the reasoning on like why like

[11:55](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=715s) just a reasoning like why the agent

[11:57](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=717s) think those are preferred job titles

[11:59](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=719s) like for example the state preference

[12:01](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=721s) anchors the primary title save search

[12:03](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=723s) agent uh adjacent one those kind of

[12:07](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=727s) So this is a kind of uh uh one example

[12:10](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=730s) of the user understanding agent that we

[12:13](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=733s) are optimizing.

[12:15](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=735s) So the step two is once we uh got this

[12:18](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=738s) uh agents trace and we'll run the judge

[12:22](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=742s) and to uh get a quality readings of this

[12:26](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=746s) uh of the agent uh output and uh for

[12:29](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=749s) this particular user agent uh the score

[12:32](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=752s) is over uh 15 uh task of the dimensions

[12:35](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=755s) by judge where are some um product uh

[12:38](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=758s) defender uh rubrics. So here is one

[12:41](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=761s) example of the uh judge uh output. So

[12:44](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=764s) you can say for particular um member the

[12:47](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=767s) user and we will generate a judge uh

[12:50](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=770s) score which is 2.31 and uh we will show

[12:53](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=773s) and it will also tell like this score is

[12:56](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=776s) for uh which um and [clears throat]

[12:58](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=778s) it'll show the breakdowns uh of the you

[13:01](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=781s) know the each dimensions score for

[13:03](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=783s) example the go summary dimensions and we

[13:06](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=786s) also have some other dimension for the

[13:08](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=788s) the second one is a prefer job title

[13:10](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=790s) dimensions and for each this kind of

[13:12](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=792s) dimensions we give a detailed uh score

[13:15](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=795s) uh for each dimension and also the

[13:17](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=797s) notes. So the notes is uh just the

[13:20](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=800s) rationale also the uh vertx like why uh

[13:24](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=804s) we give this score uh to this uh to this

[13:28](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=808s) particular uh dimension and for this

[13:30](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=810s) particular uh judge is the agent. So the

[13:34](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=814s) uh judge score is one to three uh scale.

[13:37](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=817s) So the score one means uh this is not a

[13:40](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=820s) high uh good is not a good yeah the

[13:45](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=825s) quality is not good and the two is a

[13:47](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=827s) fair uh quality and the three means this

[13:50](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=830s) is a very good quality. Yeah. And uh uh

[13:54](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=834s) so yeah that's our uh um judge and uh

[13:57](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=837s) after we got the judge score uh for each

[14:01](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=841s) dimension of our user understanding

[14:03](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=843s) agent and the next step is we will run

[14:06](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=846s) this uh agent reflection process. So as

[14:09](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=849s) we mentioned earlier so this is a kind

[14:11](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=851s) of uh uh process for us to understand

[14:15](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=855s) okay if this is a low score then what is

[14:17](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=857s) the root cause uh lead to the uh this

[14:21](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=861s) low score. So uh this is a so for this

[14:24](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=864s) uh step three so actually uh internally

[14:27](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=867s) there are four uh like we can call it a

[14:30](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=870s) mini steps for the agent reflection. So

[14:33](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=873s) the first step is a clustering. So we

[14:36](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=876s) are group the failing uh rows because

[14:39](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=879s) like in a production trace so we are we

[14:41](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=881s) will have a lot of like uh examples a

[14:44](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=884s) lot of users and we will group uh the

[14:48](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=888s) those users also the traces by the

[14:50](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=890s) failing uh dimension. So for this uh

[14:54](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=894s) step so this is just a pure clustering

[14:56](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=896s) uh without uh any um cost and uh once we

[15:00](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=900s) get the clustering and uh uh so the next

[15:03](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=903s) step is we call it L2 uh judge so b

[15:07](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=907s) within each uh class. So we are run the

[15:11](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=911s) um uh cause to diagnose uh each row and

[15:16](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=916s) uh and then for each row we are assign a

[15:19](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=919s) root cause and plus tag. So b like this

[15:22](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=922s) is a u uh place that we uh can like rely

[15:27](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=927s) on the to uh let us know uh uh if we

[15:31](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=931s) think uh this is a for example this

[15:34](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=934s) dimension has a low score is that due to

[15:36](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=936s) the prompt issue uh agent issue harness

[15:38](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=938s) issue code issue or like even the judge

[15:41](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=941s) issue or rubric issue. So they will be

[15:44](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=944s) covered by the root cause and tag uh

[15:46](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=946s) here. So uh the then once we get the

[15:51](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=951s) perole level root cause and the tag so

[15:53](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=953s) the next step is to merge the uh issues

[15:56](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=956s) because uh in the second step uh so we

[16:00](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=960s) it's just a we is based right so so the

[16:05](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=965s) root cost is more like the free tax but

[16:08](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=968s) uh uh some root cause uh even the tax is

[16:12](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=972s) not um uh exactly the same but they may

[16:16](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=976s) just describe very similar thing or uh

[16:19](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=979s) so that's why like we want to do this

[16:21](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=981s) kind of uh dduping so we merge the

[16:24](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=984s) issues if we think uh the root cause is

[16:28](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=988s) uh near uh duplicates so uh I can give

[16:31](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=991s) one example here so for example uh for

[16:34](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=994s) one row for one member uh the root cause

[16:37](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=997s) can be uh the members uh profile data is

[16:41](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1001s) uh missing uh and another row uh uh so

[16:45](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1005s) the arm can

[16:47](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1007s) the profile data is uh not showing the

[16:50](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1010s) full picture of this users uh like a

[16:53](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1013s) browsing history or whatever. So b like

[16:55](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1015s) they are uh both of them are describing

[16:58](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1018s) the same thing. So the profile data is

[17:00](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1020s) not supporting the uh this agent output.

[17:04](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1024s) So uh but they just use a different

[17:06](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1026s) text. So that's we are like that's why

[17:09](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1029s) we need this um kind like a dduplication

[17:12](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1032s) uh step to merge the similar issues so

[17:15](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1035s) we can get some uh canonical uh issue I

[17:17](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1037s) say the roo uh labels

[17:21](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1041s) and uh once we finish the step three so

[17:23](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1043s) the next step is to based on the uh

[17:26](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1046s) canonical issues we found we are group

[17:28](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1048s) and rank them by the uh frequency and uh

[17:31](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1051s) this step uh there's no

[17:34](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1054s) am required so this is just to help us

[17:37](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1057s) to understand uh what is the major

[17:40](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1060s) issues bottlenecks uh in our uh AI

[17:43](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1063s) system and uh once we uh do the group

[17:46](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1066s) and ranking and we are generated the uh

[17:48](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1068s) scorec card for the human to uh get uh

[17:51](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1071s) to just uh get a human to act on to uh

[17:56](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1076s) continue uh iterate our uh AI system and

[17:59](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1079s) all uh and also we are uh uh like

[18:02](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1082s) basically just a cache I say save those

[18:05](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1085s) root cause uh somewhere and it will

[18:08](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1088s) continue use this for the self question

[18:10](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1090s) loop. So that's why so you can see uh

[18:13](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1093s) like there's a arrow here like a go back

[18:16](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1096s) to the uh L2 judge. So be like it will

[18:19](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1099s) guide us on like what is basically the

[18:22](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1102s) pri prior diagnose prime on the next

[18:24](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1104s) turn. Uh so that is easier for the uh

[18:27](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1107s) like to judge to identify the issues uh

[18:30](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1110s) in future runs. So that's uh uh like the

[18:34](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1114s) what we do for the agent uh reflection.

[18:37](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1117s) Uh so the next step is a generator edits

[18:40](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1120s) proposal because from the step three uh

[18:43](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1123s) the previous slide we know uh we already

[18:45](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1125s) know okay so what is the root cause

[18:47](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1127s) right why like the agent fails so the

[18:50](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1130s) next step is naturally like given uh

[18:53](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1133s) those root cause those issues we want to

[18:55](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1135s) fix so what what can we do right what is

[18:59](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1139s) u new edits we want to propose and uh

[19:02](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1142s) here like

[19:04](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1144s) we have uh again like this kind of we

[19:07](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1147s) call it the proposal uh life cycle. So

[19:09](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1149s) based on the judge uh evidence and the

[19:12](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1152s) reflection artifact we will propose the

[19:15](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1155s) input and uh we will run uh some

[19:18](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1158s) methodology. So uh for this project uh

[19:21](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1161s) we are using the uh JPA to uh given the

[19:25](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1165s) uh you know the uh lookout and to uh

[19:28](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1168s) propose the new prompt change and uh and

[19:32](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1172s) like given a trace sample we do it in a

[19:35](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1175s) mini batch uh fashion and uh and

[19:38](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1178s) generated the new uh prompt updates

[19:40](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1180s) proposal and once we generate the promp

[19:43](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1183s) uh updates proposal we'll uh select the

[19:46](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1186s) window prompts and do the uh validation

[19:49](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1189s) validation uh set

[19:52](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1192s) and uh once we uh found a good prompt

[19:55](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1195s) candidates and we run the uh you know

[19:58](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1198s) the validation set and we think uh it is

[20:01](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1201s) uh become uh it is a good uh prompt

[20:04](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1204s) updates and uh we are just um updates

[20:08](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1208s) our prompt in our system. So this will

[20:10](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1210s) be and into the production and this will

[20:13](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1213s) become our new uh kind of like a

[20:15](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1215s) baseline and we will continue to uh do

[20:17](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1217s) the uh like the future uh RSI uh

[20:21](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1221s) iterations. So yeah that's uh kind of

[20:24](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1224s) like the and uh yeah that's a kind of

[20:26](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1226s) like the entire life cycle of this RSI

[20:29](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1229s) uh iteration. And uh next this slide uh

[20:33](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1233s) we want to show our primary focus for

[20:35](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1235s) this RSI work. So we so currently we

[20:39](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1239s) have three major focuses. So the first

[20:41](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1241s) is the quality, second is automation and

[20:44](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1244s) the third is a production uh updates. So

[20:47](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1247s) quality uh like we are focused on the

[20:50](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1250s) different uh uh things uh on the

[20:53](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1253s) quality. So um and we uh did many uh

[20:56](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1256s) iterations to trying to improve the uh

[20:59](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1259s) quality of our agent uh judge and this

[21:03](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1263s) RSI loop. For example, we did extensive

[21:06](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1266s) human annotations so that we can improve

[21:09](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1269s) the judge both the L1 judge and the L2

[21:12](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1272s) judge uh quality and we did uh many

[21:15](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1275s) study and analysis on assess agent judge

[21:18](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1278s) uh uh variance so that we can uh make

[21:21](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1281s) sure we can kind of mitigate the impact

[21:24](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1284s) from the non nondeterministic nature of

[21:27](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1287s) the LM and uh uh the third one is we

[21:30](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1290s) push quality updates to both Asian judge

[21:33](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1293s) and RSI reflection uh based on our

[21:36](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1296s) annotation and the variance analysis uh

[21:40](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1300s) mentioned uh above. So that's on the CI

[21:43](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1303s) side and the second focus is uh

[21:45](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1305s) automation uh because as we mentioned so

[21:48](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1308s) there are seven steps in the RSI

[21:51](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1311s) iteration and uh so that's why like here

[21:54](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1314s) like I we build a uh this kind like

[21:57](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1317s) workflow so that we can kick off this

[22:00](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1320s) workflow with some uh cadence for weekly

[22:03](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1323s) or bi-weekly or monthly based on the

[22:05](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1325s) needs uh so that it can u automatically

[22:08](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1328s) running and keep suggesting the uh like

[22:11](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1331s) the prompt updates uh with some uh

[22:13](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1333s) cadence and the third focus is

[22:16](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1336s) production uh updates. So definitely

[22:18](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1338s) like we want to uh build a uh useful uh

[22:22](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1342s) RSI uh system that can help uh keep

[22:26](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1346s) continuously uh improve our uh you know

[22:28](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1348s) product. So that's why like we the third

[22:31](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1351s) focus uh for the our our uh work is to

[22:35](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1355s) make updates to the current agent in

[22:37](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1357s) production with human in the loop uh

[22:40](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1360s) like for now to uh ensure like the

[22:43](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1363s) production uh uh quality uh is there but

[22:46](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1366s) later like once we think uh the quality

[22:49](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1369s) is good enough like we gradually uh uh

[22:53](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1373s) like reduce the human effort and uh and

[22:57](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1377s) hopefully we can achieve a higher

[22:59](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1379s) automation uh kind of uh state.

[23:03](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1383s) Uh cool. Yeah. Next, I will hand over uh

[23:06](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1386s) to Saloni to uh share more details about

[23:09](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1389s) the outcomes and uh learnings of our our

[23:13](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1393s) work.

[23:16](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1396s) I will share my screen.

[23:18](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1398s) Sure. Yeah.

[23:29](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1409s) Um, can you see my screen?

[23:31](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1411s) Yeah. Yeah.

[23:33](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1413s) Yeah. Um, so yeah. Hi, I'm Salon. And so

[23:36](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1416s) far we described how the RSI system is

[23:39](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1419s) designed. Uh, but one of the most

[23:40](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1420s) interesting aspects of this work has

[23:42](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1422s) been understanding what happens when we

[23:45](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1425s) actually try to make this process

[23:47](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1427s) reliable. And in particular, we learned

[23:49](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1429s) that the quality of the evaluator is

[23:52](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1432s) just as important as the quality of the

[23:54](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1434s) optimization system. So let's walk

[23:56](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1436s) through some of those um findings. Um so

[24:00](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1440s) there are two major outcomes we want to

[24:01](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1441s) highlight. So the first is that we

[24:03](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1443s) successfully like deployed our initial

[24:05](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1445s) RSI improvements into uh production

[24:08](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1448s) which was like um landing a first PR

[24:11](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1451s) which was optimized by Japa and through

[24:13](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1453s) the loop which we just described. Um so

[24:15](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1455s) these changes went through human review

[24:17](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1457s) and quantitative measurement before

[24:19](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1459s) deployment and it was an important

[24:21](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1461s) milestone in moving RSI like beyond just

[24:23](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1463s) an experimental workflow. The second

[24:26](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1466s) major improvement was around like um

[24:28](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1468s) evaluation quality where we invested

[24:30](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1470s) substantial effort and like human

[24:32](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1472s) annotations for the user understanding

[24:35](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1475s) judge and as well as like analyzing

[24:36](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1476s) where its judgment sort of like

[24:38](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1478s) disagreed with like human reviewers. And

[24:41](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1481s) this process revealed several recurring

[24:43](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1483s) weaknesses in both our evaluation

[24:45](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1485s) rubrics as well as our like our

[24:46](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1486s) reflection system. And by investigating

[24:49](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1489s) those disagreements, we were able to

[24:50](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1490s) like make targeted improvements which

[24:53](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1493s) further strengthen the reliability as

[24:55](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1495s) well as like reduce the evaluation

[24:56](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1496s) variance which we have. And uh before we

[24:59](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1499s) could trust a system to like improve an

[25:01](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1501s) agent automatically, we had to

[25:02](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1502s) understand whether the feedback driving

[25:04](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1504s) those improvements was also itself

[25:06](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1506s) trustworthy. And this was like a major

[25:09](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1509s) area of our engineering work. Um to so

[25:12](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1512s) to understand the quality of our like um

[25:14](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1514s) L1 which was our evaluation judge. We

[25:16](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1516s) analyzed about 180 to 200 human reviews.

[25:20](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1520s) We were particularly interested in how

[25:22](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1522s) the judge agreed with human reviewers

[25:24](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1524s) both across different score categories

[25:26](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1526s) and across the individual evaluation

[25:29](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1529s) dimensions. Starting with the chart on

[25:31](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1531s) the left, we see basically that um for

[25:33](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1533s) outputs we assigned a score of like one

[25:36](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1536s) um a full judgment alignment was

[25:38](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1538s) approximately 55% and for score two it

[25:41](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1541s) was around 57%. But for score three the

[25:44](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1544s) alignment increased to approximately

[25:46](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1546s) like 80%. Which indicates an interesting

[25:49](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1549s) pattern which was that the judge was

[25:51](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1551s) considerably like better at recognizing

[25:52](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1552s) stronger outputs than it was at like

[25:55](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1555s) distinguishing between different sort of

[25:57](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1557s) like weaker outputs. And we found that

[25:59](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1559s) many of its uh lowest scores were like

[26:02](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1562s) also overly harsh. And um now looking at

[26:05](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1565s) the chart on right agreement also varied

[26:08](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1568s) significantly across like different

[26:09](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1569s) dimensions where higher preferred

[26:12](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1572s) dimensions like titles had like the

[26:14](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1574s) weakest observed alignment as well as

[26:15](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1575s) like uh domains or like the expertise or

[26:18](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1578s) area of work was around like 47% and the

[26:22](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1582s) sort of like work type was around 50%.

[26:25](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1585s) which told us like the evaluation

[26:27](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1587s) quality was uniform across all the tasks

[26:29](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1589s) which we cared about and certain

[26:31](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1591s) dimensions required more clearer

[26:33](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1593s) definitions better evidence handling

[26:35](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1595s) rules as well as like more explicit

[26:37](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1597s) scoring guidance.

[26:40](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1600s) Um so when we look more closely at the

[26:42](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1602s) human annotations several recurring

[26:43](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1603s) patterns emerged. the first involved

[26:46](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1606s) like using job titles. We found cases

[26:48](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1608s) where the judge would accept like a

[26:50](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1610s) return title because it was plausible

[26:52](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1612s) would but would miss the fact that the

[26:53](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1613s) agent had omitted an expected um signal

[26:57](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1617s) or like an ignored and basically an

[26:59](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1619s) ignored and important intent signal. Um

[27:02](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1622s) in other words, sometimes the judge was

[27:03](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1623s) sometimes evaluating the plausibility

[27:05](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1625s) rather than the completeness of the

[27:07](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1627s) facts which were stated. Uh the second

[27:09](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1629s) issue was primarily around primarily

[27:11](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1631s) around the fields of expertise. Here the

[27:14](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1634s) distinction between like a domain,

[27:15](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1635s) function and a skill wasn't like always

[27:17](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1637s) clear where um a domain could be like um

[27:20](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1640s) say software engineering and a function

[27:23](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1643s) or a role could be like for example a

[27:25](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1645s) machine learning engineer while skill

[27:27](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1647s) being uh machine learning. Um the rubric

[27:29](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1649s) also like lack consistent partial credit

[27:32](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1652s) rules for output containing like a

[27:34](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1654s) mixture of like correct and incorrect

[27:35](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1655s) information. And another issue was

[27:38](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1658s) involving around formatting and

[27:40](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1660s) inference. So in some cases the judge

[27:42](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1662s) penalized harmless formatting

[27:43](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1663s) differences. In other words, uh it made

[27:46](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1666s) it inconsistent assumptions about what

[27:48](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1668s) could be inferred from like a member's

[27:50](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1670s) existing employment information too. We

[27:53](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1673s) also examined like the distribution of

[27:55](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1675s) overly harsh judgments. Out of like the

[27:57](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1677s) 37 um judgments identified as too harsh,

[28:00](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1680s) 31 of them involved outputs that had

[28:03](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1683s) been assigned a score of one, which

[28:05](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1685s) suggested that the boundaries between

[28:07](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1687s) our scores needed to be like

[28:08](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1688s) recalibrated.

[28:10](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1690s) Um, another recurring source of

[28:13](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1693s) disagreement was whether an inferred

[28:14](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1694s) member's preference should be considered

[28:16](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1696s) valid when the evidence wasn't like

[28:18](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1698s) explicitly stated. And the broader

[28:21](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1701s) lesson here was that many evaluation

[28:22](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1702s) disagreements were not simply like model

[28:24](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1704s) periods. They were reflecting

[28:26](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1706s) ambiguities in the rubric as well as

[28:27](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1707s) like the underlying evaluation policy.

[28:31](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1711s) Um, based on these findings, we

[28:32](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1712s) introduced several targeted improvements

[28:34](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1714s) to our evaluation rubrics. One was like

[28:37](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1717s) basically fixing harmless formatting

[28:39](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1719s) differences. Um, previously like a value

[28:42](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1722s) such as like you know full-time could

[28:44](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1724s) receive an incorrect score because of

[28:46](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1726s) like capitalization or incorrect

[28:47](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1727s) formatting. We updated the rubrics so

[28:50](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1730s) that these differences would not be

[28:51](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1731s) treated as like semantic failures.

[28:53](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1733s) Similarly, we introduced normalization

[28:55](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1735s) guidance for like titles. Um we also

[28:58](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1738s) clarified how the judge should handle

[29:00](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1740s) concurrent employment like you know sort

[29:02](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1742s) of impairing seniority as well as like

[29:05](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1745s) impairing geographic references of the

[29:07](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1747s) user and more broadly we recalibrated

[29:09](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1749s) the rules uh separating low scores from

[29:11](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1751s) like partially correct outputs and these

[29:14](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1754s) changes resulted like resulted in like

[29:16](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1756s) measurable improvements uh where human

[29:18](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1758s) alignment increased from like 57% to

[29:20](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1760s) like 64%

[29:22](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1762s) and um at the same time the mean error

[29:24](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1764s) decreased from like 96 to 74 What we're

[29:27](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1767s) trying to achieve here is to get our

[29:28](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1768s) system as high or like as highly

[29:31](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1771s) aligning to humans as much as possible.

[29:34](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1774s) And what we find particularly

[29:35](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1775s) interesting here is that we didn't

[29:36](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1776s) achieve these improvements by only like

[29:38](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1778s) handling the model, but we also improved

[29:40](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1780s) specifications of what like the correct

[29:42](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1782s) evaluation should look like. That is

[29:43](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1783s) like rubric design as well as like um

[29:47](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1787s) evidences which our evaluator is like

[29:49](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1789s) spitting out. Um

[29:52](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1792s) moving forward uh we perform like a

[29:54](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1794s) similar human review analysis for our

[29:56](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1796s) reflection agent which was described

[29:58](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1798s) earlier in our pipeline. We should

[30:00](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1800s) recall that our L2 takes an evaluation

[30:02](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1802s) failure and then it tries to identify

[30:04](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1804s) the root cause. For example, it

[30:07](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1807s) basically identifies was the problem

[30:08](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1808s) caused by the agent itself by the

[30:10](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1810s) evaluation rubric or by some other issue

[30:12](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1812s) with the available context. What we

[30:15](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1815s) discovered was that root cause

[30:16](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1816s) attribution was substantially harder

[30:18](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1818s) than like simply identifying a failure.

[30:21](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1821s) Human reviewers agreed more often when

[30:24](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1824s) L2 attributed a problem to like the

[30:27](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1827s) underlying model, but agreement was much

[30:30](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1830s) lower for other causes, particularly

[30:31](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1831s) like rubric related issues. Um, one

[30:34](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1834s) important issue was that our original

[30:37](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1837s) reflection taxonomy didn't include a way

[30:39](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1839s) to explicitly say that or Elvin judge

[30:41](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1841s) could have been wrong. Um, as a result,

[30:43](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1843s) sometimes the reflection agent fell back

[30:46](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1846s) to blaming the rubric even when the real

[30:48](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1848s) problem was like an incorrect um, L1

[30:50](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1850s) judgment. Um, and like another issue was

[30:53](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1853s) like the uh, reflection agent sometimes

[30:56](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1856s) used the L1 judge's rationale without

[30:58](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1858s) like independently validating or

[30:59](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1859s) independently thinking over it against

[31:01](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1861s) the agents output. And um we exposed an

[31:06](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1866s) important risk here which was like a

[31:07](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1867s) reflection system can produce convincing

[31:09](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1869s) explanations while still like

[31:11](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1871s) identifying misidentifying like the

[31:13](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1873s) underlying problem and if we use those

[31:16](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1876s) explanations to generate prompt changes

[31:18](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1878s) we could like end up modifying like the

[31:19](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1879s) wrong component of the system. So

[31:21](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1881s) improving the L2 uh like reflection

[31:24](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1884s) chart like became our next priority.

[31:28](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1888s) Um so to talk about some of the quality

[31:30](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1890s) improvements um these findings led us to

[31:33](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1893s) like focus on improving both layers of

[31:35](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1895s) evaluation. At L1 the goal was to make

[31:38](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1898s) sure that the scoring was more accurate

[31:40](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1900s) and more better aligned with human

[31:42](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1902s) judgment. And at L2 the goal was to

[31:44](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1904s) improve the reliability of the root

[31:46](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1906s) cause diagnosis.

[31:50](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1910s) And um the next this slide basically

[31:53](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1913s) summarizes some of like the changes we

[31:54](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1914s) made to the system and the impact that

[31:56](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1916s) we observed. We introduced like several

[31:58](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1918s) targeted changes to address the um the

[32:01](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1921s) reflection failures

[32:03](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1923s) um identified through like human review

[32:06](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1926s) and um

[32:09](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1929s) we we were basically like trying to

[32:11](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1931s) measure the impact here in terms of like

[32:12](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1932s) how do we translate all this evaluation

[32:15](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1935s) and reflection work into like measurable

[32:17](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1937s) impact. So there are two levels to

[32:19](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1939s) consider here. The first is like

[32:21](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1941s) improving the reliability of the RSI

[32:23](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1943s) process itself. So by improving L1 we

[32:25](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1945s) make the um evaluation scores more

[32:28](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1948s) stable and like more better aligned with

[32:30](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1950s) human expectations and by improving L2

[32:33](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1953s) we increase like the uh like the root

[32:35](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1955s) cause diagnosis as we mentioned earlier.

[32:38](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1958s) So we also use like majority voting sort

[32:40](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1960s) of and like held out evaluations to like

[32:42](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1962s) help mitigate um variance from like

[32:46](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1966s) individual uh LLM judgments and together

[32:49](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1969s) these improvements support like a more

[32:51](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1971s) reliable optimization process.

[32:54](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1974s) Um importantly like one of the observed

[32:57](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1977s) improvement here was about the required

[32:58](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1978s) adoption threshold and the reported

[33:00](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1980s) confidence interval as well. Uh the

[33:03](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1983s) second level of impact is where we want

[33:05](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1985s) to measure the RSI where the RSI

[33:07](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1987s) adoption grows. This includes the

[33:09](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1989s) percentage of like agent prompt changes

[33:11](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1991s) that can reliably be generated and

[33:12](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1992s) validated through recursive

[33:14](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1994s) self-improvements in like downstream

[33:16](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1996s) response quality and also productivity

[33:18](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=1998s) gains for engineering for engineers like

[33:20](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2000s) maintaining these systems. Um those

[33:22](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2002s) broader metrics are a part of how we

[33:24](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2004s) want to like basically think about

[33:26](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2006s) evaluation evaluating these systems at

[33:28](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2008s) scale because the ultimate goal isn't to

[33:30](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2010s) like maximize the number of like

[33:32](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2012s) automated prompt edits. It's to like

[33:34](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2014s) produce measurable improvements in like

[33:36](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2016s) agent quality while maintaining some

[33:38](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2018s) sort of a confidence that those changes

[33:40](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2020s) are like genuinely beneficial. And this

[33:43](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2023s) brings up brings us to like the final

[33:45](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2025s) part of our talk which is how do we

[33:47](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2027s) scale these um capabilities

[33:50](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2030s) um which is like through an acceleration

[33:52](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2032s) plan. So so far we focused on like one

[33:55](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2035s) implementation of RSI for user

[33:57](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2037s) understanding but like the broader

[33:58](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2038s) opportunity is like much larger. If we

[34:01](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2041s) are able to like um kind of like

[34:03](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2043s) standardize evaluation, reflection and

[34:05](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2045s) optimization, we could potentially uh

[34:08](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2048s) reuse this loop for like or the cycle

[34:10](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2050s) for like many other agent teams. And um

[34:14](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2054s) moving forward like um to understand the

[34:17](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2057s) end toend evals the current state is

[34:19](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2059s) like this diagram basically shows

[34:20](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2060s) progression of capabilities that we

[34:22](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2062s) believe are like necessary to support

[34:23](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2063s) RSI at scale. At the foundation, we

[34:26](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2066s) obviously have like the base agent where

[34:28](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2068s) most teams already have agents that

[34:29](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2069s) perform like useful tasks whether that's

[34:31](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2071s) like question answering, summarizing,

[34:33](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2073s) chat agents um or like just completing

[34:36](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2076s) workflows as such. The next layer is

[34:37](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2077s) agent evaluation and this is where we

[34:39](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2079s) begin to like see gaps because some

[34:41](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2081s) teams have built their own bespoke

[34:43](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2083s) evaluation workflows. Other rely on like

[34:45](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2085s) one-off analysis or like manually

[34:47](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2087s) inspecting outputs. But only a few of

[34:50](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2090s) them have like relatively standardized

[34:52](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2092s) reliable evaluation processes that can

[34:54](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2094s) be continuously reused. And above um

[34:58](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2098s) evaluation we have agentic reflection

[35:00](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2100s) which means being able to systematically

[35:01](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2101s) investigate or do some sort of like root

[35:04](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2104s) cause diagnosis. Um and finally at the

[35:06](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2106s) top we have the RSI loop itself which

[35:08](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2108s) connects evaluation and reflection to

[35:10](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2110s) like optimization and validation. The

[35:13](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2113s) the important observation here is that

[35:15](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2115s) these layers like they depend on one

[35:17](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2117s) another and you can't build like a

[35:19](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2119s) trustworthy loop without like reliable

[35:21](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2121s) diagnosis. That's why we see like

[35:24](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2124s) leveling up evaluation infrastructure as

[35:26](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2126s) a prerequisite to like scaling RSI and

[35:29](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2129s) rather than asking every team to like

[35:31](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2131s) independently build all these

[35:33](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2133s) components, there's an opportunity over

[35:34](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2134s) here to like provide some sort of like a

[35:36](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2136s) shared um infrastructure.

[35:39](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2139s) So uh what would it like basically take

[35:41](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2141s) to make RSI like a reusable capability

[35:43](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2143s) across an enterprise is like uh we've

[35:46](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2146s) identified like five strong areas. First

[35:48](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2148s) is like standardizing evaluation

[35:50](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2150s) infrastructure. So we need reusable

[35:52](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2152s) evaluation frameworks, metrics, judges

[35:54](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2154s) which can be calibrated with human

[35:56](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2156s) feedback and um the second is like

[35:58](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2158s) automating feedback loops. So we want

[36:00](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2160s) systems that can continuously collect

[36:02](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2162s) execution traces, detect recurring

[36:04](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2164s) failures and perform some sort of loop

[36:06](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2166s) case analysis.

[36:08](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2168s) Um the third one is we need to scale um

[36:10](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2170s) automated optimization that includes

[36:12](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2172s) prompt optimization, agent reflection.

[36:15](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2175s) So instead of making like ad hoc

[36:17](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2177s) changes, we want improvement processes

[36:19](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2179s) where candidates where candidate updates

[36:21](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2181s) can be like generated and tested in more

[36:23](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2183s) like a repeatable way. Um the fourth

[36:25](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2185s) area is like validation and trust. So

[36:27](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2187s) any automated improvement needs evidence

[36:29](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2189s) that it actually you know works which

[36:32](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2192s) means like having robust hold out

[36:34](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2194s) evaluations and some sort of regression

[36:36](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2196s) testing. And finally, we want to like

[36:38](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2198s) obviously democratize RSI across teams

[36:40](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2200s) where the goal is to develop reusable

[36:42](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2202s) components and self-service infra so

[36:44](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2204s) that teams can adopt these capabilities

[36:46](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2206s) without having to build an entire RSI

[36:49](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2209s) pipeline themselves which is what our

[36:51](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2211s) broader vision is. And I think that this

[36:54](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2214s) is an important shift in how we approach

[36:55](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2215s) the engineering of like AI systems.

[36:59](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2219s) Um so in recursive self-improvement we

[37:02](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2222s) basically just to summarize we have like

[37:04](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2224s) two key components which is the

[37:05](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2225s) improvement engine and the target agent.

[37:07](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2227s) The target agent in the AI system is

[37:09](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2229s) something we're trying to make better

[37:10](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2230s) while the improvement engine is you know

[37:12](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2232s) responsible for driving that change by

[37:15](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2235s) evaluating performance identifying

[37:17](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2237s) failures root causes and proposing

[37:19](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2239s) improvements. But the objective of the

[37:21](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2241s) RSI system isn't simply to improve the

[37:23](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2243s) agents performance at like a given point

[37:25](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2245s) in time. It's to like accelerate how

[37:27](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2247s) quickly the agent gets better. And if we

[37:30](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2250s) think of like the agent performance as a

[37:32](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2252s) function of time, we are interested in

[37:34](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2254s) increasing its first like its first

[37:36](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2256s) derivative. If we think about it

[37:38](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2258s) essentially like making the performance

[37:40](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2260s) um improvement curve steeper and in our

[37:43](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2263s) case like the target agent is our user

[37:45](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2265s) understanding agent while the

[37:46](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2266s) improvement engine is like the RSI

[37:48](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2268s) pipeline which brings together

[37:50](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2270s) evaluation, reflection and firmed

[37:52](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2272s) optimization.

[37:54](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2274s) Um before I close, we want to like

[37:56](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2276s) acknowledge the people who helped make

[37:58](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2278s) this work possible. This has been like

[38:00](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2280s) an collaborative effort across the core

[38:02](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2282s) team, our evaluation team and our

[38:04](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2284s) engineering partners and leadership. Uh

[38:06](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2286s) we we particularly like to thank all the

[38:08](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2288s) team members and collaborators listed

[38:10](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2290s) here for their contributions.

[38:12](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2292s) Um a lot of the work we have described

[38:14](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2294s) for evaluation quality to reflection and

[38:16](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2296s) production required collaborations

[38:18](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2298s) across these different areas of

[38:20](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2300s) expertise and we're grateful for the

[38:22](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2302s) support and um feedback we have

[38:24](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2304s) received. Um

[38:27](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2307s) yep so thank you and uh I guess we just

[38:30](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2310s) want to conclude by saying that um most

[38:33](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2313s) important part of like improvement

[38:35](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2315s) processes is like being able to you know

[38:37](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2317s) trust the changes we are making to make

[38:39](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2319s) the system better. So that's some of the

[38:41](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2321s) directions we're exploring with um

[38:43](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2323s) recursive self-improvement here. So

[38:45](https://www.youtube.com/watch?v=yH7_SIIgI-Y&t=2325s) yeah, thank you so much for listening.
