[
  {
    "startMs": 1199,
    "text": "Uh hello everyone. Uh so my name is uh"
  },
  {
    "startMs": 4160,
    "text": "Zui and today uh together with uh Saloni"
  },
  {
    "startMs": 7440,
    "text": "so we are going to present the recursive"
  },
  {
    "startMs": 9760,
    "text": "self-improvement uh project that we are"
  },
  {
    "startMs": 12160,
    "text": "doing at linking and for this RSI work"
  },
  {
    "startMs": 15519,
    "text": "we focus on the user understanding and"
  },
  {
    "startMs": 19039,
    "text": "yeah super excited to share the uh RSI"
  },
  {
    "startMs": 22240,
    "text": "work we did at linking. So first uh just"
  },
  {
    "startMs": 25439,
    "text": "a high level or overview on what is a"
  },
  {
    "startMs": 27760,
    "text": "recursive self-improvement."
  },
  {
    "startMs": 30080,
    "text": "So the recursive self-improvement is a"
  },
  {
    "startMs": 32000,
    "text": "build around some uh AI system and uh"
  },
  {
    "startMs": 34880,
    "text": "this is the purpose of doing this is we"
  },
  {
    "startMs": 37440,
    "text": "want to get a system that"
  },
  {
    "startMs": 39600,
    "text": "self-improvements uh self-improves. So"
  },
  {
    "startMs": 42160,
    "text": "it can learns from issues in production"
  },
  {
    "startMs": 44399,
    "text": "trace and that work. So the AI system"
  },
  {
    "startMs": 47760,
    "text": "can be some uh agent and I linking. So"
  },
  {
    "startMs": 50879,
    "text": "one example is uh our career uh agent to"
  },
  {
    "startMs": 54320,
    "text": "help uh our uh linking users members to"
  },
  {
    "startMs": 57680,
    "text": "achieve their career goals and from the"
  },
  {
    "startMs": 60480,
    "text": "agent so uh and there's uh many daily"
  },
  {
    "startMs": 63920,
    "text": "development work that we discover the"
  },
  {
    "startMs": 66320,
    "text": "issues we want to address and also we"
  },
  {
    "startMs": 69360,
    "text": "will get the production traces and that"
  },
  {
    "startMs": 72400,
    "text": "we can uh so they like we can log the"
  },
  {
    "startMs": 74720,
    "text": "issues we found during the production"
  },
  {
    "startMs": 76720,
    "text": "traffic and once we have this"
  },
  {
    "startMs": 79360,
    "text": "informations So we will have some"
  },
  {
    "startMs": 81119,
    "text": "evaluation engine for example the ar"
  },
  {
    "startMs": 83280,
    "text": "judge and to uh check our uh traces and"
  },
  {
    "startMs": 87280,
    "text": "to identify the issues and gives"
  },
  {
    "startMs": 89360,
    "text": "feedbacks and based on the feedbacks and"
  },
  {
    "startMs": 92320,
    "text": "we will uh use those feedback to keep uh"
  },
  {
    "startMs": 95600,
    "text": "continuously uh iterate our system. So"
  },
  {
    "startMs": 98960,
    "text": "we build this kind of like a loop to"
  },
  {
    "startMs": 101040,
    "text": "keep uh improve our um AI system. And uh"
  },
  {
    "startMs": 105280,
    "text": "one key thing we want to uh call out is"
  },
  {
    "startMs": 107920,
    "text": "uh during this uh recursive self"
  },
  {
    "startMs": 109759,
    "text": "improvement process the evaluation is"
  },
  {
    "startMs": 112720,
    "text": "very critical and it must be objective"
  },
  {
    "startMs": 115200,
    "text": "and well aligned with the user feedback"
  },
  {
    "startMs": 117520,
    "text": "because this is a key to uh like"
  },
  {
    "startMs": 119759,
    "text": "identify where the bottleneck is and"
  },
  {
    "startMs": 121840,
    "text": "guide the next step for the uh system AI"
  },
  {
    "startMs": 124719,
    "text": "system improvement."
  },
  {
    "startMs": 126960,
    "text": "And uh next we want to uh uh like"
  },
  {
    "startMs": 129759,
    "text": "describe like show uh what kind of like"
  },
  {
    "startMs": 132319,
    "text": "uh which component of the AI system we"
  },
  {
    "startMs": 134959,
    "text": "can uh optimize. Uh so here take the"
  },
  {
    "startMs": 138239,
    "text": "harness uh as as an example or like as"
  },
  {
    "startMs": 141760,
    "text": "orchestrator. So there are different"
  },
  {
    "startMs": 143680,
    "text": "components we can uh use we can uh"
  },
  {
    "startMs": 146959,
    "text": "optimize by using the RSI. So for"
  },
  {
    "startMs": 149440,
    "text": "example the first is a planning and"
  },
  {
    "startMs": 151440,
    "text": "logics which is used to orchestrator"
  },
  {
    "startMs": 154239,
    "text": "orchestrates execution and make a"
  },
  {
    "startMs": 156400,
    "text": "decision uh past and the second is a"
  },
  {
    "startMs": 158959,
    "text": "context and the memory where we preserve"
  },
  {
    "startMs": 161760,
    "text": "and manage the session state and the"
  },
  {
    "startMs": 164319,
    "text": "third one is a two interface that"
  },
  {
    "startMs": 166800,
    "text": "interacts with external and internal"
  },
  {
    "startMs": 168560,
    "text": "APIs and data sets. The last uh last but"
  },
  {
    "startMs": 171760,
    "text": "not least is artifact storage uh that"
  },
  {
    "startMs": 174800,
    "text": "where we stores outputs and the"
  },
  {
    "startMs": 176400,
    "text": "generation logs and so they like uh and"
  },
  {
    "startMs": 179680,
    "text": "we are on top of those we definitely"
  },
  {
    "startMs": 181519,
    "text": "have some like evaluation uh logic and"
  },
  {
    "startMs": 184159,
    "text": "uh so that to help us to evaluate uh"
  },
  {
    "startMs": 186480,
    "text": "each um component and help us to keep uh"
  },
  {
    "startMs": 189519,
    "text": "improve the uh AI system and we want to"
  },
  {
    "startMs": 193440,
    "text": "uh like uh uh and another thing to uh"
  },
  {
    "startMs": 196080,
    "text": "keep in mind that all those uh like"
  },
  {
    "startMs": 198879,
    "text": "harness is a powered by the base uh iron"
  },
  {
    "startMs": 202080,
    "text": "model but for the RSI uh so it's"
  },
  {
    "startMs": 206159,
    "text": "currently the optimize of this base"
  },
  {
    "startMs": 209280,
    "text": "model uh itself is not in uh scope but"
  },
  {
    "startMs": 212720,
    "text": "we focus on the uh harness optimization"
  },
  {
    "startMs": 216480,
    "text": "uh first"
  },
  {
    "startMs": 218319,
    "text": "and uh here we want to uh like show uh"
  },
  {
    "startMs": 221440,
    "text": "like this is the ladders that we can"
  },
  {
    "startMs": 223760,
    "text": "climb to optimize the uh AI system by"
  },
  {
    "startMs": 226560,
    "text": "using the uh RSI. So uh at the beginning"
  },
  {
    "startMs": 230159,
    "text": "so typically so this is a start from"
  },
  {
    "startMs": 232400,
    "text": "simple uh process at the beginning we"
  },
  {
    "startMs": 234560,
    "text": "can focus on the prompts uh updates"
  },
  {
    "startMs": 237120,
    "text": "because this is a most uh this is the"
  },
  {
    "startMs": 239599,
    "text": "easiest and the most straightforward way"
  },
  {
    "startMs": 242239,
    "text": "to uh keep iterate our AI system and"
  },
  {
    "startMs": 246080,
    "text": "after that uh we can uh continue to"
  },
  {
    "startMs": 248879,
    "text": "optimize the structured context of the"
  },
  {
    "startMs": 251599,
    "text": "AI system and uh uh the and the for the"
  },
  {
    "startMs": 255599,
    "text": "this is the L1 uh ladder and for the L2"
  },
  {
    "startMs": 258239,
    "text": "tool ladder and which is after the"
  },
  {
    "startMs": 260400,
    "text": "structure context optimization. We can"
  },
  {
    "startMs": 263120,
    "text": "uh expand our scope to design the auto"
  },
  {
    "startMs": 266560,
    "text": "automatically design the entire uh"
  },
  {
    "startMs": 268720,
    "text": "workflow and uh uh so this is kind of"
  },
  {
    "startMs": 271919,
    "text": "like uh like a meta information uh"
  },
  {
    "startMs": 274880,
    "text": "design and uh after that and we can"
  },
  {
    "startMs": 277840,
    "text": "automize the harness code uh itself and"
  },
  {
    "startMs": 280880,
    "text": "as at the end we can uh go very"
  },
  {
    "startMs": 283520,
    "text": "compactly to optimize the optimizer code"
  },
  {
    "startMs": 286160,
    "text": "uh itself. So this is kind of the"
  },
  {
    "startMs": 288479,
    "text": "different optimization ladders uh with"
  },
  {
    "startMs": 291600,
    "text": "increasing complexity and uh uh for our"
  },
  {
    "startMs": 295120,
    "text": "project. So currently uh our active"
  },
  {
    "startMs": 297520,
    "text": "focus is a prompt uh optimization but uh"
  },
  {
    "startMs": 301280,
    "text": "uh definitely like moving forward we are"
  },
  {
    "startMs": 303120,
    "text": "extend our scope to uh more and more"
  },
  {
    "startMs": 306080,
    "text": "complicated uh optimization uh in future"
  },
  {
    "startMs": 309840,
    "text": "and uh here we want to show our uh uh"
  },
  {
    "startMs": 313199,
    "text": "RSI uh strategy. So we break down our uh"
  },
  {
    "startMs": 317039,
    "text": "RSI uh uh process uh into seven uh"
  },
  {
    "startMs": 320800,
    "text": "steps. So the first step is to uh run"
  },
  {
    "startMs": 323919,
    "text": "the agent and uh generate some uh"
  },
  {
    "startMs": 326479,
    "text": "samples. So basically like for our"
  },
  {
    "startMs": 329360,
    "text": "career agent uh that we are optimizing"
  },
  {
    "startMs": 331840,
    "text": "for. So given a query we'll generate"
  },
  {
    "startMs": 334400,
    "text": "some uh output and we will lock those uh"
  },
  {
    "startMs": 337440,
    "text": "process in the uh traces for example the"
  },
  {
    "startMs": 340000,
    "text": "last miss traces and uh once so this is"
  },
  {
    "startMs": 343520,
    "text": "basically just the uh like the uh kind"
  },
  {
    "startMs": 345919,
    "text": "of the agent uh trajectories and the"
  },
  {
    "startMs": 348240,
    "text": "traces that we can uh uh use to uh"
  },
  {
    "startMs": 351120,
    "text": "optimize and once we got those"
  },
  {
    "startMs": 353840,
    "text": "information the next step step two is we"
  },
  {
    "startMs": 356880,
    "text": "uh runs through the evation uh engine"
  },
  {
    "startMs": 359600,
    "text": "for example the judge and to uh get some"
  },
  {
    "startMs": 362800,
    "text": "judge uh output to tell us uh where we"
  },
  {
    "startMs": 366080,
    "text": "are uh good at and we are where we are"
  },
  {
    "startMs": 368479,
    "text": "not good at. So to get the uh uh future"
  },
  {
    "startMs": 371840,
    "text": "iteration and then uh once we get the uh"
  },
  {
    "startMs": 375759,
    "text": "um judge output then we go uh we we are"
  },
  {
    "startMs": 379280,
    "text": "going to the RSI components and uh"
  },
  {
    "startMs": 382240,
    "text": "starting from the uh step three. So this"
  },
  {
    "startMs": 384479,
    "text": "is a kind of the RSI in the inside the"
  },
  {
    "startMs": 386639,
    "text": "RSI loop and the first step is we will"
  },
  {
    "startMs": 390080,
    "text": "uh we will do the agent reflection uh"
  },
  {
    "startMs": 393120,
    "text": "sometime we also call it a root causing"
  },
  {
    "startMs": 395120,
    "text": "so they like based on the judge uh"
  },
  {
    "startMs": 397199,
    "text": "output we want to understand what is a"
  },
  {
    "startMs": 399919,
    "text": "root cause uh that we don't have a good"
  },
  {
    "startMs": 403360,
    "text": "uh output for example is this is a agent"
  },
  {
    "startMs": 405840,
    "text": "issue a model issue u prompt issue or"
  },
  {
    "startMs": 409120,
    "text": "like a two uh two two issue or some"
  },
  {
    "startMs": 411759,
    "text": "other issues or sometimes it's Even it"
  },
  {
    "startMs": 414319,
    "text": "can even be a judge issue or rubric"
  },
  {
    "startMs": 416560,
    "text": "issue. And uh once we got the root cause"
  },
  {
    "startMs": 419440,
    "text": "and uh uh we will uh the next step is to"
  },
  {
    "startMs": 422720,
    "text": "uh based on the root cause uh we"
  },
  {
    "startMs": 425360,
    "text": "identified we want uh we can generate"
  },
  {
    "startMs": 427840,
    "text": "some uh addits proposal. So uh currently"
  },
  {
    "startMs": 431280,
    "text": "the addits proposal are just uh prompt"
  },
  {
    "startMs": 434160,
    "text": "but as we are climbing the climbing the"
  },
  {
    "startMs": 436479,
    "text": "ladder uh for the SI loop. So in future"
  },
  {
    "startMs": 439520,
    "text": "this uh additis can also be some like a"
  },
  {
    "startMs": 442400,
    "text": "new design of the workflow the harness"
  },
  {
    "startMs": 444479,
    "text": "code or even the optimizer optimizer"
  },
  {
    "startMs": 447520,
    "text": "code and uh uh so yeah that's uh step"
  },
  {
    "startMs": 451280,
    "text": "four to generate the edits proposal and"
  },
  {
    "startMs": 453759,
    "text": "the next step is to uh validate our"
  },
  {
    "startMs": 457199,
    "text": "edits because uh some edits can be uh"
  },
  {
    "startMs": 460240,
    "text": "good some can be bad uh so we will use"
  },
  {
    "startMs": 462880,
    "text": "the same uh evaluation engine for"
  },
  {
    "startMs": 464800,
    "text": "example the same judge with the same"
  },
  {
    "startMs": 467120,
    "text": "rubrics to uh to uh evaluate uh our"
  },
  {
    "startMs": 470479,
    "text": "addit validation and uh uh once we found"
  },
  {
    "startMs": 474160,
    "text": "some like a good candidates and we will"
  },
  {
    "startMs": 476479,
    "text": "uh naturally the next step is to uh"
  },
  {
    "startMs": 479039,
    "text": "evolve our uh prompt to adapt the uh the"
  },
  {
    "startMs": 483360,
    "text": "uh the proposed edits uh we have and uh"
  },
  {
    "startMs": 487280,
    "text": "uh once we uh once the prompt is uh"
  },
  {
    "startMs": 489599,
    "text": "updated. So the very last step which is"
  },
  {
    "startMs": 492160,
    "text": "a step seven here is we run the"
  },
  {
    "startMs": 494479,
    "text": "evaluation uh again on a hold up side"
  },
  {
    "startMs": 497759,
    "text": "and to confirm uh our uh this kind of"
  },
  {
    "startMs": 500560,
    "text": "the uh new system is indeed performs"
  },
  {
    "startMs": 503120,
    "text": "better uh than the old system after the"
  },
  {
    "startMs": 506080,
    "text": "SI optimization and uh uh this will and"
  },
  {
    "startMs": 510560,
    "text": "also like uh after this hold out"
  },
  {
    "startMs": 513200,
    "text": "validation uh like we will make the uh"
  },
  {
    "startMs": 516560,
    "text": "prompt updates and this will become our"
  },
  {
    "startMs": 519680,
    "text": "uh new baseline and uh and we are uh in"
  },
  {
    "startMs": 523200,
    "text": "future we will like continue to do this"
  },
  {
    "startMs": 526000,
    "text": "uh RSI uh iteration to keep improving uh"
  },
  {
    "startMs": 529360,
    "text": "our system uh with some uh cadence."
  },
  {
    "startMs": 533360,
    "text": "Uh okay. Yeah. Next uh like uh we are we"
  },
  {
    "startMs": 535839,
    "text": "can uh like dig deeper into uh each step"
  },
  {
    "startMs": 539600,
    "text": "so that we can get uh so so to help you"
  },
  {
    "startMs": 542000,
    "text": "to understand more details on what we"
  },
  {
    "startMs": 544000,
    "text": "are uh doing here. for example, what is"
  },
  {
    "startMs": 545839,
    "text": "the agent we are optimizing and what is"
  },
  {
    "startMs": 548800,
    "text": "like a steps uh like a detail steps uh"
  },
  {
    "startMs": 551360,
    "text": "in each uh this high level steps to uh"
  },
  {
    "startMs": 554800,
    "text": "help us uh to help you to understand"
  },
  {
    "startMs": 556560,
    "text": "more of of our uh SI work here. So the"
  },
  {
    "startMs": 560160,
    "text": "first step I I want to show uh so this"
  },
  {
    "startMs": 563040,
    "text": "is the first uh the step we generated"
  },
  {
    "startMs": 565200,
    "text": "the output and the generated trace and"
  },
  {
    "startMs": 567680,
    "text": "uh so first we want to I want to uh help"
  },
  {
    "startMs": 570160,
    "text": "you to understand the agent uh we are"
  },
  {
    "startMs": 572320,
    "text": "optimizing here. So this is a uh user uh"
  },
  {
    "startMs": 575600,
    "text": "understanding uh agent uh at linking. So"
  },
  {
    "startMs": 579279,
    "text": "uh so like you can see we uh in this"
  },
  {
    "startMs": 581760,
    "text": "slide we are showing the input and the"
  },
  {
    "startMs": 583680,
    "text": "output of this uh agent. So the input is"
  },
  {
    "startMs": 586880,
    "text": "just a simply member ID also it's just a"
  },
  {
    "startMs": 590240,
    "text": "user ID and uh uh by uh given user ID we"
  },
  {
    "startMs": 594560,
    "text": "can fetch uh some information uh of this"
  },
  {
    "startMs": 597279,
    "text": "user for example this user search"
  },
  {
    "startMs": 599120,
    "text": "history uh profile data uh and also some"
  },
  {
    "startMs": 602800,
    "text": "like a location data and the preference"
  },
  {
    "startMs": 604640,
    "text": "data those kind of things and we have"
  },
  {
    "startMs": 607120,
    "text": "this user understanding agent to uh like"
  },
  {
    "startMs": 609839,
    "text": "take those users uh history or behavior"
  },
  {
    "startMs": 612640,
    "text": "data profile data as a input and uh do"
  },
  {
    "startMs": 615600,
    "text": "the uh do a lot like this kind of um uh"
  },
  {
    "startMs": 618480,
    "text": "agentic uh synthesize steps and we"
  },
  {
    "startMs": 621519,
    "text": "generate the uh output. So for this"
  },
  {
    "startMs": 624160,
    "text": "output we call it is a uh memor sending"
  },
  {
    "startMs": 626880,
    "text": "or the user understanding uh output. So"
  },
  {
    "startMs": 629600,
    "text": "here is a high level uh structure of the"
  },
  {
    "startMs": 632399,
    "text": "uh this output is so given a member. So"
  },
  {
    "startMs": 635760,
    "text": "we will generate some uh fields to"
  },
  {
    "startMs": 638720,
    "text": "describe uh this uh member based on our"
  },
  {
    "startMs": 641440,
    "text": "agent understanding. For example, what"
  },
  {
    "startMs": 643440,
    "text": "is the preferred uh job title uh of this"
  },
  {
    "startMs": 646000,
    "text": "member? This is just one uh field of"
  },
  {
    "startMs": 648720,
    "text": "this uh this user understanding uh"
  },
  {
    "startMs": 650959,
    "text": "output. So for this one, this is just a"
  },
  {
    "startMs": 654000,
    "text": "example of the preferred job titles. So"
  },
  {
    "startMs": 657200,
    "text": "it can be some like a network manager,"
  },
  {
    "startMs": 659279,
    "text": "account manager those kind of things and"
  },
  {
    "startMs": 661440,
    "text": "we'll give a confidence on like how"
  },
  {
    "startMs": 663839,
    "text": "confidence this uh agent think uh like"
  },
  {
    "startMs": 667360,
    "text": "this is indeed the preferred job titles"
  },
  {
    "startMs": 670160,
    "text": "for this user and uh also uh we'll get"
  },
  {
    "startMs": 673920,
    "text": "uh generate some uh citation data. So"
  },
  {
    "startMs": 676079,
    "text": "they like this is the uh like why the"
  },
  {
    "startMs": 678320,
    "text": "this is where we can tell why the agent"
  },
  {
    "startMs": 681040,
    "text": "think those uh titles is a preferred job"
  },
  {
    "startMs": 684240,
    "text": "title for this particular uh member. Uh"
  },
  {
    "startMs": 687040,
    "text": "for example it could be uh the reason"
  },
  {
    "startMs": 689279,
    "text": "could be like uh in the users uh save"
  },
  {
    "startMs": 692399,
    "text": "the search history like those job titles"
  },
  {
    "startMs": 695680,
    "text": "uh appears very frequently. So the agent"
  },
  {
    "startMs": 698160,
    "text": "think oh this is a strong evidence uh"
  },
  {
    "startMs": 700480,
    "text": "that to support that the preferred job"
  },
  {
    "startMs": 702480,
    "text": "title is a network uh manager uh for"
  },
  {
    "startMs": 704880,
    "text": "this uh for this uh user and uh uh as a"
  },
  {
    "startMs": 709360,
    "text": "uh and uh finally this agent can also"
  },
  {
    "startMs": 712160,
    "text": "output the reasoning on like why like"
  },
  {
    "startMs": 715839,
    "text": "just a reasoning like why the agent"
  },
  {
    "startMs": 717760,
    "text": "think those are preferred job titles"
  },
  {
    "startMs": 719680,
    "text": "like for example the state preference"
  },
  {
    "startMs": 721440,
    "text": "anchors the primary title save search"
  },
  {
    "startMs": 723600,
    "text": "agent uh adjacent one those kind of"
  },
  {
    "startMs": 727040,
    "text": "So this is a kind of uh uh one example"
  },
  {
    "startMs": 730000,
    "text": "of the user understanding agent that we"
  },
  {
    "startMs": 733120,
    "text": "are optimizing."
  },
  {
    "startMs": 735360,
    "text": "So the step two is once we uh got this"
  },
  {
    "startMs": 738480,
    "text": "uh agents trace and we'll run the judge"
  },
  {
    "startMs": 742480,
    "text": "and to uh get a quality readings of this"
  },
  {
    "startMs": 746240,
    "text": "uh of the agent uh output and uh for"
  },
  {
    "startMs": 749440,
    "text": "this particular user agent uh the score"
  },
  {
    "startMs": 752000,
    "text": "is over uh 15 uh task of the dimensions"
  },
  {
    "startMs": 755360,
    "text": "by judge where are some um product uh"
  },
  {
    "startMs": 758320,
    "text": "defender uh rubrics. So here is one"
  },
  {
    "startMs": 761279,
    "text": "example of the uh judge uh output. So"
  },
  {
    "startMs": 764720,
    "text": "you can say for particular um member the"
  },
  {
    "startMs": 767920,
    "text": "user and we will generate a judge uh"
  },
  {
    "startMs": 770639,
    "text": "score which is 2.31 and uh we will show"
  },
  {
    "startMs": 773920,
    "text": "and it will also tell like this score is"
  },
  {
    "startMs": 776000,
    "text": "for uh which um and [clears throat]"
  },
  {
    "startMs": 778720,
    "text": "it'll show the breakdowns uh of the you"
  },
  {
    "startMs": 781760,
    "text": "know the each dimensions score for"
  },
  {
    "startMs": 783600,
    "text": "example the go summary dimensions and we"
  },
  {
    "startMs": 786399,
    "text": "also have some other dimension for the"
  },
  {
    "startMs": 788160,
    "text": "the second one is a prefer job title"
  },
  {
    "startMs": 790399,
    "text": "dimensions and for each this kind of"
  },
  {
    "startMs": 792959,
    "text": "dimensions we give a detailed uh score"
  },
  {
    "startMs": 795839,
    "text": "uh for each dimension and also the"
  },
  {
    "startMs": 797760,
    "text": "notes. So the notes is uh just the"
  },
  {
    "startMs": 800160,
    "text": "rationale also the uh vertx like why uh"
  },
  {
    "startMs": 804399,
    "text": "we give this score uh to this uh to this"
  },
  {
    "startMs": 808240,
    "text": "particular uh dimension and for this"
  },
  {
    "startMs": 810959,
    "text": "particular uh judge is the agent. So the"
  },
  {
    "startMs": 814399,
    "text": "uh judge score is one to three uh scale."
  },
  {
    "startMs": 817519,
    "text": "So the score one means uh this is not a"
  },
  {
    "startMs": 820800,
    "text": "high uh good is not a good yeah the"
  },
  {
    "startMs": 825519,
    "text": "quality is not good and the two is a"
  },
  {
    "startMs": 827680,
    "text": "fair uh quality and the three means this"
  },
  {
    "startMs": 830240,
    "text": "is a very good quality. Yeah. And uh uh"
  },
  {
    "startMs": 834000,
    "text": "so yeah that's our uh um judge and uh"
  },
  {
    "startMs": 837360,
    "text": "after we got the judge score uh for each"
  },
  {
    "startMs": 841199,
    "text": "dimension of our user understanding"
  },
  {
    "startMs": 843360,
    "text": "agent and the next step is we will run"
  },
  {
    "startMs": 846320,
    "text": "this uh agent reflection process. So as"
  },
  {
    "startMs": 849839,
    "text": "we mentioned earlier so this is a kind"
  },
  {
    "startMs": 851839,
    "text": "of uh uh process for us to understand"
  },
  {
    "startMs": 855600,
    "text": "okay if this is a low score then what is"
  },
  {
    "startMs": 857920,
    "text": "the root cause uh lead to the uh this"
  },
  {
    "startMs": 861199,
    "text": "low score. So uh this is a so for this"
  },
  {
    "startMs": 864639,
    "text": "uh step three so actually uh internally"
  },
  {
    "startMs": 867120,
    "text": "there are four uh like we can call it a"
  },
  {
    "startMs": 870240,
    "text": "mini steps for the agent reflection. So"
  },
  {
    "startMs": 873680,
    "text": "the first step is a clustering. So we"
  },
  {
    "startMs": 876560,
    "text": "are group the failing uh rows because"
  },
  {
    "startMs": 879199,
    "text": "like in a production trace so we are we"
  },
  {
    "startMs": 881600,
    "text": "will have a lot of like uh examples a"
  },
  {
    "startMs": 884160,
    "text": "lot of users and we will group uh the"
  },
  {
    "startMs": 888000,
    "text": "those users also the traces by the"
  },
  {
    "startMs": 890639,
    "text": "failing uh dimension. So for this uh"
  },
  {
    "startMs": 894240,
    "text": "step so this is just a pure clustering"
  },
  {
    "startMs": 896639,
    "text": "uh without uh any um cost and uh once we"
  },
  {
    "startMs": 900480,
    "text": "get the clustering and uh uh so the next"
  },
  {
    "startMs": 903519,
    "text": "step is we call it L2 uh judge so b"
  },
  {
    "startMs": 907519,
    "text": "within each uh class. So we are run the"
  },
  {
    "startMs": 911120,
    "text": "um uh cause to diagnose uh each row and"
  },
  {
    "startMs": 916399,
    "text": "uh and then for each row we are assign a"
  },
  {
    "startMs": 919040,
    "text": "root cause and plus tag. So b like this"
  },
  {
    "startMs": 922720,
    "text": "is a u uh place that we uh can like rely"
  },
  {
    "startMs": 927120,
    "text": "on the to uh let us know uh uh if we"
  },
  {
    "startMs": 931839,
    "text": "think uh this is a for example this"
  },
  {
    "startMs": 934399,
    "text": "dimension has a low score is that due to"
  },
  {
    "startMs": 936160,
    "text": "the prompt issue uh agent issue harness"
  },
  {
    "startMs": 938800,
    "text": "issue code issue or like even the judge"
  },
  {
    "startMs": 941279,
    "text": "issue or rubric issue. So they will be"
  },
  {
    "startMs": 944079,
    "text": "covered by the root cause and tag uh"
  },
  {
    "startMs": 946959,
    "text": "here. So uh the then once we get the"
  },
  {
    "startMs": 951040,
    "text": "perole level root cause and the tag so"
  },
  {
    "startMs": 953519,
    "text": "the next step is to merge the uh issues"
  },
  {
    "startMs": 956959,
    "text": "because uh in the second step uh so we"
  },
  {
    "startMs": 960240,
    "text": "it's just a we is based right so so the"
  },
  {
    "startMs": 965519,
    "text": "root cost is more like the free tax but"
  },
  {
    "startMs": 968880,
    "text": "uh uh some root cause uh even the tax is"
  },
  {
    "startMs": 972480,
    "text": "not um uh exactly the same but they may"
  },
  {
    "startMs": 976480,
    "text": "just describe very similar thing or uh"
  },
  {
    "startMs": 979519,
    "text": "so that's why like we want to do this"
  },
  {
    "startMs": 981360,
    "text": "kind of uh dduping so we merge the"
  },
  {
    "startMs": 984880,
    "text": "issues if we think uh the root cause is"
  },
  {
    "startMs": 988240,
    "text": "uh near uh duplicates so uh I can give"
  },
  {
    "startMs": 991600,
    "text": "one example here so for example uh for"
  },
  {
    "startMs": 994800,
    "text": "one row for one member uh the root cause"
  },
  {
    "startMs": 997680,
    "text": "can be uh the members uh profile data is"
  },
  {
    "startMs": 1001600,
    "text": "uh missing uh and another row uh uh so"
  },
  {
    "startMs": 1005839,
    "text": "the arm can"
  },
  {
    "startMs": 1007120,
    "text": "the profile data is uh not showing the"
  },
  {
    "startMs": 1010720,
    "text": "full picture of this users uh like a"
  },
  {
    "startMs": 1013199,
    "text": "browsing history or whatever. So b like"
  },
  {
    "startMs": 1015360,
    "text": "they are uh both of them are describing"
  },
  {
    "startMs": 1018079,
    "text": "the same thing. So the profile data is"
  },
  {
    "startMs": 1020320,
    "text": "not supporting the uh this agent output."
  },
  {
    "startMs": 1024400,
    "text": "So uh but they just use a different"
  },
  {
    "startMs": 1026720,
    "text": "text. So that's we are like that's why"
  },
  {
    "startMs": 1029199,
    "text": "we need this um kind like a dduplication"
  },
  {
    "startMs": 1032160,
    "text": "uh step to merge the similar issues so"
  },
  {
    "startMs": 1035199,
    "text": "we can get some uh canonical uh issue I"
  },
  {
    "startMs": 1037919,
    "text": "say the roo uh labels"
  },
  {
    "startMs": 1041039,
    "text": "and uh once we finish the step three so"
  },
  {
    "startMs": 1043679,
    "text": "the next step is to based on the uh"
  },
  {
    "startMs": 1046000,
    "text": "canonical issues we found we are group"
  },
  {
    "startMs": 1048240,
    "text": "and rank them by the uh frequency and uh"
  },
  {
    "startMs": 1051760,
    "text": "this step uh there's no"
  },
  {
    "startMs": 1054640,
    "text": "am required so this is just to help us"
  },
  {
    "startMs": 1057840,
    "text": "to understand uh what is the major"
  },
  {
    "startMs": 1060240,
    "text": "issues bottlenecks uh in our uh AI"
  },
  {
    "startMs": 1063039,
    "text": "system and uh once we uh do the group"
  },
  {
    "startMs": 1066320,
    "text": "and ranking and we are generated the uh"
  },
  {
    "startMs": 1068400,
    "text": "scorec card for the human to uh get uh"
  },
  {
    "startMs": 1071840,
    "text": "to just uh get a human to act on to uh"
  },
  {
    "startMs": 1076000,
    "text": "continue uh iterate our uh AI system and"
  },
  {
    "startMs": 1079760,
    "text": "all uh and also we are uh uh like"
  },
  {
    "startMs": 1082799,
    "text": "basically just a cache I say save those"
  },
  {
    "startMs": 1085600,
    "text": "root cause uh somewhere and it will"
  },
  {
    "startMs": 1088480,
    "text": "continue use this for the self question"
  },
  {
    "startMs": 1090559,
    "text": "loop. So that's why so you can see uh"
  },
  {
    "startMs": 1093200,
    "text": "like there's a arrow here like a go back"
  },
  {
    "startMs": 1096080,
    "text": "to the uh L2 judge. So be like it will"
  },
  {
    "startMs": 1099440,
    "text": "guide us on like what is basically the"
  },
  {
    "startMs": 1102080,
    "text": "pri prior diagnose prime on the next"
  },
  {
    "startMs": 1104640,
    "text": "turn. Uh so that is easier for the uh"
  },
  {
    "startMs": 1107679,
    "text": "like to judge to identify the issues uh"
  },
  {
    "startMs": 1110640,
    "text": "in future runs. So that's uh uh like the"
  },
  {
    "startMs": 1114240,
    "text": "what we do for the agent uh reflection."
  },
  {
    "startMs": 1117600,
    "text": "Uh so the next step is a generator edits"
  },
  {
    "startMs": 1120880,
    "text": "proposal because from the step three uh"
  },
  {
    "startMs": 1123200,
    "text": "the previous slide we know uh we already"
  },
  {
    "startMs": 1125600,
    "text": "know okay so what is the root cause"
  },
  {
    "startMs": 1127520,
    "text": "right why like the agent fails so the"
  },
  {
    "startMs": 1130640,
    "text": "next step is naturally like given uh"
  },
  {
    "startMs": 1133520,
    "text": "those root cause those issues we want to"
  },
  {
    "startMs": 1135919,
    "text": "fix so what what can we do right what is"
  },
  {
    "startMs": 1139200,
    "text": "u new edits we want to propose and uh"
  },
  {
    "startMs": 1142320,
    "text": "here like"
  },
  {
    "startMs": 1144160,
    "text": "we have uh again like this kind of we"
  },
  {
    "startMs": 1147280,
    "text": "call it the proposal uh life cycle. So"
  },
  {
    "startMs": 1149840,
    "text": "based on the judge uh evidence and the"
  },
  {
    "startMs": 1152480,
    "text": "reflection artifact we will propose the"
  },
  {
    "startMs": 1155280,
    "text": "input and uh we will run uh some"
  },
  {
    "startMs": 1158240,
    "text": "methodology. So uh for this project uh"
  },
  {
    "startMs": 1161600,
    "text": "we are using the uh JPA to uh given the"
  },
  {
    "startMs": 1165360,
    "text": "uh you know the uh lookout and to uh"
  },
  {
    "startMs": 1168559,
    "text": "propose the new prompt change and uh and"
  },
  {
    "startMs": 1172000,
    "text": "like given a trace sample we do it in a"
  },
  {
    "startMs": 1175440,
    "text": "mini batch uh fashion and uh and"
  },
  {
    "startMs": 1178320,
    "text": "generated the new uh prompt updates"
  },
  {
    "startMs": 1180720,
    "text": "proposal and once we generate the promp"
  },
  {
    "startMs": 1183360,
    "text": "uh updates proposal we'll uh select the"
  },
  {
    "startMs": 1186080,
    "text": "window prompts and do the uh validation"
  },
  {
    "startMs": 1189120,
    "text": "validation uh set"
  },
  {
    "startMs": 1192480,
    "text": "and uh once we uh found a good prompt"
  },
  {
    "startMs": 1195919,
    "text": "candidates and we run the uh you know"
  },
  {
    "startMs": 1198320,
    "text": "the validation set and we think uh it is"
  },
  {
    "startMs": 1201280,
    "text": "uh become uh it is a good uh prompt"
  },
  {
    "startMs": 1204400,
    "text": "updates and uh we are just um updates"
  },
  {
    "startMs": 1208240,
    "text": "our prompt in our system. So this will"
  },
  {
    "startMs": 1210559,
    "text": "be and into the production and this will"
  },
  {
    "startMs": 1213039,
    "text": "become our new uh kind of like a"
  },
  {
    "startMs": 1215200,
    "text": "baseline and we will continue to uh do"
  },
  {
    "startMs": 1217919,
    "text": "the uh like the future uh RSI uh"
  },
  {
    "startMs": 1221280,
    "text": "iterations. So yeah that's uh kind of"
  },
  {
    "startMs": 1224240,
    "text": "like the and uh yeah that's a kind of"
  },
  {
    "startMs": 1226240,
    "text": "like the entire life cycle of this RSI"
  },
  {
    "startMs": 1229840,
    "text": "uh iteration. And uh next this slide uh"
  },
  {
    "startMs": 1233039,
    "text": "we want to show our primary focus for"
  },
  {
    "startMs": 1235760,
    "text": "this RSI work. So we so currently we"
  },
  {
    "startMs": 1239200,
    "text": "have three major focuses. So the first"
  },
  {
    "startMs": 1241600,
    "text": "is the quality, second is automation and"
  },
  {
    "startMs": 1244159,
    "text": "the third is a production uh updates. So"
  },
  {
    "startMs": 1247039,
    "text": "quality uh like we are focused on the"
  },
  {
    "startMs": 1250240,
    "text": "different uh uh things uh on the"
  },
  {
    "startMs": 1253360,
    "text": "quality. So um and we uh did many uh"
  },
  {
    "startMs": 1256720,
    "text": "iterations to trying to improve the uh"
  },
  {
    "startMs": 1259280,
    "text": "quality of our agent uh judge and this"
  },
  {
    "startMs": 1263360,
    "text": "RSI loop. For example, we did extensive"
  },
  {
    "startMs": 1266000,
    "text": "human annotations so that we can improve"
  },
  {
    "startMs": 1269280,
    "text": "the judge both the L1 judge and the L2"
  },
  {
    "startMs": 1272080,
    "text": "judge uh quality and we did uh many"
  },
  {
    "startMs": 1275760,
    "text": "study and analysis on assess agent judge"
  },
  {
    "startMs": 1278720,
    "text": "uh uh variance so that we can uh make"
  },
  {
    "startMs": 1281440,
    "text": "sure we can kind of mitigate the impact"
  },
  {
    "startMs": 1284480,
    "text": "from the non nondeterministic nature of"
  },
  {
    "startMs": 1287360,
    "text": "the LM and uh uh the third one is we"
  },
  {
    "startMs": 1290720,
    "text": "push quality updates to both Asian judge"
  },
  {
    "startMs": 1293440,
    "text": "and RSI reflection uh based on our"
  },
  {
    "startMs": 1296880,
    "text": "annotation and the variance analysis uh"
  },
  {
    "startMs": 1300000,
    "text": "mentioned uh above. So that's on the CI"
  },
  {
    "startMs": 1303600,
    "text": "side and the second focus is uh"
  },
  {
    "startMs": 1305919,
    "text": "automation uh because as we mentioned so"
  },
  {
    "startMs": 1308880,
    "text": "there are seven steps in the RSI"
  },
  {
    "startMs": 1311520,
    "text": "iteration and uh so that's why like here"
  },
  {
    "startMs": 1314559,
    "text": "like I we build a uh this kind like"
  },
  {
    "startMs": 1317520,
    "text": "workflow so that we can kick off this"
  },
  {
    "startMs": 1320240,
    "text": "workflow with some uh cadence for weekly"
  },
  {
    "startMs": 1323120,
    "text": "or bi-weekly or monthly based on the"
  },
  {
    "startMs": 1325679,
    "text": "needs uh so that it can u automatically"
  },
  {
    "startMs": 1328559,
    "text": "running and keep suggesting the uh like"
  },
  {
    "startMs": 1331280,
    "text": "the prompt updates uh with some uh"
  },
  {
    "startMs": 1333600,
    "text": "cadence and the third focus is"
  },
  {
    "startMs": 1336640,
    "text": "production uh updates. So definitely"
  },
  {
    "startMs": 1338799,
    "text": "like we want to uh build a uh useful uh"
  },
  {
    "startMs": 1342240,
    "text": "RSI uh system that can help uh keep"
  },
  {
    "startMs": 1346000,
    "text": "continuously uh improve our uh you know"
  },
  {
    "startMs": 1348720,
    "text": "product. So that's why like we the third"
  },
  {
    "startMs": 1351600,
    "text": "focus uh for the our our uh work is to"
  },
  {
    "startMs": 1355520,
    "text": "make updates to the current agent in"
  },
  {
    "startMs": 1357679,
    "text": "production with human in the loop uh"
  },
  {
    "startMs": 1360080,
    "text": "like for now to uh ensure like the"
  },
  {
    "startMs": 1363120,
    "text": "production uh uh quality uh is there but"
  },
  {
    "startMs": 1366640,
    "text": "later like once we think uh the quality"
  },
  {
    "startMs": 1369440,
    "text": "is good enough like we gradually uh uh"
  },
  {
    "startMs": 1373919,
    "text": "like reduce the human effort and uh and"
  },
  {
    "startMs": 1377120,
    "text": "hopefully we can achieve a higher"
  },
  {
    "startMs": 1379679,
    "text": "automation uh kind of uh state."
  },
  {
    "startMs": 1383840,
    "text": "Uh cool. Yeah. Next, I will hand over uh"
  },
  {
    "startMs": 1386480,
    "text": "to Saloni to uh share more details about"
  },
  {
    "startMs": 1389360,
    "text": "the outcomes and uh learnings of our our"
  },
  {
    "startMs": 1393039,
    "text": "work."
  },
  {
    "startMs": 1396880,
    "text": "I will share my screen."
  },
  {
    "startMs": 1398400,
    "text": "Sure. Yeah."
  },
  {
    "startMs": 1409039,
    "text": "Um, can you see my screen?"
  },
  {
    "startMs": 1411840,
    "text": "Yeah. Yeah."
  },
  {
    "startMs": 1413440,
    "text": "Yeah. Um, so yeah. Hi, I'm Salon. And so"
  },
  {
    "startMs": 1416000,
    "text": "far we described how the RSI system is"
  },
  {
    "startMs": 1419039,
    "text": "designed. Uh, but one of the most"
  },
  {
    "startMs": 1420720,
    "text": "interesting aspects of this work has"
  },
  {
    "startMs": 1422799,
    "text": "been understanding what happens when we"
  },
  {
    "startMs": 1425679,
    "text": "actually try to make this process"
  },
  {
    "startMs": 1427280,
    "text": "reliable. And in particular, we learned"
  },
  {
    "startMs": 1429760,
    "text": "that the quality of the evaluator is"
  },
  {
    "startMs": 1432080,
    "text": "just as important as the quality of the"
  },
  {
    "startMs": 1434240,
    "text": "optimization system. So let's walk"
  },
  {
    "startMs": 1436799,
    "text": "through some of those um findings. Um so"
  },
  {
    "startMs": 1440240,
    "text": "there are two major outcomes we want to"
  },
  {
    "startMs": 1441760,
    "text": "highlight. So the first is that we"
  },
  {
    "startMs": 1443520,
    "text": "successfully like deployed our initial"
  },
  {
    "startMs": 1445919,
    "text": "RSI improvements into uh production"
  },
  {
    "startMs": 1448400,
    "text": "which was like um landing a first PR"
  },
  {
    "startMs": 1451360,
    "text": "which was optimized by Japa and through"
  },
  {
    "startMs": 1453039,
    "text": "the loop which we just described. Um so"
  },
  {
    "startMs": 1455360,
    "text": "these changes went through human review"
  },
  {
    "startMs": 1457440,
    "text": "and quantitative measurement before"
  },
  {
    "startMs": 1459279,
    "text": "deployment and it was an important"
  },
  {
    "startMs": 1461039,
    "text": "milestone in moving RSI like beyond just"
  },
  {
    "startMs": 1463760,
    "text": "an experimental workflow. The second"
  },
  {
    "startMs": 1466159,
    "text": "major improvement was around like um"
  },
  {
    "startMs": 1468480,
    "text": "evaluation quality where we invested"
  },
  {
    "startMs": 1470559,
    "text": "substantial effort and like human"
  },
  {
    "startMs": 1472080,
    "text": "annotations for the user understanding"
  },
  {
    "startMs": 1475279,
    "text": "judge and as well as like analyzing"
  },
  {
    "startMs": 1476960,
    "text": "where its judgment sort of like"
  },
  {
    "startMs": 1478480,
    "text": "disagreed with like human reviewers. And"
  },
  {
    "startMs": 1481200,
    "text": "this process revealed several recurring"
  },
  {
    "startMs": 1483200,
    "text": "weaknesses in both our evaluation"
  },
  {
    "startMs": 1485520,
    "text": "rubrics as well as our like our"
  },
  {
    "startMs": 1486960,
    "text": "reflection system. And by investigating"
  },
  {
    "startMs": 1489360,
    "text": "those disagreements, we were able to"
  },
  {
    "startMs": 1490880,
    "text": "like make targeted improvements which"
  },
  {
    "startMs": 1493120,
    "text": "further strengthen the reliability as"
  },
  {
    "startMs": 1495120,
    "text": "well as like reduce the evaluation"
  },
  {
    "startMs": 1496799,
    "text": "variance which we have. And uh before we"
  },
  {
    "startMs": 1499600,
    "text": "could trust a system to like improve an"
  },
  {
    "startMs": 1501279,
    "text": "agent automatically, we had to"
  },
  {
    "startMs": 1502960,
    "text": "understand whether the feedback driving"
  },
  {
    "startMs": 1504400,
    "text": "those improvements was also itself"
  },
  {
    "startMs": 1506320,
    "text": "trustworthy. And this was like a major"
  },
  {
    "startMs": 1509360,
    "text": "area of our engineering work. Um to so"
  },
  {
    "startMs": 1512559,
    "text": "to understand the quality of our like um"
  },
  {
    "startMs": 1514880,
    "text": "L1 which was our evaluation judge. We"
  },
  {
    "startMs": 1516799,
    "text": "analyzed about 180 to 200 human reviews."
  },
  {
    "startMs": 1520720,
    "text": "We were particularly interested in how"
  },
  {
    "startMs": 1522240,
    "text": "the judge agreed with human reviewers"
  },
  {
    "startMs": 1524240,
    "text": "both across different score categories"
  },
  {
    "startMs": 1526960,
    "text": "and across the individual evaluation"
  },
  {
    "startMs": 1529440,
    "text": "dimensions. Starting with the chart on"
  },
  {
    "startMs": 1531120,
    "text": "the left, we see basically that um for"
  },
  {
    "startMs": 1533760,
    "text": "outputs we assigned a score of like one"
  },
  {
    "startMs": 1536960,
    "text": "um a full judgment alignment was"
  },
  {
    "startMs": 1538720,
    "text": "approximately 55% and for score two it"
  },
  {
    "startMs": 1541600,
    "text": "was around 57%. But for score three the"
  },
  {
    "startMs": 1544480,
    "text": "alignment increased to approximately"
  },
  {
    "startMs": 1546480,
    "text": "like 80%. Which indicates an interesting"
  },
  {
    "startMs": 1549360,
    "text": "pattern which was that the judge was"
  },
  {
    "startMs": 1551039,
    "text": "considerably like better at recognizing"
  },
  {
    "startMs": 1552720,
    "text": "stronger outputs than it was at like"
  },
  {
    "startMs": 1555360,
    "text": "distinguishing between different sort of"
  },
  {
    "startMs": 1557039,
    "text": "like weaker outputs. And we found that"
  },
  {
    "startMs": 1559840,
    "text": "many of its uh lowest scores were like"
  },
  {
    "startMs": 1562240,
    "text": "also overly harsh. And um now looking at"
  },
  {
    "startMs": 1565840,
    "text": "the chart on right agreement also varied"
  },
  {
    "startMs": 1568159,
    "text": "significantly across like different"
  },
  {
    "startMs": 1569600,
    "text": "dimensions where higher preferred"
  },
  {
    "startMs": 1572320,
    "text": "dimensions like titles had like the"
  },
  {
    "startMs": 1574080,
    "text": "weakest observed alignment as well as"
  },
  {
    "startMs": 1575840,
    "text": "like uh domains or like the expertise or"
  },
  {
    "startMs": 1578640,
    "text": "area of work was around like 47% and the"
  },
  {
    "startMs": 1582320,
    "text": "sort of like work type was around 50%."
  },
  {
    "startMs": 1585440,
    "text": "which told us like the evaluation"
  },
  {
    "startMs": 1587039,
    "text": "quality was uniform across all the tasks"
  },
  {
    "startMs": 1589440,
    "text": "which we cared about and certain"
  },
  {
    "startMs": 1591760,
    "text": "dimensions required more clearer"
  },
  {
    "startMs": 1593679,
    "text": "definitions better evidence handling"
  },
  {
    "startMs": 1595840,
    "text": "rules as well as like more explicit"
  },
  {
    "startMs": 1597600,
    "text": "scoring guidance."
  },
  {
    "startMs": 1600159,
    "text": "Um so when we look more closely at the"
  },
  {
    "startMs": 1602000,
    "text": "human annotations several recurring"
  },
  {
    "startMs": 1603840,
    "text": "patterns emerged. the first involved"
  },
  {
    "startMs": 1606159,
    "text": "like using job titles. We found cases"
  },
  {
    "startMs": 1608960,
    "text": "where the judge would accept like a"
  },
  {
    "startMs": 1610320,
    "text": "return title because it was plausible"
  },
  {
    "startMs": 1612080,
    "text": "would but would miss the fact that the"
  },
  {
    "startMs": 1613840,
    "text": "agent had omitted an expected um signal"
  },
  {
    "startMs": 1617120,
    "text": "or like an ignored and basically an"
  },
  {
    "startMs": 1619120,
    "text": "ignored and important intent signal. Um"
  },
  {
    "startMs": 1622000,
    "text": "in other words, sometimes the judge was"
  },
  {
    "startMs": 1623600,
    "text": "sometimes evaluating the plausibility"
  },
  {
    "startMs": 1625440,
    "text": "rather than the completeness of the"
  },
  {
    "startMs": 1627200,
    "text": "facts which were stated. Uh the second"
  },
  {
    "startMs": 1629760,
    "text": "issue was primarily around primarily"
  },
  {
    "startMs": 1631840,
    "text": "around the fields of expertise. Here the"
  },
  {
    "startMs": 1634000,
    "text": "distinction between like a domain,"
  },
  {
    "startMs": 1635440,
    "text": "function and a skill wasn't like always"
  },
  {
    "startMs": 1637279,
    "text": "clear where um a domain could be like um"
  },
  {
    "startMs": 1640720,
    "text": "say software engineering and a function"
  },
  {
    "startMs": 1643120,
    "text": "or a role could be like for example a"
  },
  {
    "startMs": 1645039,
    "text": "machine learning engineer while skill"
  },
  {
    "startMs": 1647200,
    "text": "being uh machine learning. Um the rubric"
  },
  {
    "startMs": 1649760,
    "text": "also like lack consistent partial credit"
  },
  {
    "startMs": 1652159,
    "text": "rules for output containing like a"
  },
  {
    "startMs": 1654080,
    "text": "mixture of like correct and incorrect"
  },
  {
    "startMs": 1655679,
    "text": "information. And another issue was"
  },
  {
    "startMs": 1658400,
    "text": "involving around formatting and"
  },
  {
    "startMs": 1660320,
    "text": "inference. So in some cases the judge"
  },
  {
    "startMs": 1662159,
    "text": "penalized harmless formatting"
  },
  {
    "startMs": 1663840,
    "text": "differences. In other words, uh it made"
  },
  {
    "startMs": 1666640,
    "text": "it inconsistent assumptions about what"
  },
  {
    "startMs": 1668799,
    "text": "could be inferred from like a member's"
  },
  {
    "startMs": 1670559,
    "text": "existing employment information too. We"
  },
  {
    "startMs": 1673360,
    "text": "also examined like the distribution of"
  },
  {
    "startMs": 1675360,
    "text": "overly harsh judgments. Out of like the"
  },
  {
    "startMs": 1677760,
    "text": "37 um judgments identified as too harsh,"
  },
  {
    "startMs": 1680880,
    "text": "31 of them involved outputs that had"
  },
  {
    "startMs": 1683520,
    "text": "been assigned a score of one, which"
  },
  {
    "startMs": 1685279,
    "text": "suggested that the boundaries between"
  },
  {
    "startMs": 1687039,
    "text": "our scores needed to be like"
  },
  {
    "startMs": 1688559,
    "text": "recalibrated."
  },
  {
    "startMs": 1690720,
    "text": "Um, another recurring source of"
  },
  {
    "startMs": 1693279,
    "text": "disagreement was whether an inferred"
  },
  {
    "startMs": 1694960,
    "text": "member's preference should be considered"
  },
  {
    "startMs": 1696799,
    "text": "valid when the evidence wasn't like"
  },
  {
    "startMs": 1698720,
    "text": "explicitly stated. And the broader"
  },
  {
    "startMs": 1701039,
    "text": "lesson here was that many evaluation"
  },
  {
    "startMs": 1702559,
    "text": "disagreements were not simply like model"
  },
  {
    "startMs": 1704399,
    "text": "periods. They were reflecting"
  },
  {
    "startMs": 1706080,
    "text": "ambiguities in the rubric as well as"
  },
  {
    "startMs": 1707760,
    "text": "like the underlying evaluation policy."
  },
  {
    "startMs": 1711360,
    "text": "Um, based on these findings, we"
  },
  {
    "startMs": 1712799,
    "text": "introduced several targeted improvements"
  },
  {
    "startMs": 1714480,
    "text": "to our evaluation rubrics. One was like"
  },
  {
    "startMs": 1717919,
    "text": "basically fixing harmless formatting"
  },
  {
    "startMs": 1719679,
    "text": "differences. Um, previously like a value"
  },
  {
    "startMs": 1722720,
    "text": "such as like you know full-time could"
  },
  {
    "startMs": 1724399,
    "text": "receive an incorrect score because of"
  },
  {
    "startMs": 1726080,
    "text": "like capitalization or incorrect"
  },
  {
    "startMs": 1727760,
    "text": "formatting. We updated the rubrics so"
  },
  {
    "startMs": 1730240,
    "text": "that these differences would not be"
  },
  {
    "startMs": 1731600,
    "text": "treated as like semantic failures."
  },
  {
    "startMs": 1733520,
    "text": "Similarly, we introduced normalization"
  },
  {
    "startMs": 1735360,
    "text": "guidance for like titles. Um we also"
  },
  {
    "startMs": 1738480,
    "text": "clarified how the judge should handle"
  },
  {
    "startMs": 1740159,
    "text": "concurrent employment like you know sort"
  },
  {
    "startMs": 1742799,
    "text": "of impairing seniority as well as like"
  },
  {
    "startMs": 1745200,
    "text": "impairing geographic references of the"
  },
  {
    "startMs": 1747200,
    "text": "user and more broadly we recalibrated"
  },
  {
    "startMs": 1749679,
    "text": "the rules uh separating low scores from"
  },
  {
    "startMs": 1751840,
    "text": "like partially correct outputs and these"
  },
  {
    "startMs": 1754559,
    "text": "changes resulted like resulted in like"
  },
  {
    "startMs": 1756240,
    "text": "measurable improvements uh where human"
  },
  {
    "startMs": 1758240,
    "text": "alignment increased from like 57% to"
  },
  {
    "startMs": 1760480,
    "text": "like 64%"
  },
  {
    "startMs": 1762240,
    "text": "and um at the same time the mean error"
  },
  {
    "startMs": 1764320,
    "text": "decreased from like 96 to 74 What we're"
  },
  {
    "startMs": 1767360,
    "text": "trying to achieve here is to get our"
  },
  {
    "startMs": 1768960,
    "text": "system as high or like as highly"
  },
  {
    "startMs": 1771520,
    "text": "aligning to humans as much as possible."
  },
  {
    "startMs": 1774159,
    "text": "And what we find particularly"
  },
  {
    "startMs": 1775440,
    "text": "interesting here is that we didn't"
  },
  {
    "startMs": 1776880,
    "text": "achieve these improvements by only like"
  },
  {
    "startMs": 1778480,
    "text": "handling the model, but we also improved"
  },
  {
    "startMs": 1780559,
    "text": "specifications of what like the correct"
  },
  {
    "startMs": 1782080,
    "text": "evaluation should look like. That is"
  },
  {
    "startMs": 1783760,
    "text": "like rubric design as well as like um"
  },
  {
    "startMs": 1787279,
    "text": "evidences which our evaluator is like"
  },
  {
    "startMs": 1789360,
    "text": "spitting out. Um"
  },
  {
    "startMs": 1792880,
    "text": "moving forward uh we perform like a"
  },
  {
    "startMs": 1794799,
    "text": "similar human review analysis for our"
  },
  {
    "startMs": 1796799,
    "text": "reflection agent which was described"
  },
  {
    "startMs": 1798399,
    "text": "earlier in our pipeline. We should"
  },
  {
    "startMs": 1800559,
    "text": "recall that our L2 takes an evaluation"
  },
  {
    "startMs": 1802960,
    "text": "failure and then it tries to identify"
  },
  {
    "startMs": 1804799,
    "text": "the root cause. For example, it"
  },
  {
    "startMs": 1807200,
    "text": "basically identifies was the problem"
  },
  {
    "startMs": 1808720,
    "text": "caused by the agent itself by the"
  },
  {
    "startMs": 1810480,
    "text": "evaluation rubric or by some other issue"
  },
  {
    "startMs": 1812880,
    "text": "with the available context. What we"
  },
  {
    "startMs": 1815360,
    "text": "discovered was that root cause"
  },
  {
    "startMs": 1816960,
    "text": "attribution was substantially harder"
  },
  {
    "startMs": 1818799,
    "text": "than like simply identifying a failure."
  },
  {
    "startMs": 1821360,
    "text": "Human reviewers agreed more often when"
  },
  {
    "startMs": 1824000,
    "text": "L2 attributed a problem to like the"
  },
  {
    "startMs": 1827120,
    "text": "underlying model, but agreement was much"
  },
  {
    "startMs": 1830000,
    "text": "lower for other causes, particularly"
  },
  {
    "startMs": 1831840,
    "text": "like rubric related issues. Um, one"
  },
  {
    "startMs": 1834960,
    "text": "important issue was that our original"
  },
  {
    "startMs": 1837039,
    "text": "reflection taxonomy didn't include a way"
  },
  {
    "startMs": 1839039,
    "text": "to explicitly say that or Elvin judge"
  },
  {
    "startMs": 1841360,
    "text": "could have been wrong. Um, as a result,"
  },
  {
    "startMs": 1843760,
    "text": "sometimes the reflection agent fell back"
  },
  {
    "startMs": 1846159,
    "text": "to blaming the rubric even when the real"
  },
  {
    "startMs": 1848080,
    "text": "problem was like an incorrect um, L1"
  },
  {
    "startMs": 1850240,
    "text": "judgment. Um, and like another issue was"
  },
  {
    "startMs": 1853919,
    "text": "like the uh, reflection agent sometimes"
  },
  {
    "startMs": 1856000,
    "text": "used the L1 judge's rationale without"
  },
  {
    "startMs": 1858159,
    "text": "like independently validating or"
  },
  {
    "startMs": 1859840,
    "text": "independently thinking over it against"
  },
  {
    "startMs": 1861919,
    "text": "the agents output. And um we exposed an"
  },
  {
    "startMs": 1866080,
    "text": "important risk here which was like a"
  },
  {
    "startMs": 1867440,
    "text": "reflection system can produce convincing"
  },
  {
    "startMs": 1869600,
    "text": "explanations while still like"
  },
  {
    "startMs": 1871360,
    "text": "identifying misidentifying like the"
  },
  {
    "startMs": 1873679,
    "text": "underlying problem and if we use those"
  },
  {
    "startMs": 1876240,
    "text": "explanations to generate prompt changes"
  },
  {
    "startMs": 1878000,
    "text": "we could like end up modifying like the"
  },
  {
    "startMs": 1879919,
    "text": "wrong component of the system. So"
  },
  {
    "startMs": 1881679,
    "text": "improving the L2 uh like reflection"
  },
  {
    "startMs": 1884559,
    "text": "chart like became our next priority."
  },
  {
    "startMs": 1888480,
    "text": "Um so to talk about some of the quality"
  },
  {
    "startMs": 1890880,
    "text": "improvements um these findings led us to"
  },
  {
    "startMs": 1893760,
    "text": "like focus on improving both layers of"
  },
  {
    "startMs": 1895919,
    "text": "evaluation. At L1 the goal was to make"
  },
  {
    "startMs": 1898399,
    "text": "sure that the scoring was more accurate"
  },
  {
    "startMs": 1900880,
    "text": "and more better aligned with human"
  },
  {
    "startMs": 1902480,
    "text": "judgment. And at L2 the goal was to"
  },
  {
    "startMs": 1904640,
    "text": "improve the reliability of the root"
  },
  {
    "startMs": 1906399,
    "text": "cause diagnosis."
  },
  {
    "startMs": 1910480,
    "text": "And um the next this slide basically"
  },
  {
    "startMs": 1913200,
    "text": "summarizes some of like the changes we"
  },
  {
    "startMs": 1914880,
    "text": "made to the system and the impact that"
  },
  {
    "startMs": 1916320,
    "text": "we observed. We introduced like several"
  },
  {
    "startMs": 1918799,
    "text": "targeted changes to address the um the"
  },
  {
    "startMs": 1921679,
    "text": "reflection failures"
  },
  {
    "startMs": 1923679,
    "text": "um identified through like human review"
  },
  {
    "startMs": 1926799,
    "text": "and um"
  },
  {
    "startMs": 1929039,
    "text": "we we were basically like trying to"
  },
  {
    "startMs": 1931120,
    "text": "measure the impact here in terms of like"
  },
  {
    "startMs": 1932799,
    "text": "how do we translate all this evaluation"
  },
  {
    "startMs": 1935360,
    "text": "and reflection work into like measurable"
  },
  {
    "startMs": 1937760,
    "text": "impact. So there are two levels to"
  },
  {
    "startMs": 1939760,
    "text": "consider here. The first is like"
  },
  {
    "startMs": 1941360,
    "text": "improving the reliability of the RSI"
  },
  {
    "startMs": 1943360,
    "text": "process itself. So by improving L1 we"
  },
  {
    "startMs": 1945679,
    "text": "make the um evaluation scores more"
  },
  {
    "startMs": 1948480,
    "text": "stable and like more better aligned with"
  },
  {
    "startMs": 1950399,
    "text": "human expectations and by improving L2"
  },
  {
    "startMs": 1953039,
    "text": "we increase like the uh like the root"
  },
  {
    "startMs": 1955519,
    "text": "cause diagnosis as we mentioned earlier."
  },
  {
    "startMs": 1958320,
    "text": "So we also use like majority voting sort"
  },
  {
    "startMs": 1960640,
    "text": "of and like held out evaluations to like"
  },
  {
    "startMs": 1962399,
    "text": "help mitigate um variance from like"
  },
  {
    "startMs": 1966080,
    "text": "individual uh LLM judgments and together"
  },
  {
    "startMs": 1969679,
    "text": "these improvements support like a more"
  },
  {
    "startMs": 1971360,
    "text": "reliable optimization process."
  },
  {
    "startMs": 1974480,
    "text": "Um importantly like one of the observed"
  },
  {
    "startMs": 1977039,
    "text": "improvement here was about the required"
  },
  {
    "startMs": 1978960,
    "text": "adoption threshold and the reported"
  },
  {
    "startMs": 1980559,
    "text": "confidence interval as well. Uh the"
  },
  {
    "startMs": 1983039,
    "text": "second level of impact is where we want"
  },
  {
    "startMs": 1985039,
    "text": "to measure the RSI where the RSI"
  },
  {
    "startMs": 1987440,
    "text": "adoption grows. This includes the"
  },
  {
    "startMs": 1989519,
    "text": "percentage of like agent prompt changes"
  },
  {
    "startMs": 1991279,
    "text": "that can reliably be generated and"
  },
  {
    "startMs": 1992960,
    "text": "validated through recursive"
  },
  {
    "startMs": 1994159,
    "text": "self-improvements in like downstream"
  },
  {
    "startMs": 1996559,
    "text": "response quality and also productivity"
  },
  {
    "startMs": 1998399,
    "text": "gains for engineering for engineers like"
  },
  {
    "startMs": 2000559,
    "text": "maintaining these systems. Um those"
  },
  {
    "startMs": 2002960,
    "text": "broader metrics are a part of how we"
  },
  {
    "startMs": 2004480,
    "text": "want to like basically think about"
  },
  {
    "startMs": 2006399,
    "text": "evaluation evaluating these systems at"
  },
  {
    "startMs": 2008559,
    "text": "scale because the ultimate goal isn't to"
  },
  {
    "startMs": 2010640,
    "text": "like maximize the number of like"
  },
  {
    "startMs": 2012640,
    "text": "automated prompt edits. It's to like"
  },
  {
    "startMs": 2014480,
    "text": "produce measurable improvements in like"
  },
  {
    "startMs": 2016320,
    "text": "agent quality while maintaining some"
  },
  {
    "startMs": 2018640,
    "text": "sort of a confidence that those changes"
  },
  {
    "startMs": 2020320,
    "text": "are like genuinely beneficial. And this"
  },
  {
    "startMs": 2023039,
    "text": "brings up brings us to like the final"
  },
  {
    "startMs": 2025039,
    "text": "part of our talk which is how do we"
  },
  {
    "startMs": 2027120,
    "text": "scale these um capabilities"
  },
  {
    "startMs": 2030720,
    "text": "um which is like through an acceleration"
  },
  {
    "startMs": 2032640,
    "text": "plan. So so far we focused on like one"
  },
  {
    "startMs": 2035440,
    "text": "implementation of RSI for user"
  },
  {
    "startMs": 2037120,
    "text": "understanding but like the broader"
  },
  {
    "startMs": 2038720,
    "text": "opportunity is like much larger. If we"
  },
  {
    "startMs": 2041279,
    "text": "are able to like um kind of like"
  },
  {
    "startMs": 2043279,
    "text": "standardize evaluation, reflection and"
  },
  {
    "startMs": 2045919,
    "text": "optimization, we could potentially uh"
  },
  {
    "startMs": 2048480,
    "text": "reuse this loop for like or the cycle"
  },
  {
    "startMs": 2050399,
    "text": "for like many other agent teams. And um"
  },
  {
    "startMs": 2054720,
    "text": "moving forward like um to understand the"
  },
  {
    "startMs": 2057520,
    "text": "end toend evals the current state is"
  },
  {
    "startMs": 2059280,
    "text": "like this diagram basically shows"
  },
  {
    "startMs": 2060560,
    "text": "progression of capabilities that we"
  },
  {
    "startMs": 2062079,
    "text": "believe are like necessary to support"
  },
  {
    "startMs": 2063760,
    "text": "RSI at scale. At the foundation, we"
  },
  {
    "startMs": 2066240,
    "text": "obviously have like the base agent where"
  },
  {
    "startMs": 2068079,
    "text": "most teams already have agents that"
  },
  {
    "startMs": 2069520,
    "text": "perform like useful tasks whether that's"
  },
  {
    "startMs": 2071280,
    "text": "like question answering, summarizing,"
  },
  {
    "startMs": 2073040,
    "text": "chat agents um or like just completing"
  },
  {
    "startMs": 2076000,
    "text": "workflows as such. The next layer is"
  },
  {
    "startMs": 2077760,
    "text": "agent evaluation and this is where we"
  },
  {
    "startMs": 2079679,
    "text": "begin to like see gaps because some"
  },
  {
    "startMs": 2081760,
    "text": "teams have built their own bespoke"
  },
  {
    "startMs": 2083760,
    "text": "evaluation workflows. Other rely on like"
  },
  {
    "startMs": 2085919,
    "text": "one-off analysis or like manually"
  },
  {
    "startMs": 2087520,
    "text": "inspecting outputs. But only a few of"
  },
  {
    "startMs": 2090159,
    "text": "them have like relatively standardized"
  },
  {
    "startMs": 2092159,
    "text": "reliable evaluation processes that can"
  },
  {
    "startMs": 2094000,
    "text": "be continuously reused. And above um"
  },
  {
    "startMs": 2098160,
    "text": "evaluation we have agentic reflection"
  },
  {
    "startMs": 2100079,
    "text": "which means being able to systematically"
  },
  {
    "startMs": 2101839,
    "text": "investigate or do some sort of like root"
  },
  {
    "startMs": 2104160,
    "text": "cause diagnosis. Um and finally at the"
  },
  {
    "startMs": 2106960,
    "text": "top we have the RSI loop itself which"
  },
  {
    "startMs": 2108960,
    "text": "connects evaluation and reflection to"
  },
  {
    "startMs": 2110720,
    "text": "like optimization and validation. The"
  },
  {
    "startMs": 2113839,
    "text": "the important observation here is that"
  },
  {
    "startMs": 2115359,
    "text": "these layers like they depend on one"
  },
  {
    "startMs": 2117359,
    "text": "another and you can't build like a"
  },
  {
    "startMs": 2119280,
    "text": "trustworthy loop without like reliable"
  },
  {
    "startMs": 2121839,
    "text": "diagnosis. That's why we see like"
  },
  {
    "startMs": 2124160,
    "text": "leveling up evaluation infrastructure as"
  },
  {
    "startMs": 2126320,
    "text": "a prerequisite to like scaling RSI and"
  },
  {
    "startMs": 2129440,
    "text": "rather than asking every team to like"
  },
  {
    "startMs": 2131839,
    "text": "independently build all these"
  },
  {
    "startMs": 2133040,
    "text": "components, there's an opportunity over"
  },
  {
    "startMs": 2134480,
    "text": "here to like provide some sort of like a"
  },
  {
    "startMs": 2136079,
    "text": "shared um infrastructure."
  },
  {
    "startMs": 2139280,
    "text": "So uh what would it like basically take"
  },
  {
    "startMs": 2141599,
    "text": "to make RSI like a reusable capability"
  },
  {
    "startMs": 2143680,
    "text": "across an enterprise is like uh we've"
  },
  {
    "startMs": 2146400,
    "text": "identified like five strong areas. First"
  },
  {
    "startMs": 2148800,
    "text": "is like standardizing evaluation"
  },
  {
    "startMs": 2150320,
    "text": "infrastructure. So we need reusable"
  },
  {
    "startMs": 2152160,
    "text": "evaluation frameworks, metrics, judges"
  },
  {
    "startMs": 2154560,
    "text": "which can be calibrated with human"
  },
  {
    "startMs": 2156079,
    "text": "feedback and um the second is like"
  },
  {
    "startMs": 2158800,
    "text": "automating feedback loops. So we want"
  },
  {
    "startMs": 2160560,
    "text": "systems that can continuously collect"
  },
  {
    "startMs": 2162320,
    "text": "execution traces, detect recurring"
  },
  {
    "startMs": 2164800,
    "text": "failures and perform some sort of loop"
  },
  {
    "startMs": 2166480,
    "text": "case analysis."
  },
  {
    "startMs": 2168160,
    "text": "Um the third one is we need to scale um"
  },
  {
    "startMs": 2170800,
    "text": "automated optimization that includes"
  },
  {
    "startMs": 2172880,
    "text": "prompt optimization, agent reflection."
  },
  {
    "startMs": 2175599,
    "text": "So instead of making like ad hoc"
  },
  {
    "startMs": 2177119,
    "text": "changes, we want improvement processes"
  },
  {
    "startMs": 2179440,
    "text": "where candidates where candidate updates"
  },
  {
    "startMs": 2181599,
    "text": "can be like generated and tested in more"
  },
  {
    "startMs": 2183200,
    "text": "like a repeatable way. Um the fourth"
  },
  {
    "startMs": 2185760,
    "text": "area is like validation and trust. So"
  },
  {
    "startMs": 2187760,
    "text": "any automated improvement needs evidence"
  },
  {
    "startMs": 2189760,
    "text": "that it actually you know works which"
  },
  {
    "startMs": 2192000,
    "text": "means like having robust hold out"
  },
  {
    "startMs": 2194079,
    "text": "evaluations and some sort of regression"
  },
  {
    "startMs": 2196079,
    "text": "testing. And finally, we want to like"
  },
  {
    "startMs": 2198400,
    "text": "obviously democratize RSI across teams"
  },
  {
    "startMs": 2200560,
    "text": "where the goal is to develop reusable"
  },
  {
    "startMs": 2202320,
    "text": "components and self-service infra so"
  },
  {
    "startMs": 2204960,
    "text": "that teams can adopt these capabilities"
  },
  {
    "startMs": 2206880,
    "text": "without having to build an entire RSI"
  },
  {
    "startMs": 2209520,
    "text": "pipeline themselves which is what our"
  },
  {
    "startMs": 2211280,
    "text": "broader vision is. And I think that this"
  },
  {
    "startMs": 2214000,
    "text": "is an important shift in how we approach"
  },
  {
    "startMs": 2215760,
    "text": "the engineering of like AI systems."
  },
  {
    "startMs": 2219280,
    "text": "Um so in recursive self-improvement we"
  },
  {
    "startMs": 2222160,
    "text": "basically just to summarize we have like"
  },
  {
    "startMs": 2224240,
    "text": "two key components which is the"
  },
  {
    "startMs": 2225599,
    "text": "improvement engine and the target agent."
  },
  {
    "startMs": 2227760,
    "text": "The target agent in the AI system is"
  },
  {
    "startMs": 2229680,
    "text": "something we're trying to make better"
  },
  {
    "startMs": 2230960,
    "text": "while the improvement engine is you know"
  },
  {
    "startMs": 2232720,
    "text": "responsible for driving that change by"
  },
  {
    "startMs": 2235119,
    "text": "evaluating performance identifying"
  },
  {
    "startMs": 2237040,
    "text": "failures root causes and proposing"
  },
  {
    "startMs": 2239119,
    "text": "improvements. But the objective of the"
  },
  {
    "startMs": 2241280,
    "text": "RSI system isn't simply to improve the"
  },
  {
    "startMs": 2243839,
    "text": "agents performance at like a given point"
  },
  {
    "startMs": 2245839,
    "text": "in time. It's to like accelerate how"
  },
  {
    "startMs": 2247760,
    "text": "quickly the agent gets better. And if we"
  },
  {
    "startMs": 2250320,
    "text": "think of like the agent performance as a"
  },
  {
    "startMs": 2252640,
    "text": "function of time, we are interested in"
  },
  {
    "startMs": 2254640,
    "text": "increasing its first like its first"
  },
  {
    "startMs": 2256720,
    "text": "derivative. If we think about it"
  },
  {
    "startMs": 2258160,
    "text": "essentially like making the performance"
  },
  {
    "startMs": 2260800,
    "text": "um improvement curve steeper and in our"
  },
  {
    "startMs": 2263760,
    "text": "case like the target agent is our user"
  },
  {
    "startMs": 2265520,
    "text": "understanding agent while the"
  },
  {
    "startMs": 2266720,
    "text": "improvement engine is like the RSI"
  },
  {
    "startMs": 2268480,
    "text": "pipeline which brings together"
  },
  {
    "startMs": 2270560,
    "text": "evaluation, reflection and firmed"
  },
  {
    "startMs": 2272320,
    "text": "optimization."
  },
  {
    "startMs": 2274640,
    "text": "Um before I close, we want to like"
  },
  {
    "startMs": 2276800,
    "text": "acknowledge the people who helped make"
  },
  {
    "startMs": 2278400,
    "text": "this work possible. This has been like"
  },
  {
    "startMs": 2280240,
    "text": "an collaborative effort across the core"
  },
  {
    "startMs": 2282079,
    "text": "team, our evaluation team and our"
  },
  {
    "startMs": 2284160,
    "text": "engineering partners and leadership. Uh"
  },
  {
    "startMs": 2286480,
    "text": "we we particularly like to thank all the"
  },
  {
    "startMs": 2288640,
    "text": "team members and collaborators listed"
  },
  {
    "startMs": 2290480,
    "text": "here for their contributions."
  },
  {
    "startMs": 2292720,
    "text": "Um a lot of the work we have described"
  },
  {
    "startMs": 2294560,
    "text": "for evaluation quality to reflection and"
  },
  {
    "startMs": 2296880,
    "text": "production required collaborations"
  },
  {
    "startMs": 2298400,
    "text": "across these different areas of"
  },
  {
    "startMs": 2300640,
    "text": "expertise and we're grateful for the"
  },
  {
    "startMs": 2302560,
    "text": "support and um feedback we have"
  },
  {
    "startMs": 2304240,
    "text": "received. Um"
  },
  {
    "startMs": 2307119,
    "text": "yep so thank you and uh I guess we just"
  },
  {
    "startMs": 2310800,
    "text": "want to conclude by saying that um most"
  },
  {
    "startMs": 2313920,
    "text": "important part of like improvement"
  },
  {
    "startMs": 2315440,
    "text": "processes is like being able to you know"
  },
  {
    "startMs": 2317359,
    "text": "trust the changes we are making to make"
  },
  {
    "startMs": 2319359,
    "text": "the system better. So that's some of the"
  },
  {
    "startMs": 2321599,
    "text": "directions we're exploring with um"
  },
  {
    "startMs": 2323520,
    "text": "recursive self-improvement here. So"
  },
  {
    "startMs": 2325200,
    "text": "yeah, thank you so much for listening."
  }
]
