[
  {
    "startMs": 17920,
    "text": "Uh, hey guys, this is Ara. I'm a"
  },
  {
    "startMs": 20560,
    "text": "founding engineer at Cline and today I'm"
  },
  {
    "startMs": 22560,
    "text": "going to be talking about recursively"
  },
  {
    "startMs": 24080,
    "text": "self-improving coding agents. So, this"
  },
  {
    "startMs": 26880,
    "text": "is a very interesting conversation in"
  },
  {
    "startMs": 30480,
    "text": "the sense that everyone wants to"
  },
  {
    "startMs": 32320,
    "text": "understand what RSI is, how we"
  },
  {
    "startMs": 34079,
    "text": "accomplish it, how we go through it, and"
  },
  {
    "startMs": 35520,
    "text": "how it affects our career. So I'm going"
  },
  {
    "startMs": 37520,
    "text": "to start with giving you just three"
  },
  {
    "startMs": 38879,
    "text": "goals of this entire conversation. Just"
  },
  {
    "startMs": 40879,
    "text": "three simple goals and that would help"
  },
  {
    "startMs": 43840,
    "text": "you understand like what what we get out"
  },
  {
    "startMs": 45360,
    "text": "of it and what the future uh for this"
  },
  {
    "startMs": 47200,
    "text": "looks like. So the goal number first is"
  },
  {
    "startMs": 49360,
    "text": "that understanding the intent of"
  },
  {
    "startMs": 50800,
    "text": "frontier labs right everyone who's"
  },
  {
    "startMs": 52559,
    "text": "outside of the frontier labs is always"
  },
  {
    "startMs": 53920,
    "text": "trying to speculate like hey what what"
  },
  {
    "startMs": 55840,
    "text": "is the topic in open or what is deep"
  },
  {
    "startMs": 57600,
    "text": "mind going to come out with or what what"
  },
  {
    "startMs": 58719,
    "text": "are they trying to make and what's the"
  },
  {
    "startMs": 60879,
    "text": "future going to look like who has"
  },
  {
    "startMs": 62000,
    "text": "cracked the RSI. So like this kind of"
  },
  {
    "startMs": 64479,
    "text": "helps you get get a rough outline of"
  },
  {
    "startMs": 66799,
    "text": "what their road map looks like. And the"
  },
  {
    "startMs": 68720,
    "text": "second thing is that like if the models"
  },
  {
    "startMs": 70880,
    "text": "are improving so fast like how do you"
  },
  {
    "startMs": 72320,
    "text": "think about your career or the next few"
  },
  {
    "startMs": 74320,
    "text": "years right that is really what we want"
  },
  {
    "startMs": 76159,
    "text": "to understand. And we want to understand"
  },
  {
    "startMs": 77439,
    "text": "like okay things are changing so fast"
  },
  {
    "startMs": 80479,
    "text": "like the ground underneath us is"
  },
  {
    "startMs": 82400,
    "text": "shifting so quickly right now like it's"
  },
  {
    "startMs": 84960,
    "text": "like uh"
  },
  {
    "startMs": 86880,
    "text": "there used to be a time when you could"
  },
  {
    "startMs": 88240,
    "text": "like demonstrate your side projects on"
  },
  {
    "startMs": 91439,
    "text": "your personal website or GitHub and now"
  },
  {
    "startMs": 93360,
    "text": "it's just like you could literally"
  },
  {
    "startMs": 94560,
    "text": "oneshot them using any Frontier coding"
  },
  {
    "startMs": 97040,
    "text": "engine. And with that it's like it's"
  },
  {
    "startMs": 99600,
    "text": "like oh how how do we demonstrate proof"
  },
  {
    "startMs": 101600,
    "text": "of work? How do you demonstrate, you"
  },
  {
    "startMs": 103439,
    "text": "know, that that that we know what we're"
  },
  {
    "startMs": 105439,
    "text": "doing and how do we think of our career"
  },
  {
    "startMs": 107119,
    "text": "for the next coming years, right? So,"
  },
  {
    "startMs": 109200,
    "text": "it's a very interesting times that we're"
  },
  {
    "startMs": 111200,
    "text": "living in. And then the third and the"
  },
  {
    "startMs": 112720,
    "text": "most important goal of the conversation"
  },
  {
    "startMs": 114079,
    "text": "is escaping the permanent underclass,"
  },
  {
    "startMs": 116399,
    "text": "right? So, I know this seems like a"
  },
  {
    "startMs": 118719,
    "text": "joke, but this really is a very valid"
  },
  {
    "startMs": 121119,
    "text": "possibility that people are"
  },
  {
    "startMs": 122479,
    "text": "contemplating. Everyone's like, \"Oh,"
  },
  {
    "startMs": 124079,
    "text": "well, if everything that we want to be"
  },
  {
    "startMs": 126399,
    "text": "doing can potentially easily be done by"
  },
  {
    "startMs": 128399,
    "text": "a frontier model, like what are we what"
  },
  {
    "startMs": 129840,
    "text": "are we good for?\" And like where does"
  },
  {
    "startMs": 131599,
    "text": "software go? Where where where does the"
  },
  {
    "startMs": 133360,
    "text": "future of technology going to go? And"
  },
  {
    "startMs": 134800,
    "text": "what are we going to do as as things"
  },
  {
    "startMs": 136160,
    "text": "evolve so fast, right? So I think that"
  },
  {
    "startMs": 139360,
    "text": "there's like a lot of like"
  },
  {
    "startMs": 140480,
    "text": "fear-mongering but also there's like"
  },
  {
    "startMs": 142319,
    "text": "some very valid merge where people are"
  },
  {
    "startMs": 145200,
    "text": "concerned that like if all the revenue"
  },
  {
    "startMs": 147040,
    "text": "of the planet is going to go to um a few"
  },
  {
    "startMs": 150000,
    "text": "hardware companies that make frontier"
  },
  {
    "startMs": 151840,
    "text": "GPUs and then a few few uh AI companies"
  },
  {
    "startMs": 155360,
    "text": "that that make the best models like how"
  },
  {
    "startMs": 158000,
    "text": "are we how are the rest of us going to"
  },
  {
    "startMs": 159519,
    "text": "thrive and what what are we going to do"
  },
  {
    "startMs": 161040,
    "text": "here? So, I think I'll I'll walk you"
  },
  {
    "startMs": 163599,
    "text": "through a few examples and a few ideas"
  },
  {
    "startMs": 165120,
    "text": "that will like really really help you"
  },
  {
    "startMs": 167440,
    "text": "reconsider the career choices and and"
  },
  {
    "startMs": 169840,
    "text": "and the life choices you're making and"
  },
  {
    "startMs": 171760,
    "text": "what your problems you're picking up and"
  },
  {
    "startMs": 173519,
    "text": "then and then that will help you like"
  },
  {
    "startMs": 175200,
    "text": "realign"
  },
  {
    "startMs": 176720,
    "text": "what it is that we should be working on"
  },
  {
    "startMs": 178400,
    "text": "and what it is that you you know would"
  },
  {
    "startMs": 180959,
    "text": "would that you could work on that will"
  },
  {
    "startMs": 182879,
    "text": "stand the test of time and help you both"
  },
  {
    "startMs": 185040,
    "text": "just not just like on a psychological"
  },
  {
    "startMs": 186640,
    "text": "but like financial and other levels as"
  },
  {
    "startMs": 188080,
    "text": "well. So uh to give you a very precise"
  },
  {
    "startMs": 190239,
    "text": "example, let's start with something very"
  },
  {
    "startMs": 191680,
    "text": "precise that I had done. So a couple"
  },
  {
    "startMs": 194080,
    "text": "months ago I uh came up with recursive"
  },
  {
    "startMs": 198400,
    "text": "shampoo improvements on a coding"
  },
  {
    "startMs": 201200,
    "text": "harness. So this was the first public"
  },
  {
    "startMs": 203440,
    "text": "example of uh an automatic eval"
  },
  {
    "startMs": 206159,
    "text": "improvement. So to give you an example"
  },
  {
    "startMs": 208400,
    "text": "um we used to do this thing where we"
  },
  {
    "startMs": 210159,
    "text": "would measure the quality of a coding"
  },
  {
    "startMs": 212959,
    "text": "agent using evaluation scores. So on"
  },
  {
    "startMs": 215599,
    "text": "Terminal-Bench 2.1 Kimi K3 with client"
  },
  {
    "startMs": 218480,
    "text": "would score um usually like you know um"
  },
  {
    "startMs": 222239,
    "text": "60 70% and then depending on like how"
  },
  {
    "startMs": 225280,
    "text": "good the model is like you get better"
  },
  {
    "startMs": 226799,
    "text": "scores on on Terminal-Bench 2.1. So we"
  },
  {
    "startMs": 229440,
    "text": "noticed that when Kimi K3 came out, our"
  },
  {
    "startMs": 231280,
    "text": "score was 77.5%. Which is a good score,"
  },
  {
    "startMs": 233599,
    "text": "but we were like, \"Oh, is this is this"
  },
  {
    "startMs": 235440,
    "text": "truly is it truly the best score that"
  },
  {
    "startMs": 237840,
    "text": "this model can get on this benchmark?"
  },
  {
    "startMs": 240000,
    "text": "And also like what does the model"
  },
  {
    "startMs": 242239,
    "text": "release paper claim about what their"
  },
  {
    "startMs": 244080,
    "text": "best score is?\" So the model's release"
  },
  {
    "startMs": 245680,
    "text": "paper said that they could score up to"
  },
  {
    "startMs": 247760,
    "text": "85% or even more. So we were just like,"
  },
  {
    "startMs": 249920,
    "text": "okay, okay, so we are not getting the"
  },
  {
    "startMs": 251920,
    "text": "sort of score. And often what we do with"
  },
  {
    "startMs": 254000,
    "text": "model releases in client is that we try"
  },
  {
    "startMs": 255760,
    "text": "to like optimize your score and try to"
  },
  {
    "startMs": 257440,
    "text": "get the best out of the model. So when"
  },
  {
    "startMs": 260400,
    "text": "we started hill climbing on the on the"
  },
  {
    "startMs": 262880,
    "text": "model like we we realized this we"
  },
  {
    "startMs": 265040,
    "text": "realized that like you know there's a"
  },
  {
    "startMs": 267040,
    "text": "lot of steps that we do says that we do"
  },
  {
    "startMs": 268880,
    "text": "it gracefully says that we pursue the"
  },
  {
    "startMs": 270560,
    "text": "right kind of problems and do it the"
  },
  {
    "startMs": 271919,
    "text": "right way. And we started thinking like"
  },
  {
    "startMs": 275280,
    "text": "okay if we there's a lot of ground work"
  },
  {
    "startMs": 277040,
    "text": "here how do we automate it and this is a"
  },
  {
    "startMs": 279120,
    "text": "demonstration of like how we automated"
  },
  {
    "startMs": 280639,
    "text": "it. So I'll I'll just leave that as a"
  },
  {
    "startMs": 282479,
    "text": "placeholder for now and we'll get get"
  },
  {
    "startMs": 283840,
    "text": "back to it later where we talk about in"
  },
  {
    "startMs": 285680,
    "text": "detail of like how does like this"
  },
  {
    "startMs": 287520,
    "text": "automatic improvement loop of a client"
  },
  {
    "startMs": 289759,
    "text": "harness work. So this basically means"
  },
  {
    "startMs": 292000,
    "text": "that like you have a coding agent"
  },
  {
    "startMs": 293040,
    "text": "harness that could just like basically"
  },
  {
    "startMs": 294320,
    "text": "improve itself and become better on on"
  },
  {
    "startMs": 296560,
    "text": "the same set of problems. So let's start"
  },
  {
    "startMs": 299199,
    "text": "with the ground terms. What is recursive"
  },
  {
    "startMs": 300880,
    "text": "self improvement? Recursive"
  },
  {
    "startMs": 301919,
    "text": "self-improvement is the idea that an AI"
  },
  {
    "startMs": 303680,
    "text": "system that helps [snorts] build a more"
  },
  {
    "startMs": 306160,
    "text": "capable successor which repeats the"
  },
  {
    "startMs": 308240,
    "text": "process you know until you hit"
  },
  {
    "startMs": 310320,
    "text": "artificial general intelligence right"
  },
  {
    "startMs": 311840,
    "text": "you have better systems you have better"
  },
  {
    "startMs": 312960,
    "text": "improvements you have a better system"
  },
  {
    "startMs": 314320,
    "text": "and then you know you have singularity"
  },
  {
    "startMs": 316400,
    "text": "so a classic example is like a"
  },
  {
    "startMs": 318000,
    "text": "self-reing coding agent right"
  },
  {
    "startMs": 319919,
    "text": "self-reving coding agent is just like a"
  },
  {
    "startMs": 321680,
    "text": "coding agent that can improve itself so"
  },
  {
    "startMs": 323759,
    "text": "you know this this is all like a pie in"
  },
  {
    "startMs": 325600,
    "text": "the sky idea like it's like it's like"
  },
  {
    "startMs": 327199,
    "text": "you know yeah sure this looks cool but"
  },
  {
    "startMs": 328800,
    "text": "how how did this even come out to be"
  },
  {
    "startMs": 330320,
    "text": "like how did we even get to here? So if"
  },
  {
    "startMs": 333919,
    "text": "you notice in topics road map in 2021 in"
  },
  {
    "startMs": 336479,
    "text": "2021's road map they very specifically"
  },
  {
    "startMs": 338639,
    "text": "said that uh uh they wanted to develop a"
  },
  {
    "startMs": 342240,
    "text": "general AI assistant that can interact"
  },
  {
    "startMs": 343759,
    "text": "conversationally with a user to create"
  },
  {
    "startMs": 346080,
    "text": "edit and explain software. It would"
  },
  {
    "startMs": 348000,
    "text": "display all the capabilities of a"
  },
  {
    "startMs": 349600,
    "text": "software engineer colleague. It would"
  },
  {
    "startMs": 350960,
    "text": "write functions. It would identify bugs."
  },
  {
    "startMs": 352320,
    "text": "It would design an API discuss style and"
  },
  {
    "startMs": 354880,
    "text": "best practices. Right? So we don't"
  },
  {
    "startMs": 356720,
    "text": "expect to reach this goal right away but"
  },
  {
    "startMs": 358160,
    "text": "it's in the direction which we're"
  },
  {
    "startMs": 359199,
    "text": "striving. It's interesting how this was"
  },
  {
    "startMs": 361280,
    "text": "more than five years ago. More than five"
  },
  {
    "startMs": 363199,
    "text": "years ago, people had predicted that"
  },
  {
    "startMs": 365280,
    "text": "there exist a day where you will have"
  },
  {
    "startMs": 367360,
    "text": "like a full-blown journal assistant that"
  },
  {
    "startMs": 369280,
    "text": "can basically act like a colleague,"
  },
  {
    "startMs": 370479,
    "text": "which we do have right now. Um, then in"
  },
  {
    "startMs": 373199,
    "text": "2022, Chad Deputy got released. We all"
  },
  {
    "startMs": 375280,
    "text": "know how that happened. It was it was a"
  },
  {
    "startMs": 377199,
    "text": "fun moment. It was like the first time"
  },
  {
    "startMs": 379039,
    "text": "things got sparked for us. Um, then in"
  },
  {
    "startMs": 382080,
    "text": "2023, 2024, there was the first time"
  },
  {
    "startMs": 384639,
    "text": "chat models could call functions. These"
  },
  {
    "startMs": 387440,
    "text": "functions would be simple functions like"
  },
  {
    "startMs": 389120,
    "text": "hey what's the weather or you know"
  },
  {
    "startMs": 391199,
    "text": "what's the what's you know what's this"
  },
  {
    "startMs": 393520,
    "text": "computation or what this computations"
  },
  {
    "startMs": 395360,
    "text": "result in python um and then finally in"
  },
  {
    "startMs": 398080,
    "text": "2024 when s 345 was really easy that was"
  },
  {
    "startMs": 400880,
    "text": "when the agents could like truly call"
  },
  {
    "startMs": 402560,
    "text": "like actual tools the way we call like"
  },
  {
    "startMs": 405680,
    "text": "think of tools so it's like reading a"
  },
  {
    "startMs": 407120,
    "text": "file editing a file you know reading"
  },
  {
    "startMs": 409600,
    "text": "searching something on web uh listing"
  },
  {
    "startMs": 411440,
    "text": "listing a folder searching and grapping"
  },
  {
    "startMs": 413440,
    "text": "through you know uh a file system things"
  },
  {
    "startMs": 415840,
    "text": "like that. So, and in 2024, we had the"
  },
  {
    "startMs": 419280,
    "text": "intuition that like chatbot, what if"
  },
  {
    "startMs": 421120,
    "text": "chatbots could actually talk to each"
  },
  {
    "startMs": 422800,
    "text": "other, you know? So, then that's was the"
  },
  {
    "startMs": 424479,
    "text": "first time when we had like, you know,"
  },
  {
    "startMs": 426639,
    "text": "uh multi-agent conversations. Now, at"
  },
  {
    "startMs": 428560,
    "text": "the time, these things were very janky,"
  },
  {
    "startMs": 430000,
    "text": "right? like they were just like taped"
  },
  {
    "startMs": 431680,
    "text": "together in this like weird weird uh"
  },
  {
    "startMs": 434880,
    "text": "bizarre ways where you have just like"
  },
  {
    "startMs": 436560,
    "text": "these classes and these links and then"
  },
  {
    "startMs": 438479,
    "text": "like these hierarchies of classes and it"
  },
  {
    "startMs": 440800,
    "text": "was like it was all like kind of like a"
  },
  {
    "startMs": 442160,
    "text": "very janky attempt to like make us think"
  },
  {
    "startMs": 444720,
    "text": "that like these these models could like"
  },
  {
    "startMs": 446560,
    "text": "have these pipeline of flows of"
  },
  {
    "startMs": 447840,
    "text": "responses that can all communicate with"
  },
  {
    "startMs": 449199,
    "text": "each other but a [snorts] lot of it was"
  },
  {
    "startMs": 451199,
    "text": "like mostly deterministic and it was"
  },
  {
    "startMs": 453120,
    "text": "like you know it was not as free flowing"
  },
  {
    "startMs": 454880,
    "text": "as it is right now. So, and you know, we"
  },
  {
    "startMs": 458639,
    "text": "started getting the intuition. So, this"
  },
  {
    "startMs": 460080,
    "text": "was the autogen project, but like in"
  },
  {
    "startMs": 461520,
    "text": "only in 2025, we we [snorts] started"
  },
  {
    "startMs": 463599,
    "text": "getting the like what if the agents just"
  },
  {
    "startMs": 465120,
    "text": "kept getting better and faster and"
  },
  {
    "startMs": 466400,
    "text": "cheaper, right? What if that just kept"
  },
  {
    "startMs": 468080,
    "text": "happening? And then in 2026, uh we we"
  },
  {
    "startMs": 471440,
    "text": "had the intuition that like what if what"
  },
  {
    "startMs": 474080,
    "text": "if the agent could improve itself, which"
  },
  {
    "startMs": 476000,
    "text": "is like the final, you know, the final"
  },
  {
    "startMs": 478160,
    "text": "part of the whole process. So, let me"
  },
  {
    "startMs": 480240,
    "text": "give an example like what does it mean"
  },
  {
    "startMs": 481440,
    "text": "for an agent to improve itself, right?"
  },
  {
    "startMs": 483120,
    "text": "So like it's like a lot of times when we"
  },
  {
    "startMs": 485199,
    "text": "think of like evaluating an agent, we"
  },
  {
    "startMs": 487759,
    "text": "initially used to think of evaluating an"
  },
  {
    "startMs": 489360,
    "text": "agent in the form of like just a single"
  },
  {
    "startMs": 491280,
    "text": "turn conversation. So we would tell we"
  },
  {
    "startMs": 493440,
    "text": "would tell like um um agents that like"
  },
  {
    "startMs": 496879,
    "text": "ask a simple question like how many toes"
  },
  {
    "startMs": 499039,
    "text": "does a dash animal has like it could be"
  },
  {
    "startMs": 501440,
    "text": "let's say how many toes does a cat have"
  },
  {
    "startMs": 503120,
    "text": "and then al would go through and the"
  },
  {
    "startMs": 504800,
    "text": "element would like prepare response and"
  },
  {
    "startMs": 506400,
    "text": "then we would get the response. So like"
  },
  {
    "startMs": 507840,
    "text": "a cat has maybe 10 toes or 18 toes,"
  },
  {
    "startMs": 510080,
    "text": "however many. I don't I actually don't"
  },
  {
    "startMs": 511520,
    "text": "know how many toes the cat has. But"
  },
  {
    "startMs": 513200,
    "text": "anyway, um yeah. So then we create the"
  },
  {
    "startMs": 517279,
    "text": "response as like a yes or no. But like"
  },
  {
    "startMs": 518880,
    "text": "when you work with an agent, there's a"
  },
  {
    "startMs": 520479,
    "text": "ton of things that you're doing. So let"
  },
  {
    "startMs": 522159,
    "text": "me give an example. Let me give an"
  },
  {
    "startMs": 523440,
    "text": "example. So for example, you're going to"
  },
  {
    "startMs": 525040,
    "text": "ask an agent like hey set up this MCP"
  },
  {
    "startMs": 527120,
    "text": "for me. So now it's not just like it's"
  },
  {
    "startMs": 529600,
    "text": "not just like you know a trivial oneable"
  },
  {
    "startMs": 532320,
    "text": "like thing like it will like search"
  },
  {
    "startMs": 533760,
    "text": "online and go oh what's the MCP? what is"
  },
  {
    "startMs": 536399,
    "text": "the config? So, I'll like edit and"
  },
  {
    "startMs": 538160,
    "text": "download some soft files. Like maybe I"
  },
  {
    "startMs": 540000,
    "text": "need to add database update the MCB"
  },
  {
    "startMs": 542160,
    "text": "tool. Maybe I need to read and edit some"
  },
  {
    "startMs": 544240,
    "text": "files. Maybe I need to run some bash"
  },
  {
    "startMs": 546000,
    "text": "commands. Right? So, you do all of that"
  },
  {
    "startMs": 548000,
    "text": "and then the agent response like, hey, I"
  },
  {
    "startMs": 549680,
    "text": "did it. And then you grab that with a"
  },
  {
    "startMs": 551680,
    "text": "bunch of tests to gauge like, hey, did"
  },
  {
    "startMs": 553839,
    "text": "the agent actually accomplish the thing"
  },
  {
    "startMs": 555680,
    "text": "that it said it would do. So to do that"
  },
  {
    "startMs": 558640,
    "text": "thing the second part which is like more"
  },
  {
    "startMs": 560320,
    "text": "intricate where you're actually like"
  },
  {
    "startMs": 562399,
    "text": "truly you know testing agents like uh to"
  },
  {
    "startMs": 565440,
    "text": "their best extent like it's like you"
  },
  {
    "startMs": 567680,
    "text": "know um we use this thing called"
  },
  {
    "startMs": 569760,
    "text": "Terminal-Bench Terminal-Bench is a"
  },
  {
    "startMs": 571440,
    "text": "benchmark for you know AI agents in"
  },
  {
    "startMs": 573200,
    "text": "terminal environments and that is"
  },
  {
    "startMs": 574959,
    "text": "something that we we have a lot of like"
  },
  {
    "startMs": 577120,
    "text": "experience with at client and uh it's"
  },
  {
    "startMs": 579519,
    "text": "like a it's like one of the best ways to"
  },
  {
    "startMs": 581120,
    "text": "like really gauge like is your coding"
  },
  {
    "startMs": 582800,
    "text": "agent accomplishing uh in general the"
  },
  {
    "startMs": 585200,
    "text": "task that you're doing uh that you want"
  },
  {
    "startMs": 587200,
    "text": "to be doing with a hudded model. So"
  },
  {
    "startMs": 589440,
    "text": "there's a couple there's a many"
  },
  {
    "startMs": 590880,
    "text": "different parts of a validation suit. So"
  },
  {
    "startMs": 593279,
    "text": "ideally you want that to run in um"
  },
  {
    "startMs": 596711,
    "text": "[snorts] you know you you want that to"
  },
  {
    "startMs": 598240,
    "text": "run in um individual sandboxes. Then you"
  },
  {
    "startMs": 601200,
    "text": "have like independent graders that like"
  },
  {
    "startMs": 603120,
    "text": "check everything like how many turns it"
  },
  {
    "startMs": 605120,
    "text": "took, how many tokens it took, how much"
  },
  {
    "startMs": 606560,
    "text": "latency it took, how many how many how"
  },
  {
    "startMs": 608720,
    "text": "many tool calls did it make, what was"
  },
  {
    "startMs": 610399,
    "text": "the trajectory looking like and then"
  },
  {
    "startMs": 612000,
    "text": "grading all the results for things like"
  },
  {
    "startMs": 613680,
    "text": "hey did you not reward hack? Did you not"
  },
  {
    "startMs": 615279,
    "text": "cheat through the solution? do not just"
  },
  {
    "startMs": 616640,
    "text": "like look up the solution online things"
  },
  {
    "startMs": 618640,
    "text": "like that. Um so we we basically like a"
  },
  {
    "startMs": 622240,
    "text": "lot of these uh data sets of evaluation"
  },
  {
    "startMs": 624959,
    "text": "problems have a set of tasks. So when"
  },
  {
    "startMs": 627600,
    "text": "here's a Terminal-Bench 2.1 they have 89"
  },
  {
    "startMs": 629600,
    "text": "tasks. So what we do is like we spin up"
  },
  {
    "startMs": 631839,
    "text": "89 parallel containers where they all"
  },
  {
    "startMs": 634480,
    "text": "run online and they run in parallel and"
  },
  {
    "startMs": 636800,
    "text": "they're sandbox and then they each try"
  },
  {
    "startMs": 639120,
    "text": "to solve a problem. So then they're all"
  },
  {
    "startMs": 641440,
    "text": "uh in parallel and independent of each"
  },
  {
    "startMs": 643279,
    "text": "other and then at the end we get get a"
  },
  {
    "startMs": 645839,
    "text": "score. So our score could like give us"
  },
  {
    "startMs": 647680,
    "text": "measures like is is a is this is the"
  },
  {
    "startMs": 650560,
    "text": "model performing well and what kind of"
  },
  {
    "startMs": 652880,
    "text": "failures it has right? So like you run"
  },
  {
    "startMs": 654480,
    "text": "the evaluation you get a score. So you"
  },
  {
    "startMs": 656240,
    "text": "get an aggregate score and there"
  },
  {
    "startMs": 658320,
    "text": "aggregate score could reveal a set of"
  },
  {
    "startMs": 660079,
    "text": "failures. Okay failures could be of any"
  },
  {
    "startMs": 661680,
    "text": "kind. Failures could be hey the read"
  },
  {
    "startMs": 663680,
    "text": "file tool is not working. Sometimes the"
  },
  {
    "startMs": 665600,
    "text": "entrance is so slow that like you know"
  },
  {
    "startMs": 667600,
    "text": "the agent's working it's doing it's"
  },
  {
    "startMs": 669200,
    "text": "making progress maybe it's running a"
  },
  {
    "startMs": 670880,
    "text": "circle but it's so slow that it"
  },
  {
    "startMs": 672160,
    "text": "literally like runs out of time it runs"
  },
  {
    "startMs": 673760,
    "text": "out of like 45 minutes or where many"
  },
  {
    "startMs": 675680,
    "text": "that were given for the task and just"
  },
  {
    "startMs": 677200,
    "text": "like stop working. So once you do that"
  },
  {
    "startMs": 679839,
    "text": "once you understand hey this is what we"
  },
  {
    "startMs": 681519,
    "text": "ran this is what the failures we got we"
  },
  {
    "startMs": 683440,
    "text": "bucket the failures in different"
  },
  {
    "startMs": 684959,
    "text": "categories. So like 10 failures of you"
  },
  {
    "startMs": 686880,
    "text": "know this kind of tool called 10 10 20"
  },
  {
    "startMs": 688720,
    "text": "failures of like say inference not"
  },
  {
    "startMs": 690160,
    "text": "working or something else right and then"
  },
  {
    "startMs": 691760,
    "text": "we once we bucket the failures into"
  },
  {
    "startMs": 693360,
    "text": "different categories where like we try"
  },
  {
    "startMs": 694800,
    "text": "to make an improvement to the harness"
  },
  {
    "startMs": 696240,
    "text": "because like what if there's like"
  },
  {
    "startMs": 698079,
    "text": "something wrong with the harness that we"
  },
  {
    "startMs": 699519,
    "text": "could change such that the model score"
  },
  {
    "startMs": 701120,
    "text": "would improve right and on and on and on"
  },
  {
    "startMs": 703120,
    "text": "and you do this iteratively it takes a"
  },
  {
    "startMs": 704800,
    "text": "while it's just like you know hill"
  },
  {
    "startMs": 706240,
    "text": "climbing is like a very groundworky kind"
  },
  {
    "startMs": 708320,
    "text": "of process and you know usually usually"
  },
  {
    "startMs": 711279,
    "text": "we we took we used to take a lot of time"
  },
  {
    "startMs": 713519,
    "text": "trying to do this but it was a great way"
  },
  {
    "startMs": 715040,
    "text": "for us to like learn how to improve"
  },
  {
    "startMs": 716959,
    "text": "models. So [snorts]"
  },
  {
    "startMs": 718640,
    "text": "one way to hill climb on public eval is"
  },
  {
    "startMs": 721120,
    "text": "that like you know we we could have a"
  },
  {
    "startMs": 724240,
    "text": "thing where you know we started doing"
  },
  {
    "startMs": 726800,
    "text": "this thing where we could just like what"
  },
  {
    "startMs": 729440,
    "text": "if we could have like a longunning agent"
  },
  {
    "startMs": 731200,
    "text": "that runs for a while and automatically"
  },
  {
    "startMs": 733040,
    "text": "improves the model score and that's what"
  },
  {
    "startMs": 734720,
    "text": "we did. So we ran GPD 5.6 soul that led"
  },
  {
    "startMs": 738320,
    "text": "the work and that was like the main"
  },
  {
    "startMs": 739519,
    "text": "agent and then uh Ky K3 was just running"
  },
  {
    "startMs": 742480,
    "text": "eval. So what it was doing is that like"
  },
  {
    "startMs": 744399,
    "text": "GBD 5.6 was like hey run these events"
  },
  {
    "startMs": 747360,
    "text": "get these scores let me look at the"
  },
  {
    "startMs": 749200,
    "text": "failures let me look at where where it"
  },
  {
    "startMs": 751200,
    "text": "went wrong and then I'll make changes to"
  },
  {
    "startMs": 753200,
    "text": "client's harness and then I'll run these"
  },
  {
    "startMs": 755279,
    "text": "eval again and then I'll keep trying. So"
  },
  {
    "startMs": 757760,
    "text": "again, you can see how like very quickly"
  },
  {
    "startMs": 759600,
    "text": "this can like go off the rails because"
  },
  {
    "startMs": 761279,
    "text": "this is supposed to be like the running"
  },
  {
    "startMs": 762639,
    "text": "and evaluation itself takes like an"
  },
  {
    "startMs": 764240,
    "text": "hour. Then you have to read the"
  },
  {
    "startMs": 765680,
    "text": "feedback. Then you have to like see if"
  },
  {
    "startMs": 767760,
    "text": "you're properly interpreting the"
  },
  {
    "startMs": 768959,
    "text": "feedback to make reasonable changes to"
  },
  {
    "startMs": 770800,
    "text": "the harness that are like not reward"
  },
  {
    "startMs": 772399,
    "text": "hacking that are just like actual"
  },
  {
    "startMs": 773519,
    "text": "genuine improvements and then run the"
  },
  {
    "startMs": 774800,
    "text": "eval again. So we ran that for 17 hours"
  },
  {
    "startMs": 777519,
    "text": "and uh when we ran that for 17 hours"
  },
  {
    "startMs": 780320,
    "text": "there was like a legitimate hell"
  },
  {
    "startMs": 781680,
    "text": "climbing and what we found that was very"
  },
  {
    "startMs": 783760,
    "text": "interesting. This happened with a very"
  },
  {
    "startMs": 785839,
    "text": "detailed prompt that accomplished this"
  },
  {
    "startMs": 787920,
    "text": "is that like we had baseline score of"
  },
  {
    "startMs": 790480,
    "text": "27% and then you know we we found issues"
  },
  {
    "startMs": 794720,
    "text": "like loop detection. Loop detection is"
  },
  {
    "startMs": 796240,
    "text": "when a harness just runs in circles"
  },
  {
    "startMs": 797839,
    "text": "around a problem right uh we found like"
  },
  {
    "startMs": 800240,
    "text": "background process grinding where"
  },
  {
    "startMs": 801600,
    "text": "background processes would run too long."
  },
  {
    "startMs": 803760,
    "text": "Uh we found some file index issues and"
  },
  {
    "startMs": 805920,
    "text": "other things as well. I mean when I say"
  },
  {
    "startMs": 807839,
    "text": "we what I actually meant is GP 5.6 solve"
  },
  {
    "startMs": 810320,
    "text": "right? So that found those issues that"
  },
  {
    "startMs": 812399,
    "text": "found and bucketed those issues. It's"
  },
  {
    "startMs": 814000,
    "text": "like hey how about I make those changes"
  },
  {
    "startMs": 816240,
    "text": "in the harness and I run the eval again"
  },
  {
    "startMs": 818079,
    "text": "and it iteratively built the hill"
  },
  {
    "startMs": 819760,
    "text": "climbing process completely autonomously"
  },
  {
    "startMs": 821519,
    "text": "on its own without my interference and"
  },
  {
    "startMs": 823519,
    "text": "13 hours later when I come back to it it"
  },
  {
    "startMs": 825360,
    "text": "like it has improved the harness's"
  },
  {
    "startMs": 827120,
    "text": "performance in so many ways. So see what"
  },
  {
    "startMs": 830240,
    "text": "does that mean? What does that mean? It"
  },
  {
    "startMs": 831680,
    "text": "just it just basically means that like"
  },
  {
    "startMs": 834000,
    "text": "we're at a point where the experience"
  },
  {
    "startMs": 836959,
    "text": "could make the harness better. That like"
  },
  {
    "startMs": 838639,
    "text": "there's like a two-level learning loop"
  },
  {
    "startMs": 840240,
    "text": "where the harness does some action,"
  },
  {
    "startMs": 842639,
    "text": "interacts with the environment,"
  },
  {
    "startMs": 844079,
    "text": "understands whether or not it did the"
  },
  {
    "startMs": 845680,
    "text": "right thing or the wrong thing and gets"
  },
  {
    "startMs": 847519,
    "text": "some reward and and like almost and then"
  },
  {
    "startMs": 850240,
    "text": "has there's like a meta agent that's"
  },
  {
    "startMs": 852079,
    "text": "observing the main harness and finding"
  },
  {
    "startMs": 854880,
    "text": "the changes to it and just like"
  },
  {
    "startMs": 856560,
    "text": "incorporating those changes and"
  },
  {
    "startMs": 857839,
    "text": "improving. Now it's I think in my"
  },
  {
    "startMs": 859760,
    "text": "personal opinion it's much harder to do"
  },
  {
    "startMs": 861199,
    "text": "in real time because if you do it in"
  },
  {
    "startMs": 863040,
    "text": "real time you overindex on solving"
  },
  {
    "startMs": 865360,
    "text": "certain problems or you're just like too"
  },
  {
    "startMs": 868240,
    "text": "cartel to a specific thing. But I do"
  },
  {
    "startMs": 870560,
    "text": "think that this is something that works"
  },
  {
    "startMs": 872000,
    "text": "like on a broader scale of like working"
  },
  {
    "startMs": 873440,
    "text": "on a wide range of problems"
  },
  {
    "startMs": 876000,
    "text": "and you want to be able to like you want"
  },
  {
    "startMs": 878160,
    "text": "you want to be able to justify the"
  },
  {
    "startMs": 879600,
    "text": "changes you're making on a broader act"
  },
  {
    "startMs": 882079,
    "text": "applicability so that the changes you"
  },
  {
    "startMs": 884240,
    "text": "make here will apply to a broader range"
  },
  {
    "startMs": 886320,
    "text": "of problems as well. So what does that"
  },
  {
    "startMs": 889600,
    "text": "mean? Like what am I trying to get at"
  },
  {
    "startMs": 892399,
    "text": "here? The point I'm trying to get at is"
  },
  {
    "startMs": 894399,
    "text": "that the models are so good right now"
  },
  {
    "startMs": 895839,
    "text": "that they can if they can improve the"
  },
  {
    "startMs": 897199,
    "text": "harness, they can improve a lot of other"
  },
  {
    "startMs": 898560,
    "text": "things. So then what else can they"
  },
  {
    "startMs": 900000,
    "text": "improve? So this is this is a this is a"
  },
  {
    "startMs": 902880,
    "text": "post uh primarily from OpenI where GP2"
  },
  {
    "startMs": 905680,
    "text": "5.6 autonomously rewrote and optimized"
  },
  {
    "startMs": 908560,
    "text": "its own production kernels and those"
  },
  {
    "startMs": 910959,
    "text": "production kernel improvements remove"
  },
  {
    "startMs": 913040,
    "text": "the end to-end serving costs by 20%. you"
  },
  {
    "startMs": 916079,
    "text": "know it also had 15% higher to token"
  },
  {
    "startMs": 919279,
    "text": "generation efficiency and this was like"
  },
  {
    "startMs": 920959,
    "text": "completely done by 5.6 soul on itself,"
  },
  {
    "startMs": 924000,
    "text": "right? So what that means is that like"
  },
  {
    "startMs": 926399,
    "text": "inference is becoming much cheaper and"
  },
  {
    "startMs": 928399,
    "text": "that means that like if on if if for a"
  },
  {
    "startMs": 930720,
    "text": "company like OpenAI if you're having 20%"
  },
  {
    "startMs": 933360,
    "text": "low lower serving cost for a frontier"
  },
  {
    "startMs": 935920,
    "text": "model that could be a billions of"
  },
  {
    "startMs": 937680,
    "text": "dollars of improvement in price and you"
  },
  {
    "startMs": 939360,
    "text": "could you could do that as well for your"
  },
  {
    "startMs": 940880,
    "text": "own models. Um another interesting thing"
  },
  {
    "startMs": 944160,
    "text": "is that like when people have kernel"
  },
  {
    "startMs": 945680,
    "text": "writing uh competitions so core auto is"
  },
  {
    "startMs": 947839,
    "text": "a company which has a kernel writing"
  },
  {
    "startMs": 949759,
    "text": "competitions and you'll notice that like"
  },
  {
    "startMs": 952240,
    "text": "they they mentioned that like the person"
  },
  {
    "startMs": 954720,
    "text": "who won the kernel writing competition"
  },
  {
    "startMs": 956480,
    "text": "the explic said that I have never"
  },
  {
    "startMs": 958560,
    "text": "handwritten a GPU operator it's all ALM"
  },
  {
    "startMs": 961040,
    "text": "and agent driven so notice that like how"
  },
  {
    "startMs": 963839,
    "text": "like the best uh inference improvements"
  },
  {
    "startMs": 966399,
    "text": "kernel improvements are at this point"
  },
  {
    "startMs": 968160,
    "text": "completely exclusively done by a swarm"
  },
  {
    "startMs": 970399,
    "text": "of agents or generally like agents doing"
  },
  {
    "startMs": 972959,
    "text": "some magic to you know magically improve"
  },
  {
    "startMs": 975040,
    "text": "like the kernel kernel code and have"
  },
  {
    "startMs": 977279,
    "text": "faster inference cheaper inference. Um"
  },
  {
    "startMs": 980160,
    "text": "yeah so basically like the improvement"
  },
  {
    "startMs": 983279,
    "text": "surface that you think of is not just"
  },
  {
    "startMs": 985440,
    "text": "like the harness improvement it's"
  },
  {
    "startMs": 987040,
    "text": "literally the whole surface stack right"
  },
  {
    "startMs": 988880,
    "text": "so the whole stack from like evaluation"
  },
  {
    "startMs": 991600,
    "text": "to harness improvements to training data"
  },
  {
    "startMs": 993600,
    "text": "to post training to world interaction to"
  },
  {
    "startMs": 995360,
    "text": "inference like literally the entire pip"
  },
  {
    "startMs": 998079,
    "text": "part of the pipeline like all the way"
  },
  {
    "startMs": 1000000,
    "text": "from like electricity going through your"
  },
  {
    "startMs": 1003120,
    "text": "machine to like making a network request"
  },
  {
    "startMs": 1005759,
    "text": "going through the model models doing"
  },
  {
    "startMs": 1007440,
    "text": "inference um making a response and"
  },
  {
    "startMs": 1009680,
    "text": "getting back to you like every single"
  },
  {
    "startMs": 1011120,
    "text": "part of the pipeline we can optimize it."
  },
  {
    "startMs": 1013199,
    "text": "So the whole RSI loop has like um"
  },
  {
    "startMs": 1015360,
    "text": "there's a couple different parts so it's"
  },
  {
    "startMs": 1016800,
    "text": "like pre-training um uh neatronic"
  },
  {
    "startMs": 1019279,
    "text": "examples of like you know AI curated uh"
  },
  {
    "startMs": 1021600,
    "text": "pre-training data that was good um"
  },
  {
    "startMs": 1023519,
    "text": "there's like post- training examples"
  },
  {
    "startMs": 1024959,
    "text": "which are you know where there's like"
  },
  {
    "startMs": 1027679,
    "text": "constitutional AI examples of like post-"
  },
  {
    "startMs": 1030000,
    "text": "training uh improvements then inference"
  },
  {
    "startMs": 1032079,
    "text": "I gave you the example of GP 5.6 soul"
  },
  {
    "startMs": 1034799,
    "text": "for eval and eval generation. I give you"
  },
  {
    "startMs": 1036959,
    "text": "our own example of how we improved"
  },
  {
    "startMs": 1038640,
    "text": "ourselves on our own eval. Um in general"
  },
  {
    "startMs": 1041600,
    "text": "what you want to think of is that like"
  },
  {
    "startMs": 1043520,
    "text": "anytime a model is trying to do"
  },
  {
    "startMs": 1044959,
    "text": "something and it's like not able to do"
  },
  {
    "startMs": 1046480,
    "text": "it or not able to accomplish it. So for"
  },
  {
    "startMs": 1048480,
    "text": "example like imagine you're having a"
  },
  {
    "startMs": 1050080,
    "text": "long conversation where you ask the"
  },
  {
    "startMs": 1051679,
    "text": "model something obvious. So like it's"
  },
  {
    "startMs": 1053520,
    "text": "like or something that you think is not"
  },
  {
    "startMs": 1056480,
    "text": "necessarily obvious but it's like it's"
  },
  {
    "startMs": 1058720,
    "text": "hard but you would assume that the model"
  },
  {
    "startMs": 1060480,
    "text": "could potentially do it. So to give an"
  },
  {
    "startMs": 1062080,
    "text": "example, let's say you want to make uh"
  },
  {
    "startMs": 1064240,
    "text": "add a new column to a database and then"
  },
  {
    "startMs": 1066799,
    "text": "fill that column, right? And then put a"
  },
  {
    "startMs": 1069440,
    "text": "cache in front of it. So what happens is"
  },
  {
    "startMs": 1071679,
    "text": "it does the whole thing, but for some"
  },
  {
    "startMs": 1073120,
    "text": "reason the way it pushes the cache in"
  },
  {
    "startMs": 1074960,
    "text": "front of the whole uh database like it"
  },
  {
    "startMs": 1076720,
    "text": "kind of doesn't do it right. So at that"
  },
  {
    "startMs": 1078000,
    "text": "point you're like, \"Oh no, set this"
  },
  {
    "startMs": 1080000,
    "text": "cache differently. Set this as the TTL.\""
  },
  {
    "startMs": 1082000,
    "text": "And then you give like a correction"
  },
  {
    "startMs": 1083280,
    "text": "signal and then that kind of like fixes"
  },
  {
    "startMs": 1085840,
    "text": "your problem and you kind of go about"
  },
  {
    "startMs": 1087120,
    "text": "your day."
  },
  {
    "startMs": 1088160,
    "text": "The way I want to think of it is that I"
  },
  {
    "startMs": 1089840,
    "text": "want to think of like every single time"
  },
  {
    "startMs": 1091600,
    "text": "like a human correction to the digital"
  },
  {
    "startMs": 1094320,
    "text": "landscape is a part of a delta that"
  },
  {
    "startMs": 1098480,
    "text": "needs to be covered. And what that means"
  },
  {
    "startMs": 1101120,
    "text": "is that like every single time you're"
  },
  {
    "startMs": 1102720,
    "text": "finding like the edge of model"
  },
  {
    "startMs": 1104320,
    "text": "capabilities of like how far it can push"
  },
  {
    "startMs": 1106080,
    "text": "for infants, how much it can push for"
  },
  {
    "startMs": 1107440,
    "text": "this, how much it can push for, you"
  },
  {
    "startMs": 1109760,
    "text": "know, solving your problems. You want to"
  },
  {
    "startMs": 1112320,
    "text": "find that delta and we want to like push"
  },
  {
    "startMs": 1114160,
    "text": "that frontier a little bit further. a"
  },
  {
    "startMs": 1116160,
    "text": "little bit further and that's what post"
  },
  {
    "startMs": 1118080,
    "text": "training does, right? So post training"
  },
  {
    "startMs": 1119520,
    "text": "is is this is in that post training we"
  },
  {
    "startMs": 1122960,
    "text": "find these like really interesting and"
  },
  {
    "startMs": 1125039,
    "text": "really hard long horizon problems for uh"
  },
  {
    "startMs": 1127200,
    "text": "for coding agents or for other kinds of"
  },
  {
    "startMs": 1128720,
    "text": "problems and then just like slowly hill"
  },
  {
    "startMs": 1130320,
    "text": "climb their models on on those kind of"
  },
  {
    "startMs": 1132559,
    "text": "problems. So anything that has a"
  },
  {
    "startMs": 1134240,
    "text": "verifiable loop can be optimized but is"
  },
  {
    "startMs": 1138080,
    "text": "this is not just limited to software"
  },
  {
    "startMs": 1140400,
    "text": "like literally like the physical world"
  },
  {
    "startMs": 1142080,
    "text": "itself could have a verifiable loop. So"
  },
  {
    "startMs": 1144960,
    "text": "to give an example like let me let me"
  },
  {
    "startMs": 1146640,
    "text": "give you a very funny kind of bizarre"
  },
  {
    "startMs": 1148559,
    "text": "example where like it's like you you"
  },
  {
    "startMs": 1151679,
    "text": "look at this and u you um"
  },
  {
    "startMs": 1155679,
    "text": "um look at this uh like robot and it's"
  },
  {
    "startMs": 1158160,
    "text": "it's typing something and it's like"
  },
  {
    "startMs": 1160559,
    "text": "really really like interesting what the"
  },
  {
    "startMs": 1162960,
    "text": "robot is typing."
  },
  {
    "startMs": 1172799,
    "text": "So notice how like notice how the robot"
  },
  {
    "startMs": 1174960,
    "text": "is itself typing like how it wants to"
  },
  {
    "startMs": 1177200,
    "text": "like solve robotics and imagine if the"
  },
  {
    "startMs": 1179600,
    "text": "robot itself could like interpret the"
  },
  {
    "startMs": 1181600,
    "text": "results of of like the answers and just"
  },
  {
    "startMs": 1184640,
    "text": "like keep improving then that means that"
  },
  {
    "startMs": 1186559,
    "text": "like this is you know basically like an"
  },
  {
    "startMs": 1188400,
    "text": "ASI robot. Another example that I could"
  },
  {
    "startMs": 1190480,
    "text": "give you from my uh personal life is"
  },
  {
    "startMs": 1193120,
    "text": "that like um I have this um chip that"
  },
  {
    "startMs": 1196320,
    "text": "has the ability to see and has a speaker"
  },
  {
    "startMs": 1199120,
    "text": "and has a microphone inside of it. So I"
  },
  {
    "startMs": 1200880,
    "text": "was trying to set up this uh with my"
  },
  {
    "startMs": 1203440,
    "text": "machine and it was um a gift that I had"
  },
  {
    "startMs": 1207440,
    "text": "for my girlfriend. And uh what I found"
  },
  {
    "startMs": 1209360,
    "text": "very interesting about this is as I was"
  },
  {
    "startMs": 1211520,
    "text": "trying to set it up, it was very slow."
  },
  {
    "startMs": 1213360,
    "text": "It was slow to respond. It had higher"
  },
  {
    "startMs": 1214960,
    "text": "latency. It had an agent in and of"
  },
  {
    "startMs": 1216480,
    "text": "itself. Right? So what I did was I"
  },
  {
    "startMs": 1218640,
    "text": "connected the step to my um computer and"
  },
  {
    "startMs": 1222480,
    "text": "I let um I let um coding agents run on"
  },
  {
    "startMs": 1227039,
    "text": "this hardware and basically run on this"
  },
  {
    "startMs": 1229679,
    "text": "hardware automatically make tests to it."
  },
  {
    "startMs": 1232880,
    "text": "So like my computer would talk to my"
  },
  {
    "startMs": 1235760,
    "text": "hardware give it some tests like hey do"
  },
  {
    "startMs": 1238080,
    "text": "this do that and just like what do you"
  },
  {
    "startMs": 1239840,
    "text": "see right now? So then it would notice"
  },
  {
    "startMs": 1242480,
    "text": "the end to end latency of like the"
  },
  {
    "startMs": 1244080,
    "text": "inference response of like how fast uh"
  },
  {
    "startMs": 1246400,
    "text": "my hardware is responding and then um"
  },
  {
    "startMs": 1249360,
    "text": "what I found is that like it was able to"
  },
  {
    "startMs": 1251360,
    "text": "do the breakdown of like every single"
  },
  {
    "startMs": 1253679,
    "text": "case and scenario where there was like"
  },
  {
    "startMs": 1255520,
    "text": "high latency um slower response or or"
  },
  {
    "startMs": 1259600,
    "text": "just the algorithm wasn't right and I"
  },
  {
    "startMs": 1261919,
    "text": "could then set up a recursive loop where"
  },
  {
    "startMs": 1265039,
    "text": "both the firmware could be updated the"
  },
  {
    "startMs": 1267520,
    "text": "um the tool calls the inference model"
  },
  {
    "startMs": 1270240,
    "text": "used um the speech to text model used"
  },
  {
    "startMs": 1272799,
    "text": "all of those could be improved"
  },
  {
    "startMs": 1273840,
    "text": "automatically with like an OpenRouter"
  },
  {
    "startMs": 1275679,
    "text": "key which I'd given to a coding agent"
  },
  {
    "startMs": 1277520,
    "text": "and then overnight with self-running"
  },
  {
    "startMs": 1279360,
    "text": "experiments with the hardware uh my um"
  },
  {
    "startMs": 1282720,
    "text": "this tiny little piece of beautiful"
  },
  {
    "startMs": 1284080,
    "text": "hardware that I have was like deeply"
  },
  {
    "startMs": 1286080,
    "text": "optimized so was this also like even"
  },
  {
    "startMs": 1288480,
    "text": "hardware could be a part of the RSI loop"
  },
  {
    "startMs": 1290400,
    "text": "so what happens when you connect these"
  },
  {
    "startMs": 1292080,
    "text": "loops right that's the next question"
  },
  {
    "startMs": 1293280,
    "text": "like you could have cheaper inference"
  },
  {
    "startMs": 1295200,
    "text": "you could have like more experiments"
  },
  {
    "startMs": 1296559,
    "text": "budget you have like you know better"
  },
  {
    "startMs": 1298960,
    "text": "data and better models. So, is it"
  },
  {
    "startMs": 1300960,
    "text": "singularity? It's a very question to ask"
  },
  {
    "startMs": 1303520,
    "text": "and I think that the part of the parts"
  },
  {
    "startMs": 1304880,
    "text": "of the loop are already there, right?"
  },
  {
    "startMs": 1306240,
    "text": "Like we're we we are really we really"
  },
  {
    "startMs": 1308480,
    "text": "are very close to parts of the RSI loop,"
  },
  {
    "startMs": 1311360,
    "text": "right? And what do we do like what do we"
  },
  {
    "startMs": 1314000,
    "text": "do about this and it's like how do we"
  },
  {
    "startMs": 1315520,
    "text": "think of that in terms of like our"
  },
  {
    "startMs": 1317840,
    "text": "career and how do you think of that in"
  },
  {
    "startMs": 1319280,
    "text": "terms of you know our lives and what do"
  },
  {
    "startMs": 1320799,
    "text": "we want to do next? So here's my advice"
  },
  {
    "startMs": 1322880,
    "text": "to everyone who is watching this like"
  },
  {
    "startMs": 1324720,
    "text": "please please please please please move"
  },
  {
    "startMs": 1326400,
    "text": "closer to the RS right so it's like"
  },
  {
    "startMs": 1329200,
    "text": "right now like people are working on all"
  },
  {
    "startMs": 1331039,
    "text": "these like different kinds of software"
  },
  {
    "startMs": 1332559,
    "text": "which is like you know a agents for you"
  },
  {
    "startMs": 1334480,
    "text": "know construction agents for this agents"
  },
  {
    "startMs": 1336000,
    "text": "for that and it's like I think that"
  },
  {
    "startMs": 1337200,
    "text": "you're you know what the way I see those"
  },
  {
    "startMs": 1341280,
    "text": "things is that like I think of those as"
  },
  {
    "startMs": 1342640,
    "text": "like you're missing the forest for the"
  },
  {
    "startMs": 1344080,
    "text": "for the trees you really are and it's"
  },
  {
    "startMs": 1345919,
    "text": "like you have this like very interesting"
  },
  {
    "startMs": 1349440,
    "text": "godlike technology that can change parts"
  },
  {
    "startMs": 1351679,
    "text": "of humanity. And then we're not"
  },
  {
    "startMs": 1354000,
    "text": "ambitious enough to pursue the best"
  },
  {
    "startMs": 1355679,
    "text": "problems. We really are not. And we're"
  },
  {
    "startMs": 1357840,
    "text": "just like I think that it's not just"
  },
  {
    "startMs": 1359760,
    "text": "like the problem isn't just that we're"
  },
  {
    "startMs": 1361919,
    "text": "not ambitious and we're also like what"
  },
  {
    "startMs": 1363679,
    "text": "we're pursuing. A lot of those things"
  },
  {
    "startMs": 1366080,
    "text": "are just like one model release away"
  },
  {
    "startMs": 1367919,
    "text": "from like being completely like wiped"
  },
  {
    "startMs": 1369600,
    "text": "out, right? So like it's also like it's"
  },
  {
    "startMs": 1371919,
    "text": "also like it may not be that"
  },
  {
    "startMs": 1373679,
    "text": "intellectually simulating because like"
  },
  {
    "startMs": 1375120,
    "text": "all you're essentially all we're"
  },
  {
    "startMs": 1376400,
    "text": "essentially doing is like sitting all"
  },
  {
    "startMs": 1377520,
    "text": "day. It's like hey cloud do this for me."
  },
  {
    "startMs": 1378799,
    "text": "Hey cloud do this for me. And I don't"
  },
  {
    "startMs": 1380080,
    "text": "know if that's like I don't know if"
  },
  {
    "startMs": 1381200,
    "text": "that's like the most intellectual"
  },
  {
    "startMs": 1382720,
    "text": "stimulating or like or like um"
  },
  {
    "startMs": 1385039,
    "text": "thoughtful use of a person's time like"
  },
  {
    "startMs": 1387120,
    "text": "maybe maybe you know we're supposed to"
  },
  {
    "startMs": 1388720,
    "text": "be solving harder and more uh more"
  },
  {
    "startMs": 1390480,
    "text": "frontier problems and I think that like"
  },
  {
    "startMs": 1392080,
    "text": "it's like the ceiling is like our"
  },
  {
    "startMs": 1393760,
    "text": "ambition. So instead of like working on"
  },
  {
    "startMs": 1395919,
    "text": "these kind of things my suggestion to"
  },
  {
    "startMs": 1397440,
    "text": "people is to like you know do things"
  },
  {
    "startMs": 1400240,
    "text": "that of course like goes without saying"
  },
  {
    "startMs": 1402159,
    "text": "but you should say things you believe in"
  },
  {
    "startMs": 1403520,
    "text": "and you know I'm just a guy on the"
  },
  {
    "startMs": 1404960,
    "text": "internet but do things that help you"
  },
  {
    "startMs": 1407280,
    "text": "move closer to the RSI loop right? So"
  },
  {
    "startMs": 1409520,
    "text": "like anything that just like the end"
  },
  {
    "startMs": 1411120,
    "text": "to-end pipeline of like hey this is"
  },
  {
    "startMs": 1414480,
    "text": "where like the models are lacking like"
  },
  {
    "startMs": 1416320,
    "text": "it's like whether that's inference"
  },
  {
    "startMs": 1417919,
    "text": "whether that's post training whether"
  },
  {
    "startMs": 1419039,
    "text": "that's literally like physical"
  },
  {
    "startMs": 1420240,
    "text": "interaction whether that could be you"
  },
  {
    "startMs": 1422480,
    "text": "know send sending end to-end pipelines"
  },
  {
    "startMs": 1424720,
    "text": "of truly any kind you know whether"
  },
  {
    "startMs": 1426480,
    "text": "that's hardware problems whether that's"
  },
  {
    "startMs": 1428240,
    "text": "um networking problems whether that's um"
  },
  {
    "startMs": 1430480,
    "text": "building a sense of community things"
  },
  {
    "startMs": 1431760,
    "text": "that just you know um things that just"
  },
  {
    "startMs": 1434159,
    "text": "like help um you as an engineer get"
  },
  {
    "startMs": 1437360,
    "text": "closer to the benefit benefits of the"
  },
  {
    "startMs": 1439919,
    "text": "improvements um in uh in u um in in the"
  },
  {
    "startMs": 1444240,
    "text": "model. So this is a very interesting"
  },
  {
    "startMs": 1445919,
    "text": "slide from Sasha topic. So um um they"
  },
  {
    "startMs": 1450159,
    "text": "mentioned that like you want to have"
  },
  {
    "startMs": 1453120,
    "text": "like this you're in this space where"
  },
  {
    "startMs": 1455679,
    "text": "it's like you know you only you only"
  },
  {
    "startMs": 1457039,
    "text": "have like 1 to five years left to"
  },
  {
    "startMs": 1458400,
    "text": "contribute to luxury to the world what"
  },
  {
    "startMs": 1459840,
    "text": "do you do right so it's like um they"
  },
  {
    "startMs": 1461919,
    "text": "they give great suggestions where like"
  },
  {
    "startMs": 1463600,
    "text": "prefer ambitious targeted and past"
  },
  {
    "startMs": 1465600,
    "text": "collaborations right and you know in"
  },
  {
    "startMs": 1468799,
    "text": "general like the time scale like all the"
  },
  {
    "startMs": 1470559,
    "text": "timelines that you give to yourself like"
  },
  {
    "startMs": 1472000,
    "text": "they should be just like completely"
  },
  {
    "startMs": 1473039,
    "text": "squeezed out should be completely"
  },
  {
    "startMs": 1474320,
    "text": "squeezed out because it's like a lot of"
  },
  {
    "startMs": 1475679,
    "text": "times and you also have to think through"
  },
  {
    "startMs": 1477279,
    "text": "that like if if the model if the problem"
  },
  {
    "startMs": 1479039,
    "text": "that you're working on is too ambitious"
  },
  {
    "startMs": 1480400,
    "text": "like it's like is it probably is it"
  },
  {
    "startMs": 1483120,
    "text": "probably better to wait for just like"
  },
  {
    "startMs": 1484400,
    "text": "the next next frontier model to come out"
  },
  {
    "startMs": 1486480,
    "text": "or not like that's that's something you"
  },
  {
    "startMs": 1488080,
    "text": "also have to weigh through right and uh"
  },
  {
    "startMs": 1491679,
    "text": "you you ideally want to work on problems"
  },
  {
    "startMs": 1493840,
    "text": "that benefit from scale and you know you"
  },
  {
    "startMs": 1496159,
    "text": "also at the same time want to avoid"
  },
  {
    "startMs": 1497440,
    "text": "projects that can be solved by sale"
  },
  {
    "startMs": 1498720,
    "text": "alone because then you know if even if"
  },
  {
    "startMs": 1500480,
    "text": "you like it's like if you're working on"
  },
  {
    "startMs": 1501840,
    "text": "a problem and then the next model"
  },
  {
    "startMs": 1503039,
    "text": "release comes out it solves your problem"
  },
  {
    "startMs": 1504080,
    "text": "so that's not that's probably not a good"
  },
  {
    "startMs": 1505440,
    "text": "thing to do so it's very like you have"
  },
  {
    "startMs": 1507440,
    "text": "to like think philosophically of like"
  },
  {
    "startMs": 1509360,
    "text": "what is the endeavor that benefits uh"
  },
  {
    "startMs": 1512480,
    "text": "greatly from auto improvements but like"
  },
  {
    "startMs": 1514000,
    "text": "isn't also completely wiped out from it"
  },
  {
    "startMs": 1516000,
    "text": "and then ideally like it's like if the"
  },
  {
    "startMs": 1518559,
    "text": "if the endeavor could also uh be a part"
  },
  {
    "startMs": 1520960,
    "text": "of the the like the improvement pipeline"
  },
  {
    "startMs": 1524480,
    "text": "uh that could be amazing. So you see a"
  },
  {
    "startMs": 1526640,
    "text": "lot of companies raising on say um"
  },
  {
    "startMs": 1529520,
    "text": "making data pipelines and there's a lot"
  },
  {
    "startMs": 1531840,
    "text": "of money there. There's a a lot of"
  },
  {
    "startMs": 1534000,
    "text": "companies working on physical hardware"
  },
  {
    "startMs": 1535360,
    "text": "and stuff. So I think if you're a"
  },
  {
    "startMs": 1536640,
    "text": "software engineer watching these, it's"
  },
  {
    "startMs": 1538000,
    "text": "like I think that if you're if you feel"
  },
  {
    "startMs": 1540159,
    "text": "like you're like here the delta of your"
  },
  {
    "startMs": 1542640,
    "text": "intellect and ability in your"
  },
  {
    "startMs": 1544480,
    "text": "day-to-day, it's like not contributing"
  },
  {
    "startMs": 1546960,
    "text": "as in like you have the intellect, you"
  },
  {
    "startMs": 1549039,
    "text": "can make a difference. But like if cloud"
  },
  {
    "startMs": 1550960,
    "text": "is just like oneshotting all those"
  },
  {
    "startMs": 1552799,
    "text": "problems like in your day-to-day and"
  },
  {
    "startMs": 1554080,
    "text": "you're just like kind of like becoming a"
  },
  {
    "startMs": 1555279,
    "text": "me proxy, you should work to think"
  },
  {
    "startMs": 1556720,
    "text": "through. Take a step back and think"
  },
  {
    "startMs": 1557840,
    "text": "through. like okay where are the"
  },
  {
    "startMs": 1559760,
    "text": "problems where where you're actually"
  },
  {
    "startMs": 1561279,
    "text": "truly uh contributing and that they will"
  },
  {
    "startMs": 1563520,
    "text": "stand the test of time and then that"
  },
  {
    "startMs": 1565200,
    "text": "kind of reflection and realignment right"
  },
  {
    "startMs": 1567279,
    "text": "now would I think like have like very"
  },
  {
    "startMs": 1569840,
    "text": "interesting consequences for you um in"
  },
  {
    "startMs": 1572320,
    "text": "the long term um so yeah so thank you so"
  },
  {
    "startMs": 1575520,
    "text": "much you know thank you so much for your"
  },
  {
    "startMs": 1577039,
    "text": "time like I think that uh I think that"
  },
  {
    "startMs": 1579520,
    "text": "if you're watching this and if you've"
  },
  {
    "startMs": 1580799,
    "text": "gone this far I think that you're"
  },
  {
    "startMs": 1582559,
    "text": "definitely contemplating like one of the"
  },
  {
    "startMs": 1584880,
    "text": "most interesting improvements in the"
  },
  {
    "startMs": 1586240,
    "text": "course of humanity And uh you know"
  },
  {
    "startMs": 1588880,
    "text": "there's many philosophical arguments"
  },
  {
    "startMs": 1590400,
    "text": "here. Some people might say hey you know"
  },
  {
    "startMs": 1592720,
    "text": "I I think I'm maybe I'm off on the"
  },
  {
    "startMs": 1595520,
    "text": "timelines or maybe maybe I'm I'm just"
  },
  {
    "startMs": 1597360,
    "text": "being too pessimistic or maybe I'm being"
  },
  {
    "startMs": 1599039,
    "text": "too optimistic. But I would love to hear"
  },
  {
    "startMs": 1601200,
    "text": "all those thoughts. So this is you know"
  },
  {
    "startMs": 1602559,
    "text": "my QR code. Please feel free to scan it"
  },
  {
    "startMs": 1604880,
    "text": "and let me know all your thoughts. Uh"
  },
  {
    "startMs": 1606559,
    "text": "thank you so much. Very kind of you to"
  },
  {
    "startMs": 1607840,
    "text": "give me your time."
  }
]
