[
  {
    "startMs": 1309,
    "text": "[music]"
  },
  {
    "startMs": 12559,
    "text": "So, I'm going to be talking today about"
  },
  {
    "startMs": 14719,
    "text": "software factories and uh specifically"
  },
  {
    "startMs": 18720,
    "text": "like an infrastructure approach to"
  },
  {
    "startMs": 20160,
    "text": "software factories. Um, a little bit"
  },
  {
    "startMs": 23199,
    "text": "about me though before we start. So, I'm"
  },
  {
    "startMs": 25039,
    "text": "Zach. Uh, I've been an engineer for"
  },
  {
    "startMs": 28560,
    "text": "really for a long time at this point. I"
  },
  {
    "startMs": 30160,
    "text": "was a principal engineer at Google. I"
  },
  {
    "startMs": 31679,
    "text": "used to lead engineering on the Google"
  },
  {
    "startMs": 32960,
    "text": "Doc Suite. Helped scale that up to"
  },
  {
    "startMs": 35440,
    "text": "hundreds of millions and I think maybe"
  },
  {
    "startMs": 37120,
    "text": "now over a billion people who use it."
  },
  {
    "startMs": 39360,
    "text": "I'm currently the founder and CEO at"
  },
  {
    "startMs": 41600,
    "text": "Warp. Uh, Warp is a company in the"
  },
  {
    "startMs": 44160,
    "text": "agentic development space. Our current"
  },
  {
    "startMs": 46160,
    "text": "focus is on helping companies build"
  },
  {
    "startMs": 49200,
    "text": "their own internal software factories."
  },
  {
    "startMs": 51039,
    "text": "So, we have a platform called Warp"
  },
  {
    "startMs": 53280,
    "text": "Factories which we do that with. We're"
  },
  {
    "startMs": 55360,
    "text": "also uh pretty well known for Warp"
  },
  {
    "startMs": 57199,
    "text": "Terminal, which is an Agentic terminal."
  },
  {
    "startMs": 59760,
    "text": "It's open source, around 65,000 GitHub"
  },
  {
    "startMs": 62000,
    "text": "stars, almost a million active"
  },
  {
    "startMs": 63359,
    "text": "developers. Um, our focus as a company"
  },
  {
    "startMs": 66799,
    "text": "has always been helping developers ship"
  },
  {
    "startMs": 69600,
    "text": "better software more quickly."
  },
  {
    "startMs": 72320,
    "text": "Just to set some context on where we're"
  },
  {
    "startMs": 75760,
    "text": "at uh in terms of agentic development."
  },
  {
    "startMs": 78320,
    "text": "So most companies that I speak with uh"
  },
  {
    "startMs": 82159,
    "text": "are at this like phase of what I think"
  },
  {
    "startMs": 84159,
    "text": "of as interactive prompting. And so"
  },
  {
    "startMs": 86320,
    "text": "they've deployed tools like cloud code"
  },
  {
    "startMs": 88320,
    "text": "or codeex to their engineers. Uh"
  },
  {
    "startMs": 90640,
    "text": "engineers are working on their laptops."
  },
  {
    "startMs": 92400,
    "text": "They are they might have a bunch of"
  },
  {
    "startMs": 93920,
    "text": "these going but generally they're like"
  },
  {
    "startMs": 95920,
    "text": "interactively prompting agents to build"
  },
  {
    "startMs": 97920,
    "text": "features and fix bugs. And where uh most"
  },
  {
    "startMs": 102320,
    "text": "companies I speak with want to get to is"
  },
  {
    "startMs": 104880,
    "text": "something that is more like automated"
  },
  {
    "startMs": 106399,
    "text": "development. So"
  },
  {
    "startMs": 108640,
    "text": "the reason for that is really there's"
  },
  {
    "startMs": 112079,
    "text": "this um you know they want to get to a"
  },
  {
    "startMs": 114560,
    "text": "world where they are deploying this"
  },
  {
    "startMs": 116880,
    "text": "transformational technology which is"
  },
  {
    "startMs": 118719,
    "text": "agents and LLMs to get the most possible"
  },
  {
    "startMs": 121920,
    "text": "business value at the lowest possible"
  },
  {
    "startMs": 124399,
    "text": "cost in a way that is governed and"
  },
  {
    "startMs": 127439,
    "text": "secure and they're running up against"
  },
  {
    "startMs": 129599,
    "text": "the limits of interactive agents which"
  },
  {
    "startMs": 131360,
    "text": "are like you you're bottlenecked by a"
  },
  {
    "startMs": 134160,
    "text": "person running them. Um it's hard to"
  },
  {
    "startMs": 137200,
    "text": "control costs if everyone has their own"
  },
  {
    "startMs": 138800,
    "text": "setup and it's hard to govern. And so"
  },
  {
    "startMs": 141599,
    "text": "just to double click on that I think for"
  },
  {
    "startMs": 143360,
    "text": "this audience like local agents have"
  },
  {
    "startMs": 145760,
    "text": "this security problem where uh if there"
  },
  {
    "startMs": 149440,
    "text": "is a problem or compromise of a local"
  },
  {
    "startMs": 151920,
    "text": "agent then it's basically has the"
  },
  {
    "startMs": 154239,
    "text": "ability to take over whatever the user"
  },
  {
    "startMs": 156400,
    "text": "who signed in on their laptop is. And so"
  },
  {
    "startMs": 158640,
    "text": "that's not a good thing. Um local agents"
  },
  {
    "startMs": 161200,
    "text": "are also hard to govern. And so right"
  },
  {
    "startMs": 162720,
    "text": "now we're in this situation where you"
  },
  {
    "startMs": 165200,
    "text": "know everyone is kind of configuring"
  },
  {
    "startMs": 166560,
    "text": "their own MCPs, their own skills, their"
  },
  {
    "startMs": 168640,
    "text": "own tools. And it's like you can't see"
  },
  {
    "startMs": 170560,
    "text": "what's going on. You can't audit it. And"
  },
  {
    "startMs": 172640,
    "text": "so the solution uh to both these"
  },
  {
    "startMs": 175599,
    "text": "problems is to sandbox your agents and"
  },
  {
    "startMs": 178160,
    "text": "move them to the cloud. And the product"
  },
  {
    "startMs": 180720,
    "text": "category that is emerging to support"
  },
  {
    "startMs": 183040,
    "text": "this uh is what um you know we call a"
  },
  {
    "startMs": 186000,
    "text": "cloud software factory. [snorts] And so"
  },
  {
    "startMs": 188239,
    "text": "this this term is kind of overloaded,"
  },
  {
    "startMs": 190239,
    "text": "but what it means to me is basically"
  },
  {
    "startMs": 195040,
    "text": "every company is going to set up a loop"
  },
  {
    "startMs": 197360,
    "text": "in the cloud where"
  },
  {
    "startMs": 199840,
    "text": "um a combination of humans and agents do"
  },
  {
    "startMs": 203519,
    "text": "work. And so humans are responsible for"
  },
  {
    "startMs": 206640,
    "text": "putting work in, agents will do"
  },
  {
    "startMs": 209840,
    "text": "different parts of the software life"
  },
  {
    "startMs": 211280,
    "text": "cycle. also a triage. They'll work with"
  },
  {
    "startMs": 213280,
    "text": "the human on a spec. They'll do"
  },
  {
    "startMs": 215680,
    "text": "implementation. They will do computer"
  },
  {
    "startMs": 218080,
    "text": "use verification. And as a as a"
  },
  {
    "startMs": 221200,
    "text": "business, what you're going to want to"
  },
  {
    "startMs": 222480,
    "text": "do is set these up internally. Define"
  },
  {
    "startMs": 226239,
    "text": "all the rules for how humans and agents"
  },
  {
    "startMs": 227942,
    "text": "[music]"
  },
  {
    "startMs": 228640,
    "text": "interact and then get to a spot where"
  },
  {
    "startMs": 230879,
    "text": "you can really, really measure and"
  },
  {
    "startMs": 233120,
    "text": "improve over time. Because getting this"
  },
  {
    "startMs": 235599,
    "text": "concept right is going to be the key"
  },
  {
    "startMs": 238480,
    "text": "driver to how your organization ships"
  },
  {
    "startMs": 240400,
    "text": "software."
  },
  {
    "startMs": 242159,
    "text": "Um at at warp like our our belief is"
  },
  {
    "startMs": 244959,
    "text": "that you should think of this as"
  },
  {
    "startMs": 246239,
    "text": "infrastructure. So not as like a product"
  },
  {
    "startMs": 248480,
    "text": "or an AI teammate but really like an"
  },
  {
    "startMs": 250959,
    "text": "infrastructure stack that you can build"
  },
  {
    "startMs": 253200,
    "text": "on top of and scale. And the key aspects"
  },
  {
    "startMs": 256479,
    "text": "of this are like it should be open, it"
  },
  {
    "startMs": 259120,
    "text": "should be adaptable and you should think"
  },
  {
    "startMs": 260720,
    "text": "of your factories as something that you"
  },
  {
    "startMs": 262560,
    "text": "define in code. And so by open I mean"
  },
  {
    "startMs": 267040,
    "text": "you want a a factory model that can"
  },
  {
    "startMs": 269360,
    "text": "support different uh different models"
  },
  {
    "startMs": 271759,
    "text": "whether it's open weight or frontier."
  },
  {
    "startMs": 273680,
    "text": "You want one that can support different"
  },
  {
    "startMs": 275120,
    "text": "harnesses and the reason is just like"
  },
  {
    "startMs": 277919,
    "text": "the best models and harnesses are"
  },
  {
    "startMs": 279600,
    "text": "constantly changing. You want adaptable"
  },
  {
    "startMs": 282240,
    "text": "means like the factory should work with"
  },
  {
    "startMs": 283919,
    "text": "your existing infrastructure. And then"
  },
  {
    "startMs": 286160,
    "text": "finally defined in code means like your"
  },
  {
    "startMs": 289040,
    "text": "factory should be a thing uh that has"
  },
  {
    "startMs": 293280,
    "text": "like a strict definition and that"
  },
  {
    "startMs": 295040,
    "text": "definition should include like what are"
  },
  {
    "startMs": 297360,
    "text": "the workflows, what are the skills, what"
  },
  {
    "startMs": 299120,
    "text": "are the MCPs, where do these run and"
  },
  {
    "startMs": 301520,
    "text": "when you define it in code and version"
  },
  {
    "startMs": 303680,
    "text": "control it then you get the ability to"
  },
  {
    "startMs": 305759,
    "text": "like test different versions of your"
  },
  {
    "startMs": 307759,
    "text": "factory. You get the ability for coding"
  },
  {
    "startMs": 309360,
    "text": "agents to update your factory. So these"
  },
  {
    "startMs": 311840,
    "text": "are some key characteristics."
  },
  {
    "startMs": 314240,
    "text": "Uh I'll say another thing that I'm"
  },
  {
    "startMs": 316160,
    "text": "hearing from from businesses is like AI"
  },
  {
    "startMs": 319199,
    "text": "sovereignty matters. And so the concept"
  },
  {
    "startMs": 321759,
    "text": "here is like as you think about"
  },
  {
    "startMs": 324000,
    "text": "factories, you should think about uh"
  },
  {
    "startMs": 326639,
    "text": "owning all the data that goes in and all"
  },
  {
    "startMs": 329360,
    "text": "the data that comes out as well."
  },
  {
    "startMs": 332320,
    "text": "You should think about how do you have"
  },
  {
    "startMs": 333600,
    "text": "these agents running on your own"
  },
  {
    "startMs": 335600,
    "text": "infrastructure and own compute. Um this"
  },
  {
    "startMs": 337520,
    "text": "is what most larger enterprises that we"
  },
  {
    "startMs": 339840,
    "text": "work with want."
  },
  {
    "startMs": 341840,
    "text": "How do you provide your own inference?"
  },
  {
    "startMs": 344400,
    "text": "And then finally, like I said, any"
  },
  {
    "startMs": 345919,
    "text": "model, any harness. And so really"
  },
  {
    "startMs": 348400,
    "text": "preserve optionality and treat um treat"
  },
  {
    "startMs": 351440,
    "text": "these software factories as something"
  },
  {
    "startMs": 352880,
    "text": "that's very key to your business and"
  },
  {
    "startMs": 355360,
    "text": "that you want to, you know, mostly take"
  },
  {
    "startMs": 357840,
    "text": "ownership of the core ingredients of."
  },
  {
    "startMs": 361199,
    "text": "Um, another really important point here"
  },
  {
    "startMs": 363440,
    "text": "is like as you think about building your"
  },
  {
    "startMs": 366240,
    "text": "software factory, like if you were to"
  },
  {
    "startMs": 368720,
    "text": "look on Twitter on any given day, you"
  },
  {
    "startMs": 370560,
    "text": "would probably see different model like"
  },
  {
    "startMs": 373440,
    "text": "someone would say like PI is the best or"
  },
  {
    "startMs": 374960,
    "text": "someone say Grock or like there's all"
  },
  {
    "startMs": 376800,
    "text": "these different things in the world"
  },
  {
    "startMs": 378240,
    "text": "around like what is best and you might"
  },
  {
    "startMs": 379919,
    "text": "even see benchmarks like Sweetbench or"
  },
  {
    "startMs": 381600,
    "text": "Terminal Bench. But the world that you"
  },
  {
    "startMs": 383680,
    "text": "want to get to um as you automate more"
  },
  {
    "startMs": 386960,
    "text": "development is actually being"
  },
  {
    "startMs": 389120,
    "text": "measurement first on your own data. And"
  },
  {
    "startMs": 391520,
    "text": "so you should whatever factory model you"
  },
  {
    "startMs": 393759,
    "text": "approach have the ability to score"
  },
  {
    "startMs": 396639,
    "text": "meaning like eval how your coding agents"
  },
  {
    "startMs": 398560,
    "text": "are doing. uh feed those scores into"
  },
  {
    "startMs": 401919,
    "text": "agents that can improve your factory and"
  },
  {
    "startMs": 404319,
    "text": "then also create your own benchmarks."
  },
  {
    "startMs": 406560,
    "text": "Meaning like be able to take all of your"
  },
  {
    "startMs": 408880,
    "text": "real coding agent tasks and try them"
  },
  {
    "startMs": 411280,
    "text": "against different models and measure how"
  },
  {
    "startMs": 412960,
    "text": "it goes."
  },
  {
    "startMs": 415759,
    "text": "So I'll also talk a little bit about"
  },
  {
    "startMs": 417199,
    "text": "like um how engineering changes in this"
  },
  {
    "startMs": 421120,
    "text": "factory world."
  },
  {
    "startMs": 422479,
    "text": "[snorts] And so the way that I I conceive of this"
  },
  {
    "startMs": 425280,
    "text": "and talk about it is you will still have"
  },
  {
    "startMs": 427120,
    "text": "engineers who are very much responsible"
  },
  {
    "startMs": 429680,
    "text": "for building useful things for your"
  },
  {
    "startMs": 432080,
    "text": "customers. And so they'll build be"
  },
  {
    "startMs": 433919,
    "text": "building the features, they'll be fixing"
  },
  {
    "startMs": 435199,
    "text": "bugs, they'll be doing product"
  },
  {
    "startMs": 436319,
    "text": "engineering and they'll be doing it like"
  },
  {
    "startMs": 438319,
    "text": "through the factory. So they put work"
  },
  {
    "startMs": 440000,
    "text": "into the factory. But there's a second"
  },
  {
    "startMs": 442400,
    "text": "kind of engineering which I call factory"
  },
  {
    "startMs": 445120,
    "text": "engineering which is uh either uh your"
  },
  {
    "startMs": 449520,
    "text": "platform team or just your general"
  },
  {
    "startMs": 451120,
    "text": "engineers will have to get good at like"
  },
  {
    "startMs": 454639,
    "text": "trying to make the factory better. So"
  },
  {
    "startMs": 457120,
    "text": "improving the skills,"
  },
  {
    "startMs": 459440,
    "text": "setting the context, changing the model"
  },
  {
    "startMs": 461840,
    "text": "mix, all of these things are going to"
  },
  {
    "startMs": 463440,
    "text": "become more important as engineers are"
  },
  {
    "startMs": 465759,
    "text": "like building and improving the system"
  },
  {
    "startMs": 468319,
    "text": "that actually builds the software."
  },
  {
    "startMs": 471520,
    "text": "So, I'm going to I'm going to show what"
  },
  {
    "startMs": 473120,
    "text": "some of this looks like in practice. And"
  },
  {
    "startMs": 474800,
    "text": "again, this is going to be in the"
  },
  {
    "startMs": 476560,
    "text": "context of our own product, Warp"
  },
  {
    "startMs": 478080,
    "text": "Factories. But these general concepts, I"
  },
  {
    "startMs": 480960,
    "text": "think, will translate to however you're"
  },
  {
    "startMs": 483120,
    "text": "thinking about building your factories."
  },
  {
    "startMs": 486160,
    "text": "I'll try to leave a little bit of time"
  },
  {
    "startMs": 487360,
    "text": "for questions at the end if I can."
  },
  {
    "startMs": 490080,
    "text": "So"
  },
  {
    "startMs": 492639,
    "text": "I'm going to start with um the view of"
  },
  {
    "startMs": 496560,
    "text": "the factory that the factory engineer or"
  },
  {
    "startMs": 499680,
    "text": "the manager has. And so this is like"
  },
  {
    "startMs": 503520,
    "text": "if you are"
  },
  {
    "startMs": 505599,
    "text": "someone who is trying to get better uh"
  },
  {
    "startMs": 508639,
    "text": "productivity across your team, you're"
  },
  {
    "startMs": 510879,
    "text": "going to want some view of like what is"
  },
  {
    "startMs": 514000,
    "text": "happening from a productivity standpoint"
  },
  {
    "startMs": 515839,
    "text": "like how automated is my factory? Uh"
  },
  {
    "startMs": 518800,
    "text": "you're going to want a view of um what's"
  },
  {
    "startMs": 521440,
    "text": "it what's the cycle time. So like a"
  },
  {
    "startMs": 523279,
    "text": "dorometric view. You're going to want a"
  },
  {
    "startMs": 525519,
    "text": "view of uh spending. This might take a"
  },
  {
    "startMs": 527680,
    "text": "second to load. Um but basically you"
  },
  {
    "startMs": 530320,
    "text": "want all this data tracked. And the way"
  },
  {
    "startMs": 533360,
    "text": "that you get these views is by having a"
  },
  {
    "startMs": 537920,
    "text": "system where everything that your team"
  },
  {
    "startMs": 541519,
    "text": "is doing, I don't know if you all can"
  },
  {
    "startMs": 543839,
    "text": "see this, but everything that's"
  },
  {
    "startMs": 545600,
    "text": "happening on our team, and this is"
  },
  {
    "startMs": 547279,
    "text": "Warp's internal software development"
  },
  {
    "startMs": 549760,
    "text": "that I'm leaking here, but it's like I"
  },
  {
    "startMs": 552320,
    "text": "can see what I can see what every"
  },
  {
    "startMs": 554640,
    "text": "engineer on our team is doing, and I can"
  },
  {
    "startMs": 556480,
    "text": "see what all the automations that we're"
  },
  {
    "startMs": 557839,
    "text": "running are doing. And not just can I"
  },
  {
    "startMs": 560320,
    "text": "see them like I can see all the"
  },
  {
    "startMs": 561680,
    "text": "artifacts that they're creating, but all"
  },
  {
    "startMs": 563920,
    "text": "this is getting stored. And so all these"
  },
  {
    "startMs": 566240,
    "text": "agent traces, you know, I I could"
  },
  {
    "startMs": 568640,
    "text": "actually go in I could see what this"
  },
  {
    "startMs": 570800,
    "text": "this one is doing. I could go in and"
  },
  {
    "startMs": 572640,
    "text": "like see the actual conversation that"
  },
  {
    "startMs": 574480,
    "text": "someone is doing. But the important"
  },
  {
    "startMs": 576080,
    "text": "thing here is that um all this data is"
  },
  {
    "startMs": 579519,
    "text": "getting stored and what your coding"
  },
  {
    "startMs": 581760,
    "text": "agents are doing is the raw material for"
  },
  {
    "startMs": 584080,
    "text": "improving them. And so, uh, you might be"
  },
  {
    "startMs": 586959,
    "text": "wondering, well, how do how do engineers"
  },
  {
    "startMs": 589200,
    "text": "actually do this work? Like, how are"
  },
  {
    "startMs": 590640,
    "text": "they putting stuff in? Um, and the"
  },
  {
    "startMs": 592880,
    "text": "answer on this is that they put it in"
  },
  {
    "startMs": 595600,
    "text": "through the tools that they're already"
  },
  {
    "startMs": 596959,
    "text": "using. So, on our team, they're using"
  },
  {
    "startMs": 599360,
    "text": "Slack uh to do a lot of the work. Um,"
  },
  {
    "startMs": 602080,
    "text": "they put in uh agentic work through our"
  },
  {
    "startMs": 604880,
    "text": "task tracker, which we use linear, but"
  },
  {
    "startMs": 606560,
    "text": "it could be Jira. They have the same"
  },
  {
    "startMs": 608640,
    "text": "context whether they're uh responding to"
  },
  {
    "startMs": 611200,
    "text": "a GitHub PR comment. And so the the idea"
  },
  {
    "startMs": 614480,
    "text": "here is like the product engineers are"
  },
  {
    "startMs": 616000,
    "text": "using the tools that they're already"
  },
  {
    "startMs": 617279,
    "text": "using, but all the work goes into this"
  },
  {
    "startMs": 619360,
    "text": "centralized location where you get a"
  },
  {
    "startMs": 621040,
    "text": "centralized view and all this stuff gets"
  },
  {
    "startMs": 623200,
    "text": "recorded over time by recording"
  },
  {
    "startMs": 625839,
    "text": "everything. Uh just to go back to this"
  },
  {
    "startMs": 628800,
    "text": "dashboard for a second. Um actually let"
  },
  {
    "startMs": 632079,
    "text": "me let me not do that for one second."
  },
  {
    "startMs": 633440,
    "text": "The other thing that's important here"
  },
  {
    "startMs": 634560,
    "text": "that I I talked about earlier is like"
  },
  {
    "startMs": 636880,
    "text": "the factory workflow itself defining it"
  },
  {
    "startMs": 640000,
    "text": "as code. And so this is um our kind of"
  },
  {
    "startMs": 643839,
    "text": "like default factory workflow where"
  },
  {
    "startMs": 646720,
    "text": "think of it as like workflows in we have"
  },
  {
    "startMs": 649440,
    "text": "this orchestrator agent that will figure"
  },
  {
    "startMs": 651920,
    "text": "out okay should this next go to a triage"
  },
  {
    "startMs": 654079,
    "text": "step should it go to a spec step an"
  },
  {
    "startMs": 655760,
    "text": "implementation step code review the"
  },
  {
    "startMs": 658480,
    "text": "actual details of this need to be very"
  },
  {
    "startMs": 660320,
    "text": "very flexible. Um if you click into one"
  },
  {
    "startMs": 663360,
    "text": "of these you can see like essentially"
  },
  {
    "startMs": 666560,
    "text": "you are providing an agent definition"
  },
  {
    "startMs": 669519,
    "text": "there's uh which is like what are all"
  },
  {
    "startMs": 671600,
    "text": "the secrets the MCPS the agent prompt"
  },
  {
    "startMs": 674240,
    "text": "the model all this stuff the the cool"
  },
  {
    "startMs": 677440,
    "text": "thing though that I would emphasize that"
  },
  {
    "startMs": 680399,
    "text": "however you're approaching this you"
  },
  {
    "startMs": 681920,
    "text": "probably want to do is that there's an"
  },
  {
    "startMs": 684079,
    "text": "actual code repo that sits behind this"
  },
  {
    "startMs": 687279,
    "text": "where if you go and like look in that"
  },
  {
    "startMs": 689279,
    "text": "repo you can see like I that earlier, a"
  },
  {
    "startMs": 692240,
    "text": "definition of the entire factory at any"
  },
  {
    "startMs": 695279,
    "text": "given point in time. And so that"
  },
  {
    "startMs": 696959,
    "text": "definition includes, think of it as like"
  },
  {
    "startMs": 699760,
    "text": "everything that goes into the process of"
  },
  {
    "startMs": 701440,
    "text": "automating development. Um, it's your"
  },
  {
    "startMs": 704240,
    "text": "it's your uh agent definitions, it's"
  },
  {
    "startMs": 707440,
    "text": "your automations, your benchmarks, it's"
  },
  {
    "startMs": 709040,
    "text": "where these agents run, the skills, the"
  },
  {
    "startMs": 710560,
    "text": "web hooks, everything. So everything"
  },
  {
    "startMs": 712160,
    "text": "gets stored in in a repo that it allows"
  },
  {
    "startMs": 715200,
    "text": "for automation. Um, and I'll I'll I'll"
  },
  {
    "startMs": 718959,
    "text": "show a little more why that matters in a"
  },
  {
    "startMs": 720560,
    "text": "second."
  },
  {
    "startMs": 721360,
    "text": "The other thing uh just to return to"
  },
  {
    "startMs": 723680,
    "text": "this view for a second because I think"
  },
  {
    "startMs": 724959,
    "text": "it's interesting. You get these metrics"
  },
  {
    "startMs": 727200,
    "text": "around productivity and spend but you"
  },
  {
    "startMs": 729519,
    "text": "also want to set up something where"
  },
  {
    "startMs": 731760,
    "text": "there are like we call them scores. You"
  },
  {
    "startMs": 734320,
    "text": "could think of it as like online evals."
  },
  {
    "startMs": 737200,
    "text": "And what what you're looking at here is"
  },
  {
    "startMs": 740079,
    "text": "basically we have a set of agents that"
  },
  {
    "startMs": 743440,
    "text": "are looking at all of these runs that"
  },
  {
    "startMs": 745839,
    "text": "are going through our system. So"
  },
  {
    "startMs": 747519,
    "text": "remember I go back here. It's like you"
  },
  {
    "startMs": 748880,
    "text": "can see everything that everyone's doing"
  },
  {
    "startMs": 750079,
    "text": "on our team. You then have a scoring"
  },
  {
    "startMs": 752800,
    "text": "mechanism that goes and looks at those"
  },
  {
    "startMs": 755680,
    "text": "runs and sort of figures out by using"
  },
  {
    "startMs": 759120,
    "text": "LLM as a judge whether or not um those"
  },
  {
    "startMs": 763600,
    "text": "runs are good or not. And so for"
  },
  {
    "startMs": 766000,
    "text": "instance we have a scorer here that is"
  },
  {
    "startMs": 769040,
    "text": "looking for agents that are writing"
  },
  {
    "startMs": 770240,
    "text": "redundant tests which is a common"
  },
  {
    "startMs": 772480,
    "text": "failure mode with these coding agents."
  },
  {
    "startMs": 774560,
    "text": "And so for the most part, our our agents"
  },
  {
    "startMs": 776639,
    "text": "seem to be doing pretty well here. But"
  },
  {
    "startMs": 778320,
    "text": "if you look back in time, you'll see"
  },
  {
    "startMs": 779600,
    "text": "there's some set of them where an agent"
  },
  {
    "startMs": 781760,
    "text": "is deciding, okay, this was this thing"
  },
  {
    "startMs": 784160,
    "text": "wrote tests that didn't need to be"
  },
  {
    "startMs": 785600,
    "text": "written. And so every one of these"
  },
  {
    "startMs": 787920,
    "text": "failures is an opportunity for improving"
  },
  {
    "startMs": 791680,
    "text": "how your system works. And so as you"
  },
  {
    "startMs": 794720,
    "text": "gather this information, and you know,"
  },
  {
    "startMs": 797120,
    "text": "you could go look at this as a person"
  },
  {
    "startMs": 798800,
    "text": "and grade it, but it's much more"
  },
  {
    "startMs": 800000,
    "text": "efficient to have agents do it. As you"
  },
  {
    "startMs": 802240,
    "text": "gather this information, you can then"
  },
  {
    "startMs": 804320,
    "text": "use it to feed what we call a"
  },
  {
    "startMs": 806959,
    "text": "self-improvement loop. And so the idea"
  },
  {
    "startMs": 809680,
    "text": "behind a self-improvement loop is that"
  },
  {
    "startMs": 812560,
    "text": "uh agents will go and they'll look at"
  },
  {
    "startMs": 814160,
    "text": "all these failure modes. They'll"
  },
  {
    "startMs": 816320,
    "text": "aggregate them. They will look for"
  },
  {
    "startMs": 818639,
    "text": "patterns and then they will come up with"
  },
  {
    "startMs": 820959,
    "text": "ideas for how do you improve the"
  },
  {
    "startMs": 823839,
    "text": "underlying factory definition so that"
  },
  {
    "startMs": 826320,
    "text": "going forward you don't have this same"
  },
  {
    "startMs": 828240,
    "text": "failure mode. And so you know for"
  },
  {
    "startMs": 832000,
    "text": "instance um I don't know I'm going to"
  },
  {
    "startMs": 833760,
    "text": "pick a random one here. Um this is like"
  },
  {
    "startMs": 838639,
    "text": "read matching repository skills before"
  },
  {
    "startMs": 840639,
    "text": "implementing. The key thing here is like"
  },
  {
    "startMs": 843279,
    "text": "it's saying I saw this fail 37 times and"
  },
  {
    "startMs": 847360,
    "text": "then it's saying okay we can actually um"
  },
  {
    "startMs": 851760,
    "text": "oops we can actually fix this"
  },
  {
    "startMs": 855279,
    "text": "um with a PR"
  },
  {
    "startMs": 858800,
    "text": "to the factory definition."
  },
  {
    "startMs": 861519,
    "text": "So this is like closing the loop where"
  },
  {
    "startMs": 864320,
    "text": "the agent has identified a problem then"
  },
  {
    "startMs": 866000,
    "text": "it's suggesting a change to actually"
  },
  {
    "startMs": 868320,
    "text": "drive a better result in the factory. So"
  },
  {
    "startMs": 870399,
    "text": "this is like a a core concept I think as"
  },
  {
    "startMs": 873360,
    "text": "you think of like how do you really"
  },
  {
    "startMs": 874800,
    "text": "improve development it's it's about"
  },
  {
    "startMs": 877519,
    "text": "creating these loops where agents can"
  },
  {
    "startMs": 879440,
    "text": "identify problems and then fix them."
  },
  {
    "startMs": 882000,
    "text": "There's one more thing that I want to"
  },
  {
    "startMs": 883440,
    "text": "show and then maybe I'll take a couple"
  },
  {
    "startMs": 885360,
    "text": "questions. Um I mentioned early on like"
  },
  {
    "startMs": 889920,
    "text": "you can't really trust the public"
  },
  {
    "startMs": 892000,
    "text": "benchmarks [clears throat]"
  },
  {
    "startMs": 893360,
    "text": "as like a way of understanding how good"
  },
  {
    "startMs": 895440,
    "text": "a certain model or harness is going to"
  },
  {
    "startMs": 897600,
    "text": "be on your own workflows. And so what"
  },
  {
    "startMs": 899839,
    "text": "you want to get to a point where you can"
  },
  {
    "startMs": 901600,
    "text": "do is basically take any past run. So we"
  },
  {
    "startMs": 906880,
    "text": "can do this like with any anything that"
  },
  {
    "startMs": 909199,
    "text": "we've done in our system. I could go"
  },
  {
    "startMs": 910720,
    "text": "into it here and I have the um"
  },
  {
    "startMs": 914639,
    "text": "sorry I picked a bad example. I um"
  },
  {
    "startMs": 920880,
    "text": "uh maybe I'm picking a bad example. The"
  },
  {
    "startMs": 922560,
    "text": "the idea here is like you can basically"
  },
  {
    "startMs": 924560,
    "text": "take past runs. I don't know why I can't"
  },
  {
    "startMs": 928480,
    "text": "do this here and say okay I want to add"
  },
  {
    "startMs": 930880,
    "text": "this to a benchmark suite and what this"
  },
  {
    "startMs": 933600,
    "text": "is doing is say okay this is like a"
  },
  {
    "startMs": 935120,
    "text": "representative thing that I did in my"
  },
  {
    "startMs": 936720,
    "text": "factory that could be useful to measure"
  },
  {
    "startMs": 940079,
    "text": "by testing it with different model"
  },
  {
    "startMs": 941920,
    "text": "configurations and so think of like"
  },
  {
    "startMs": 943920,
    "text": "building up a a suite of like front-end"
  },
  {
    "startMs": 945920,
    "text": "tasks and the question that you're"
  },
  {
    "startMs": 947680,
    "text": "trying to answer is like what's the best"
  },
  {
    "startMs": 950000,
    "text": "model for me to use for front-end tasks"
  },
  {
    "startMs": 951920,
    "text": "and like you might look on Twitter but"
  },
  {
    "startMs": 953600,
    "text": "you might also just be like let"
  },
  {
    "startMs": 955839,
    "text": "actually test. And so what you can do or"
  },
  {
    "startMs": 959920,
    "text": "should try to set up for your own system"
  },
  {
    "startMs": 962320,
    "text": "and this is built into warp factory. So"
  },
  {
    "startMs": 964079,
    "text": "I'll show it here is basically curate"
  },
  {
    "startMs": 967199,
    "text": "one of these benchmarks and what what it"
  },
  {
    "startMs": 971120,
    "text": "does is it will replay your real past"
  },
  {
    "startMs": 975120,
    "text": "coding agent tasks and then generate for"
  },
  {
    "startMs": 978720,
    "text": "you what looks like a basically a cost"
  },
  {
    "startMs": 981680,
    "text": "versus quality trade-off graph. And so"
  },
  {
    "startMs": 984320,
    "text": "this is something you'll see when um you"
  },
  {
    "startMs": 986800,
    "text": "know the public benchmarks come out, but"
  },
  {
    "startMs": 988320,
    "text": "this is on your own data. And so in in"
  },
  {
    "startMs": 991519,
    "text": "this case, for example, like we could"
  },
  {
    "startMs": 993199,
    "text": "see GLM53 flash uh was 93% correct on"
  },
  {
    "startMs": 998959,
    "text": "these tasks at a cost of $2. GPT56 soul"
  },
  {
    "startMs": 1002720,
    "text": "was like 95% correct, but it cost three"
  },
  {
    "startMs": 1005519,
    "text": "times as much. And so you can take this"
  },
  {
    "startMs": 1008160,
    "text": "information and then build a model"
  },
  {
    "startMs": 1009839,
    "text": "routing strategy that you have real"
  },
  {
    "startMs": 1012560,
    "text": "confidence is not just a generic one but"
  },
  {
    "startMs": 1016079,
    "text": "for your actual code your stack is"
  },
  {
    "startMs": 1019759,
    "text": "actually providing value. And so um this"
  },
  {
    "startMs": 1022720,
    "text": "is like I think a good example of like"
  },
  {
    "startMs": 1026720,
    "text": "closing the loop. And then just to go"
  },
  {
    "startMs": 1028959,
    "text": "back I'm going to finish up here in a"
  },
  {
    "startMs": 1030880,
    "text": "second."
  },
  {
    "startMs": 1032799,
    "text": "The"
  },
  {
    "startMs": 1035199,
    "text": "the sort of like way that we think about"
  },
  {
    "startMs": 1037600,
    "text": "this is like at the bottom of your"
  },
  {
    "startMs": 1039760,
    "text": "stack, it's defined in code. It's think"
  },
  {
    "startMs": 1042319,
    "text": "of a data layer. What's the data going"
  },
  {
    "startMs": 1044160,
    "text": "in and out? Where do these agents run?"
  },
  {
    "startMs": 1046640,
    "text": "You really want to sandbox that. You"
  },
  {
    "startMs": 1049440,
    "text": "want inference that's flexible. You want"
  },
  {
    "startMs": 1052240,
    "text": "a self-improvement layer that gives you"
  },
  {
    "startMs": 1054640,
    "text": "something like benchmark scoring. You"
  },
  {
    "startMs": 1056960,
    "text": "want orchestration so you can have"
  },
  {
    "startMs": 1058400,
    "text": "multi-step workflows. And then you want"
  },
  {
    "startMs": 1060080,
    "text": "access from all these existing tools."
  },
  {
    "startMs": 1062320,
    "text": "And so that's like the infrastructure"
  },
  {
    "startMs": 1064640,
    "text": "that I think is going to it's going to"
  },
  {
    "startMs": 1066720,
    "text": "look something like this. Um no matter"
  },
  {
    "startMs": 1069679,
    "text": "how you approach this problem uh if"
  },
  {
    "startMs": 1072080,
    "text": "you're interested in our approach"
  },
  {
    "startMs": 1073120,
    "text": "obviously we're we're we would love to"
  },
  {
    "startMs": 1075039,
    "text": "talk to you. But let me pause there."
  },
  {
    "startMs": 1078080,
    "text": "Hopefully this was interesting to folks."
  },
  {
    "startMs": 1079840,
    "text": "And I might have time for one question."
  },
  {
    "startMs": 1082400,
    "text": "I don't know. Or I might have to go."
  },
  {
    "startMs": 1086080,
    "text": "Yes. What what uh what advice do you"
  },
  {
    "startMs": 1088640,
    "text": "have for testing and verification of"
  },
  {
    "startMs": 1091120,
    "text": "code at the factory?"
  },
  {
    "startMs": 1092799,
    "text": "Awesome question. So the you know one"
  },
  {
    "startMs": 1095760,
    "text": "thing that could happen if you do this"
  },
  {
    "startMs": 1097520,
    "text": "approach is you get like a pile up of"
  },
  {
    "startMs": 1099280,
    "text": "PRs right and so you want some way of"
  },
  {
    "startMs": 1101200,
    "text": "like getting them through the big things"
  },
  {
    "startMs": 1103360,
    "text": "that I advise are spec driven"
  },
  {
    "startMs": 1105440,
    "text": "development. So you can actually test um"
  },
  {
    "startMs": 1108559,
    "text": "make sure that the changes are adherent"
  },
  {
    "startMs": 1110720,
    "text": "to a spec. And the other thing is"
  },
  {
    "startMs": 1112640,
    "text": "computer use verification which I didn't"
  },
  {
    "startMs": 1114640,
    "text": "show but actually um you know if you can"
  },
  {
    "startMs": 1117440,
    "text": "have agents that verify or produce"
  },
  {
    "startMs": 1120240,
    "text": "evidence from the user standpoint that"
  },
  {
    "startMs": 1122000,
    "text": "the thing works that can go a long way"
  },
  {
    "startMs": 1124559,
    "text": "in terms of like lessening the code"
  },
  {
    "startMs": 1126160,
    "text": "review burden. So those are the those"
  },
  {
    "startMs": 1127919,
    "text": "are the two things that I recommend."
  },
  {
    "startMs": 1130559,
    "text": "Yes."
  },
  {
    "startMs": 1130960,
    "text": "So how do you plan um the budgets when"
  },
  {
    "startMs": 1133520,
    "text": "we are starting a project? How do you"
  },
  {
    "startMs": 1137200,
    "text": "plan these verification costs?"
  },
  {
    "startMs": 1139520,
    "text": "Yeah. So the the way that we do this is"
  },
  {
    "startMs": 1142080,
    "text": "like we give people the tools to measure"
  },
  {
    "startMs": 1144400,
    "text": "like what's your cost per PR and then"
  },
  {
    "startMs": 1147200,
    "text": "what percent of the cost is going"
  },
  {
    "startMs": 1148640,
    "text": "towards like scoring or benchmarking or"
  },
  {
    "startMs": 1150480,
    "text": "stuff like that. Uh and then you can"
  },
  {
    "startMs": 1152640,
    "text": "kind of extrapolate it out like are you"
  },
  {
    "startMs": 1154559,
    "text": "willing to spend x amount on a project"
  },
  {
    "startMs": 1156960,
    "text": "or whatever. But the the key the key"
  },
  {
    "startMs": 1159520,
    "text": "thing um that that I want people to"
  },
  {
    "startMs": 1161679,
    "text": "leave if there's one idea is like really"
  },
  {
    "startMs": 1164240,
    "text": "set up something that you can measure"
  },
  {
    "startMs": 1165600,
    "text": "and improve. And once you can truly"
  },
  {
    "startMs": 1167679,
    "text": "measure in a closed loop system, you can"
  },
  {
    "startMs": 1170160,
    "text": "you can do all of these things in a way"
  },
  {
    "startMs": 1171600,
    "text": "that's just never been possible before."
  },
  {
    "startMs": 1176640,
    "text": "Any other questions?"
  },
  {
    "startMs": 1178160,
    "text": "Question in the back."
  },
  {
    "startMs": 1178880,
    "text": "Ycess."
  },
  {
    "startMs": 1188960,
    "text": "Yeah. So, so the question was around uh"
  },
  {
    "startMs": 1191840,
    "text": "how do we do cost per PR? So this is"
  },
  {
    "startMs": 1193840,
    "text": "data curation"
  },
  {
    "startMs": 1194880,
    "text": "the the data curation"
  },
  {
    "startMs": 1197679,
    "text": "um are you so I just want to make sure"
  },
  {
    "startMs": 1199760,
    "text": "we're talking about the same thing. So"
  },
  {
    "startMs": 1200960,
    "text": "you're talking about like when I"
  },
  {
    "startMs": 1203600,
    "text": "for the the benchmarking or for like our"
  },
  {
    "startMs": 1206400,
    "text": "general"
  },
  {
    "startMs": 1210320,
    "text": "Okay. Yeah. So the way that we we do"
  },
  {
    "startMs": 1212559,
    "text": "this is like you can either build up a"
  },
  {
    "startMs": 1214400,
    "text": "benchmark set by hand which a lot of"
  },
  {
    "startMs": 1216720,
    "text": "people will do. They'll you know and"
  },
  {
    "startMs": 1218160,
    "text": "that's kind of what we recommend to"
  },
  {
    "startMs": 1219440,
    "text": "start. uh we will pick like for ours we"
  },
  {
    "startMs": 1222400,
    "text": "pick like a set of sort of small medium"
  },
  {
    "startMs": 1224640,
    "text": "hard tasks whatever you want uh to be"
  },
  {
    "startMs": 1227679,
    "text": "representative or you can actually ask"
  },
  {
    "startMs": 1230960,
    "text": "the factory to do it because the factory"
  },
  {
    "startMs": 1233039,
    "text": "is aware of all the past sessions and so"
  },
  {
    "startMs": 1235360,
    "text": "and everything is stored in code so it's"
  },
  {
    "startMs": 1237919,
    "text": "possible to have the factory itself look"
  },
  {
    "startMs": 1240240,
    "text": "at prior sessions and identify"
  },
  {
    "startMs": 1242720,
    "text": "interesting ones for a benchmark set"
  },
  {
    "startMs": 1244720,
    "text": "which is pretty cool"
  },
  {
    "startMs": 1247120,
    "text": "y"
  },
  {
    "startMs": 1247600,
    "text": "how is correctness evaluated on chart."
  },
  {
    "startMs": 1250640,
    "text": "Yeah. So everything in this chart is"
  },
  {
    "startMs": 1252640,
    "text": "driven by LLM as a judge which we we may"
  },
  {
    "startMs": 1256559,
    "text": "I think another way of doing it is human"
  },
  {
    "startMs": 1258159,
    "text": "grading which makes total sense. We just"
  },
  {
    "startMs": 1259919,
    "text": "haven't added that yet. So I would"
  },
  {
    "startMs": 1261200,
    "text": "suggest that as well. But basically"
  },
  {
    "startMs": 1263840,
    "text": "anything that is defined uh in the"
  },
  {
    "startMs": 1266799,
    "text": "system as a scorer. So you define these"
  },
  {
    "startMs": 1269760,
    "text": "scores which basically grade how well"
  },
  {
    "startMs": 1271760,
    "text": "something is doing can be used as a"
  },
  {
    "startMs": 1273280,
    "text": "benchmark evaluation criteria."
  },
  {
    "startMs": 1277039,
    "text": "Yeah."
  },
  {
    "startMs": 1278720,
    "text": "this work platform is this open source"
  },
  {
    "startMs": 1281520,
    "text": "or is this only for enterprise?"
  },
  {
    "startMs": 1283760,
    "text": "So it is is not open source. It's it's"
  },
  {
    "startMs": 1287760,
    "text": "like very flexible and like like the"
  },
  {
    "startMs": 1290000,
    "text": "idea is to make it open so you can bring"
  },
  {
    "startMs": 1291919,
    "text": "any existing tools into it. It's not"
  },
  {
    "startMs": 1294640,
    "text": "only for enterprise like this is for"
  },
  {
    "startMs": 1296799,
    "text": "small teams and you can try it out on"
  },
  {
    "startMs": 1298480,
    "text": "your personal project. It is best for"
  },
  {
    "startMs": 1301039,
    "text": "teams though I will say because like if"
  },
  {
    "startMs": 1303600,
    "text": "you're um if you're an it's just"
  },
  {
    "startMs": 1305520,
    "text": "overkill for an individual like you're"
  },
  {
    "startMs": 1307039,
    "text": "probably gonna want to use like your"
  },
  {
    "startMs": 1308000,
    "text": "cloud code sub or something but once you"
  },
  {
    "startMs": 1310240,
    "text": "have a team of like maybe 10 or more"
  },
  {
    "startMs": 1311760,
    "text": "engineers having a system like this is"
  },
  {
    "startMs": 1313440,
    "text": "very valuable."
  }
]
