[
  {
    "startMs": 1309,
    "text": "[music]"
  },
  {
    "startMs": 12719,
    "text": "All right, let's get started. Apologies"
  },
  {
    "startMs": 15360,
    "text": "for the delay, but I'm really excited to"
  },
  {
    "startMs": 17600,
    "text": "be here. I'm Mingshan, VP of engineering"
  },
  {
    "startMs": 20960,
    "text": "focused on AI at Ironclad. And today"
  },
  {
    "startMs": 24320,
    "text": "I'll be telling you about something"
  },
  {
    "startMs": 25840,
    "text": "that's probably on top of many of your"
  },
  {
    "startMs": 28720,
    "text": "mind. uh how to control and optimize for"
  },
  {
    "startMs": 32320,
    "text": "your AI token spend. Can I get a get a"
  },
  {
    "startMs": 35600,
    "text": "quick show of hands that this is a"
  },
  {
    "startMs": 37760,
    "text": "relevant topic?"
  },
  {
    "startMs": 40160,
    "text": "Okay, awesome. I appreciate that."
  },
  {
    "startMs": 45920,
    "text": "So, we have all heard a few sensational"
  },
  {
    "startMs": 48960,
    "text": "stories from the media. There's an"
  },
  {
    "startMs": 51440,
    "text": "interesting Amazon story where an"
  },
  {
    "startMs": 53360,
    "text": "employee just created kind of a"
  },
  {
    "startMs": 55440,
    "text": "voluntary dashboard and everyone start"
  },
  {
    "startMs": 58239,
    "text": "tracking their own AI token usage. I'm"
  },
  {
    "startMs": 61120,
    "text": "not sure there's explicit encouragement"
  },
  {
    "startMs": 63039,
    "text": "from the leadership, but the effect is"
  },
  {
    "startMs": 65680,
    "text": "you know engineers some of the engineers"
  },
  {
    "startMs": 67520,
    "text": "started competing with each other in"
  },
  {
    "startMs": 69600,
    "text": "maximizing their token usage and get to"
  },
  {
    "startMs": 72159,
    "text": "the top of the so-called leaderboard."
  },
  {
    "startMs": 74479,
    "text": "There's a similar story from Meta and"
  },
  {
    "startMs": 77200,
    "text": "then another even more sensational story"
  },
  {
    "startMs": 79119,
    "text": "about some companies spending $500"
  },
  {
    "startMs": 81600,
    "text": "million on cloud oops within a month. So"
  },
  {
    "startMs": 86400,
    "text": "while these may not be happening in your"
  },
  {
    "startMs": 88720,
    "text": "companies today, the threats, the risks"
  },
  {
    "startMs": 91360,
    "text": "are real. How do we think about the"
  },
  {
    "startMs": 94159,
    "text": "policies? How do we measure the cost?"
  },
  {
    "startMs": 96720,
    "text": "And how do we control and optimize for"
  },
  {
    "startMs": 98720,
    "text": "it?"
  },
  {
    "startMs": 100000,
    "text": "So one initial learning I want to share"
  },
  {
    "startMs": 102240,
    "text": "is it is really important to have"
  },
  {
    "startMs": 105200,
    "text": "dashboard that track every team every"
  },
  {
    "startMs": 108000,
    "text": "individual's token usage and cost but"
  },
  {
    "startMs": 111280,
    "text": "that should not be positioned as a"
  },
  {
    "startMs": 113439,
    "text": "leaderboard. We think of the the usage"
  },
  {
    "startMs": 117119,
    "text": "dashboard more as a smoke detector. If"
  },
  {
    "startMs": 120240,
    "text": "there are local pockets of teams or"
  },
  {
    "startMs": 122240,
    "text": "individuals that don't use much AI token"
  },
  {
    "startMs": 125040,
    "text": "that might be a signal worth"
  },
  {
    "startMs": 126799,
    "text": "investigating. But beyond that certainly"
  },
  {
    "startMs": 129440,
    "text": "we don't want to create even indirect"
  },
  {
    "startMs": 132319,
    "text": "incentive to maximize the token usage"
  },
  {
    "startMs": 135120,
    "text": "itself."
  },
  {
    "startMs": 138640,
    "text": "So how do we think about it then? First"
  },
  {
    "startMs": 141360,
    "text": "I want to make sure that we position"
  },
  {
    "startMs": 143520,
    "text": "this talk for those of you whose teams"
  },
  {
    "startMs": 146959,
    "text": "have already gone through the hump of"
  },
  {
    "startMs": 149280,
    "text": "getting AI adopted. If you're still in"
  },
  {
    "startMs": 152560,
    "text": "the initial process of provisioning easy"
  },
  {
    "startMs": 156480,
    "text": "access to your engineers or encouraging"
  },
  {
    "startMs": 159680,
    "text": "the teams and individuals to adopt, then"
  },
  {
    "startMs": 162560,
    "text": "you may not be ready to implement some"
  },
  {
    "startMs": 165519,
    "text": "of the ideas for controlling and"
  },
  {
    "startMs": 167440,
    "text": "optimizing for cost. But that's okay."
  },
  {
    "startMs": 170000,
    "text": "This could still be a good discussion."
  },
  {
    "startMs": 171920,
    "text": "And frankly, we just got over that hump"
  },
  {
    "startMs": 174319,
    "text": "over the last couple quarters. So this"
  },
  {
    "startMs": 176560,
    "text": "is a very topical subject that every"
  },
  {
    "startMs": 179360,
    "text": "engineering leader I believe is"
  },
  {
    "startMs": 181200,
    "text": "navigating. So I would love to start"
  },
  {
    "startMs": 182800,
    "text": "that dialogue with you all today to"
  },
  {
    "startMs": 184720,
    "text": "explore the best practices. Can I get a"
  },
  {
    "startMs": 187760,
    "text": "quick show of hand for those of you"
  },
  {
    "startMs": 189519,
    "text": "whose teams have gone over the initial"
  },
  {
    "startMs": 191920,
    "text": "adoption phase now you are starting to"
  },
  {
    "startMs": 194480,
    "text": "seriously worry about the cost. Okay, I"
  },
  {
    "startMs": 197040,
    "text": "see roughly half of the hands raised."
  },
  {
    "startMs": 198879,
    "text": "Thank you. So let's talk about then how"
  },
  {
    "startMs": 202000,
    "text": "we can control and how we can optimize"
  },
  {
    "startMs": 205040,
    "text": "what we call the trusted throughput as a"
  },
  {
    "startMs": 207920,
    "text": "kind of a proxy metric as a way to"
  },
  {
    "startMs": 210640,
    "text": "measure your ROI. But before that, just"
  },
  {
    "startMs": 213519,
    "text": "for those of you who are in the process"
  },
  {
    "startMs": 215360,
    "text": "of still increasing adoption, one lesson"
  },
  {
    "startMs": 218480,
    "text": "we learned is to after the kind of the"
  },
  {
    "startMs": 221120,
    "text": "top down leadership push is to sit down"
  },
  {
    "startMs": 223840,
    "text": "with the individual teams and uh the"
  },
  {
    "startMs": 227120,
    "text": "individuals who may be resistant or"
  },
  {
    "startMs": 229040,
    "text": "struggling with adoption, understand"
  },
  {
    "startMs": 231360,
    "text": "where they came from. For example, there"
  },
  {
    "startMs": 233200,
    "text": "are some legitimate concerns that I"
  },
  {
    "startMs": 235120,
    "text": "heard, you know, people say, \"Hey, I"
  },
  {
    "startMs": 237120,
    "text": "used to really take pride and joy in"
  },
  {
    "startMs": 239280,
    "text": "handcrafting the code and now a lot of"
  },
  {
    "startMs": 241760,
    "text": "the joy and the pride got taken away and"
  },
  {
    "startMs": 245280,
    "text": "replaced with me reviewing AI slop code,"
  },
  {
    "startMs": 248560,
    "text": "right? So that doesn't sound like a very"
  },
  {
    "startMs": 250319,
    "text": "satisfying professional activity and"
  },
  {
    "startMs": 252959,
    "text": "that's where we need to kind of dig down"
  },
  {
    "startMs": 254640,
    "text": "and understand what are still the kind"
  },
  {
    "startMs": 256720,
    "text": "of the high impact and uh engineering"
  },
  {
    "startMs": 259919,
    "text": "tasks technical work that we can help"
  },
  {
    "startMs": 261840,
    "text": "our engineers continue to grow"
  },
  {
    "startMs": 263199,
    "text": "themselves in the era of the AI."
  },
  {
    "startMs": 268479,
    "text": "So I wanted to share with you a bit more"
  },
  {
    "startMs": 271280,
    "text": "about what we at ironclad does and"
  },
  {
    "startMs": 273440,
    "text": "there's an interesting connection"
  },
  {
    "startMs": 275040,
    "text": "actually within how we think about"
  },
  {
    "startMs": 276800,
    "text": "optimizing for engineering AI token"
  },
  {
    "startMs": 279280,
    "text": "usage. So, ironclad is a legal"
  },
  {
    "startMs": 282320,
    "text": "contracting AI companies AI company. We"
  },
  {
    "startMs": 285840,
    "text": "build AI features and native AI products"
  },
  {
    "startMs": 289280,
    "text": "to help lawyers, procurement and other"
  },
  {
    "startMs": 292400,
    "text": "business users move forward new"
  },
  {
    "startMs": 294960,
    "text": "contracts, move them forward faster with"
  },
  {
    "startMs": 297919,
    "text": "controlled risk. What that means is"
  },
  {
    "startMs": 300800,
    "text": "building trust is the number one"
  },
  {
    "startMs": 302720,
    "text": "priority with our AI product features"
  },
  {
    "startMs": 305040,
    "text": "and products. And for the prior speak uh"
  },
  {
    "startMs": 307680,
    "text": "speaker speaker, she did a wonderful job"
  },
  {
    "startMs": 309600,
    "text": "telling you about the importance of"
  },
  {
    "startMs": 311520,
    "text": "trust and how to build it in their"
  },
  {
    "startMs": 313360,
    "text": "domain. In our ironclad product domain,"
  },
  {
    "startMs": 316240,
    "text": "it often means lawyers especially, but"
  },
  {
    "startMs": 319360,
    "text": "other persona as well taking the time to"
  },
  {
    "startMs": 321440,
    "text": "kind of test the water and see if they"
  },
  {
    "startMs": 323680,
    "text": "can trust the AI output. For example,"
  },
  {
    "startMs": 326080,
    "text": "they may feed our conversational search"
  },
  {
    "startMs": 328800,
    "text": "a set of contracts they are firmly"
  },
  {
    "startMs": 330880,
    "text": "familiar with and they run a search and"
  },
  {
    "startMs": 333440,
    "text": "see if the output is towards the"
  },
  {
    "startMs": 335280,
    "text": "expectation. If so, they may expand on"
  },
  {
    "startMs": 338400,
    "text": "searching for things they don't know"
  },
  {
    "startMs": 339759,
    "text": "about or apply other workflows using AI"
  },
  {
    "startMs": 342960,
    "text": "to solve other things like redlinining"
  },
  {
    "startMs": 344720,
    "text": "the contract um and you know finding"
  },
  {
    "startMs": 347840,
    "text": "anomalies and so on. And so similarly"
  },
  {
    "startMs": 351440,
    "text": "using AI and making sure AI is"
  },
  {
    "startMs": 354000,
    "text": "delivering high engineering value also"
  },
  {
    "startMs": 356160,
    "text": "involves a you know a sequence of steps"
  },
  {
    "startMs": 359440,
    "text": "in gaining trust from the internal"
  },
  {
    "startMs": 361840,
    "text": "engineers the leadership as well with as"
  },
  {
    "startMs": 364400,
    "text": "uh with our customers. So this is the"
  },
  {
    "startMs": 366560,
    "text": "focus of our talk today"
  },
  {
    "startMs": 371280,
    "text": "and this probably will not come as a"
  },
  {
    "startMs": 373520,
    "text": "surprise here. The goal is not to"
  },
  {
    "startMs": 377280,
    "text": "minimizing or not even necessarily to"
  },
  {
    "startMs": 379680,
    "text": "reduce token spend. So here we kind of"
  },
  {
    "startMs": 382080,
    "text": "use the word it's not about austerity."
  },
  {
    "startMs": 384560,
    "text": "It's about further improving the ROI of"
  },
  {
    "startMs": 386960,
    "text": "the token spend."
  },
  {
    "startMs": 389280,
    "text": "So how do we do that? Here we propose um"
  },
  {
    "startMs": 392400,
    "text": "a concept we call trusted throughput. So"
  },
  {
    "startMs": 396080,
    "text": "the trusted throughput comes from having"
  },
  {
    "startMs": 399120,
    "text": "the code reviewed and validated"
  },
  {
    "startMs": 402000,
    "text": "internally and ultimately validated in"
  },
  {
    "startMs": 405039,
    "text": "customer uh in customer deployments."
  },
  {
    "startMs": 410639,
    "text": "So how do we go and how do we think"
  },
  {
    "startMs": 414479,
    "text": "about controlling the cost and uh"
  },
  {
    "startMs": 417840,
    "text": "measuring and in turn optimizing the"
  },
  {
    "startMs": 420080,
    "text": "ROI? The first step is I'm pretty"
  },
  {
    "startMs": 423039,
    "text": "confident that all of you your teams who"
  },
  {
    "startMs": 425520,
    "text": "have been adopting AI have been"
  },
  {
    "startMs": 426960,
    "text": "measuring the cost. If you're using a"
  },
  {
    "startMs": 429360,
    "text": "single tool like claw code or codeex"
  },
  {
    "startMs": 432080,
    "text": "then you tend to get very rich analytics"
  },
  {
    "startMs": 434479,
    "text": "from the vendor's dashboard already. If"
  },
  {
    "startMs": 436960,
    "text": "you're like us who use a combination of"
  },
  {
    "startMs": 438960,
    "text": "these different coding tools then we"
  },
  {
    "startMs": 441440,
    "text": "basically use AI to build simple"
  },
  {
    "startMs": 443919,
    "text": "dashboards and pipelines to extract such"
  },
  {
    "startMs": 446639,
    "text": "vendor data. So we can kind of"
  },
  {
    "startMs": 448560,
    "text": "crossorrelate them. Then we can break it"
  },
  {
    "startMs": 450639,
    "text": "down, aggregate and then break down by"
  },
  {
    "startMs": 452639,
    "text": "per team, per individual, what is their"
  },
  {
    "startMs": 455280,
    "text": "cost usage across all of these uh tools."
  },
  {
    "startMs": 459360,
    "text": "So that's the first step for measuring"
  },
  {
    "startMs": 460960,
    "text": "cost. Now one pitfall I have seen and we"
  },
  {
    "startMs": 464880,
    "text": "wanted to caution everybody is to then"
  },
  {
    "startMs": 467120,
    "text": "jump from measuring cost to start"
  },
  {
    "startMs": 470160,
    "text": "reducing or minimizing the cost, right?"
  },
  {
    "startMs": 472160,
    "text": "Cutting cost. We think that is"
  },
  {
    "startMs": 474000,
    "text": "premature. Instead, the other important"
  },
  {
    "startMs": 476560,
    "text": "side of the equation for ROI is to"
  },
  {
    "startMs": 478879,
    "text": "measure value. How much value are we"
  },
  {
    "startMs": 481440,
    "text": "getting from burning the tokens? Once we"
  },
  {
    "startMs": 484240,
    "text": "can measure the cost and value side, we"
  },
  {
    "startMs": 486639,
    "text": "understand ROI and then to improve ROI,"
  },
  {
    "startMs": 489680,
    "text": "we want to find and then fix the"
  },
  {
    "startMs": 491759,
    "text": "bottlenecks. In the next couple slides,"
  },
  {
    "startMs": 494000,
    "text": "I'm going to introduce two new"
  },
  {
    "startMs": 495759,
    "text": "bottlenecks we identify in this whole"
  },
  {
    "startMs": 497919,
    "text": "new software development life cycle"
  },
  {
    "startMs": 499840,
    "text": "where code generation now becomes"
  },
  {
    "startMs": 501919,
    "text": "abundant thanks to AI. But the pressure"
  },
  {
    "startMs": 504400,
    "text": "is now getting pushed down to code"
  },
  {
    "startMs": 506800,
    "text": "review and continuous integration CICD"
  },
  {
    "startMs": 509919,
    "text": "the merging the code. So we'll talk"
  },
  {
    "startMs": 511599,
    "text": "about that and finally we'll put"
  },
  {
    "startMs": 513440,
    "text": "together these ideas into a pra"
  },
  {
    "startMs": 515120,
    "text": "pragmatic framework of how we think"
  },
  {
    "startMs": 517200,
    "text": "about optimizing the ROI and thus the"
  },
  {
    "startMs": 519919,
    "text": "leverage in using AI."
  },
  {
    "startMs": 525279,
    "text": "Okay. So this is kind of just a slide in"
  },
  {
    "startMs": 528640,
    "text": "building or using the vendor dashboard"
  },
  {
    "startMs": 530640,
    "text": "to measure the cost. And again we want"
  },
  {
    "startMs": 532800,
    "text": "to caution that here the main goal for"
  },
  {
    "startMs": 535680,
    "text": "regularly reviewing the dashboard is to"
  },
  {
    "startMs": 538080,
    "text": "see a if there's still adoption gap"
  },
  {
    "startMs": 540240,
    "text": "within individual pockets of teams or"
  },
  {
    "startMs": 542560,
    "text": "the individual engineers and b if there"
  },
  {
    "startMs": 545040,
    "text": "are any sudden surprises in kind of the"
  },
  {
    "startMs": 548000,
    "text": "usage burst and if so understand what's"
  },
  {
    "startMs": 550240,
    "text": "been happening if they're legitimate and"
  },
  {
    "startMs": 552720,
    "text": "then also compare teams"
  },
  {
    "startMs": 555519,
    "text": "contextually. So this is important. We"
  },
  {
    "startMs": 557839,
    "text": "don't control just the AI usage per se"
  },
  {
    "startMs": 560320,
    "text": "because for example a platform"
  },
  {
    "startMs": 561839,
    "text": "infrastructure team the way they use AI"
  },
  {
    "startMs": 564240,
    "text": "and the way they get value may be"
  },
  {
    "startMs": 565680,
    "text": "different from the UI team. So we need"
  },
  {
    "startMs": 567440,
    "text": "to take the context into consideration."
  },
  {
    "startMs": 570240,
    "text": "All of such review analysis is to help"
  },
  {
    "startMs": 572560,
    "text": "us extract learnings. So there's a"
  },
  {
    "startMs": 574720,
    "text": "self-learning loop that we can then feed"
  },
  {
    "startMs": 576560,
    "text": "back into institutional best practices."
  },
  {
    "startMs": 579839,
    "text": "What we don't want to use the dashboards"
  },
  {
    "startMs": 582080,
    "text": "are to kind of stack rank people, right?"
  },
  {
    "startMs": 584640,
    "text": "making it a a leaderboard and somehow"
  },
  {
    "startMs": 587040,
    "text": "reward maximization."
  },
  {
    "startMs": 589200,
    "text": "There's an interesting analogy I want to"
  },
  {
    "startMs": 590880,
    "text": "draw with uh a traditional edge"
  },
  {
    "startMs": 593760,
    "text": "productivity metric called lines of"
  },
  {
    "startMs": 595360,
    "text": "code. So I believe all of you will be"
  },
  {
    "startMs": 598000,
    "text": "tracking that metric but it wouldn't be"
  },
  {
    "startMs": 600399,
    "text": "wise to use that metric as the key goal"
  },
  {
    "startMs": 602800,
    "text": "to measure engine velocity because if we"
  },
  {
    "startMs": 605760,
    "text": "want productive and high quality engine"
  },
  {
    "startMs": 608320,
    "text": "work one can argue that removing code is"
  },
  {
    "startMs": 611200,
    "text": "even better. So, LOC line of code is an"
  },
  {
    "startMs": 614399,
    "text": "important metric but not something we"
  },
  {
    "startMs": 616079,
    "text": "want to directly optimize for. Same"
  },
  {
    "startMs": 618000,
    "text": "thing for the token usage and spend."
  },
  {
    "startMs": 622320,
    "text": "So, that that gets us to the notion of"
  },
  {
    "startMs": 625040,
    "text": "trusted throughput. How do we think"
  },
  {
    "startMs": 627440,
    "text": "about that? How do we define that?"
  },
  {
    "startMs": 629680,
    "text": "First, I want to kind of share the"
  },
  {
    "startMs": 631040,
    "text": "quantified uh side of the things. What"
  },
  {
    "startMs": 633200,
    "text": "are the metrics that kind of we have"
  },
  {
    "startMs": 634800,
    "text": "been involving in defining and tracking."
  },
  {
    "startMs": 637440,
    "text": "So we talked about line of code is"
  },
  {
    "startMs": 639519,
    "text": "clearly not a good way to measure if AI"
  },
  {
    "startMs": 642320,
    "text": "is you know generating a lot of value."
  },
  {
    "startMs": 644720,
    "text": "So the next evolution can be let's count"
  },
  {
    "startMs": 647200,
    "text": "the number of open PRs pull requests."
  },
  {
    "startMs": 650880,
    "text": "The intuition being engineers are using"
  },
  {
    "startMs": 653839,
    "text": "AI to generate a lot more code. So let's"
  },
  {
    "startMs": 655920,
    "text": "measure the open PR. So clearly we see a"
  },
  {
    "startMs": 659120,
    "text": "big kind of inflection in the open PR"
  },
  {
    "startMs": 662160,
    "text": "count. But eventually as we as I assume"
  },
  {
    "startMs": 665440,
    "text": "everyone would agree over the time even"
  },
  {
    "startMs": 667839,
    "text": "though people may do oneoff you know R&D"
  },
  {
    "startMs": 670399,
    "text": "work to try out things without lending"
  },
  {
    "startMs": 672480,
    "text": "them but eventually we're all measured"
  },
  {
    "startMs": 674399,
    "text": "by the code we ship. So therefore we"
  },
  {
    "startMs": 677519,
    "text": "evolved from tracking the open PR count"
  },
  {
    "startMs": 680720,
    "text": "to tracking the merge PR count. So"
  },
  {
    "startMs": 682959,
    "text": "that's an improvement."
  },
  {
    "startMs": 685040,
    "text": "But the next question is not every"
  },
  {
    "startMs": 687040,
    "text": "merged PR is equal. There can be a PO"
  },
  {
    "startMs": 690399,
    "text": "with only 10 lines of code that takes"
  },
  {
    "startMs": 692560,
    "text": "forever that finds and fix a concurrency"
  },
  {
    "startMs": 694720,
    "text": "bug or there can be a thousand line kind"
  },
  {
    "startMs": 697360,
    "text": "of boilerplate code that just takes a"
  },
  {
    "startMs": 699519,
    "text": "lot of time to then kind of generate and"
  },
  {
    "startMs": 701600,
    "text": "review but otherwise it's not necessary"
  },
  {
    "startMs": 703600,
    "text": "adding as much business value."
  },
  {
    "startMs": 706320,
    "text": "So as such we then started kind of"
  },
  {
    "startMs": 708480,
    "text": "tagging each merged PR with some sort of"
  },
  {
    "startMs": 711440,
    "text": "complexity score. There's no traditional"
  },
  {
    "startMs": 714480,
    "text": "definition of what that means. We looked"
  },
  {
    "startMs": 716320,
    "text": "at the literature a bit. So we just took"
  },
  {
    "startMs": 718079,
    "text": "a pragmatic approach of giving AI a"
  },
  {
    "startMs": 720959,
    "text": "well-crafted prompt and then we feed the"
  },
  {
    "startMs": 723360,
    "text": "PR into basically one or two M and say"
  },
  {
    "startMs": 726639,
    "text": "score the complexity based on t-shirt"
  },
  {
    "startMs": 728639,
    "text": "size. So I the idea being if you use AI"
  },
  {
    "startMs": 731839,
    "text": "to generate a more complex PR we"
  },
  {
    "startMs": 733920,
    "text": "consider that as being more valuable"
  },
  {
    "startMs": 735839,
    "text": "basically that's how we kind of add a"
  },
  {
    "startMs": 737360,
    "text": "weightage to each merged PR but that's"
  },
  {
    "startMs": 740160,
    "text": "not the end of the journey that's still"
  },
  {
    "startMs": 741760,
    "text": "something we're going to evolve keep"
  },
  {
    "startMs": 743360,
    "text": "evolving and I would love to discuss"
  },
  {
    "startMs": 745040,
    "text": "with everyone on kind of how we end up"
  },
  {
    "startMs": 747440,
    "text": "creating defining a set of metrics that"
  },
  {
    "startMs": 750000,
    "text": "kind of approximate the value AI is"
  },
  {
    "startMs": 752399,
    "text": "generating."
  },
  {
    "startMs": 754000,
    "text": "Now let's look at the qualitative view."
  },
  {
    "startMs": 756639,
    "text": "What we think about the way we would"
  },
  {
    "startMs": 758880,
    "text": "define trusted throughput is a high"
  },
  {
    "startMs": 761279,
    "text": "quality output that's interested by both"
  },
  {
    "startMs": 763920,
    "text": "internal engineering and leadership and"
  },
  {
    "startMs": 766240,
    "text": "external customers. We think they come"
  },
  {
    "startMs": 768480,
    "text": "from three buckets."
  },
  {
    "startMs": 770639,
    "text": "The first bucket is all of the objective"
  },
  {
    "startMs": 773600,
    "text": "metrics that we run with checking the"
  },
  {
    "startMs": 775680,
    "text": "test coverage whether uh all of the"
  },
  {
    "startMs": 778240,
    "text": "predefined security checks are passing."
  },
  {
    "startMs": 780880,
    "text": "Do we go through the regular canarying"
  },
  {
    "startMs": 783040,
    "text": "practice as we roll out features safely"
  },
  {
    "startMs": 784880,
    "text": "and so on. In addition, we complement"
  },
  {
    "startMs": 788560,
    "text": "the subjective objective metrics with"
  },
  {
    "startMs": 791279,
    "text": "our subjective human judgment. So that's"
  },
  {
    "startMs": 793760,
    "text": "where the code review, the design review"
  },
  {
    "startMs": 796079,
    "text": "come in to look at the code quality,"
  },
  {
    "startMs": 798320,
    "text": "clarity, maintenance, architecture fit"
  },
  {
    "startMs": 800720,
    "text": "and so on. And then finally we want to"
  },
  {
    "startMs": 803600,
    "text": "make sure through all of these internal"
  },
  {
    "startMs": 805360,
    "text": "objective and subjective check when the"
  },
  {
    "startMs": 807440,
    "text": "rubber meets the road how customer"
  },
  {
    "startMs": 810399,
    "text": "perceive the changes are there"
  },
  {
    "startMs": 812639,
    "text": "production fire that lead to ro"
  },
  {
    "startMs": 814480,
    "text": "rollbacks do customers complain have"
  },
  {
    "startMs": 817279,
    "text": "tickets that talk about usability uh"
  },
  {
    "startMs": 819600,
    "text": "friction uh bugs and so on. So these are"
  },
  {
    "startMs": 822240,
    "text": "the three buckets that together form"
  },
  {
    "startMs": 824320,
    "text": "what we think is trusted throughput from"
  },
  {
    "startMs": 826639,
    "text": "engineering."
  },
  {
    "startMs": 832160,
    "text": "Okay. So now let's talk about from a"
  },
  {
    "startMs": 834399,
    "text": "software deploy deployment life cycle"
  },
  {
    "startMs": 836880,
    "text": "perspective where we observe the new"
  },
  {
    "startMs": 839279,
    "text": "bottlenecks are as I mentioned earlier"
  },
  {
    "startMs": 842800,
    "text": "AI code generation is making PR creation"
  },
  {
    "startMs": 846800,
    "text": "abundant. So now the the bottleneck from"
  },
  {
    "startMs": 850639,
    "text": "kind of the whole life cycle perspective"
  },
  {
    "startMs": 852560,
    "text": "gets shifted onto re review and they're"
  },
  {
    "startMs": 855360,
    "text": "subsequently merging the PR. Does that"
  },
  {
    "startMs": 857920,
    "text": "resonate?"
  },
  {
    "startMs": 860240,
    "text": "I see some heads nodding. So this is"
  },
  {
    "startMs": 863760,
    "text": "where we spend time on figuring out how"
  },
  {
    "startMs": 865839,
    "text": "we can further improve the review"
  },
  {
    "startMs": 867839,
    "text": "process as well as the continuous"
  },
  {
    "startMs": 869839,
    "text": "integration the CI process. So we will"
  },
  {
    "startMs": 872240,
    "text": "dive into these two topics in the next"
  },
  {
    "startMs": 874000,
    "text": "couple slides here. I just want to say a"
  },
  {
    "startMs": 877360,
    "text": "potential anti-attern anti-solution is"
  },
  {
    "startMs": 880399,
    "text": "that hey if the CI infrastructure gets"
  },
  {
    "startMs": 883519,
    "text": "overloaded then a workar around by"
  },
  {
    "startMs": 885839,
    "text": "engineers to stop splitting PR just"
  },
  {
    "startMs": 888639,
    "text": "start submitting large PR for review and"
  },
  {
    "startMs": 890959,
    "text": "submission because if it takes an hour"
  },
  {
    "startMs": 893279,
    "text": "to run all of your regression test and"
  },
  {
    "startMs": 895120,
    "text": "submit it I don't want to break my PR"
  },
  {
    "startMs": 897199,
    "text": "into 10 right which might take 10 hours"
  },
  {
    "startMs": 899760,
    "text": "however this in our view can be pretty"
  },
  {
    "startMs": 902880,
    "text": "risky because it makes the human review"
  },
  {
    "startMs": 906240,
    "text": "overhead higher it also reduce the"
  },
  {
    "startMs": 909199,
    "text": "quality of the review because the human"
  },
  {
    "startMs": 910720,
    "text": "attention can be spread thin so that is"
  },
  {
    "startMs": 912800,
    "text": "an anti-attern I wanted to caution"
  },
  {
    "startMs": 917760,
    "text": "so for code review the key principle we"
  },
  {
    "startMs": 921440,
    "text": "use is to make sure we onboard AI"
  },
  {
    "startMs": 923839,
    "text": "tooling as the first level of defense"
  },
  {
    "startMs": 926160,
    "text": "they don't replace human reviewers but"
  },
  {
    "startMs": 928160,
    "text": "we want to offload human reviewers as"
  },
  {
    "startMs": 930480,
    "text": "much as possible let the AI review take"
  },
  {
    "startMs": 932959,
    "text": "care of simpler things like coding style"
  },
  {
    "startMs": 935360,
    "text": "issues or if there's a missing test"
  },
  {
    "startMs": 938240,
    "text": "coverage. So, make sure the author gets"
  },
  {
    "startMs": 941360,
    "text": "through all of them before then the"
  },
  {
    "startMs": 943440,
    "text": "review gets routed to a human reviewer."
  },
  {
    "startMs": 946320,
    "text": "And this way our human engineers can"
  },
  {
    "startMs": 948880,
    "text": "focus on applying their deep judgment on"
  },
  {
    "startMs": 951440,
    "text": "aspects that are somewhat subjective"
  },
  {
    "startMs": 953680,
    "text": "like if the code is good, if the"
  },
  {
    "startMs": 955680,
    "text": "architecture is sound, if the code uh uh"
  },
  {
    "startMs": 959199,
    "text": "passes kind of the security uh the"
  },
  {
    "startMs": 962320,
    "text": "security design and so on. so that in"
  },
  {
    "startMs": 964720,
    "text": "the end our engineering team can take"
  },
  {
    "startMs": 967040,
    "text": "the final accountability."
  },
  {
    "startMs": 971600,
    "text": "Now let's look at CI. So I assume all of"
  },
  {
    "startMs": 974639,
    "text": "you deploy some form of CI uh CI/CD and"
  },
  {
    "startMs": 978639,
    "text": "what we're seeing is thanks to AI now"
  },
  {
    "startMs": 981839,
    "text": "making it much easier to generate code"
  },
  {
    "startMs": 984079,
    "text": "as splitting code into smaller but more"
  },
  {
    "startMs": 987120,
    "text": "PRs it puts a lot more pressure on the"
  },
  {
    "startMs": 990000,
    "text": "CI and this is something that uh if we"
  },
  {
    "startMs": 993680,
    "text": "don't address uh at a company level"
  },
  {
    "startMs": 996240,
    "text": "individual engineers can be struggling"
  },
  {
    "startMs": 998079,
    "text": "because that means they have to waste"
  },
  {
    "startMs": 999920,
    "text": "their human time babysitting the PR to"
  },
  {
    "startMs": 1002560,
    "text": "get merged. If they run into flaky test"
  },
  {
    "startMs": 1005199,
    "text": "then they have to manually they hit"
  },
  {
    "startMs": 1007040,
    "text": "rerun it's very frustrating or they can"
  },
  {
    "startMs": 1009519,
    "text": "recruit an AI agent to babysit and kind"
  },
  {
    "startMs": 1012320,
    "text": "of do a loop but that in turn waste AI"
  },
  {
    "startMs": 1014560,
    "text": "token as well. So these are not these"
  },
  {
    "startMs": 1016720,
    "text": "are just workarounds not perfect"
  },
  {
    "startMs": 1018480,
    "text": "solution and also tend to make engineers"
  },
  {
    "startMs": 1021440,
    "text": "feel a little bit lower morale a little"
  },
  {
    "startMs": 1023120,
    "text": "bit more frustrated. So what we what we"
  },
  {
    "startMs": 1026559,
    "text": "are doing is kind of we put more uh"
  },
  {
    "startMs": 1029360,
    "text": "developer experience uh platform kind of"
  },
  {
    "startMs": 1032240,
    "text": "engineering to invest into reducing"
  },
  {
    "startMs": 1035679,
    "text": "removing the flaky test improving the CI"
  },
  {
    "startMs": 1038720,
    "text": "infrastructure and the key thing here is"
  },
  {
    "startMs": 1041199,
    "text": "to also define and measure the right"
  },
  {
    "startMs": 1044319,
    "text": "metrics for example uh the work clock"
  },
  {
    "startMs": 1047039,
    "text": "time between when a peer is ready to"
  },
  {
    "startMs": 1049840,
    "text": "submit till when it's submitted right if"
  },
  {
    "startMs": 1052559,
    "text": "a typical CR uh run takes an hour. Does"
  },
  {
    "startMs": 1056000,
    "text": "the typical PR submission take two or"
  },
  {
    "startMs": 1057919,
    "text": "three hours? In which case, that's a red"
  },
  {
    "startMs": 1059679,
    "text": "flag and also the number of times a PR"
  },
  {
    "startMs": 1062480,
    "text": "needs to get retrieded for passing"
  },
  {
    "startMs": 1064480,
    "text": "through the test. So, these are the key"
  },
  {
    "startMs": 1066400,
    "text": "metrics that we are using to measure our"
  },
  {
    "startMs": 1068320,
    "text": "developer experiences and the relevant"
  },
  {
    "startMs": 1070640,
    "text": "team who is focused on improving uh"
  },
  {
    "startMs": 1073360,
    "text": "these uh the developer experience."
  },
  {
    "startMs": 1078480,
    "text": "So with all of the analysis and ideas"
  },
  {
    "startMs": 1080960,
    "text": "here we uh want to share kind of the a"
  },
  {
    "startMs": 1084320,
    "text": "pragmatic framework of how we can then"
  },
  {
    "startMs": 1087520,
    "text": "measure and optimize token usage. It has"
  },
  {
    "startMs": 1090480,
    "text": "three aspects. The first one is set the"
  },
  {
    "startMs": 1093440,
    "text": "right set of guards across setting the"
  },
  {
    "startMs": 1095840,
    "text": "budget and quota tracking usage defining"
  },
  {
    "startMs": 1099600,
    "text": "anomalies so that no users uh leaders"
  },
  {
    "startMs": 1102480,
    "text": "can get notified if something feels"
  },
  {
    "startMs": 1104480,
    "text": "wrong. This is complementaryary to still"
  },
  {
    "startMs": 1107120,
    "text": "regular human review which can catch"
  },
  {
    "startMs": 1109280,
    "text": "other interesting patterns or learnings"
  },
  {
    "startMs": 1111840,
    "text": "and feedback into the institutional"
  },
  {
    "startMs": 1113679,
    "text": "knowledge base."
  },
  {
    "startMs": 1115840,
    "text": "Let me just couple that with the third"
  },
  {
    "startMs": 1117600,
    "text": "item here which is the learning loop we"
  },
  {
    "startMs": 1120160,
    "text": "talk about as our leadership work with"
  },
  {
    "startMs": 1123039,
    "text": "individuals to define these guard rails"
  },
  {
    "startMs": 1125520,
    "text": "review the metrics and then refine"
  },
  {
    "startMs": 1127520,
    "text": "that's how we kind of close the learning"
  },
  {
    "startMs": 1129280,
    "text": "loop. In addition to that, we want to"
  },
  {
    "startMs": 1132480,
    "text": "work with our teams, individual"
  },
  {
    "startMs": 1134400,
    "text": "engineers to continue to search for and"
  },
  {
    "startMs": 1136559,
    "text": "if needed innovate on the best practices"
  },
  {
    "startMs": 1139039,
    "text": "of how to use AI, how to use AI to build"
  },
  {
    "startMs": 1142640,
    "text": "products and also use it internally. For"
  },
  {
    "startMs": 1145360,
    "text": "example,"
  },
  {
    "startMs": 1147280,
    "text": "some engineers may be writing an agentic"
  },
  {
    "startMs": 1149760,
    "text": "loop as part of the harness when they"
  },
  {
    "startMs": 1151760,
    "text": "use cloud code. after they generated"
  },
  {
    "startMs": 1154000,
    "text": "initial PR they go and loop around and"
  },
  {
    "startMs": 1156640,
    "text": "say try and pass the set of tests and"
  },
  {
    "startMs": 1159520,
    "text": "then if some tests don't pass just auto"
  },
  {
    "startMs": 1162080,
    "text": "fix the test or the code and retry. One"
  },
  {
    "startMs": 1165440,
    "text": "thing to watch out for is to put a limit"
  },
  {
    "startMs": 1167679,
    "text": "on the number of loop steps to make sure"
  },
  {
    "startMs": 1170080,
    "text": "if things go out of control we don't"
  },
  {
    "startMs": 1172640,
    "text": "waste too many tokens on that. Another"
  },
  {
    "startMs": 1175919,
    "text": "example is prompt caching. This is"
  },
  {
    "startMs": 1178080,
    "text": "becoming increasingly more prevalent by"
  },
  {
    "startMs": 1179840,
    "text": "the commercial uh model vendors where"
  },
  {
    "startMs": 1184160,
    "text": "what they advise is if you send a prompt"
  },
  {
    "startMs": 1188400,
    "text": "with the same prefix they could optimize"
  },
  {
    "startMs": 1191280,
    "text": "how they process the prefix of the"
  },
  {
    "startMs": 1193360,
    "text": "prompt. What that means then as a user"
  },
  {
    "startMs": 1196400,
    "text": "to those is that we want to encourage"
  },
  {
    "startMs": 1198880,
    "text": "our users to structure their prompt that"
  },
  {
    "startMs": 1201440,
    "text": "way. For example, if your prompt"
  },
  {
    "startMs": 1203120,
    "text": "consists of a system prompt followed by"
  },
  {
    "startMs": 1205360,
    "text": "a user prompt, you want to put the"
  },
  {
    "startMs": 1206720,
    "text": "system prompt that's fixed at the top"
  },
  {
    "startMs": 1209280,
    "text": "and the varying content at the bottom."
  },
  {
    "startMs": 1212240,
    "text": "Context pruning is also important. We"
  },
  {
    "startMs": 1214000,
    "text": "want to kind of drill it into each"
  },
  {
    "startMs": 1215679,
    "text": "individual users kind of new kind of"
  },
  {
    "startMs": 1218000,
    "text": "muscle memory. So they are aware that as"
  },
  {
    "startMs": 1220320,
    "text": "they build out the context through a"
  },
  {
    "startMs": 1222000,
    "text": "longer chat session, they would be"
  },
  {
    "startMs": 1224000,
    "text": "mindful of summarizing the context and"
  },
  {
    "startMs": 1226480,
    "text": "make sure that the token usage is"
  },
  {
    "startMs": 1228400,
    "text": "efficient that way. There are"
  },
  {
    "startMs": 1229679,
    "text": "increasingly more tools like claw code"
  },
  {
    "startMs": 1231600,
    "text": "that will automatically manage and"
  },
  {
    "startMs": 1233200,
    "text": "compact the context for you. And so this"
  },
  {
    "startMs": 1235600,
    "text": "increases the token usage efficiency but"
  },
  {
    "startMs": 1238000,
    "text": "also increase the quality of AI output."
  },
  {
    "startMs": 1240320,
    "text": "There are other ideas we're exploring as"
  },
  {
    "startMs": 1242080,
    "text": "well."
  },
  {
    "startMs": 1248559,
    "text": "So I know we're at time so this is"
  },
  {
    "startMs": 1250320,
    "text": "towards the end of the talk. There is"
  },
  {
    "startMs": 1252640,
    "text": "sometimes we also face build versus by"
  },
  {
    "startMs": 1254720,
    "text": "decision. The principle is simple for"
  },
  {
    "startMs": 1257200,
    "text": "things that are non- differentiating"
  },
  {
    "startMs": 1259120,
    "text": "like IDE CI infrastructure we want to"
  },
  {
    "startMs": 1262159,
    "text": "buy. But then for things that are"
  },
  {
    "startMs": 1264080,
    "text": "specific to our context like how we"
  },
  {
    "startMs": 1266640,
    "text": "would generate high quality PR for small"
  },
  {
    "startMs": 1269360,
    "text": "bug fixes versus building a new UI"
  },
  {
    "startMs": 1271440,
    "text": "feature for refactoring and so on. We"
  },
  {
    "startMs": 1273679,
    "text": "have our internal playbook which is a"
  },
  {
    "startMs": 1275760,
    "text": "set of well-crafted AI prompts. So we"
  },
  {
    "startMs": 1278960,
    "text": "save that and share across our team. So"
  },
  {
    "startMs": 1281120,
    "text": "that gets reused and enhanced. So that's"
  },
  {
    "startMs": 1283360,
    "text": "something we must build internally. When"
  },
  {
    "startMs": 1285600,
    "text": "it comes to case to case though,"
  },
  {
    "startMs": 1287039,
    "text": "sometimes it's still a bit ambiguous"
  },
  {
    "startMs": 1288720,
    "text": "like we're trying to build what we call"
  },
  {
    "startMs": 1290640,
    "text": "builder agent. That's like a cloud-based"
  },
  {
    "startMs": 1292640,
    "text": "code generation that wrap the cloud"
  },
  {
    "startMs": 1294799,
    "text": "codec and so on. While we know there are"
  },
  {
    "startMs": 1297280,
    "text": "also other vendors out there that we're"
  },
  {
    "startMs": 1299039,
    "text": "still exploring. So we love to exchange"
  },
  {
    "startMs": 1300720,
    "text": "thoughts on that."
  },
  {
    "startMs": 1303520,
    "text": "So then to summarize here are a couple"
  },
  {
    "startMs": 1305760,
    "text": "key lessons as we went through the last"
  },
  {
    "startMs": 1307600,
    "text": "couple quarters of journey. I wanted to"
  },
  {
    "startMs": 1309440,
    "text": "share so that hopefully you could kind"
  },
  {
    "startMs": 1310880,
    "text": "of accelerate your process there. If I"
  },
  {
    "startMs": 1313200,
    "text": "were to summarize these three things I"
  },
  {
    "startMs": 1314799,
    "text": "would it's about learning planning ahead"
  },
  {
    "startMs": 1317440,
    "text": "and learn from other people's stories"
  },
  {
    "startMs": 1319520,
    "text": "mistakes. So what that means is think"
  },
  {
    "startMs": 1321919,
    "text": "about build respences by early on as you"
  },
  {
    "startMs": 1325280,
    "text": "are encouraging more code gen think"
  },
  {
    "startMs": 1327520,
    "text": "about how that impact your code review"
  },
  {
    "startMs": 1329679,
    "text": "and CI and how you can address these new"
  },
  {
    "startMs": 1332080,
    "text": "bottlenecks. And finally, continue to"
  },
  {
    "startMs": 1334400,
    "text": "define and instrument your system to get"
  },
  {
    "startMs": 1336799,
    "text": "the right metrics to measure the health"
  },
  {
    "startMs": 1338880,
    "text": "of your CI system and the whole"
  },
  {
    "startMs": 1341120,
    "text": "developer experience in general."
  },
  {
    "startMs": 1344080,
    "text": "So that's it for the talk. We believe"
  },
  {
    "startMs": 1346080,
    "text": "that this is the golden era of AI where"
  },
  {
    "startMs": 1349039,
    "text": "maximizing token ROI is the key for"
  },
  {
    "startMs": 1352400,
    "text": "every team success. And with that, I"
  },
  {
    "startMs": 1354799,
    "text": "just want to end with saying we are"
  },
  {
    "startMs": 1356240,
    "text": "hiring. I know this is engineering"
  },
  {
    "startMs": 1358240,
    "text": "leadership crowd but if you know of"
  },
  {
    "startMs": 1359760,
    "text": "someone who is interested in building"
  },
  {
    "startMs": 1361120,
    "text": "cutting edge legal contracting AI we"
  },
  {
    "startMs": 1363919,
    "text": "would love to talk. Thank you."
  }
]
