[
  {
    "startMs": 80,
    "text": "Hello and welcome to our ag engineer San"
  },
  {
    "startMs": 3679,
    "text": "Francisco talk. My name is Steven Chin."
  },
  {
    "startMs": 5359,
    "text": "I'm joined by Cassandra Chin who's the"
  },
  {
    "startMs": 7440,
    "text": "creator of the t trace tune framework."
  },
  {
    "startMs": 12080,
    "text": "Hi, I'm Cassandra Chin. I built trace"
  },
  {
    "startMs": 14880,
    "text": "tune because I wanted to improve PRs. I"
  },
  {
    "startMs": 18080,
    "text": "see a lot of developers doing great"
  },
  {
    "startMs": 20160,
    "text": "things. The speed of innovation is as"
  },
  {
    "startMs": 22560,
    "text": "fast as it has ever been with AI"
  },
  {
    "startMs": 24480,
    "text": "technologies. But a lot of poll requests"
  },
  {
    "startMs": 27039,
    "text": "are getting rejected. And it's not"
  },
  {
    "startMs": 29279,
    "text": "because the code is bad. It's because"
  },
  {
    "startMs": 31119,
    "text": "the PRs often contain incorrect"
  },
  {
    "startMs": 34079,
    "text": "information or are written poorly."
  },
  {
    "startMs": 37760,
    "text": "So hopefully with Tune, we can improve"
  },
  {
    "startMs": 39760,
    "text": "that. It also has the option to add a"
  },
  {
    "startMs": 42239,
    "text": "fun comic to help explain your pull"
  },
  {
    "startMs": 44719,
    "text": "requests in a funny way."
  },
  {
    "startMs": 47440,
    "text": "Cool. And um"
  },
  {
    "startMs": 48879,
    "text": "what we're going to do today is we're"
  },
  {
    "startMs": 50160,
    "text": "going to go through a little bit about"
  },
  {
    "startMs": 52399,
    "text": "kind of the common issues of PRs, how to"
  },
  {
    "startMs": 54640,
    "text": "improve the quality, how to improve"
  },
  {
    "startMs": 56160,
    "text": "token efficiency on this, and hopefully"
  },
  {
    "startMs": 58879,
    "text": "help everyone to be a better AI engineer"
  },
  {
    "startMs": 62079,
    "text": "for your organization. Like our main"
  },
  {
    "startMs": 65280,
    "text": "character, Alex,"
  },
  {
    "startMs": 67600,
    "text": "who all he wants in life is to get an AI"
  },
  {
    "startMs": 70960,
    "text": "engineer hoodie."
  },
  {
    "startMs": 73280,
    "text": "That's a really neat hoodie. I have a"
  },
  {
    "startMs": 75360,
    "text": "lot of AI engineer t-shirts, but I don't"
  },
  {
    "startMs": 78159,
    "text": "have the hoodie yet."
  },
  {
    "startMs": 81200,
    "text": "Get switched to make this AI engineer"
  },
  {
    "startMs": 82880,
    "text": "hoodie. Everyone"
  },
  {
    "startMs": 84159,
    "text": "everyone needs an"
  },
  {
    "startMs": 85360,
    "text": "ready. And um this is enough motivation"
  },
  {
    "startMs": 88799,
    "text": "that hopefully he's he's going to do an"
  },
  {
    "startMs": 90880,
    "text": "amazing talk at the conference about his"
  },
  {
    "startMs": 93920,
    "text": "use case study on what he's doing in his"
  },
  {
    "startMs": 95840,
    "text": "company to improve the quality of PRs."
  },
  {
    "startMs": 99040,
    "text": "Here"
  },
  {
    "startMs": 100799,
    "text": "is a picture of the token factory where"
  },
  {
    "startMs": 104240,
    "text": "Alex works. So, um, Alex works together"
  },
  {
    "startMs": 108799,
    "text": "with a bunch of agents. He has, um, of"
  },
  {
    "startMs": 112079,
    "text": "course, finance looking over his"
  },
  {
    "startMs": 113200,
    "text": "shoulder because we we all spend too"
  },
  {
    "startMs": 114720,
    "text": "much tokens and they want to make sure"
  },
  {
    "startMs": 116640,
    "text": "we're keeping up with costs. Um, his"
  },
  {
    "startMs": 119280,
    "text": "buddy there, Pip, he's walking with"
  },
  {
    "startMs": 121119,
    "text": "carries all the evidence and looks into"
  },
  {
    "startMs": 123360,
    "text": "the the archives and what's actually"
  },
  {
    "startMs": 125280,
    "text": "happening. And then, u Meera does all"
  },
  {
    "startMs": 127600,
    "text": "the reviews. So, she's she's keeping"
  },
  {
    "startMs": 129440,
    "text": "track of things."
  },
  {
    "startMs": 131440,
    "text": "It looks like Pip left his station empty"
  },
  {
    "startMs": 139200,
    "text": "part of the factory"
  },
  {
    "startMs": 141680,
    "text": "until until he sees what Alex produced"
  },
  {
    "startMs": 144560,
    "text": "for a PR. Oh, no."
  },
  {
    "startMs": 147360,
    "text": "That's a very long ledger."
  },
  {
    "startMs": 150319,
    "text": "A lot to review."
  },
  {
    "startMs": 153599,
    "text": "A lot of code. So, it's it's hard. And"
  },
  {
    "startMs": 156400,
    "text": "Merror's Merror's going through"
  },
  {
    "startMs": 158160,
    "text": "everything for the fine tune comb. She"
  },
  {
    "startMs": 159760,
    "text": "actually is red lines. Found some issues"
  },
  {
    "startMs": 163680,
    "text": "with the PR. And this is a a real PR. I"
  },
  {
    "startMs": 168319,
    "text": "think this is actually one of your trace"
  },
  {
    "startMs": 169360,
    "text": "tune PRs where"
  },
  {
    "startMs": 171840,
    "text": "it needs to have exact preview links for"
  },
  {
    "startMs": 174000,
    "text": "the URLs, command attribution,"
  },
  {
    "startMs": 177040,
    "text": "make sure the earlier work is um"
  },
  {
    "startMs": 179440,
    "text": "qualified as well and explain all of"
  },
  {
    "startMs": 182000,
    "text": "this in detail in the PR record to be"
  },
  {
    "startMs": 184080,
    "text": "accepted by Mera's very critical review."
  },
  {
    "startMs": 187360,
    "text": "This was interesting cuz as I was"
  },
  {
    "startMs": 189200,
    "text": "building trace tune, I was actually like"
  },
  {
    "startMs": 192720,
    "text": "as I was building it, I was using trace"
  },
  {
    "startMs": 194720,
    "text": "tune to test it. So I would have trace"
  },
  {
    "startMs": 197120,
    "text": "tune build the PRs and as I improved the"
  },
  {
    "startMs": 199680,
    "text": "PRs it would like get better."
  },
  {
    "startMs": 203280,
    "text": "You got amazing comics along the way too"
  },
  {
    "startMs": 205200,
    "text": "just to have fun while you were building"
  },
  {
    "startMs": 208239,
    "text": "stuff."
  },
  {
    "startMs": 209840,
    "text": "Addictive."
  },
  {
    "startMs": 211567,
    "text": "[laughter]"
  },
  {
    "startMs": 214000,
    "text": "So um"
  },
  {
    "startMs": 216000,
    "text": "you can see that the repeated PRs which"
  },
  {
    "startMs": 219360,
    "text": "Alex is doing to try to get good quality"
  },
  {
    "startMs": 221280,
    "text": "results are not working out well. is"
  },
  {
    "startMs": 224400,
    "text": "working out for his token billionaire"
  },
  {
    "startMs": 226319,
    "text": "card. Do do you know anything about the"
  },
  {
    "startMs": 227680,
    "text": "token billionaire program?"
  },
  {
    "startMs": 229200,
    "text": "Yeah, so actually I am a token"
  },
  {
    "startMs": 231280,
    "text": "billionaire. I spent lots and lots of"
  },
  {
    "startMs": 234480,
    "text": "tokens to build trace tune."
  },
  {
    "startMs": 239360,
    "text": "That's kind of unfortunate. Was that was"
  },
  {
    "startMs": 241920,
    "text": "that all my money?"
  },
  {
    "startMs": 243760,
    "text": "Uh don't sweat the details."
  },
  {
    "startMs": 246000,
    "text": "Okay."
  },
  {
    "startMs": 247920,
    "text": "So, um I I think there's a lot of good"
  },
  {
    "startMs": 250640,
    "text": "uses for for tokens and for spending"
  },
  {
    "startMs": 252879,
    "text": "more AI work. Um maybe we're doing"
  },
  {
    "startMs": 255120,
    "text": "repeated PRs is not where you want to"
  },
  {
    "startMs": 256880,
    "text": "spend your token"
  },
  {
    "startMs": 259040,
    "text": "and definitely the finance department's"
  },
  {
    "startMs": 260799,
    "text": "not happy with this. So, he wants a"
  },
  {
    "startMs": 262960,
    "text": "results and he wants to make sure that"
  },
  {
    "startMs": 264720,
    "text": "the PRs are accurate, are working and he"
  },
  {
    "startMs": 268160,
    "text": "has they don't have need to keep getting"
  },
  {
    "startMs": 269759,
    "text": "repeated again and again by reviewing"
  },
  {
    "startMs": 272320,
    "text": "the logs and reconstructing the evidence"
  },
  {
    "startMs": 274400,
    "text": "repeatedly."
  },
  {
    "startMs": 276000,
    "text": "And it's interesting cuz normally when"
  },
  {
    "startMs": 279040,
    "text": "you're just like uh coding on your"
  },
  {
    "startMs": 281840,
    "text": "agent, you're in your session and you"
  },
  {
    "startMs": 283919,
    "text": "ask it to write a PR, it just reads your"
  },
  {
    "startMs": 288400,
    "text": "session log to try to figure out what it"
  },
  {
    "startMs": 292080,
    "text": "should have done using its inflated"
  },
  {
    "startMs": 294240,
    "text": "context window."
  },
  {
    "startMs": 296240,
    "text": "It actually often like as it's churning"
  },
  {
    "startMs": 299120,
    "text": "it compresses its context window. So it"
  },
  {
    "startMs": 301680,
    "text": "gets more and more inaccurate as it goes"
  },
  {
    "startMs": 303440,
    "text": "along."
  },
  {
    "startMs": 305120,
    "text": "So instead, what we're doing here is"
  },
  {
    "startMs": 306800,
    "text": "with trace tune, we're having it read"
  },
  {
    "startMs": 309840,
    "text": "through uh different sessions on your"
  },
  {
    "startMs": 312320,
    "text": "computer to try to really figure out"
  },
  {
    "startMs": 314639,
    "text": "what you built and what happened."
  },
  {
    "startMs": 318560,
    "text": "Yeah. And I think like like that concept"
  },
  {
    "startMs": 320479,
    "text": "of um looking through the audit trails,"
  },
  {
    "startMs": 322960,
    "text": "looking through like what the agents"
  },
  {
    "startMs": 324080,
    "text": "have actually done is a lot of what you"
  },
  {
    "startMs": 326880,
    "text": "were doing with the Showtale project you"
  },
  {
    "startMs": 328880,
    "text": "were doing for education."
  },
  {
    "startMs": 331280,
    "text": "Yeah. I actually started"
  },
  {
    "startMs": 333840,
    "text": "this project with this showtail project"
  },
  {
    "startMs": 336720,
    "text": "because there was this like academic"
  },
  {
    "startMs": 338560,
    "text": "research study I was supporting. But"
  },
  {
    "startMs": 341039,
    "text": "basically the issue they hit is students"
  },
  {
    "startMs": 343840,
    "text": "were creating programs but there was no"
  },
  {
    "startMs": 346240,
    "text": "way to trace back the actual work and"
  },
  {
    "startMs": 348880,
    "text": "effort they did cuz with AI coding the"
  },
  {
    "startMs": 352560,
    "text": "what you submit is the final work the"
  },
  {
    "startMs": 354800,
    "text": "program but most of the work is the"
  },
  {
    "startMs": 357199,
    "text": "prompting and everything that came"
  },
  {
    "startMs": 359280,
    "text": "before and that doesn't get submitted"
  },
  {
    "startMs": 361600,
    "text": "with the program and I use the"
  },
  {
    "startMs": 364960,
    "text": "technology and theory behind that to be"
  },
  {
    "startMs": 367280,
    "text": "like well I can also improve poll"
  },
  {
    "startMs": 369840,
    "text": "requests with all this data collection."
  },
  {
    "startMs": 374639,
    "text": "Cool. And I think that improvement in"
  },
  {
    "startMs": 377039,
    "text": "pull requests is something that benefits"
  },
  {
    "startMs": 378800,
    "text": "everybody because now you can find the"
  },
  {
    "startMs": 381440,
    "text": "evidence like um logging the change"
  },
  {
    "startMs": 384000,
    "text": "directory before the or the CWD before"
  },
  {
    "startMs": 386479,
    "text": "the CD runs. explaining why maybe other"
  },
  {
    "startMs": 389680,
    "text": "test data is leaking into other repos"
  },
  {
    "startMs": 392479,
    "text": "and explaining what's happening behind"
  },
  {
    "startMs": 394880,
    "text": "the scenes on the PR rather than what"
  },
  {
    "startMs": 397520,
    "text": "most AIPRs read like which is a line by"
  },
  {
    "startMs": 400400,
    "text": "line explanation of the diffs since"
  },
  {
    "startMs": 402639,
    "text": "that's what they build the PRs off of"
  },
  {
    "startMs": 408560,
    "text": "but incorrect facts can still pollute"
  },
  {
    "startMs": 411520,
    "text": "the information um can cause a problem"
  },
  {
    "startMs": 414240,
    "text": "with conflicting PRs and then still give"
  },
  {
    "startMs": 417759,
    "text": "you back inaccurate results."
  },
  {
    "startMs": 420880,
    "text": "So maybe we have a solution for this."
  },
  {
    "startMs": 422800,
    "text": "What What do you What do you think? Do"
  },
  {
    "startMs": 424479,
    "text": "you have a solution, Cassandra?"
  },
  {
    "startMs": 426319,
    "text": "Yeah, everyone should use Trace June"
  },
  {
    "startMs": 428479,
    "text": "because of the comics."
  },
  {
    "startMs": 431199,
    "text": "So because of the comics, not because it"
  },
  {
    "startMs": 433199,
    "text": "gives you better PRs, right?"
  },
  {
    "startMs": 435039,
    "text": "Well, I started with the comics and I"
  },
  {
    "startMs": 437759,
    "text": "kind of went backwards and like, oh, I"
  },
  {
    "startMs": 440319,
    "text": "need to actually make the PRs better."
  },
  {
    "startMs": 443919,
    "text": "Well, hopefully better PRs, better"
  },
  {
    "startMs": 446319,
    "text": "comics, and maybe better token savings"
  },
  {
    "startMs": 450800,
    "text": "if by the end of this. Maybe"
  },
  {
    "startMs": 453280,
    "text": "it should be."
  },
  {
    "startMs": 456000,
    "text": "So, um, trade students an open source"
  },
  {
    "startMs": 458160,
    "text": "project. It lets you improve the quality"
  },
  {
    "startMs": 460319,
    "text": "of your PRs by pulling in agent context"
  },
  {
    "startMs": 463120,
    "text": "by pulling in information about the the"
  },
  {
    "startMs": 466240,
    "text": "code and then having an independent"
  },
  {
    "startMs": 468240,
    "text": "writer independent reviewer build better"
  },
  {
    "startMs": 470880,
    "text": "quality PRs plus optionally giving you"
  },
  {
    "startMs": 474080,
    "text": "an amazing comic which what do you think"
  },
  {
    "startMs": 477280,
    "text": "the best feature is here Cassandra?"
  },
  {
    "startMs": 479680,
    "text": "I think it's obviously the comics."
  },
  {
    "startMs": 482560,
    "text": "This is one of the earlier comics from"
  },
  {
    "startMs": 484319,
    "text": "when I was using traune to build it."
  },
  {
    "startMs": 490560,
    "text": "That's good. And then you you you made"
  },
  {
    "startMs": 492080,
    "text": "sure that there's new characters every"
  },
  {
    "startMs": 493599,
    "text": "episode, not none of this repeating"
  },
  {
    "startMs": 495199,
    "text": "characters."
  },
  {
    "startMs": 496560,
    "text": "Yeah, that's what this PR was about."
  },
  {
    "startMs": 502000,
    "text": "Okay. And then this is a description of"
  },
  {
    "startMs": 504400,
    "text": "the original trace tune flow. So it"
  },
  {
    "startMs": 507120,
    "text": "finds all the traces for a particular"
  },
  {
    "startMs": 508879,
    "text": "branch. It researches them, reviews the"
  },
  {
    "startMs": 511680,
    "text": "description, adds more context, and kind"
  },
  {
    "startMs": 513440,
    "text": "of goes in a loop until it gets all of"
  },
  {
    "startMs": 516320,
    "text": "the evidence it needs to make a a"
  },
  {
    "startMs": 518240,
    "text": "quality PR. And um subjectively, but"
  },
  {
    "startMs": 522080,
    "text": "also through um PRB bench, which is"
  },
  {
    "startMs": 524399,
    "text": "testing framework, um gives you better"
  },
  {
    "startMs": 527519,
    "text": "quality, better research, lower um rate"
  },
  {
    "startMs": 531200,
    "text": "of inaccuracies in the PRs. I think at"
  },
  {
    "startMs": 533839,
    "text": "this point the PRs were already better"
  },
  {
    "startMs": 536399,
    "text": "than if you don't use trace dune, but I"
  },
  {
    "startMs": 538720,
    "text": "found that it still makes some mistakes"
  },
  {
    "startMs": 540959,
    "text": "occasionally."
  },
  {
    "startMs": 542720,
    "text": "You can see at the end that it tries to"
  },
  {
    "startMs": 544320,
    "text": "revise with more context. I think that's"
  },
  {
    "startMs": 546640,
    "text": "where the mistakes come in."
  },
  {
    "startMs": 549551,
    "text": "[snorts]"
  },
  {
    "startMs": 550720,
    "text": "Okay. Well, Alex Alex thinks it's good"
  },
  {
    "startMs": 552560,
    "text": "enough. He's going to try to solve the"
  },
  {
    "startMs": 554080,
    "text": "token crisis the factory with trace tune"
  },
  {
    "startMs": 556240,
    "text": "and see if this will solve all of his"
  },
  {
    "startMs": 557680,
    "text": "problems. Polish checked. Alex looks"
  },
  {
    "startMs": 562160,
    "text": "very very happy"
  },
  {
    "startMs": 564480,
    "text": "but Mera doesn't look that happy."
  },
  {
    "startMs": 566800,
    "text": "It looks like we still hit a mistake"
  },
  {
    "startMs": 568959,
    "text": "here like I was saying earlier. So the"
  },
  {
    "startMs": 571920,
    "text": "original PR claim was no later passing"
  },
  {
    "startMs": 574240,
    "text": "run is recorded."
  },
  {
    "startMs": 576800,
    "text": "And the problem here is the in the"
  },
  {
    "startMs": 579760,
    "text": "actual trace it shows it running a bun"
  },
  {
    "startMs": 583519,
    "text": "test at the same time as the bun xtsc"
  },
  {
    "startMs": 588720,
    "text": "and you can't run those two at the same"
  },
  {
    "startMs": 590560,
    "text": "time. They have to be run sequentially"
  },
  {
    "startMs": 592959,
    "text": "for the test to pass."
  },
  {
    "startMs": 595200,
    "text": "Yeah, exactly. So he claims test passing"
  },
  {
    "startMs": 598320,
    "text": "and all the checks were run and actually"
  },
  {
    "startMs": 601040,
    "text": "that's not what the code did. So poor"
  },
  {
    "startMs": 604640,
    "text": "Alex. Another failed PR, a very angry"
  },
  {
    "startMs": 608080,
    "text": "badger with all the token cost. Um kind"
  },
  {
    "startMs": 612160,
    "text": "of kind of makes me feel like Monday"
  },
  {
    "startMs": 614000,
    "text": "morning with the Neo Forj finance"
  },
  {
    "startMs": 615680,
    "text": "department looking at my token spend"
  },
  {
    "startMs": 617279,
    "text": "each week. But they're they're very"
  },
  {
    "startMs": 619200,
    "text": "forgiving, [laughter]"
  },
  {
    "startMs": 621519,
    "text": "especially when it's a important project"
  },
  {
    "startMs": 624000,
    "text": "like like this. So maybe Alex can save"
  },
  {
    "startMs": 628240,
    "text": "some cost by using cheaper writers is"
  },
  {
    "startMs": 630079,
    "text": "the number one thing finance asked me to"
  },
  {
    "startMs": 631680,
    "text": "do. They're like, \"Hey, hey, no more"
  },
  {
    "startMs": 633279,
    "text": "opus on it. Haiku, do something cheaper,"
  },
  {
    "startMs": 636240,
    "text": "please. Does this work?\""
  },
  {
    "startMs": 640000,
    "text": "I feel like the quality is a bit lower"
  },
  {
    "startMs": 642160,
    "text": "when you do it like that. Just reducing"
  },
  {
    "startMs": 644560,
    "text": "the cost doesn't quite help."
  },
  {
    "startMs": 646959,
    "text": "Yeah. So, we were trying to get better"
  },
  {
    "startMs": 648240,
    "text": "quality and we we ended up getting worse"
  },
  {
    "startMs": 650399,
    "text": "quality by just using cheaper writers."
  },
  {
    "startMs": 653519,
    "text": "And"
  },
  {
    "startMs": 655279,
    "text": "obviously, Alex is in a bind here,"
  },
  {
    "startMs": 657040,
    "text": "right? He's he's spent too much on"
  },
  {
    "startMs": 658560,
    "text": "revisions. He doesn't have a solution"
  },
  {
    "startMs": 661279,
    "text": "for reducing token costs. He's not"
  },
  {
    "startMs": 662720,
    "text": "getting the results he wants out of his"
  },
  {
    "startMs": 664000,
    "text": "PRs. But"
  },
  {
    "startMs": 667680,
    "text": "maybe there's a there's something Pip"
  },
  {
    "startMs": 669760,
    "text": "can help us with. Pip Pip has a friend."
  },
  {
    "startMs": 672880,
    "text": "I wonder who this friend could be."
  },
  {
    "startMs": 675600,
    "text": "Oh, well, I I think I think you know"
  },
  {
    "startMs": 678399,
    "text": "from my previous talks that that this"
  },
  {
    "startMs": 680480,
    "text": "friend is definitely going to be a"
  },
  {
    "startMs": 682959,
    "text": "graph."
  },
  {
    "startMs": 684880,
    "text": "You're the graph guy. We're doing"
  },
  {
    "startMs": 686560,
    "text": "another graph again."
  },
  {
    "startMs": 688320,
    "text": "Yeah. Yeah. There's always a new graph,"
  },
  {
    "startMs": 690079,
    "text": "but this is not the same old graph. This"
  },
  {
    "startMs": 693040,
    "text": "is not the the Neo Forj database. This"
  },
  {
    "startMs": 696560,
    "text": "is our new friend Archie,"
  },
  {
    "startMs": 699920,
    "text": "who is a embedded database written in"
  },
  {
    "startMs": 703040,
    "text": "Rust called relate that actually is"
  },
  {
    "startMs": 706320,
    "text": "being launched this week"
  },
  {
    "startMs": 709680,
    "text": "at the nodes conference and also this"
  },
  {
    "startMs": 711680,
    "text": "talk at a engineer in San Francisco."
  },
  {
    "startMs": 714880,
    "text": "And it's an open- source um database"
  },
  {
    "startMs": 718160,
    "text": "which allows us to get a lot of the"
  },
  {
    "startMs": 719680,
    "text": "benefits of more accuracy, multihop"
  },
  {
    "startMs": 723839,
    "text": "queries, more explainable traces without"
  },
  {
    "startMs": 727440,
    "text": "having the heavyweight approach. So it's"
  },
  {
    "startMs": 729279,
    "text": "kind of an agent focused graph database"
  },
  {
    "startMs": 732240,
    "text": "specifically for these sort of embedded"
  },
  {
    "startMs": 734720,
    "text": "use cases."
  },
  {
    "startMs": 738399,
    "text": "Okay, so let's see how Archie can help"
  },
  {
    "startMs": 740240,
    "text": "us out. So before we were relying a lot"
  },
  {
    "startMs": 743440,
    "text": "on on agents and loops of agents to"
  },
  {
    "startMs": 745680,
    "text": "resolve the problem. But graph databases"
  },
  {
    "startMs": 748560,
    "text": "can store relationships can find like"
  },
  {
    "startMs": 751600,
    "text": "like different things which are"
  },
  {
    "startMs": 753279,
    "text": "clustered build a brief which connects"
  },
  {
    "startMs": 757040,
    "text": "all this information together and then"
  },
  {
    "startMs": 759360,
    "text": "give the agents even a cheaper agent"
  },
  {
    "startMs": 761519,
    "text": "better information and more context to"
  },
  {
    "startMs": 763440,
    "text": "give a good drafting decision. And the"
  },
  {
    "startMs": 765200,
    "text": "good thing about Archie is he has zero"
  },
  {
    "startMs": 767839,
    "text": "LLM costs. It's all local compute cost"
  },
  {
    "startMs": 771360,
    "text": "and storage and it's a fully immemory"
  },
  {
    "startMs": 773519,
    "text": "database and rest so it's highly"
  },
  {
    "startMs": 774800,
    "text": "optimized. So we get this providence"
  },
  {
    "startMs": 778880,
    "text": "about all these submitted changes and"
  },
  {
    "startMs": 780480,
    "text": "this evidence graph basically for free"
  },
  {
    "startMs": 783440,
    "text": "compared to all the LLM and token costs"
  },
  {
    "startMs": 785760,
    "text": "which we were doing before. And this"
  },
  {
    "startMs": 788000,
    "text": "means our our buddy Alex doesn't need to"
  },
  {
    "startMs": 790399,
    "text": "research. She doesn't need to go through"
  },
  {
    "startMs": 791760,
    "text": "and find all the facts again and go"
  },
  {
    "startMs": 793360,
    "text": "through the sources"
  },
  {
    "startMs": 795279,
    "text": "because relate is going to do all that"
  },
  {
    "startMs": 798079,
    "text": "hard work for him and build an evidence"
  },
  {
    "startMs": 800240,
    "text": "packet that he can use for trace tune."
  },
  {
    "startMs": 803600,
    "text": "So all sounds really great. Maybe we do"
  },
  {
    "startMs": 805600,
    "text": "need more graphs."
  },
  {
    "startMs": 808240,
    "text": "That's I'm surprised to hear that from"
  },
  {
    "startMs": 810000,
    "text": "you after all the graph talks you've had"
  },
  {
    "startMs": 811680,
    "text": "to put up with. Okay, but"
  },
  {
    "startMs": 814720,
    "text": "we can do this and make it even better"
  },
  {
    "startMs": 817040,
    "text": "with the neighborhood brief. So"
  },
  {
    "startMs": 818480,
    "text": "basically taking all of the the Y the"
  },
  {
    "startMs": 820800,
    "text": "code and the checks putting attributions"
  },
  {
    "startMs": 823200,
    "text": "and relationships between those and then"
  },
  {
    "startMs": 825120,
    "text": "returning back a connected set of"
  },
  {
    "startMs": 826959,
    "text": "information as a packet. And by doing"
  },
  {
    "startMs": 829200,
    "text": "this then when the agent is drafting up"
  },
  {
    "startMs": 831839,
    "text": "the PR it has all the connected facts"
  },
  {
    "startMs": 834800,
    "text": "with their sources together in a single"
  },
  {
    "startMs": 837440,
    "text": "source. And that makes the writer draft"
  },
  {
    "startMs": 841360,
    "text": "more accurate with fewer passes and it"
  },
  {
    "startMs": 844639,
    "text": "also makes the reviewer's job much"
  },
  {
    "startMs": 846800,
    "text": "easier. So we don't need an expensive"
  },
  {
    "startMs": 848560,
    "text": "reviewer. We can save by using a few"
  },
  {
    "startMs": 851680,
    "text": "haiku reviewers instead of a um sonnet"
  },
  {
    "startMs": 854800,
    "text": "reviewer or or a luna in the case of"
  },
  {
    "startMs": 856399,
    "text": "codeex to find come up with the findings"
  },
  {
    "startMs": 859360,
    "text": "check those against the draft redo the"
  },
  {
    "startMs": 861519,
    "text": "writing and then we retain the higher"
  },
  {
    "startMs": 864000,
    "text": "quality writer a a sonnet or terror"
  },
  {
    "startMs": 867440,
    "text": "class writer to make sure we have a high"
  },
  {
    "startMs": 869120,
    "text": "quality brief that will pass human"
  },
  {
    "startMs": 870720,
    "text": "approval."
  },
  {
    "startMs": 873279,
    "text": "When we were building this, it felt very"
  },
  {
    "startMs": 875120,
    "text": "counterintuitive to have the writer be"
  },
  {
    "startMs": 877279,
    "text": "high higher quality than the reviewers."
  },
  {
    "startMs": 880079,
    "text": "It actually it it's common that people"
  },
  {
    "startMs": 883040,
    "text": "always want to have the planner, the"
  },
  {
    "startMs": 884639,
    "text": "reviewer, the checker to be the highest"
  },
  {
    "startMs": 886880,
    "text": "quality model. But if you're doing high"
  },
  {
    "startMs": 890560,
    "text": "quality writing, a lower model, no"
  },
  {
    "startMs": 892720,
    "text": "matter how many iterations and how much"
  },
  {
    "startMs": 894480,
    "text": "feedback you give it, can't do it. It"
  },
  {
    "startMs": 896800,
    "text": "just turns in things into a checklist"
  },
  {
    "startMs": 898480,
    "text": "and makes it very mechanical. So to give"
  },
  {
    "startMs": 900720,
    "text": "writing which feels humanlike, you do"
  },
  {
    "startMs": 903040,
    "text": "need a higher quality model. And it's"
  },
  {
    "startMs": 905120,
    "text": "it's actually the the reviewing and the"
  },
  {
    "startMs": 906959,
    "text": "cross-checking step which is more"
  },
  {
    "startMs": 908320,
    "text": "mechanical. So it's a little"
  },
  {
    "startMs": 909279,
    "text": "counterintuitive, but that works out"
  },
  {
    "startMs": 911760,
    "text": "quite well. It's also a little"
  },
  {
    "startMs": 914160,
    "text": "counterintuitive that um our badger has"
  },
  {
    "startMs": 917279,
    "text": "a thumbs up coming out of his chest, but"
  },
  {
    "startMs": 919440,
    "text": "let's let's ignore the high quality AI"
  },
  {
    "startMs": 921360,
    "text": "artwork."
  },
  {
    "startMs": 925199,
    "text": "Okay, so we've checked all the claims"
  },
  {
    "startMs": 926720,
    "text": "against the evidence. We've looked for"
  },
  {
    "startMs": 928880,
    "text": "missing findings with our cheaper"
  },
  {
    "startMs": 931120,
    "text": "reviewer. Did the focus work and then"
  },
  {
    "startMs": 935360,
    "text": "found the things which were connected"
  },
  {
    "startMs": 937440,
    "text": "information to give a better draft and"
  },
  {
    "startMs": 940000,
    "text": "review"
  },
  {
    "startMs": 941600,
    "text": "for mirror."
  },
  {
    "startMs": 944160,
    "text": "And here's the output. So it has the the"
  },
  {
    "startMs": 946560,
    "text": "Y. So log CWD and that describes the"
  },
  {
    "startMs": 949920,
    "text": "shell before the CD runs. What's the"
  },
  {
    "startMs": 952399,
    "text": "exception?"
  },
  {
    "startMs": 954000,
    "text": "So the CD targets in repo or the command"
  },
  {
    "startMs": 956800,
    "text": "mentions a route. What passed? It has"
  },
  {
    "startMs": 958880,
    "text": "type check plus 71 test checked on"
  },
  {
    "startMs": 961519,
    "text": "submitted content not before."
  },
  {
    "startMs": 964560,
    "text": "And then it starts in correlate.ts"
  },
  {
    "startMs": 966959,
    "text": "inspecting the leading CD exception."
  },
  {
    "startMs": 970000,
    "text": "And Meera looks pretty happy with these"
  },
  {
    "startMs": 972480,
    "text": "results, right?"
  },
  {
    "startMs": 974639,
    "text": "Yeah. This might be the first scene"
  },
  {
    "startMs": 976240,
    "text": "where she's genuinely happy."
  },
  {
    "startMs": 979360,
    "text": "I think she was kind of happy with the"
  },
  {
    "startMs": 981040,
    "text": "the AI engineer sweatshirt before she"
  },
  {
    "startMs": 983839,
    "text": "realized that Alex actually was a was a"
  },
  {
    "startMs": 985839,
    "text": "goof. Okay, so here are some of the"
  },
  {
    "startMs": 988800,
    "text": "actual data and results from the"
  },
  {
    "startMs": 992160,
    "text": "sideby-side analysis using PRBench,"
  },
  {
    "startMs": 994399,
    "text": "which is a um pull request benchmark,"
  },
  {
    "startMs": 997120,
    "text": "which uses"
  },
  {
    "startMs": 998880,
    "text": "audit traces plus code and um LM judges"
  },
  {
    "startMs": 1002959,
    "text": "to review the quality of the work. Of"
  },
  {
    "startMs": 1005360,
    "text": "course, reviewed by a human as well for"
  },
  {
    "startMs": 1007120,
    "text": "accuracy. And you can see that on both"
  },
  {
    "startMs": 1009519,
    "text": "Claude and Codeex, you get a significant"
  },
  {
    "startMs": 1012480,
    "text": "quality improvements on a scale of 1 to"
  },
  {
    "startMs": 1014160,
    "text": "100. The judges by and large almost"
  },
  {
    "startMs": 1017759,
    "text": "unilaterally liked the trace tune plus"
  },
  {
    "startMs": 1021279,
    "text": "relate quality better. Um on a 1 to 100"
  },
  {
    "startMs": 1025120,
    "text": "scale it was like 74 to 76 82 to 86. So"
  },
  {
    "startMs": 1028480,
    "text": "plus two plus 4 point."
  },
  {
    "startMs": 1030400,
    "text": "These tests contributed to my token"
  },
  {
    "startMs": 1032559,
    "text": "billionaire card because we were running"
  },
  {
    "startMs": 1035280,
    "text": "lots and lots of tests. We created this"
  },
  {
    "startMs": 1037600,
    "text": "side project called PRB bench just for"
  },
  {
    "startMs": 1039760,
    "text": "testing and it we would have different"
  },
  {
    "startMs": 1042160,
    "text": "agents test over different poll requests"
  },
  {
    "startMs": 1044959,
    "text": "and compare the results. Initially I had"
  },
  {
    "startMs": 1048319,
    "text": "it like running like sequentially. It'll"
  },
  {
    "startMs": 1052240,
    "text": "do one test and finish it then do the"
  },
  {
    "startMs": 1054160,
    "text": "next but then we decided to run them all"
  },
  {
    "startMs": 1056640,
    "text": "in parallel. It's a very quick way to"
  },
  {
    "startMs": 1059200,
    "text": "reach high spend."
  },
  {
    "startMs": 1060880,
    "text": "Yeah. So that's kind of funny. I I at at"
  },
  {
    "startMs": 1063360,
    "text": "at at the current testing rate with the"
  },
  {
    "startMs": 1065360,
    "text": "number of use cases and tests and the"
  },
  {
    "startMs": 1067039,
    "text": "benchmark um we can fully saturate my my"
  },
  {
    "startMs": 1071520,
    "text": "high-end gaming computer with as many"
  },
  {
    "startMs": 1074720,
    "text": "threads as Linux can handle"
  },
  {
    "startMs": 1076160,
    "text": "simultaneously"
  },
  {
    "startMs": 1077760,
    "text": "and and of course the token spend is is"
  },
  {
    "startMs": 1080960,
    "text": "slight slightly higher than what we'd"
  },
  {
    "startMs": 1082720,
    "text": "like. But it's all for the good of"
  },
  {
    "startMs": 1084559,
    "text": "improving the quality of pull requests"
  },
  {
    "startMs": 1086400,
    "text": "for the world. So you all get the"
  },
  {
    "startMs": 1088799,
    "text": "benefit of all these test runs and"
  },
  {
    "startMs": 1090240,
    "text": "having the highest quality improvements"
  },
  {
    "startMs": 1093120,
    "text": "and you save on tokens. So we've spent"
  },
  {
    "startMs": 1095679,
    "text": "the tokens for you in advance because"
  },
  {
    "startMs": 1099600,
    "text": "trace tune with the graph solution takes"
  },
  {
    "startMs": 1102880,
    "text": "about half as many tokens as it did"
  },
  {
    "startMs": 1105919,
    "text": "previously and which is an amazing"
  },
  {
    "startMs": 1108080,
    "text": "result from the graph because now you're"
  },
  {
    "startMs": 1109840,
    "text": "premputing you're getting all this"
  },
  {
    "startMs": 1111200,
    "text": "evidence graph construct in advance."
  },
  {
    "startMs": 1114559,
    "text": "The writing's cheaper, the reviewing's"
  },
  {
    "startMs": 1117360,
    "text": "cheaper with a higher quality result."
  },
  {
    "startMs": 1121440,
    "text": "I wasn't sure about the whole graph"
  },
  {
    "startMs": 1123360,
    "text": "thing initially, but after seeing the"
  },
  {
    "startMs": 1125600,
    "text": "cost tank almost in half, like it's not"
  },
  {
    "startMs": 1128799,
    "text": "even a question, it's making this"
  },
  {
    "startMs": 1130720,
    "text": "program work."
  },
  {
    "startMs": 1133679,
    "text": "And this is something we did for the"
  },
  {
    "startMs": 1135280,
    "text": "Trey Stone project, but you can do this"
  },
  {
    "startMs": 1136799,
    "text": "for pretty much any project where you"
  },
  {
    "startMs": 1139919,
    "text": "have data, you have relationships, you"
  },
  {
    "startMs": 1141600,
    "text": "have evidence, you have things which"
  },
  {
    "startMs": 1142880,
    "text": "you're trying to to compare because"
  },
  {
    "startMs": 1146000,
    "text": "agents are not that good at doing the"
  },
  {
    "startMs": 1148559,
    "text": "data analysis at doing all of the"
  },
  {
    "startMs": 1150400,
    "text": "evaluation and the curation. So if you"
  },
  {
    "startMs": 1153280,
    "text": "give them a help at the front end by"
  },
  {
    "startMs": 1155760,
    "text": "creating a a custom graph layer with an"
  },
  {
    "startMs": 1158080,
    "text": "in-memory database like relate suddenly"
  },
  {
    "startMs": 1161039,
    "text": "your costs go down your quality goes up"
  },
  {
    "startMs": 1163360,
    "text": "and you too"
  },
  {
    "startMs": 1166400,
    "text": "can be a token billionaire like our"
  },
  {
    "startMs": 1168080,
    "text": "friend Alex. Well I guess it's a now"
  },
  {
    "startMs": 1170720,
    "text": "he's using it as a bookmark."
  },
  {
    "startMs": 1173039,
    "text": "I think that's a good use for it. It's a"
  },
  {
    "startMs": 1174960,
    "text": "really high quality bookmark. It's"
  },
  {
    "startMs": 1176960,
    "text": "heavier than a credit card."
  },
  {
    "startMs": 1180640,
    "text": "It is. It is a nice surprise for um"
  },
  {
    "startMs": 1182799,
    "text": "spending all those tokens on PR Bench."
  },
  {
    "startMs": 1186559,
    "text": "So, finance is happy. Lower drafting"
  },
  {
    "startMs": 1189200,
    "text": "costs that they can reasonably fund."
  },
  {
    "startMs": 1192000,
    "text": "Alex is happy. He has his new bookmark."
  },
  {
    "startMs": 1194240,
    "text": "key is now has a great case study"
  },
  {
    "startMs": 1197760,
    "text": "with the connections doing neighborhood"
  },
  {
    "startMs": 1200000,
    "text": "retrievable of facts so that it's a"
  },
  {
    "startMs": 1202320,
    "text": "better brief focusing the review so the"
  },
  {
    "startMs": 1205120,
    "text": "reviewers get the same compact brief ma"
  },
  {
    "startMs": 1207840,
    "text": "gets exactly the evidence she needs and"
  },
  {
    "startMs": 1209840,
    "text": "then here's Alex on stage at um AI"
  },
  {
    "startMs": 1213520,
    "text": "engineer somewhere in the world better"
  },
  {
    "startMs": 1215919,
    "text": "PR's lower drafting cost with a graph"
  },
  {
    "startMs": 1220080,
    "text": "plus open-source"
  },
  {
    "startMs": 1222320,
    "text": "trace to builds, which also gives you"
  },
  {
    "startMs": 1225039,
    "text": "amazing comics."
  },
  {
    "startMs": 1230559,
    "text": "Okay, so big round of applause for our"
  },
  {
    "startMs": 1233039,
    "text": "friend Alex, who's now earned the AI"
  },
  {
    "startMs": 1236640,
    "text": "engineer hoodie."
  },
  {
    "startMs": 1238720,
    "text": "I think that's definitely a wardrobe"
  },
  {
    "startMs": 1240640,
    "text": "upgrade for him."
  },
  {
    "startMs": 1242720,
    "text": "Yeah, I don't think the pocket protector"
  },
  {
    "startMs": 1244400,
    "text": "look was that great for him. So, good"
  },
  {
    "startMs": 1247600,
    "text": "for him. I think we'll see Swixs on"
  },
  {
    "startMs": 1250000,
    "text": "stage with one of these hoodies soon. He"
  },
  {
    "startMs": 1251919,
    "text": "he can't he can't not do it now that we"
  },
  {
    "startMs": 1254240,
    "text": "shamed him into doing it at his own"
  },
  {
    "startMs": 1255679,
    "text": "conference. The token billion program is"
  },
  {
    "startMs": 1258159,
    "text": "amazing. A big kudos to the team for"
  },
  {
    "startMs": 1260880,
    "text": "coming up with ways to incentivize and"
  },
  {
    "startMs": 1263200,
    "text": "reward people who are on the bleeding"
  },
  {
    "startMs": 1264480,
    "text": "edge of AI usage and learning how to"
  },
  {
    "startMs": 1267919,
    "text": "make it really effective for the rest of"
  },
  {
    "startMs": 1269600,
    "text": "us. And um"
  },
  {
    "startMs": 1272960,
    "text": "now you too can have better PRs. Impress"
  },
  {
    "startMs": 1276720,
    "text": "your co-workers by having"
  },
  {
    "startMs": 1279280,
    "text": "wellressearched"
  },
  {
    "startMs": 1280880,
    "text": "effectively written and lowerc cost PRs"
  },
  {
    "startMs": 1283360,
    "text": "with a comic"
  },
  {
    "startMs": 1285280,
    "text": "with a comic. So for more information go"
  },
  {
    "startMs": 1287679,
    "text": "to trace tune.com"
  },
  {
    "startMs": 1289840,
    "text": "learn about trace tune. Give it a try."
  },
  {
    "startMs": 1291679,
    "text": "It works with cloud. It works with"
  },
  {
    "startMs": 1292960,
    "text": "codecs. It works with all your favorite"
  },
  {
    "startMs": 1294320,
    "text": "AI tools. And if you want to do these"
  },
  {
    "startMs": 1297440,
    "text": "same sort of amazing results on your own"
  },
  {
    "startMs": 1299120,
    "text": "project, try out Relate, our new"
  },
  {
    "startMs": 1302240,
    "text": "Rustbased embedded open-source graph"
  },
  {
    "startMs": 1304559,
    "text": "database, which brings context to your"
  },
  {
    "startMs": 1307280,
    "text": "agents and helps you build more"
  },
  {
    "startMs": 1308880,
    "text": "efficient applications that just work"
  },
  {
    "startMs": 1311440,
    "text": "better, are cheaper, and give you better"
  },
  {
    "startMs": 1313919,
    "text": "results. So, thank you all for joining"
  },
  {
    "startMs": 1315840,
    "text": "our AI engineer San Francisco talk. And"
  },
  {
    "startMs": 1319360,
    "text": "if you're watching this talk this week"
  },
  {
    "startMs": 1321280,
    "text": "at San Francisco,"
  },
  {
    "startMs": 1324080,
    "text": "what should you do this weekend with"
  },
  {
    "startMs": 1325919,
    "text": "your kids on Saturday? Cassandra,"
  },
  {
    "startMs": 1328559,
    "text": "we are running an AI engineer kids day."
  },
  {
    "startMs": 1331600,
    "text": "So, anyone who's available, just sign"
  },
  {
    "startMs": 1334000,
    "text": "your kids up. We're teaching kids"
  },
  {
    "startMs": 1335679,
    "text": "workshops. Kids will be learning how to"
  },
  {
    "startMs": 1338000,
    "text": "do AI programming to make their own"
  },
  {
    "startMs": 1340080,
    "text": "games. And I think we're also doing a"
  },
  {
    "startMs": 1343360,
    "text": "No, it's just the gaming one."
  },
  {
    "startMs": 1346400,
    "text": "No, we'll do we'll do the robots, too."
  },
  {
    "startMs": 1348240,
    "text": "Okay. Yeah, we'll also be doing robots."
  },
  {
    "startMs": 1350480,
    "text": "So, kids will get to control these"
  },
  {
    "startMs": 1352080,
    "text": "little soda can robots using AI."
  },
  {
    "startMs": 1355840,
    "text": "Yeah. So, a lot of fun. Physical AI"
  },
  {
    "startMs": 1358559,
    "text": "gaming AI all in one place. Um, sign up"
  },
  {
    "startMs": 1361840,
    "text": "at AI engineer kids day and um, you"
  },
  {
    "startMs": 1365120,
    "text": "know, bring your kids, bring your"
  },
  {
    "startMs": 1366320,
    "text": "nieces, bring your nephews, and let's"
  },
  {
    "startMs": 1368559,
    "text": "train the next generation of AI"
  },
  {
    "startMs": 1370480,
    "text": "engineers."
  },
  {
    "startMs": 1373039,
    "text": "All right. Thanks so much and enjoy the"
  },
  {
    "startMs": 1374720,
    "text": "rest of the conference."
  }
]
