# Your PR Says the Tests Passed. Prove It. - Stephen Chin & Cassandra Chin

Stephen Chin, Neo4j · Cassandra Chin

[Talk page](https://ai.engineer/talks/gbeuC0ud1p4-your-pr-says-tests-passed-prove-it) · [Video](https://www.youtube.com/watch?v=gbeuC0ud1p4)

## Your PR Says the Tests Passed. Prove It.

A session log can contain both a test command and a type check and still leave a reviewer wondering what passed. Cassandra Chin and Stephen Chin follow a Tracetoon pull request from that gap to a description that identifies which code was checked and where to start reading.

### What sent the pull request back

[0:00](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=0s)

Cassandra Chin built Tracetoon after seeing good code turned away in review because its pull request description got the facts wrong. The tool writes a description and can turn the change into a comic. In the story she and Stephen Chin tell, Alex works in a token factory, Pip looks after the evidence, finance watches the bill, and Alex would dearly like to earn an AI engineer hoodie.

Alex submits a long ledger of changes. What could a reviewer still need? The redlines, drawn from a real PR, ask for exact preview links, attribution for the commands, and a qualification about work done earlier. Each request gives the reviewer something to check. They can open the particular preview, trace a command to its source, and distinguish earlier work from what this PR delivers.

Cassandra used Tracetoon to write PRs for Tracetoon itself, so each change was another opportunity to read the output and improve it. In the factory, every trip back through review sends Alex into the logs to reconstruct the evidence again. The growing bill was the part Cassandra could vouch for personally. Building and testing Tracetoon had earned her a token billionaire card.

### Recovering the work behind a branch

[4:36](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=276s)

Ask a coding agent to write a PR at the end of a long session and it can draw on the conversation it just had. Two things can fall out of that account. The agent may have compressed its context along the way, losing particulars, and some of the work may have happened in another session altogether. Tracetoon reads traces across sessions on the developer’s computer to recover what happened on the branch.

The idea came from an education study Cassandra supported. A student could hand in a working program while leaving much of the intellectual work, including the prompts that shaped it, behind in an agent session. A code diff has a similar blind spot. It can show a change to how a tool handles the shell’s current working directory (`CWD`). The history can explain why the directory recorded *before* `cd` runs matters to someone investigating how test data from one repository got associated with another.

Searching more sessions also gives the writer more opportunities to pick up a fact from the wrong PR. Tracetoon pairs the agent context with the code and gives writing and review to independent agents. The comic is optional, though Cassandra started there and worked backward to improving the descriptions. One early comic accompanied a PR about giving each episode new characters.

### Two commands in a trace, one unsupported claim

[8:22](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=502s)

The original Tracetoon gathered the traces for a branch, researched them and drafted a description. A reviewer could ask for more context, sending the writer back for another pass. Both Cassandra’s experience and the project’s PR Bench comparisons favored these descriptions over the ones produced without Tracetoon. She suspected that the final revision, when the writer tried to incorporate still more context, introduced some of the remaining mistakes.

Consider the passing-run dispute. The original PR said there was no record of a later passing run. Alex’s account nevertheless credited the change with passing tests and completing all the checks. What did the trace establish? It put `bun test` and the TypeScript type check at the same time. In this case the two checks needed to run sequentially for the test to pass. The trace accounted for both invocations, but their timing left the claimed successful sequence unproven.

Finance suggested using a cheaper writer. When the team tried that, description quality fell. The writer still had to reconstruct what happened and decide how far the evidence supported its wording. A lower price per token left that work intact.

### One neighborhood brief for the writer and reviewers

[11:15](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=675s)

The next version moved evidence gathering ahead of drafting. Relate, an open-source embedded graph database written in Rust, stores relationships that an agent would otherwise have to rediscover in the logs. A query can follow several connections to collect related facts about a change. The in-memory graph uses local compute and storage. Following those connections takes no model tokens.

Tracetoon calls the result a *neighborhood brief*. For the change at hand, it brings the rationale, the code and the checks into one packet, preserving their sources and how they relate. What reaches the reviewer? The writer supplies the draft, and the brief reaches the reviewer directly as well. Those two inputs let the reviewer check in both directions, tracing a sentence back to its source or finding an important fact that never made it into the draft.

Cassandra had expected the reviewer to need the stronger model. Giving both roles the brief changed where the team could economize.

- **Writing.** The team kept a stronger writer, such as **Sonnet**, to turn the evidence into an explanation a human could follow. With a weaker writer, successive rounds of feedback tended to turn the description into a checklist.
- **Reviewing.** Several cheaper reviewers, such as **Haiku** models, could compare claims with the brief and return specific findings to the writer for revision. Cross-checking a claim against a known source was a narrower job than independently researching the branch.

Which inputs reach the reviewer?
Reviewers receive the writer’s account and the source material used to draft it. A finding can therefore point the writer back to the evidence behind a correction.
- Neighborhood brief — Relate brings the rationale, code and checks together with their sources.
- Writer’s draft — A stronger writer explains the change and what was checked.
- Focused reviewers — Reviewers compare claims with the sources and look for omissions.
- Writer’s revision — The writer incorporates the findings into the description for human review.
- Neighborhood brief → Writer’s draft (Facts for drafting)
- Neighborhood brief → Focused reviewers (Same facts for verification)
- Writer’s draft → Focused reviewers (Claims to check)
- Focused reviewers → Writer’s revision (Findings to address)

### What the revised PR lets a human check

[15:25](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=925s)

The revised description returns to the working-directory problem. A log entry records where the shell was *before* it executes a leading `cd`. One case the description calls out is a leading `cd` into the repository. The shell can start outside the repository and still do its work inside it, so the initial directory alone could lead a reviewer to attribute the command to the wrong project. The PR points to the code that handles this case.

What passed, and on which code? The revised output reports a successful type check and 71 passing tests on the submitted content. An earlier pass would describe an earlier version. A human reviewer now has both a verification result that applies to this submission and a specific place to begin reading its implementation, the leading-`cd` exception in `correlate.ts`.

### What changed on PR Bench

[16:28](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=988s)

How much did the brief improve descriptions across PRs? The team built PR Bench to compare agents on different pull requests. Language-model judges assessed descriptions against audit traces and code, with a human also checking accuracy. Across the Claude and Codex comparisons, the judges generally preferred Tracetoon with Relate. On a 1–100 quality scale, the two reported changes were about 74 to 76 and 82 to 86, gains of two and four points in the team’s development runs.

Generating those comparisons was expensive. Cassandra initially let one benchmark run finish before starting the next. Here the cases were independent, unlike the two checks within Alex’s PR, so the team could run them in parallel. The workload could saturate Stephen’s high-end gaming computer. It was also a remarkably quick way to spend tokens.

For generating a description, token use moved in the other direction. In the team’s comparisons, Tracetoon with the graph used roughly half as many tokens as the previous workflow. Precomputing the evidence spared the writer and reviewers repeated searches while leaving the stronger writer in place. Cassandra had been skeptical about adding a graph. The reduction in spend persuaded her to keep it.

### Where the pattern travels

[18:53](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1133s)

Stephen’s suggestion for other projects starts with a question about repeated work. Which records does an agent have to find and connect every time someone asks it to write or check a claim? Those relationships are candidates for a local graph. In Tracetoon, retrieving the facts connected to the submitted change gave the writer a compact brief and the reviewers a common set of sources. Alex gets his hoodie. The token billionaire card finds a second use as a bookmark.

For developers working on a pull request, Cassandra’s open-source Tracetoon works with Claude and Codex and includes the optional comic. For developers building agents of their own, Stephen invited them to try Relate as the embedded database for connecting those agents’ evidence. [Try Tracetoon](https://tracetoon.com/).

Cassandra and Stephen closed by inviting families at the San Francisco conference to Saturday’s AI Engineer Kids Day. After a brief disagreement about whether robots were on the program, they confirmed both activities. Kids would use AI to program their own games and to control little soda-can robots.

## Key takeaways

- A passing-test claim needs a command, a result and the version of the code it checked. When checks must run sequentially, their timing is part of the evidence too.
- A reviewer with the writer’s source brief can check in both directions. Does the evidence support each claim? Did the draft leave out a finding the evidence supports?
- Tracetoon kept a stronger writer and used cheaper models for focused review. With evidence assembled locally before drafting, description scores rose by about two and four points in the team’s two PR Bench comparisons, while the workflow used roughly half as many tokens as before.

## Slides

### Warning: This Pull Request Contains Graphic Content

A paper scroll unrolls across the factory floor, giving the pull request in the title a comic sense of scale. Stephen and Cassandra Chin introduce a graph as a way to make a pull request easier to review.

[0:02](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=2s) [The subtitle reads “A PR crisis. A graph. A better way to review.” Beside it, a character holds a long paper scroll at a factory gate.](https://assets.ai.engineer/sites/aiecode2025/sha256/90d25281210ee4b1940facb38346471428321ab3ef5c18cd0d6536a835369770.webp)

Warning: This Pull Request Contains Graphic Content

A PR crisis. A graph. A better way to review.

Stephen Chin · Cassandra Chin

Interface text:
neo4j
Stephen Chin

### Make the next PR easy to trust. Easy to review.

The comic shows the team gathered around a sealed envelope. Chin suggests using a comic in a pull request to help reviewers see how the evidence fits together. He points to TraceToon and to Relate, an embedded graph database.

[21:15](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1275s) [“Connect the evidence. Give the reviewer a useful story.” Two resource links appear below. In the illustration, people, animals and robots gather around an envelope.](https://assets.ai.engineer/sites/aiecode2025/sha256/b730adacf4fd6a3cadda4156fd3a0f08092fcad9173440d190a4a86a4b5c8044.webp)

Make the next PR easy to trust. Easy to review.

Connect the evidence. Give the reviewer a useful story.

tracetoon.com
relatedb.ai

AI Engineer

Interface text:
neo4j
Stephen Chin

## Full transcript

[0:00](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=0s) Hello and welcome to our ag engineer San

[0:03](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=3s) Francisco talk. My name is Steven Chin.

[0:05](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=5s) I'm joined by Cassandra Chin who's the

[0:07](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=7s) creator of the t trace tune framework.

[0:12](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=12s) Hi, I'm Cassandra Chin. I built trace

[0:14](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=14s) tune because I wanted to improve PRs. I

[0:18](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=18s) see a lot of developers doing great

[0:20](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=20s) things. The speed of innovation is as

[0:22](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=22s) fast as it has ever been with AI

[0:24](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=24s) technologies. But a lot of poll requests

[0:27](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=27s) are getting rejected. And it's not

[0:29](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=29s) because the code is bad. It's because

[0:31](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=31s) the PRs often contain incorrect

[0:34](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=34s) information or are written poorly.

[0:37](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=37s) So hopefully with Tune, we can improve

[0:39](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=39s) that. It also has the option to add a

[0:42](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=42s) fun comic to help explain your pull

[0:44](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=44s) requests in a funny way.

[0:47](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=47s) Cool. And um

[0:48](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=48s) what we're going to do today is we're

[0:50](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=50s) going to go through a little bit about

[0:52](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=52s) kind of the common issues of PRs, how to

[0:54](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=54s) improve the quality, how to improve

[0:56](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=56s) token efficiency on this, and hopefully

[0:58](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=58s) help everyone to be a better AI engineer

[1:02](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=62s) for your organization. Like our main

[1:05](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=65s) character, Alex,

[1:07](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=67s) who all he wants in life is to get an AI

[1:10](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=70s) engineer hoodie.

[1:13](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=73s) That's a really neat hoodie. I have a

[1:15](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=75s) lot of AI engineer t-shirts, but I don't

[1:18](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=78s) have the hoodie yet.

[1:21](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=81s) Get switched to make this AI engineer

[1:22](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=82s) hoodie. Everyone

[1:24](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=84s) everyone needs an

[1:25](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=85s) ready. And um this is enough motivation

[1:28](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=88s) that hopefully he's he's going to do an

[1:30](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=90s) amazing talk at the conference about his

[1:33](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=93s) use case study on what he's doing in his

[1:35](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=95s) company to improve the quality of PRs.

[1:39](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=99s) Here

[1:40](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=100s) is a picture of the token factory where

[1:44](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=104s) Alex works. So, um, Alex works together

[1:48](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=108s) with a bunch of agents. He has, um, of

[1:52](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=112s) course, finance looking over his

[1:53](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=113s) shoulder because we we all spend too

[1:54](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=114s) much tokens and they want to make sure

[1:56](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=116s) we're keeping up with costs. Um, his

[1:59](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=119s) buddy there, Pip, he's walking with

[2:01](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=121s) carries all the evidence and looks into

[2:03](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=123s) the the archives and what's actually

[2:05](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=125s) happening. And then, u Meera does all

[2:07](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=127s) the reviews. So, she's she's keeping

[2:09](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=129s) track of things.

[2:11](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=131s) It looks like Pip left his station empty

[2:19](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=139s) part of the factory

[2:21](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=141s) until until he sees what Alex produced

[2:24](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=144s) for a PR. Oh, no.

[2:27](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=147s) That's a very long ledger.

[2:30](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=150s) A lot to review.

[2:33](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=153s) A lot of code. So, it's it's hard. And

[2:36](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=156s) Merror's Merror's going through

[2:38](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=158s) everything for the fine tune comb. She

[2:39](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=159s) actually is red lines. Found some issues

[2:43](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=163s) with the PR. And this is a a real PR. I

[2:48](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=168s) think this is actually one of your trace

[2:49](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=169s) tune PRs where

[2:51](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=171s) it needs to have exact preview links for

[2:54](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=174s) the URLs, command attribution,

[2:57](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=177s) make sure the earlier work is um

[2:59](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=179s) qualified as well and explain all of

[3:02](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=182s) this in detail in the PR record to be

[3:04](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=184s) accepted by Mera's very critical review.

[3:07](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=187s) This was interesting cuz as I was

[3:09](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=189s) building trace tune, I was actually like

[3:12](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=192s) as I was building it, I was using trace

[3:14](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=194s) tune to test it. So I would have trace

[3:17](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=197s) tune build the PRs and as I improved the

[3:19](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=199s) PRs it would like get better.

[3:23](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=203s) You got amazing comics along the way too

[3:25](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=205s) just to have fun while you were building

[3:28](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=208s) stuff.

[3:29](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=209s) Addictive.

[3:31](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=211s) [laughter]

[3:34](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=214s) So um

[3:36](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=216s) you can see that the repeated PRs which

[3:39](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=219s) Alex is doing to try to get good quality

[3:41](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=221s) results are not working out well. is

[3:44](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=224s) working out for his token billionaire

[3:46](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=226s) card. Do do you know anything about the

[3:47](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=227s) token billionaire program?

[3:49](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=229s) Yeah, so actually I am a token

[3:51](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=231s) billionaire. I spent lots and lots of

[3:54](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=234s) tokens to build trace tune.

[3:59](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=239s) That's kind of unfortunate. Was that was

[4:01](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=241s) that all my money?

[4:03](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=243s) Uh don't sweat the details.

[4:06](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=246s) Okay.

[4:07](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=247s) So, um I I think there's a lot of good

[4:10](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=250s) uses for for tokens and for spending

[4:12](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=252s) more AI work. Um maybe we're doing

[4:15](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=255s) repeated PRs is not where you want to

[4:16](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=256s) spend your token

[4:19](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=259s) and definitely the finance department's

[4:20](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=260s) not happy with this. So, he wants a

[4:22](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=262s) results and he wants to make sure that

[4:24](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=264s) the PRs are accurate, are working and he

[4:28](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=268s) has they don't have need to keep getting

[4:29](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=269s) repeated again and again by reviewing

[4:32](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=272s) the logs and reconstructing the evidence

[4:34](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=274s) repeatedly.

[4:36](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=276s) And it's interesting cuz normally when

[4:39](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=279s) you're just like uh coding on your

[4:41](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=281s) agent, you're in your session and you

[4:43](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=283s) ask it to write a PR, it just reads your

[4:48](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=288s) session log to try to figure out what it

[4:52](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=292s) should have done using its inflated

[4:54](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=294s) context window.

[4:56](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=296s) It actually often like as it's churning

[4:59](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=299s) it compresses its context window. So it

[5:01](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=301s) gets more and more inaccurate as it goes

[5:03](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=303s) along.

[5:05](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=305s) So instead, what we're doing here is

[5:06](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=306s) with trace tune, we're having it read

[5:09](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=309s) through uh different sessions on your

[5:12](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=312s) computer to try to really figure out

[5:14](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=314s) what you built and what happened.

[5:18](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=318s) Yeah. And I think like like that concept

[5:20](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=320s) of um looking through the audit trails,

[5:22](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=322s) looking through like what the agents

[5:24](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=324s) have actually done is a lot of what you

[5:26](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=326s) were doing with the Showtale project you

[5:28](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=328s) were doing for education.

[5:31](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=331s) Yeah. I actually started

[5:33](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=333s) this project with this showtail project

[5:36](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=336s) because there was this like academic

[5:38](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=338s) research study I was supporting. But

[5:41](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=341s) basically the issue they hit is students

[5:43](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=343s) were creating programs but there was no

[5:46](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=346s) way to trace back the actual work and

[5:48](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=348s) effort they did cuz with AI coding the

[5:52](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=352s) what you submit is the final work the

[5:54](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=354s) program but most of the work is the

[5:57](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=357s) prompting and everything that came

[5:59](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=359s) before and that doesn't get submitted

[6:01](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=361s) with the program and I use the

[6:04](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=364s) technology and theory behind that to be

[6:07](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=367s) like well I can also improve poll

[6:09](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=369s) requests with all this data collection.

[6:14](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=374s) Cool. And I think that improvement in

[6:17](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=377s) pull requests is something that benefits

[6:18](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=378s) everybody because now you can find the

[6:21](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=381s) evidence like um logging the change

[6:24](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=384s) directory before the or the CWD before

[6:26](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=386s) the CD runs. explaining why maybe other

[6:29](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=389s) test data is leaking into other repos

[6:32](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=392s) and explaining what's happening behind

[6:34](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=394s) the scenes on the PR rather than what

[6:37](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=397s) most AIPRs read like which is a line by

[6:40](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=400s) line explanation of the diffs since

[6:42](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=402s) that's what they build the PRs off of

[6:48](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=408s) but incorrect facts can still pollute

[6:51](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=411s) the information um can cause a problem

[6:54](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=414s) with conflicting PRs and then still give

[6:57](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=417s) you back inaccurate results.

[7:00](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=420s) So maybe we have a solution for this.

[7:02](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=422s) What What do you What do you think? Do

[7:04](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=424s) you have a solution, Cassandra?

[7:06](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=426s) Yeah, everyone should use Trace June

[7:08](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=428s) because of the comics.

[7:11](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=431s) So because of the comics, not because it

[7:13](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=433s) gives you better PRs, right?

[7:15](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=435s) Well, I started with the comics and I

[7:17](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=437s) kind of went backwards and like, oh, I

[7:20](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=440s) need to actually make the PRs better.

[7:23](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=443s) Well, hopefully better PRs, better

[7:26](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=446s) comics, and maybe better token savings

[7:30](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=450s) if by the end of this. Maybe

[7:33](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=453s) it should be.

[7:36](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=456s) So, um, trade students an open source

[7:38](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=458s) project. It lets you improve the quality

[7:40](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=460s) of your PRs by pulling in agent context

[7:43](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=463s) by pulling in information about the the

[7:46](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=466s) code and then having an independent

[7:48](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=468s) writer independent reviewer build better

[7:50](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=470s) quality PRs plus optionally giving you

[7:54](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=474s) an amazing comic which what do you think

[7:57](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=477s) the best feature is here Cassandra?

[7:59](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=479s) I think it's obviously the comics.

[8:02](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=482s) This is one of the earlier comics from

[8:04](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=484s) when I was using traune to build it.

[8:10](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=490s) That's good. And then you you you made

[8:12](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=492s) sure that there's new characters every

[8:13](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=493s) episode, not none of this repeating

[8:15](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=495s) characters.

[8:16](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=496s) Yeah, that's what this PR was about.

[8:22](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=502s) Okay. And then this is a description of

[8:24](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=504s) the original trace tune flow. So it

[8:27](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=507s) finds all the traces for a particular

[8:28](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=508s) branch. It researches them, reviews the

[8:31](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=511s) description, adds more context, and kind

[8:33](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=513s) of goes in a loop until it gets all of

[8:36](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=516s) the evidence it needs to make a a

[8:38](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=518s) quality PR. And um subjectively, but

[8:42](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=522s) also through um PRB bench, which is

[8:44](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=524s) testing framework, um gives you better

[8:47](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=527s) quality, better research, lower um rate

[8:51](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=531s) of inaccuracies in the PRs. I think at

[8:53](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=533s) this point the PRs were already better

[8:56](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=536s) than if you don't use trace dune, but I

[8:58](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=538s) found that it still makes some mistakes

[9:00](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=540s) occasionally.

[9:02](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=542s) You can see at the end that it tries to

[9:04](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=544s) revise with more context. I think that's

[9:06](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=546s) where the mistakes come in.

[9:09](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=549s) [snorts]

[9:10](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=550s) Okay. Well, Alex Alex thinks it's good

[9:12](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=552s) enough. He's going to try to solve the

[9:14](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=554s) token crisis the factory with trace tune

[9:16](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=556s) and see if this will solve all of his

[9:17](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=557s) problems. Polish checked. Alex looks

[9:22](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=562s) very very happy

[9:24](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=564s) but Mera doesn't look that happy.

[9:26](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=566s) It looks like we still hit a mistake

[9:28](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=568s) here like I was saying earlier. So the

[9:31](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=571s) original PR claim was no later passing

[9:34](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=574s) run is recorded.

[9:36](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=576s) And the problem here is the in the

[9:39](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=579s) actual trace it shows it running a bun

[9:43](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=583s) test at the same time as the bun xtsc

[9:48](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=588s) and you can't run those two at the same

[9:50](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=590s) time. They have to be run sequentially

[9:52](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=592s) for the test to pass.

[9:55](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=595s) Yeah, exactly. So he claims test passing

[9:58](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=598s) and all the checks were run and actually

[10:01](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=601s) that's not what the code did. So poor

[10:04](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=604s) Alex. Another failed PR, a very angry

[10:08](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=608s) badger with all the token cost. Um kind

[10:12](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=612s) of kind of makes me feel like Monday

[10:14](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=614s) morning with the Neo Forj finance

[10:15](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=615s) department looking at my token spend

[10:17](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=617s) each week. But they're they're very

[10:19](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=619s) forgiving, [laughter]

[10:21](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=621s) especially when it's a important project

[10:24](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=624s) like like this. So maybe Alex can save

[10:28](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=628s) some cost by using cheaper writers is

[10:30](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=630s) the number one thing finance asked me to

[10:31](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=631s) do. They're like, "Hey, hey, no more

[10:33](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=633s) opus on it. Haiku, do something cheaper,

[10:36](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=636s) please. Does this work?"

[10:40](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=640s) I feel like the quality is a bit lower

[10:42](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=642s) when you do it like that. Just reducing

[10:44](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=644s) the cost doesn't quite help.

[10:46](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=646s) Yeah. So, we were trying to get better

[10:48](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=648s) quality and we we ended up getting worse

[10:50](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=650s) quality by just using cheaper writers.

[10:53](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=653s) And

[10:55](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=655s) obviously, Alex is in a bind here,

[10:57](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=657s) right? He's he's spent too much on

[10:58](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=658s) revisions. He doesn't have a solution

[11:01](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=661s) for reducing token costs. He's not

[11:02](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=662s) getting the results he wants out of his

[11:04](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=664s) PRs. But

[11:07](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=667s) maybe there's a there's something Pip

[11:09](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=669s) can help us with. Pip Pip has a friend.

[11:12](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=672s) I wonder who this friend could be.

[11:15](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=675s) Oh, well, I I think I think you know

[11:18](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=678s) from my previous talks that that this

[11:20](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=680s) friend is definitely going to be a

[11:22](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=682s) graph.

[11:24](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=684s) You're the graph guy. We're doing

[11:26](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=686s) another graph again.

[11:28](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=688s) Yeah. Yeah. There's always a new graph,

[11:30](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=690s) but this is not the same old graph. This

[11:33](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=693s) is not the the Neo Forj database. This

[11:36](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=696s) is our new friend Archie,

[11:39](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=699s) who is a embedded database written in

[11:43](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=703s) Rust called relate that actually is

[11:46](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=706s) being launched this week

[11:49](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=709s) at the nodes conference and also this

[11:51](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=711s) talk at a engineer in San Francisco.

[11:54](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=714s) And it's an open- source um database

[11:58](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=718s) which allows us to get a lot of the

[11:59](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=719s) benefits of more accuracy, multihop

[12:03](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=723s) queries, more explainable traces without

[12:07](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=727s) having the heavyweight approach. So it's

[12:09](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=729s) kind of an agent focused graph database

[12:12](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=732s) specifically for these sort of embedded

[12:14](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=734s) use cases.

[12:18](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=738s) Okay, so let's see how Archie can help

[12:20](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=740s) us out. So before we were relying a lot

[12:23](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=743s) on on agents and loops of agents to

[12:25](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=745s) resolve the problem. But graph databases

[12:28](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=748s) can store relationships can find like

[12:31](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=751s) like different things which are

[12:33](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=753s) clustered build a brief which connects

[12:37](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=757s) all this information together and then

[12:39](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=759s) give the agents even a cheaper agent

[12:41](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=761s) better information and more context to

[12:43](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=763s) give a good drafting decision. And the

[12:45](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=765s) good thing about Archie is he has zero

[12:47](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=767s) LLM costs. It's all local compute cost

[12:51](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=771s) and storage and it's a fully immemory

[12:53](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=773s) database and rest so it's highly

[12:54](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=774s) optimized. So we get this providence

[12:58](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=778s) about all these submitted changes and

[13:00](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=780s) this evidence graph basically for free

[13:03](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=783s) compared to all the LLM and token costs

[13:05](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=785s) which we were doing before. And this

[13:08](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=788s) means our our buddy Alex doesn't need to

[13:10](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=790s) research. She doesn't need to go through

[13:11](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=791s) and find all the facts again and go

[13:13](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=793s) through the sources

[13:15](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=795s) because relate is going to do all that

[13:18](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=798s) hard work for him and build an evidence

[13:20](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=800s) packet that he can use for trace tune.

[13:23](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=803s) So all sounds really great. Maybe we do

[13:25](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=805s) need more graphs.

[13:28](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=808s) That's I'm surprised to hear that from

[13:30](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=810s) you after all the graph talks you've had

[13:31](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=811s) to put up with. Okay, but

[13:34](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=814s) we can do this and make it even better

[13:37](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=817s) with the neighborhood brief. So

[13:38](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=818s) basically taking all of the the Y the

[13:40](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=820s) code and the checks putting attributions

[13:43](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=823s) and relationships between those and then

[13:45](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=825s) returning back a connected set of

[13:46](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=826s) information as a packet. And by doing

[13:49](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=829s) this then when the agent is drafting up

[13:51](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=831s) the PR it has all the connected facts

[13:54](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=834s) with their sources together in a single

[13:57](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=837s) source. And that makes the writer draft

[14:01](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=841s) more accurate with fewer passes and it

[14:04](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=844s) also makes the reviewer's job much

[14:06](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=846s) easier. So we don't need an expensive

[14:08](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=848s) reviewer. We can save by using a few

[14:11](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=851s) haiku reviewers instead of a um sonnet

[14:14](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=854s) reviewer or or a luna in the case of

[14:16](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=856s) codeex to find come up with the findings

[14:19](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=859s) check those against the draft redo the

[14:21](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=861s) writing and then we retain the higher

[14:24](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=864s) quality writer a a sonnet or terror

[14:27](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=867s) class writer to make sure we have a high

[14:29](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=869s) quality brief that will pass human

[14:30](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=870s) approval.

[14:33](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=873s) When we were building this, it felt very

[14:35](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=875s) counterintuitive to have the writer be

[14:37](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=877s) high higher quality than the reviewers.

[14:40](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=880s) It actually it it's common that people

[14:43](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=883s) always want to have the planner, the

[14:44](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=884s) reviewer, the checker to be the highest

[14:46](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=886s) quality model. But if you're doing high

[14:50](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=890s) quality writing, a lower model, no

[14:52](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=892s) matter how many iterations and how much

[14:54](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=894s) feedback you give it, can't do it. It

[14:56](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=896s) just turns in things into a checklist

[14:58](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=898s) and makes it very mechanical. So to give

[15:00](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=900s) writing which feels humanlike, you do

[15:03](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=903s) need a higher quality model. And it's

[15:05](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=905s) it's actually the the reviewing and the

[15:06](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=906s) cross-checking step which is more

[15:08](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=908s) mechanical. So it's a little

[15:09](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=909s) counterintuitive, but that works out

[15:11](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=911s) quite well. It's also a little

[15:14](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=914s) counterintuitive that um our badger has

[15:17](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=917s) a thumbs up coming out of his chest, but

[15:19](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=919s) let's let's ignore the high quality AI

[15:21](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=921s) artwork.

[15:25](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=925s) Okay, so we've checked all the claims

[15:26](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=926s) against the evidence. We've looked for

[15:28](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=928s) missing findings with our cheaper

[15:31](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=931s) reviewer. Did the focus work and then

[15:35](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=935s) found the things which were connected

[15:37](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=937s) information to give a better draft and

[15:40](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=940s) review

[15:41](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=941s) for mirror.

[15:44](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=944s) And here's the output. So it has the the

[15:46](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=946s) Y. So log CWD and that describes the

[15:49](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=949s) shell before the CD runs. What's the

[15:52](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=952s) exception?

[15:54](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=954s) So the CD targets in repo or the command

[15:56](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=956s) mentions a route. What passed? It has

[15:58](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=958s) type check plus 71 test checked on

[16:01](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=961s) submitted content not before.

[16:04](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=964s) And then it starts in correlate.ts

[16:06](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=966s) inspecting the leading CD exception.

[16:10](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=970s) And Meera looks pretty happy with these

[16:12](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=972s) results, right?

[16:14](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=974s) Yeah. This might be the first scene

[16:16](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=976s) where she's genuinely happy.

[16:19](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=979s) I think she was kind of happy with the

[16:21](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=981s) the AI engineer sweatshirt before she

[16:23](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=983s) realized that Alex actually was a was a

[16:25](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=985s) goof. Okay, so here are some of the

[16:28](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=988s) actual data and results from the

[16:32](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=992s) sideby-side analysis using PRBench,

[16:34](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=994s) which is a um pull request benchmark,

[16:37](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=997s) which uses

[16:38](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=998s) audit traces plus code and um LM judges

[16:42](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1002s) to review the quality of the work. Of

[16:45](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1005s) course, reviewed by a human as well for

[16:47](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1007s) accuracy. And you can see that on both

[16:49](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1009s) Claude and Codeex, you get a significant

[16:52](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1012s) quality improvements on a scale of 1 to

[16:54](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1014s) 100. The judges by and large almost

[16:57](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1017s) unilaterally liked the trace tune plus

[17:01](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1021s) relate quality better. Um on a 1 to 100

[17:05](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1025s) scale it was like 74 to 76 82 to 86. So

[17:08](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1028s) plus two plus 4 point.

[17:10](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1030s) These tests contributed to my token

[17:12](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1032s) billionaire card because we were running

[17:15](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1035s) lots and lots of tests. We created this

[17:17](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1037s) side project called PRB bench just for

[17:19](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1039s) testing and it we would have different

[17:22](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1042s) agents test over different poll requests

[17:24](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1044s) and compare the results. Initially I had

[17:28](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1048s) it like running like sequentially. It'll

[17:32](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1052s) do one test and finish it then do the

[17:34](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1054s) next but then we decided to run them all

[17:36](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1056s) in parallel. It's a very quick way to

[17:39](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1059s) reach high spend.

[17:40](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1060s) Yeah. So that's kind of funny. I I at at

[17:43](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1063s) at at the current testing rate with the

[17:45](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1065s) number of use cases and tests and the

[17:47](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1067s) benchmark um we can fully saturate my my

[17:51](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1071s) high-end gaming computer with as many

[17:54](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1074s) threads as Linux can handle

[17:56](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1076s) simultaneously

[17:57](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1077s) and and of course the token spend is is

[18:00](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1080s) slight slightly higher than what we'd

[18:02](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1082s) like. But it's all for the good of

[18:04](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1084s) improving the quality of pull requests

[18:06](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1086s) for the world. So you all get the

[18:08](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1088s) benefit of all these test runs and

[18:10](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1090s) having the highest quality improvements

[18:13](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1093s) and you save on tokens. So we've spent

[18:15](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1095s) the tokens for you in advance because

[18:19](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1099s) trace tune with the graph solution takes

[18:22](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1102s) about half as many tokens as it did

[18:25](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1105s) previously and which is an amazing

[18:28](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1108s) result from the graph because now you're

[18:29](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1109s) premputing you're getting all this

[18:31](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1111s) evidence graph construct in advance.

[18:34](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1114s) The writing's cheaper, the reviewing's

[18:37](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1117s) cheaper with a higher quality result.

[18:41](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1121s) I wasn't sure about the whole graph

[18:43](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1123s) thing initially, but after seeing the

[18:45](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1125s) cost tank almost in half, like it's not

[18:48](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1128s) even a question, it's making this

[18:50](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1130s) program work.

[18:53](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1133s) And this is something we did for the

[18:55](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1135s) Trey Stone project, but you can do this

[18:56](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1136s) for pretty much any project where you

[18:59](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1139s) have data, you have relationships, you

[19:01](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1141s) have evidence, you have things which

[19:02](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1142s) you're trying to to compare because

[19:06](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1146s) agents are not that good at doing the

[19:08](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1148s) data analysis at doing all of the

[19:10](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1150s) evaluation and the curation. So if you

[19:13](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1153s) give them a help at the front end by

[19:15](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1155s) creating a a custom graph layer with an

[19:18](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1158s) in-memory database like relate suddenly

[19:21](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1161s) your costs go down your quality goes up

[19:23](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1163s) and you too

[19:26](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1166s) can be a token billionaire like our

[19:28](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1168s) friend Alex. Well I guess it's a now

[19:30](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1170s) he's using it as a bookmark.

[19:33](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1173s) I think that's a good use for it. It's a

[19:34](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1174s) really high quality bookmark. It's

[19:36](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1176s) heavier than a credit card.

[19:40](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1180s) It is. It is a nice surprise for um

[19:42](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1182s) spending all those tokens on PR Bench.

[19:46](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1186s) So, finance is happy. Lower drafting

[19:49](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1189s) costs that they can reasonably fund.

[19:52](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1192s) Alex is happy. He has his new bookmark.

[19:54](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1194s) key is now has a great case study

[19:57](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1197s) with the connections doing neighborhood

[20:00](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1200s) retrievable of facts so that it's a

[20:02](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1202s) better brief focusing the review so the

[20:05](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1205s) reviewers get the same compact brief ma

[20:07](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1207s) gets exactly the evidence she needs and

[20:09](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1209s) then here's Alex on stage at um AI

[20:13](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1213s) engineer somewhere in the world better

[20:15](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1215s) PR's lower drafting cost with a graph

[20:20](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1220s) plus open-source

[20:22](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1222s) trace to builds, which also gives you

[20:25](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1225s) amazing comics.

[20:30](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1230s) Okay, so big round of applause for our

[20:33](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1233s) friend Alex, who's now earned the AI

[20:36](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1236s) engineer hoodie.

[20:38](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1238s) I think that's definitely a wardrobe

[20:40](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1240s) upgrade for him.

[20:42](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1242s) Yeah, I don't think the pocket protector

[20:44](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1244s) look was that great for him. So, good

[20:47](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1247s) for him. I think we'll see Swixs on

[20:50](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1250s) stage with one of these hoodies soon. He

[20:51](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1251s) he can't he can't not do it now that we

[20:54](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1254s) shamed him into doing it at his own

[20:55](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1255s) conference. The token billion program is

[20:58](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1258s) amazing. A big kudos to the team for

[21:00](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1260s) coming up with ways to incentivize and

[21:03](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1263s) reward people who are on the bleeding

[21:04](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1264s) edge of AI usage and learning how to

[21:07](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1267s) make it really effective for the rest of

[21:09](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1269s) us. And um

[21:12](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1272s) now you too can have better PRs. Impress

[21:16](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1276s) your co-workers by having

[21:19](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1279s) wellressearched

[21:20](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1280s) effectively written and lowerc cost PRs

[21:23](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1283s) with a comic

[21:25](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1285s) with a comic. So for more information go

[21:27](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1287s) to trace tune.com

[21:29](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1289s) learn about trace tune. Give it a try.

[21:31](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1291s) It works with cloud. It works with

[21:32](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1292s) codecs. It works with all your favorite

[21:34](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1294s) AI tools. And if you want to do these

[21:37](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1297s) same sort of amazing results on your own

[21:39](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1299s) project, try out Relate, our new

[21:42](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1302s) Rustbased embedded open-source graph

[21:44](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1304s) database, which brings context to your

[21:47](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1307s) agents and helps you build more

[21:48](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1308s) efficient applications that just work

[21:51](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1311s) better, are cheaper, and give you better

[21:53](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1313s) results. So, thank you all for joining

[21:55](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1315s) our AI engineer San Francisco talk. And

[21:59](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1319s) if you're watching this talk this week

[22:01](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1321s) at San Francisco,

[22:04](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1324s) what should you do this weekend with

[22:05](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1325s) your kids on Saturday? Cassandra,

[22:08](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1328s) we are running an AI engineer kids day.

[22:11](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1331s) So, anyone who's available, just sign

[22:14](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1334s) your kids up. We're teaching kids

[22:15](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1335s) workshops. Kids will be learning how to

[22:18](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1338s) do AI programming to make their own

[22:20](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1340s) games. And I think we're also doing a

[22:23](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1343s) No, it's just the gaming one.

[22:26](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1346s) No, we'll do we'll do the robots, too.

[22:28](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1348s) Okay. Yeah, we'll also be doing robots.

[22:30](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1350s) So, kids will get to control these

[22:32](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1352s) little soda can robots using AI.

[22:35](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1355s) Yeah. So, a lot of fun. Physical AI

[22:38](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1358s) gaming AI all in one place. Um, sign up

[22:41](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1361s) at AI engineer kids day and um, you

[22:45](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1365s) know, bring your kids, bring your

[22:46](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1366s) nieces, bring your nephews, and let's

[22:48](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1368s) train the next generation of AI

[22:50](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1370s) engineers.

[22:53](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1373s) All right. Thanks so much and enjoy the

[22:54](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1374s) rest of the conference.
