[
  {
    "startMs": 1199,
    "text": "Hey everyone, thanks for tuning in."
  },
  {
    "startMs": 3040,
    "text": "Today I'm going to walk you through how"
  },
  {
    "startMs": 4160,
    "text": "we use AI agents at Dualingo to write"
  },
  {
    "startMs": 6000,
    "text": "and ship more than 200,000 thousand"
  },
  {
    "startMs": 7759,
    "text": "lines of unit tests on iOS codebase."
  },
  {
    "startMs": 9599,
    "text": "I'll share what we built and what we"
  },
  {
    "startMs": 11040,
    "text": "learned along the way. Uh I'm Kush. I'm"
  },
  {
    "startMs": 13120,
    "text": "a staff engineer on the platform team"
  },
  {
    "startMs": 14400,
    "text": "here at Dualingo. My job is to build"
  },
  {
    "startMs": 15599,
    "text": "developer tooling and AI assisted"
  },
  {
    "startMs": 16880,
    "text": "workflows for the 100 plus engineers who"
  },
  {
    "startMs": 18320,
    "text": "contribute to our client apps. Before"
  },
  {
    "startMs": 19840,
    "text": "Duelingo, I spent close to a decade at"
  },
  {
    "startMs": 21359,
    "text": "Stripe and Uber working across a few"
  },
  {
    "startMs": 22720,
    "text": "teams there. Let's start with some"
  },
  {
    "startMs": 24880,
    "text": "background on the scale of mobile at"
  },
  {
    "startMs": 26640,
    "text": "Dualingo. Uh we ship our apps to the app"
  },
  {
    "startMs": 29039,
    "text": "store every week. Um we're also a very"
  },
  {
    "startMs": 31439,
    "text": "experiment-driven company. So at any"
  },
  {
    "startMs": 32880,
    "text": "given point of time, we have more than"
  },
  {
    "startMs": 34079,
    "text": "600 AB tests running live and our code"
  },
  {
    "startMs": 36719,
    "text": "bases are pretty mature. Both the iOS"
  },
  {
    "startMs": 38399,
    "text": "and Android apps have been around for"
  },
  {
    "startMs": 39520,
    "text": "more than 10 years at this point."
  },
  {
    "startMs": 42079,
    "text": "Here's also the number of executable"
  },
  {
    "startMs": 43680,
    "text": "lines of app code in our iOS repository."
  },
  {
    "startMs": 45680,
    "text": "So as you can see back in 2019, we were"
  },
  {
    "startMs": 47440,
    "text": "at about 200,000 lines and today we're"
  },
  {
    "startMs": 49280,
    "text": "hit close to 2 million. And as AI"
  },
  {
    "startMs": 51520,
    "text": "assisted coding picks up, you can see"
  },
  {
    "startMs": 52800,
    "text": "that curve starting to bend upward."
  },
  {
    "startMs": 55199,
    "text": "And it's not just the amount of code"
  },
  {
    "startMs": 56320,
    "text": "that's changing, it's also who's writing"
  },
  {
    "startMs": 57920,
    "text": "it. So last year about 200 people"
  },
  {
    "startMs": 60079,
    "text": "contributed to the iOS repository. And"
  },
  {
    "startMs": 61520,
    "text": "this year we've already crossed 250. And"
  },
  {
    "startMs": 63520,
    "text": "a lot of these folks aren't even iOS"
  },
  {
    "startMs": 64960,
    "text": "engineers. We have backend engineers,"
  },
  {
    "startMs": 66400,
    "text": "Android engineers, even PMs and"
  },
  {
    "startMs": 67920,
    "text": "designers all shipping code to the iOS"
  },
  {
    "startMs": 69360,
    "text": "app. And that's mostly been unlocked by"
  },
  {
    "startMs": 71600,
    "text": "AI coding tools like cursor and cloud"
  },
  {
    "startMs": 73040,
    "text": "code. Um, another thing to note about"
  },
  {
    "startMs": 75200,
    "text": "our codebase is that it's a codebase in"
  },
  {
    "startMs": 76640,
    "text": "transition. So a few years ago, we"
  },
  {
    "startMs": 78320,
    "text": "standardized on the MVVM architecture"
  },
  {
    "startMs": 79759,
    "text": "pattern. Today about half the code in"
  },
  {
    "startMs": 81600,
    "text": "our repository follows that pattern and"
  },
  {
    "startMs": 83040,
    "text": "the other half is the legacy code that"
  },
  {
    "startMs": 84400,
    "text": "we're in the process of migrating. Since"
  },
  {
    "startMs": 86320,
    "text": "the MVVM half is the easiest to test,"
  },
  {
    "startMs": 88000,
    "text": "that's where we focused on this project."
  },
  {
    "startMs": 91040,
    "text": "Here's where we started back in March of"
  },
  {
    "startMs": 92479,
    "text": "this year. We were at overall unit test"
  },
  {
    "startMs": 95040,
    "text": "coverage of 11%. The MVVM layers were a"
  },
  {
    "startMs": 97680,
    "text": "bit lower, repositories were at 11%,"
  },
  {
    "startMs": 99600,
    "text": "view models at 9, data sources at 5, and"
  },
  {
    "startMs": 101840,
    "text": "combined we are about 9% for the MVM"
  },
  {
    "startMs": 103920,
    "text": "slice of the app."
  },
  {
    "startMs": 106159,
    "text": "So why does this matter now more than it"
  },
  {
    "startMs": 108159,
    "text": "used to? As agents and humans write more"
  },
  {
    "startMs": 110640,
    "text": "code and write it faster, writing code"
  },
  {
    "startMs": 112320,
    "text": "isn't the hard part anymore. Verifying"
  },
  {
    "startMs": 113840,
    "text": "it is. So when someone puts up a PR, how"
  },
  {
    "startMs": 116000,
    "text": "do you know that it's actually safe to"
  },
  {
    "startMs": 117119,
    "text": "ship? The bottleneck is really shifting"
  },
  {
    "startMs": 118799,
    "text": "from writing to verifying. And that's"
  },
  {
    "startMs": 120880,
    "text": "where unit tests come in. They make up"
  },
  {
    "startMs": 122719,
    "text": "the base of the testing pyramid. They're"
  },
  {
    "startMs": 124079,
    "text": "pretty cheap to write and pass to run."
  },
  {
    "startMs": 125520,
    "text": "And they also act as guardrails for"
  },
  {
    "startMs": 126719,
    "text": "agents. So when an agent's change breaks"
  },
  {
    "startMs": 128640,
    "text": "a test in CI, it gets immediate feedback"
  },
  {
    "startMs": 130479,
    "text": "on what to fix. They also act as living"
  },
  {
    "startMs": 132560,
    "text": "documentation on what the code is"
  },
  {
    "startMs": 133760,
    "text": "supposed to do. And here's where we"
  },
  {
    "startMs": 135280,
    "text": "ended up 8 months later. Overall unit"
  },
  {
    "startMs": 136879,
    "text": "test coverage went from 11% to 24%."
  },
  {
    "startMs": 139200,
    "text": "Repositories went from 11 to 72. View"
  },
  {
    "startMs": 141360,
    "text": "models from 9 to 58. Data sources from 5"
  },
  {
    "startMs": 143680,
    "text": "to 41. And the combined MVM slice of the"
  },
  {
    "startMs": 145840,
    "text": "app went from 9% to 59%. So what did we"
  },
  {
    "startMs": 148800,
    "text": "actually end up building? On paper, it's"
  },
  {
    "startMs": 150560,
    "text": "a pretty simple loop. First, you"
  },
  {
    "startMs": 152000,
    "text": "identify the files that need tests. Then"
  },
  {
    "startMs": 153760,
    "text": "you write those tests with an agent. You"
  },
  {
    "startMs": 155440,
    "text": "verify that the tests are good quality"
  },
  {
    "startMs": 156720,
    "text": "and they pass CI. Then you resolve the"
  },
  {
    "startMs": 158160,
    "text": "PR, which means getting attributed by an"
  },
  {
    "startMs": 159840,
    "text": "engineer merged. And finally, it on the"
  },
  {
    "startMs": 161920,
    "text": "feedback so that the next batch of PRs"
  },
  {
    "startMs": 163280,
    "text": "is better than the last one. But in"
  },
  {
    "startMs": 165599,
    "text": "production, it's a little messier. This"
  },
  {
    "startMs": 167040,
    "text": "is what our system ended up looking"
  },
  {
    "startMs": 168239,
    "text": "like. I won't go through every box right"
  },
  {
    "startMs": 169760,
    "text": "now, but at a high level, it's a few"
  },
  {
    "startMs": 170959,
    "text": "temporal workflows that coordinate"
  },
  {
    "startMs": 172239,
    "text": "through shared state stored in S3. And"
  },
  {
    "startMs": 174560,
    "text": "every box you see here maps back to one"
  },
  {
    "startMs": 176080,
    "text": "of the five steps I just talked about."
  },
  {
    "startMs": 178560,
    "text": "Here are the building blocks that we"
  },
  {
    "startMs": 179680,
    "text": "ended up using. Temporal handles"
  },
  {
    "startMs": 181440,
    "text": "workflow orchestration and execution."
  },
  {
    "startMs": 183040,
    "text": "Cloud code acts as the coding agent. S3"
  },
  {
    "startMs": 185360,
    "text": "holds a shared state that keeps our"
  },
  {
    "startMs": 186879,
    "text": "workflows in sync. And we use GitHub for"
  },
  {
    "startMs": 188800,
    "text": "actions, labels, and PRs. Data dogs acts"
  },
  {
    "startMs": 191360,
    "text": "as observability layer where we keep an"
  },
  {
    "startMs": 192720,
    "text": "eye on the health of our pipelines."
  },
  {
    "startMs": 195440,
    "text": "If you haven't used empodoral before,"
  },
  {
    "startMs": 196720,
    "text": "it's basically a durable workflow"
  },
  {
    "startMs": 198000,
    "text": "execution engine. You write your"
  },
  {
    "startMs": 199519,
    "text": "workflows as normal Python code and if a"
  },
  {
    "startMs": 201120,
    "text": "worker crashes, the workflow just picks"
  },
  {
    "startMs": 202560,
    "text": "up right where it left off. This fits"
  },
  {
    "startMs": 204640,
    "text": "agents really well and agent session can"
  },
  {
    "startMs": 206480,
    "text": "run for seconds to hours and it can call"
  },
  {
    "startMs": 208400,
    "text": "out to tools like GitHub, CI, S3, and"
  },
  {
    "startMs": 210480,
    "text": "Claude. We also lean on a few built-in"
  },
  {
    "startMs": 212640,
    "text": "features of Temporal. There are"
  },
  {
    "startMs": 214159,
    "text": "schedules which are basically chronike"
  },
  {
    "startMs": 215599,
    "text": "schedules for our jobs. There are"
  },
  {
    "startMs": 216959,
    "text": "composible workflows so that we can fan"
  },
  {
    "startMs": 218319,
    "text": "out work in parallel. Then there's first"
  },
  {
    "startMs": 220080,
    "text": "class support for retries and timeouts"
  },
  {
    "startMs": 221599,
    "text": "and item potency so that we never kick"
  },
  {
    "startMs": 223200,
    "text": "off duplicate runs of the same file."
  },
  {
    "startMs": 225840,
    "text": "To split up this work, we split this"
  },
  {
    "startMs": 227440,
    "text": "project into four milestones. The first"
  },
  {
    "startMs": 229040,
    "text": "was local validation. Basically, is this"
  },
  {
    "startMs": 231120,
    "text": "project even viable? The second was a"
  },
  {
    "startMs": 232879,
    "text": "scheduled backfield workflow to write"
  },
  {
    "startMs": 234239,
    "text": "tests for existing code. The third was a"
  },
  {
    "startMs": 236159,
    "text": "PR life cycle workflow to actually get"
  },
  {
    "startMs": 238080,
    "text": "those PRs merged. And the fourth was a"
  },
  {
    "startMs": 239920,
    "text": "label trigger so engineers can get tests"
  },
  {
    "startMs": 241760,
    "text": "for the new code that they're writing."
  },
  {
    "startMs": 243360,
    "text": "Now, we'll go through each of these in"
  },
  {
    "startMs": 244799,
    "text": "order. Before building anything real, we"
  },
  {
    "startMs": 246640,
    "text": "wanted to do the least amount of work"
  },
  {
    "startMs": 247840,
    "text": "possible that would tell us whether an"
  },
  {
    "startMs": 249120,
    "text": "agent could actually do this job well"
  },
  {
    "startMs": 250239,
    "text": "and do it at scale. We started with a"
  },
  {
    "startMs": 252159,
    "text": "very simple local cloud code setup where"
  },
  {
    "startMs": 254319,
    "text": "we had one cloud code command that acted"
  },
  {
    "startMs": 255840,
    "text": "as an orchestrator. It ran an Xcode"
  },
  {
    "startMs": 258000,
    "text": "build script to generate a code coverage"
  },
  {
    "startMs": 259600,
    "text": "report locally. Then it fed that report"
  },
  {
    "startMs": 261440,
    "text": "through a file selection script that we"
  },
  {
    "startMs": 262720,
    "text": "wrote and then it handed each selected"
  },
  {
    "startMs": 264560,
    "text": "file off of sub two sub aents. One wrote"
  },
  {
    "startMs": 267120,
    "text": "the tests and the other one opened the"
  },
  {
    "startMs": 268479,
    "text": "PR. We ran this in a loop and generated"
  },
  {
    "startMs": 270400,
    "text": "a bunch of PRs. We basically ran this in"
  },
  {
    "startMs": 272880,
    "text": "two batches. The first patch had 17 PRs"
  },
  {
    "startMs": 274880,
    "text": "and we merged eight of them. The second"
  },
  {
    "startMs": 276240,
    "text": "had 40 PRs and we merged 19. So that's"
  },
  {
    "startMs": 278080,
    "text": "roughly a 50% merge rate. But the goal"
  },
  {
    "startMs": 279840,
    "text": "here wasn't to merge a lot of PRs. It"
  },
  {
    "startMs": 281120,
    "text": "was to learn how the agents fail and to"
  },
  {
    "startMs": 282400,
    "text": "fix those paper cuts before"
  },
  {
    "startMs": 283280,
    "text": "productionizing anything. Here are some"
  },
  {
    "startMs": 285759,
    "text": "of the failure modes that we saw. Since"
  },
  {
    "startMs": 287280,
    "text": "we're a large legacy code base with both"
  },
  {
    "startMs": 288720,
    "text": "XC test and swift testing, the agents"
  },
  {
    "startMs": 290240,
    "text": "would sometimes get confused and mix the"
  },
  {
    "startMs": 291520,
    "text": "two in one file. We fixed that by giving"
  },
  {
    "startMs": 293199,
    "text": "the agents clear rules on when to use"
  },
  {
    "startMs": 294720,
    "text": "each. We also had a giant user client"
  },
  {
    "startMs": 296800,
    "text": "singleton class. Some code passed around"
  },
  {
    "startMs": 298560,
    "text": "the concrete version of this class which"
  },
  {
    "startMs": 299680,
    "text": "is really hard to mock whereas others"
  },
  {
    "startMs": 300800,
    "text": "passed around a protocolized version. So"
  },
  {
    "startMs": 302400,
    "text": "we migrated the entire codebase over to"
  },
  {
    "startMs": 303840,
    "text": "the protocolized version and added lint"
  },
  {
    "startMs": 305680,
    "text": "rule to enforce it. This made the whole"
  },
  {
    "startMs": 307440,
    "text": "host of files easier to test for agents"
  },
  {
    "startMs": 309199,
    "text": "and for humans. We also ran into six"
  },
  {
    "startMs": 312080,
    "text": "concurrency related issues and agents"
  },
  {
    "startMs": 313919,
    "text": "writing timing based tests which are a"
  },
  {
    "startMs": 315199,
    "text": "big source of flakiness in CI. So we"
  },
  {
    "startMs": 316960,
    "text": "improved the state for both of these"
  },
  {
    "startMs": 318000,
    "text": "with better rules and examples in our"
  },
  {
    "startMs": 319440,
    "text": "testing skills."
  },
  {
    "startMs": 321280,
    "text": "Now this brings us to milestone 2. Since"
  },
  {
    "startMs": 323120,
    "text": "we were so far behind on code coverage,"
  },
  {
    "startMs": 324800,
    "text": "the biggest leverage that we saw was"
  },
  {
    "startMs": 326080,
    "text": "backfilling tests for code that was"
  },
  {
    "startMs": 327360,
    "text": "already merged in. That's where the"
  },
  {
    "startMs": 329120,
    "text": "scheduled backfill workflow comes in."
  },
  {
    "startMs": 330800,
    "text": "This is a system design diagram of the"
  },
  {
    "startMs": 332160,
    "text": "workflow as it runs in production and it"
  },
  {
    "startMs": 334080,
    "text": "essentially does two jobs. The left half"
  },
  {
    "startMs": 335840,
    "text": "is to identify which is picking the"
  },
  {
    "startMs": 337360,
    "text": "files that need tests and the right half"
  },
  {
    "startMs": 339120,
    "text": "is right which is having the agents"
  },
  {
    "startMs": 341039,
    "text": "actually write those tests. Let's start"
  },
  {
    "startMs": 343520,
    "text": "with identify. Every time a PR merges to"
  },
  {
    "startMs": 345520,
    "text": "the main branch in the iOS repo, we kick"
  },
  {
    "startMs": 346960,
    "text": "off a GitHub access job that generates"
  },
  {
    "startMs": 348560,
    "text": "an Xcode coverage report and uploads it"
  },
  {
    "startMs": 350240,
    "text": "to S3 that becomes a source of truth for"
  },
  {
    "startMs": 352320,
    "text": "the latest code coverage. The back fill"
  },
  {
    "startMs": 354160,
    "text": "workflow then fetches that report and"
  },
  {
    "startMs": 355440,
    "text": "runs it through a selector script which"
  },
  {
    "startMs": 356800,
    "text": "picks the next set of files to test."
  },
  {
    "startMs": 359280,
    "text": "Let's get a deeper a little deeper into"
  },
  {
    "startMs": 360639,
    "text": "the selector script. Its job is"
  },
  {
    "startMs": 362479,
    "text": "essentially to score every Swift file"
  },
  {
    "startMs": 363840,
    "text": "based on how confident we are that an"
  },
  {
    "startMs": 365280,
    "text": "agent can write a good test for it. The"
  },
  {
    "startMs": 367039,
    "text": "strongest signal is file type. So view"
  },
  {
    "startMs": 369039,
    "text": "models, repositories, and data sources"
  },
  {
    "startMs": 370479,
    "text": "get a 2.5x boost while legacy files get"
  },
  {
    "startMs": 372800,
    "text": "pushed down. We skipped UI code"
  },
  {
    "startMs": 374560,
    "text": "completely for now since snapshot tests"
  },
  {
    "startMs": 375840,
    "text": "are better fit there. And larger files"
  },
  {
    "startMs": 377759,
    "text": "also get a penalty. So we prioritize"
  },
  {
    "startMs": 379280,
    "text": "code that we're confident that we can"
  },
  {
    "startMs": 380639,
    "text": "test well."
  },
  {
    "startMs": 382639,
    "text": "The next step is to actually write the"
  },
  {
    "startMs": 384000,
    "text": "tests. We kick off one child workflow"
  },
  {
    "startMs": 386000,
    "text": "per file, all running in parallel. In"
  },
  {
    "startMs": 388400,
    "text": "each one of these runs, the agent clones"
  },
  {
    "startMs": 389919,
    "text": "the iOS repository, reads the testing"
  },
  {
    "startMs": 391759,
    "text": "skill that we keep in the repo, which"
  },
  {
    "startMs": 393199,
    "text": "spells out how to write a good test."
  },
  {
    "startMs": 394960,
    "text": "Then it writes the test plus any mocks"
  },
  {
    "startMs": 396720,
    "text": "and fakes it needs and opens a PR with a"
  },
  {
    "startMs": 398400,
    "text": "generated test label so we can identify"
  },
  {
    "startMs": 400080,
    "text": "it easily later."
  },
  {
    "startMs": 402560,
    "text": "These agents run on Dualingo's cloud"
  },
  {
    "startMs": 404000,
    "text": "coding agent which is a platform that"
  },
  {
    "startMs": 405280,
    "text": "also runs on Neporal. The simplest way"
  },
  {
    "startMs": 406960,
    "text": "to think about it is as an abstraction"
  },
  {
    "startMs": 408240,
    "text": "on top of cloud code and codex running"
  },
  {
    "startMs": 409520,
    "text": "in the cloud. We can invoke it from"
  },
  {
    "startMs": 411199,
    "text": "Slack, temporal or GitHub. It talks to a"
  },
  {
    "startMs": 412960,
    "text": "bunch of our internal tools for MCP and"
  },
  {
    "startMs": 414639,
    "text": "it logs a bunch of events so we can see"
  },
  {
    "startMs": 416400,
    "text": "what each agent run is doing and how"
  },
  {
    "startMs": 417680,
    "text": "much it costs. We started running these"
  },
  {
    "startMs": 420319,
    "text": "on Linux machines and then later move"
  },
  {
    "startMs": 422000,
    "text": "them to MacRunners which will become"
  },
  {
    "startMs": 423360,
    "text": "important in a minute. Here's what a"
  },
  {
    "startMs": 425840,
    "text": "generated test PR looks like. Every PR"
  },
  {
    "startMs": 427759,
    "text": "description follows the same template"
  },
  {
    "startMs": 429039,
    "text": "and it says explicitly that the PR is AI"
  },
  {
    "startMs": 430960,
    "text": "generated. The reviewer has four"
  },
  {
    "startMs": 432639,
    "text": "options. They can approve it and the PR"
  },
  {
    "startMs": 434160,
    "text": "automarges. They can push changes to it"
  },
  {
    "startMs": 436080,
    "text": "directly. They can reassign it to"
  },
  {
    "startMs": 437360,
    "text": "someone better suited or they can close"
  },
  {
    "startMs": 438800,
    "text": "it out with feedback on why it isn't"
  },
  {
    "startMs": 440000,
    "text": "mergeable."
  },
  {
    "startMs": 441680,
    "text": "Now once we had the backfl running in"
  },
  {
    "startMs": 443199,
    "text": "production we had a slightly new"
  },
  {
    "startMs": 444240,
    "text": "problem. We suddenly had tens of tens to"
  },
  {
    "startMs": 446479,
    "text": "hundreds of open PRs that nobody could"
  },
  {
    "startMs": 447840,
    "text": "manage by hand. So how do we get all of"
  },
  {
    "startMs": 449680,
    "text": "these to a mergeable state and in front"
  },
  {
    "startMs": 450880,
    "text": "of the right reviewer"
  },
  {
    "startMs": 452960,
    "text": "that's where the PR life cycle workflow"
  },
  {
    "startMs": 454479,
    "text": "comes in. You can think of it as a state"
  },
  {
    "startMs": 456080,
    "text": "machine. It runs every couple of hours"
  },
  {
    "startMs": 457759,
    "text": "looks through every open generated test"
  },
  {
    "startMs": 459360,
    "text": "PR and takes exactly one action on each."
  },
  {
    "startMs": 462080,
    "text": "It covers the next two steps of our"
  },
  {
    "startMs": 463759,
    "text": "loop. Verify which is does this PR pass"
  },
  {
    "startMs": 466000,
    "text": "CI and is it high quality and resolve"
  },
  {
    "startMs": 467840,
    "text": "which is getting a human to review it"
  },
  {
    "startMs": 469039,
    "text": "and monitor it. The logic for this"
  },
  {
    "startMs": 471360,
    "text": "workflow is a simple decision tree. If"
  },
  {
    "startMs": 472960,
    "text": "there are any merge conflicts or the PR"
  },
  {
    "startMs": 474240,
    "text": "has been open for about 3 weeks, we"
  },
  {
    "startMs": 475360,
    "text": "close it out and mark it as still. If CI"
  },
  {
    "startMs": 477199,
    "text": "is still running or a reviewer is"
  },
  {
    "startMs": 478400,
    "text": "already on it, we skip it. If CI is"
  },
  {
    "startMs": 480080,
    "text": "green and there's no reviewers, we find"
  },
  {
    "startMs": 481280,
    "text": "one assign them. If CI is red, we try to"
  },
  {
    "startMs": 483280,
    "text": "fix it. And if a reviewer rejects the"
  },
  {
    "startMs": 485440,
    "text": "PR, the file goes into 30-day cool down."
  },
  {
    "startMs": 487199,
    "text": "So, we're not spamming engineers with"
  },
  {
    "startMs": 488240,
    "text": "the same PR again and again."
  },
  {
    "startMs": 490639,
    "text": "When CI fails, the first thing we check"
  },
  {
    "startMs": 492400,
    "text": "is which job failed. If it's pre-commit,"
  },
  {
    "startMs": 494319,
    "text": "a GitHub action applies the fix. It's"
  },
  {
    "startMs": 496000,
    "text": "cheap, it's deterministic, and there's"
  },
  {
    "startMs": 497199,
    "text": "no agent needed. If it's a real test"
  },
  {
    "startMs": 498879,
    "text": "failure, we kick off another agent that"
  },
  {
    "startMs": 500319,
    "text": "reads the CI logs and pushes a fix to"
  },
  {
    "startMs": 501759,
    "text": "the PR. We cap that whole loop at five"
  },
  {
    "startMs": 503919,
    "text": "attemps. Past that we close the PR"
  },
  {
    "startMs": 505360,
    "text": "rather than trying to fix it forever."
  },
  {
    "startMs": 507840,
    "text": "Once CI is green, we need to find the"
  },
  {
    "startMs": 509280,
    "text": "right reviewer. What we do is we run git"
  },
  {
    "startMs": 511120,
    "text": "blame on the source file being tested."
  },
  {
    "startMs": 512399,
    "text": "Find the top contributor and assign it"
  },
  {
    "startMs": 514080,
    "text": "to them and turn on auto merge. If they"
  },
  {
    "startMs": 515839,
    "text": "approve it, the PR gets merged. If they"
  },
  {
    "startMs": 517360,
    "text": "rejected, the file goes into the 30-day"
  },
  {
    "startMs": 518800,
    "text": "cool down, which I previously mentioned."
  },
  {
    "startMs": 520880,
    "text": "This brings us to milestone four. Once"
  },
  {
    "startMs": 522959,
    "text": "all of this was back filling tests for"
  },
  {
    "startMs": 524320,
    "text": "existing code, we wanted to make it just"
  },
  {
    "startMs": 525760,
    "text": "as easy to get tests for the new code"
  },
  {
    "startMs": 527040,
    "text": "engineers were writing. Now any engineer"
  },
  {
    "startMs": 529440,
    "text": "can add or write test label to their own"
  },
  {
    "startMs": 530880,
    "text": "PR, it hooks into the same temporal"
  },
  {
    "startMs": 532880,
    "text": "infrastructure we just walked through."
  },
  {
    "startMs": 534160,
    "text": "The only real difference is selection."
  },
  {
    "startMs": 535680,
    "text": "Instead of the selection script, we take"
  },
  {
    "startMs": 537120,
    "text": "the file straight from the engineer's"
  },
  {
    "startMs": 538240,
    "text": "PR. The agent writes the test and you"
  },
  {
    "startMs": 540160,
    "text": "get a draft SPR link back to your"
  },
  {
    "startMs": 541519,
    "text": "original one. Now let's talk about some"
  },
  {
    "startMs": 543040,
    "text": "of the challenges that we ran into. Our"
  },
  {
    "startMs": 544880,
    "text": "agents originally ran on Linux machines"
  },
  {
    "startMs": 546560,
    "text": "which don't have X code or Swift"
  },
  {
    "startMs": 547920,
    "text": "installed. So when an agent would open a"
  },
  {
    "startMs": 549519,
    "text": "PR after about 20 or 30 minutes CI would"
  },
  {
    "startMs": 551680,
    "text": "fail frequently on lint. This wastes the"
  },
  {
    "startMs": 553760,
    "text": "agents time and CI time as well which is"
  },
  {
    "startMs": 555519,
    "text": "an increasingly scarce resource. For a"
  },
  {
    "startMs": 557519,
    "text": "while 30 13.6% of our PRs were failing"
  },
  {
    "startMs": 559680,
    "text": "due to lint and then when we dug into it"
  },
  {
    "startMs": 561279,
    "text": "62% of those failures came from just to"
  },
  {
    "startMs": 562880,
    "text": "lint rules. We fixed this in layers. The"
  },
  {
    "startMs": 565440,
    "text": "first was better examples in the prompt"
  },
  {
    "startMs": 567040,
    "text": "along with pitfalls to avoid. The second"
  },
  {
    "startMs": 568959,
    "text": "was a GitHub action that auto fixes lint"
  },
  {
    "startMs": 570640,
    "text": "errors in CI. The third was an audit of"
  },
  {
    "startMs": 572560,
    "text": "the rules that couldn't be autofixed. We"
  },
  {
    "startMs": 574320,
    "text": "essentially removed the ones that"
  },
  {
    "startMs": 575360,
    "text": "weren't high signal and this made life"
  },
  {
    "startMs": 576880,
    "text": "easier for both agents and human"
  },
  {
    "startMs": 578080,
    "text": "engineers. And finally, the real fix"
  },
  {
    "startMs": 579760,
    "text": "came when we moved the agents to run on"
  },
  {
    "startMs": 581040,
    "text": "Mac runners. So now the agents could"
  },
  {
    "startMs": 582800,
    "text": "build and lend the code locally before"
  },
  {
    "startMs": 584399,
    "text": "ever opening a PR. The second challenge"
  },
  {
    "startMs": 587120,
    "text": "was PR quality. When we assign a PR to a"
  },
  {
    "startMs": 589279,
    "text": "human, we wanted to be something they'd"
  },
  {
    "startMs": 590480,
    "text": "actually feel comfortable proving. But"
  },
  {
    "startMs": 592240,
    "text": "agents sometimes take the shortest paths"
  },
  {
    "startMs": 593680,
    "text": "to green CI. We saw agents editing"
  },
  {
    "startMs": 595760,
    "text": "production code to make a failing test"
  },
  {
    "startMs": 596959,
    "text": "pass. We also saw them modifying,"
  },
  {
    "startMs": 598320,
    "text": "deleting, or silencing existing tests."
  },
  {
    "startMs": 599920,
    "text": "And then we saw them adding long weight"
  },
  {
    "startMs": 601200,
    "text": "to tests which can make CI flaky."
  },
  {
    "startMs": 603920,
    "text": "Again, there was no single fix here, so"
  },
  {
    "startMs": 605519,
    "text": "we built three layers. The first is the"
  },
  {
    "startMs": 607440,
    "text": "deterministic rule. If a PR touches any"
  },
  {
    "startMs": 609519,
    "text": "file that isn't a test, a mock, a fake,"
  },
  {
    "startMs": 611279,
    "text": "or a stub, we close it out. The second"
  },
  {
    "startMs": 613360,
    "text": "builds on our AI code review. At"
  },
  {
    "startMs": 614959,
    "text": "Dualingo, we use Claude and Codex to"
  },
  {
    "startMs": 616480,
    "text": "review every PR. And those B bots often"
  },
  {
    "startMs": 618560,
    "text": "leave comments. So, we built an agent"
  },
  {
    "startMs": 620399,
    "text": "that resolves any P 0 comments before a"
  },
  {
    "startMs": 622160,
    "text": "human ever looks at the PR. And the"
  },
  {
    "startMs": 624320,
    "text": "third is an LLM as a judge. It's a"
  },
  {
    "startMs": 625839,
    "text": "reviewer agent built just for these test"
  },
  {
    "startMs": 627279,
    "text": "PRs. It reads the PR diff and rejects"
  },
  {
    "startMs": 629440,
    "text": "production edits, changed or deleted"
  },
  {
    "startMs": 630880,
    "text": "tests, disabled tests, and long weights."
  },
  {
    "startMs": 633040,
    "text": "And as you learn more, we keep adding"
  },
  {
    "startMs": 634399,
    "text": "additional checks to it."
  },
  {
    "startMs": 636959,
    "text": "So far, the LM as a judge has reviewed"
  },
  {
    "startMs": 638480,
    "text": "about,200 PRs and rejected 73 of them."
  },
  {
    "startMs": 641200,
    "text": "Every rejection comes with a comment"
  },
  {
    "startMs": 642480,
    "text": "like this one, saying exactly why the PR"
  },
  {
    "startMs": 644079,
    "text": "was rejected. We then read that feedback"
  },
  {
    "startMs": 646079,
    "text": "and fold it back into our prompts and"
  },
  {
    "startMs": 647440,
    "text": "skills, so the same mistakes get less"
  },
  {
    "startMs": 648959,
    "text": "likely each time. So, let's talk about"
  },
  {
    "startMs": 651440,
    "text": "what all of this added up to. Since we"
  },
  {
    "startMs": 653360,
    "text": "started, the pipeline has merged more"
  },
  {
    "startMs": 654720,
    "text": "than 700 test PRs, adding about 225,000"
  },
  {
    "startMs": 657279,
    "text": "lines of test code. Every one of those"
  },
  {
    "startMs": 659040,
    "text": "PRs was reviewed and approved by a"
  },
  {
    "startMs": 660480,
    "text": "human. MVVM test coverage went from 9%"
  },
  {
    "startMs": 663040,
    "text": "to 59% and overall coverage went from"
  },
  {
    "startMs": 664959,
    "text": "11% to 24%. Most importantly, we now"
  },
  {
    "startMs": 667680,
    "text": "have a code base so engineers and agents"
  },
  {
    "startMs": 668959,
    "text": "can ship faster and with more"
  },
  {
    "startMs": 670000,
    "text": "confidence. Here's a few more"
  },
  {
    "startMs": 672240,
    "text": "interesting stats. Since March, the"
  },
  {
    "startMs": 673760,
    "text": "pipeline has written 34% of all new test"
  },
  {
    "startMs": 675600,
    "text": "files in the repo, 44% of all new test"
  },
  {
    "startMs": 677760,
    "text": "cases, and 10% of every swift line added"
  },
  {
    "startMs": 679200,
    "text": "to the codebase. And interestingly, that"
  },
  {
    "startMs": 680959,
    "text": "share has actually dripped in the last"
  },
  {
    "startMs": 682160,
    "text": "few weeks because engineers are now"
  },
  {
    "startMs": 683279,
    "text": "writing a lot more tests themselves,"
  },
  {
    "startMs": 684560,
    "text": "which is honestly the end goal."
  },
  {
    "startMs": 687440,
    "text": "Um, and here's the coverage curve since"
  },
  {
    "startMs": 689680,
    "text": "April when the backfield started running"
  },
  {
    "startMs": 690959,
    "text": "on a schedule. You can see it bent"
  },
  {
    "startMs": 692640,
    "text": "slightly upward around June and July,"
  },
  {
    "startMs": 693920,
    "text": "which is when we really ramped up the"
  },
  {
    "startMs": 695040,
    "text": "pipeline."
  },
  {
    "startMs": 696720,
    "text": "So, what have we learned while doing"
  },
  {
    "startMs": 697920,
    "text": "this project? The first thing is that"
  },
  {
    "startMs": 699600,
    "text": "this pattern generalizes the loop we"
  },
  {
    "startMs": 701440,
    "text": "talked about identify, write, verify,"
  },
  {
    "startMs": 703120,
    "text": "resolve, iterate. It isn't specific to"
  },
  {
    "startMs": 704640,
    "text": "testing. Unit tests were just the lowest"
  },
  {
    "startMs": 706320,
    "text": "hanging fruit and any job that fits the"
  },
  {
    "startMs": 708000,
    "text": "shape can run on the same infrastructure"
  },
  {
    "startMs": 709600,
    "text": "and that's exactly what we're starting"
  },
  {
    "startMs": 710720,
    "text": "to do at the company. We're applying the"
  },
  {
    "startMs": 712320,
    "text": "same loop to modernization to"
  },
  {
    "startMs": 713920,
    "text": "modernizing screens to MVM and Swift UI,"
  },
  {
    "startMs": 715839,
    "text": "migrating code uh on Android to Corines"
  },
  {
    "startMs": 718160,
    "text": "and generating snapshot tests."
  },
  {
    "startMs": 720959,
    "text": "So before I wrap up, I want to leave you"
  },
  {
    "startMs": 722720,
    "text": "with a thought experiment. There's a new"
  },
  {
    "startMs": 724320,
    "text": "class of fast general purpose"
  },
  {
    "startMs": 725440,
    "text": "classifiers like Jev. They taken some"
  },
  {
    "startMs": 727120,
    "text": "state and output an action in"
  },
  {
    "startMs": 728240,
    "text": "milliseconds. So what if we replace a"
  },
  {
    "startMs": 730000,
    "text": "handwritten decision tree in the PR life"
  },
  {
    "startMs": 731440,
    "text": "cycle workflow with one of these? PR"
  },
  {
    "startMs": 733440,
    "text": "state goes in, the classifier picks the"
  },
  {
    "startMs": 735200,
    "text": "next action and we execute it all in"
  },
  {
    "startMs": 736720,
    "text": "many seconds."
  },
  {
    "startMs": 739120,
    "text": "I also want to thank a few folks who"
  },
  {
    "startMs": 740639,
    "text": "worked on this with me, Jesse, Gordalup,"
  },
  {
    "startMs": 742399,
    "text": "Hunter and Jiho. They built a lot of the"
  },
  {
    "startMs": 743760,
    "text": "platform pieces this project stands on"
  },
  {
    "startMs": 745040,
    "text": "and helped review a lot of this code."
  },
  {
    "startMs": 747360,
    "text": "That's it for me. Thank you so much for"
  },
  {
    "startMs": 748720,
    "text": "making it this far. If you want to go"
  },
  {
    "startMs": 749920,
    "text": "deeper, we've written about this"
  },
  {
    "startMs": 750880,
    "text": "pipeline on the dualing engineering"
  },
  {
    "startMs": 752000,
    "text": "blog. And I'd also love to hear from you"
  },
  {
    "startMs": 753440,
    "text": "if you're building something similar."
  },
  {
    "startMs": 754399,
    "text": "You can find me on LinkedIn or next"
  }
]
