# How to Build a Software Factory That Improves Itself - Zach Lloyd, Warp

Zach Lloyd, Warp

[Talk page](https://ai.engineer/talks/nKjPBAw_N_0-build-software-factory-that-improves-itself) · [Video](https://www.youtube.com/watch?v=nKjPBAw_N_0)

## How Warp Measures Whether Its Coding Agents Improve

Zach Lloyd opens Warp’s own engineering dashboard to show how a team can turn repeated mistakes into a pull request, then replay its past tasks to decide which model earns its cost.

![A circular workflow connects bug reports and new ideas with triage, specification, implementation, verification and release. People review the specification, code and product.](https://ai.engineer/api/talk-essay-media/18efecdf8df1d126ab9a5536e69a4c767db8b5ebc9d466c37cdd3260f3b94b7a)

A product idea or bug report moves through a loop with explicit points for people to review the specification, code and result.

### What disappears when the laptop closes?

[1:07](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=67s)

An engineer opens several coding agents, assigns each one a task and switches between them when they ask for input. It can be a productive afternoon. At the end of it, a team lead may still struggle to answer three questions: Which changes reached users? What did they cost? And what did one agent learn that the next engineer can reuse?

Zach Lloyd, Warp’s founder and a former engineering lead on Google Docs, describes that arrangement as interactive prompting. Each laptop has its own tools, permissions and configuration. Someone has to start each job and keep it moving. A local agent may also inherit access its particular assignment never required. Issuing more subscriptions gives the company more of these sessions, with little insight into how the work adds up.

In the alternative Lloyd calls a software factory, a person submits a feature or reports a bug. The work can pass through triage, a specification, implementation, review and verification. Some steps need a person. Others suit an agent running in an isolated cloud environment. A reviewer can see what went into a step and what came out after the person who started it has gone home.

### Put the workflow in a repository

[4:01](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=241s)

Lloyd wants the engineering team to own the definition of this process. Its repository records which agent handles a step, which tools it may use, the instructions it gets, the model it calls and where it runs. The same repository can hold benchmarks and the events that start a job. Someone reviewing a change can see precisely which part of tomorrow’s workflow will differ from today’s.

There is a business reason to spell these things out. A team might want to keep its data on its own infrastructure, pay for inference through an existing account or replace a model when another proves better for its work. Those decisions should be visible to the people responsible for the result. A stable record also gives an engineer two versions of a workflow they can test against each other.

Warp gives its product engineers familiar entry points. A task can arrive through Slack, a Linear ticket or a GitHub review comment. In the demo, an orchestrator selects the next step. A dashboard brings the runs and their artifacts together. Lloyd distinguishes the engineer shipping a customer feature from the engineer who improves the prompts, skills and routing that many such features will use. On a small team, those can be two jobs for the same person.

### When the same mistake shows up 37 times

[12:01](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=721s)

The dashboard starts with ordinary operating questions: cycle time, spend, pull requests and the number of times a person had to intervene. The interesting part lies behind the charts. A team can inspect what an agent received, the conversation as it worked and the change it left behind. That record gives a reviewer much more to go on than a green check beside a pull request.

Warp runs model-based evaluators on these traces. One looks for redundant tests. A task can produce passing code and an unnecessary test, so a passing test suite alone would miss the behavior that reviewer cares about. Storing the evaluator’s judgment with the run lets an engineer inspect the cases where it claims to have found the problem.

In a separate example, the improvement system found 37 instances of agents beginning implementation without reading the relevant skills already in the repository. Its suggested fix was a pull request against the factory definition. Lloyd opened the proposal onstage. He showed how a recurring mistake can lead to a specific edit someone can review. Whether that edit reduces the failure rate is a question for the runs that follow.

![One expanded improvement proposal recommends looking up relevant repository skills before an implementation starts.](https://ai.engineer/api/talk-essay-media/fab28ffd12a7397bf02c1b7cc8fe44195806a8d98d5e36729944dd77befd0606)

Warp’s improvement view links the proposal to read repository skills with 37 prior failures.

### A benchmark drawn from yesterday’s tickets

[14:48](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=888s)

How would the team know whether to buy a more expensive model for front-end work? A public leaderboard says little about its own repository or conventions. Lloyd wants to take representative work the team has already completed, preserve the starting conditions and replay it against competing configurations.

The live demo supplied a mundane but important detail. Lloyd tried to add a past run to a benchmark and could not do it on the example he had selected. He switched to an existing suite to show the result. In his spoken comparison, one configuration scored about 93 percent for roughly two dollars. Another reached about 95 percent and cost around three times as much. A team can decide what those last points are worth once it knows which tasks the cheaper choice misses.

A question from the audience established what “correct” meant on that chart. The scores came from an LLM judge. Human grading was something Lloyd recommended, and Warp had yet to add it to that workflow. His advice for building the first suite was equally practical: pick a few easy, medium and difficult tasks by hand. With a baseline in place, use the archive of previous sessions to look for cases it leaves out.

![The tooltip on the cost–correctness graph reports the average cost and correctness for the high configuration.](https://ai.engineer/api/talk-essay-media/2c2c63edc15bd67ebcb87f251ad1706702b4265c0b1668985dd5f4c32490b658)

The tooltip on Warp’s internal benchmark shows 94.8% correctness at $6.23 average cost for one configuration. An LLM judge supplied the scores.

### What did the user actually get?

[18:02](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1082s)

More agent throughput can leave humans staring at a longer queue of pull requests. Asked about verification, Lloyd suggests writing a specification the result can be checked against and asking an agent to produce evidence from the user’s point of view. For an interface change, a reviewer who can see the changed interaction has a more concrete starting point than a summary saying the implementation is complete. The talk describes that practice; the demonstration focused on scoring and benchmarking.

Verification costs money too. Lloyd’s suggested unit is the cost per pull request, with scoring and benchmark expenditure broken out. That lets a team estimate a project from work it recognizes and decide where another evaluation is worth paying for. It also reveals when a seemingly cheap model needs expensive supervision.

Warp’s terminal is open source. Lloyd clarified in the Q&A that the factory product he demonstrated is proprietary and designed to accommodate a team’s own tools. An individual with one interactive agent may have little need for it. The problem becomes pressing when one engineer’s change to a shared workflow can improve the next hundred tasks, and the team needs evidence to decide whether to keep it.

## Resources

- [Warp documentation](https://docs.warp.dev/) — Warp documents its terminal, cloud automation and software-factory workflows.

## Slides

### The software factory stack

[0:11](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=11s) [Warp’s software factory stack appears on Zach Lloyd’s title slide.](https://assets.ai.engineer/sites/aiecode2025/sha256/e8cc4a5b04a869bbf964d134ec9e0c2cc09f2022e52a4bf693bedb4ed27549eb.webp)

The Software Factory Stack. Zach Lloyd | Warp

### Zach Lloyd’s background

[0:22](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=22s) [Lloyd introduces his work at Google and Warp beside his portrait.](https://assets.ai.engineer/sites/aiecode2025/sha256/07764508ece8e1c2f057b9d94c74dca94b2852e495bca1e8570c14832c8ad01f.webp)

Zach Lloyd. Principal engineer at Google, lead on GDocs. 20 years of engineering. Founder of Warp.

### The evolution of agentic development

[1:10](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=70s) [Chat and autocomplete precede interactive prompting and automated development on a rising curve.](https://assets.ai.engineer/sites/aiecode2025/sha256/97c83f6410b1c4072fb58ad608a877d032a49e649b818110a9a63de0810e500e.webp)

The evolution of agentic development. Chat. Autocomplete. Interactive Prompting. Automated development.

### The limits of individual agent sessions

[1:47](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=107s) [Three limitations of ad hoc AI use sit beside coverage of corporate AI spending and governance.](https://assets.ai.engineer/sites/aiecode2025/sha256/723806d401fb5d7ec967188fda0ed45b254b5485f9af639fd8f744f3c2545b32.webp)

More AI usage != more business value. Ad hoc interactive usage, no automation. No systematic way to optimize cost and quality. No central layer for governance and control.

### Governing local agents

[2:19](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=139s) [Lloyd presents the governance problem for agents running on individual machines.](https://assets.ai.engineer/sites/aiecode2025/sha256/8d3643f39525e449ccb4d3ec6e2f294101b54340c16215761131fd7519c806a6.webp)

Local agents are ungovernable agents.

### Isolate agent work

[2:52](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=172s) [A single instruction sums up Lloyd’s proposed change to the execution environment.](https://assets.ai.engineer/sites/aiecode2025/sha256/85a97f6f72cbb2a536811d4e1fca0a70641417d66facf8472327666273a4fd30.webp)

Sandbox your agents.

### The cloud software factory loop

[2:58](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=178s) [A circular workflow connects bug reports and new ideas with triage, specification, implementation, verification and release. People review the specification, code and product.](https://assets.ai.engineer/sites/aiecode2025/sha256/18efecdf8df1d126ab9a5536e69a4c767db8b5ebc9d466c37cdd3260f3b94b7a.webp)

A cloud software factory. New product idea or bug reported. Triage. Spec. Reviewing the spec. Implement. Reviewing the code. Verify. Reviewing the product. Ship. Monitor.

### A factory defined in code

[4:00](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=240s) [Lloyd lists three desirable properties of the underlying infrastructure.](https://assets.ai.engineer/sites/aiecode2025/sha256/428386da948f0cc2903e2eee12739eb1e2775177166bbdd939290d3a83aa1e00.webp)

Software factories should be built on an infrastructure stack that is open, adaptable, and defined in code.

### Control over AI infrastructure

[5:12](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=312s) [Lloyd introduces the ownership requirements that the following slide breaks down.](https://assets.ai.engineer/sites/aiecode2025/sha256/0742256ebe91a394c4f2b60da91a578d2ed9cfb0a2a882b59184657fa5db977b.webp)

AI Sovereignty matters.

### Control over data, compute and inference

[5:44](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=344s) [The completed slide lists the infrastructure and agent choices an organization can own.](https://assets.ai.engineer/sites/aiecode2025/sha256/191bbf51ca0c56c6a1c70823f083e7426b5a92e900201b7206d5584d39bb677d.webp)

Your data. Your compute. Your inference gateway. Any model, any harness.

### Three ways to measure the system

[5:59](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=359s) [Scoring, self-improvement and benchmarking are listed as distinct measurement activities.](https://assets.ai.engineer/sites/aiecode2025/sha256/6118696a075350c02076293864f2c5fb8caeb0b26aa7fdcc91befed68b2ae924.webp)

Measurement > Vibes. Scoring vs. Self-improvement vs. Benchmarking.

### Product and factory engineers

[6:53](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=413s) [The diagram branches from a shared factory to the people shipping product changes and those improving the workflow.](https://assets.ai.engineer/sites/aiecode2025/sha256/632f4b6667d8bc49df9a1f9d28196a45677046534aeee980e958ccbf695393df.webp)

Two types of engineers will use your factory: Product Engineer. Factory Engineer.

### Measuring productivity across runs

[8:23](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=503s) [The dashboard plots human interactions, cycle time and pull requests alongside the total number of runs.](https://assets.ai.engineer/sites/aiecode2025/sha256/7925091ed357b382534c9c6ff8afe5246e8dc7e9969db5f95a97f128b82df3b7.webp)

Productivity. Spending. Scorers. Human interactions per PR. Median cycle time. Pull requests. Total runs.

### Tracking cost per pull request

[8:51](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=531s) [The spending view charts cost per PR and shows how spending varies by source.](https://assets.ai.engineer/sites/aiecode2025/sha256/848b476607cf6c7de69a851fb7ac24fc7f77184139317f28c07d118259604dd5.webp)

Spending. Cost per PR. $17.63. Cost breakdown. Spend by source. $5,251.16.

### The agents behind the workflow

[10:32](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=632s) [An orchestration view places the foreman over agents responsible for specification, triage, implementation and code review.](https://assets.ai.engineer/sites/aiecode2025/sha256/4a5be47f3abe9149012186e93426bc5fd2c467fd479814bcf0da3dc773475a93.webp)

wilson Foreman. Orchestrates the factory workflow and dispatches each gated step. code-review. implementation. spec. triage.

### Scoring redundant tests

[12:46](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=766s) [A grader’s detail view describes how it judges tests delivered in the implementation and charts its historical pass rate.](https://assets.ai.engineer/sites/aiecode2025/sha256/306d12c2f0fa90461e6e257de6b490beb5b317784a58d61365d88addbcc2052c.webp)

Redundant Tests. Whether the implementation agent’s final delivered test diff includes redundant, low-signal, or trivial tests. All-time scorer cost. Pass rate. Daily cost.

### Inspecting a grader’s individual decisions

[12:59](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=779s) [A table shows passes, failures and insufficient evidence for the redundant-tests evaluation.](https://assets.ai.engineer/sites/aiecode2025/sha256/85c38f5829668e1ec61a6c58c8a192c111b9d05e3cac29461137e53fcf23dece.webp)

Redundant Tests. Pass. Fail. No surplus tests. Some surplus tests. Insufficient evidence.

### A recurring failure becomes a proposed change

[13:59](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=839s) [One expanded improvement proposal recommends looking up relevant repository skills before an implementation starts.](https://assets.ai.engineer/sites/aiecode2025/sha256/fab28ffd12a7397bf02c1b7cc8fe44195806a8d98d5e36729944dd77befd0606.webp)

Read matching repository skills before implementing. PROBLEM. SOLUTION. 37 failed runs. Self-improvement run.

### Comparing correctness and cost on internal tasks

[16:09](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=969s) [A scatter plot compares five model configurations on WarpBench and names a lowest-cost option.](https://assets.ai.engineer/sites/aiecode2025/sha256/d07d8011b0bec0d3943db8aff3012a84539f564fafb3c30e441568ee0f4ddb30.webp)

WarpBench 0.1. Correctness vs. Average cost. Most efficient: glm 5.3 flash. Overall 0.93. Lowest cost option: glm 5.3 flash. $1.99/trial.

### The more expensive configuration

[16:18](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=978s) [The tooltip on the cost–correctness graph reports the average cost and correctness for the high configuration.](https://assets.ai.engineer/sites/aiecode2025/sha256/2c2c63edc15bd67ebcb87f251ad1706702b4265c0b1668985dd5f4c32490b658.webp)

Correctness vs Average cost. gpt-5.6-sol (high). Average cost: $6.23. Correctness: 94.8%.

### The factories stack

[17:15](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1035s) [The closing diagram builds from factories-as-code and data up through compute, inference, improvement, orchestration and access.](https://assets.ai.engineer/sites/aiecode2025/sha256/863c85335e006b32a3d99461acc59be4ae83dcab6715162aac70c5eaab26781b.webp)

The Factories Stack. 00 Factories-as-code. 01 Data & Context. 02 Compute. 03 Inference. 04 Improvement. 05 Orchestration. 06 Access.

## Full transcript

[0:01](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1s) [music]

[0:12](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=12s) So, I'm going to be talking today about

[0:14](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=14s) software factories and uh specifically

[0:18](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=18s) like an infrastructure approach to

[0:20](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=20s) software factories. Um, a little bit

[0:23](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=23s) about me though before we start. So, I'm

[0:25](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=25s) Zach. Uh, I've been an engineer for

[0:28](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=28s) really for a long time at this point. I

[0:30](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=30s) was a principal engineer at Google. I

[0:31](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=31s) used to lead engineering on the Google

[0:32](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=32s) Doc Suite. Helped scale that up to

[0:35](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=35s) hundreds of millions and I think maybe

[0:37](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=37s) now over a billion people who use it.

[0:39](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=39s) I'm currently the founder and CEO at

[0:41](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=41s) Warp. Uh, Warp is a company in the

[0:44](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=44s) agentic development space. Our current

[0:46](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=46s) focus is on helping companies build

[0:49](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=49s) their own internal software factories.

[0:51](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=51s) So, we have a platform called Warp

[0:53](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=53s) Factories which we do that with. We're

[0:55](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=55s) also uh pretty well known for Warp

[0:57](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=57s) Terminal, which is an Agentic terminal.

[0:59](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=59s) It's open source, around 65,000 GitHub

[1:02](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=62s) stars, almost a million active

[1:03](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=63s) developers. Um, our focus as a company

[1:06](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=66s) has always been helping developers ship

[1:09](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=69s) better software more quickly.

[1:12](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=72s) Just to set some context on where we're

[1:15](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=75s) at uh in terms of agentic development.

[1:18](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=78s) So most companies that I speak with uh

[1:22](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=82s) are at this like phase of what I think

[1:24](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=84s) of as interactive prompting. And so

[1:26](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=86s) they've deployed tools like cloud code

[1:28](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=88s) or codeex to their engineers. Uh

[1:30](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=90s) engineers are working on their laptops.

[1:32](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=92s) They are they might have a bunch of

[1:33](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=93s) these going but generally they're like

[1:35](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=95s) interactively prompting agents to build

[1:37](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=97s) features and fix bugs. And where uh most

[1:42](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=102s) companies I speak with want to get to is

[1:44](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=104s) something that is more like automated

[1:46](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=106s) development. So

[1:48](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=108s) the reason for that is really there's

[1:52](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=112s) this um you know they want to get to a

[1:54](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=114s) world where they are deploying this

[1:56](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=116s) transformational technology which is

[1:58](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=118s) agents and LLMs to get the most possible

[2:01](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=121s) business value at the lowest possible

[2:04](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=124s) cost in a way that is governed and

[2:07](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=127s) secure and they're running up against

[2:09](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=129s) the limits of interactive agents which

[2:11](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=131s) are like you you're bottlenecked by a

[2:14](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=134s) person running them. Um it's hard to

[2:17](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=137s) control costs if everyone has their own

[2:18](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=138s) setup and it's hard to govern. And so

[2:21](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=141s) just to double click on that I think for

[2:23](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=143s) this audience like local agents have

[2:25](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=145s) this security problem where uh if there

[2:29](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=149s) is a problem or compromise of a local

[2:31](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=151s) agent then it's basically has the

[2:34](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=154s) ability to take over whatever the user

[2:36](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=156s) who signed in on their laptop is. And so

[2:38](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=158s) that's not a good thing. Um local agents

[2:41](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=161s) are also hard to govern. And so right

[2:42](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=162s) now we're in this situation where you

[2:45](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=165s) know everyone is kind of configuring

[2:46](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=166s) their own MCPs, their own skills, their

[2:48](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=168s) own tools. And it's like you can't see

[2:50](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=170s) what's going on. You can't audit it. And

[2:52](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=172s) so the solution uh to both these

[2:55](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=175s) problems is to sandbox your agents and

[2:58](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=178s) move them to the cloud. And the product

[3:00](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=180s) category that is emerging to support

[3:03](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=183s) this uh is what um you know we call a

[3:06](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=186s) cloud software factory. [snorts] And so

[3:08](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=188s) this this term is kind of overloaded,

[3:10](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=190s) but what it means to me is basically

[3:15](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=195s) every company is going to set up a loop

[3:17](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=197s) in the cloud where

[3:19](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=199s) um a combination of humans and agents do

[3:23](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=203s) work. And so humans are responsible for

[3:26](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=206s) putting work in, agents will do

[3:29](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=209s) different parts of the software life

[3:31](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=211s) cycle. also a triage. They'll work with

[3:33](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=213s) the human on a spec. They'll do

[3:35](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=215s) implementation. They will do computer

[3:38](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=218s) use verification. And as a as a

[3:41](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=221s) business, what you're going to want to

[3:42](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=222s) do is set these up internally. Define

[3:46](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=226s) all the rules for how humans and agents

[3:47](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=227s) [music]

[3:48](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=228s) interact and then get to a spot where

[3:50](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=230s) you can really, really measure and

[3:53](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=233s) improve over time. Because getting this

[3:55](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=235s) concept right is going to be the key

[3:58](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=238s) driver to how your organization ships

[4:00](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=240s) software.

[4:02](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=242s) Um at at warp like our our belief is

[4:04](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=244s) that you should think of this as

[4:06](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=246s) infrastructure. So not as like a product

[4:08](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=248s) or an AI teammate but really like an

[4:10](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=250s) infrastructure stack that you can build

[4:13](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=253s) on top of and scale. And the key aspects

[4:16](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=256s) of this are like it should be open, it

[4:19](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=259s) should be adaptable and you should think

[4:20](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=260s) of your factories as something that you

[4:22](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=262s) define in code. And so by open I mean

[4:27](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=267s) you want a a factory model that can

[4:29](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=269s) support different uh different models

[4:31](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=271s) whether it's open weight or frontier.

[4:33](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=273s) You want one that can support different

[4:35](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=275s) harnesses and the reason is just like

[4:37](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=277s) the best models and harnesses are

[4:39](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=279s) constantly changing. You want adaptable

[4:42](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=282s) means like the factory should work with

[4:43](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=283s) your existing infrastructure. And then

[4:46](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=286s) finally defined in code means like your

[4:49](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=289s) factory should be a thing uh that has

[4:53](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=293s) like a strict definition and that

[4:55](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=295s) definition should include like what are

[4:57](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=297s) the workflows, what are the skills, what

[4:59](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=299s) are the MCPs, where do these run and

[5:01](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=301s) when you define it in code and version

[5:03](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=303s) control it then you get the ability to

[5:05](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=305s) like test different versions of your

[5:07](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=307s) factory. You get the ability for coding

[5:09](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=309s) agents to update your factory. So these

[5:11](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=311s) are some key characteristics.

[5:14](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=314s) Uh I'll say another thing that I'm

[5:16](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=316s) hearing from from businesses is like AI

[5:19](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=319s) sovereignty matters. And so the concept

[5:21](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=321s) here is like as you think about

[5:24](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=324s) factories, you should think about uh

[5:26](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=326s) owning all the data that goes in and all

[5:29](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=329s) the data that comes out as well.

[5:32](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=332s) You should think about how do you have

[5:33](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=333s) these agents running on your own

[5:35](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=335s) infrastructure and own compute. Um this

[5:37](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=337s) is what most larger enterprises that we

[5:39](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=339s) work with want.

[5:41](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=341s) How do you provide your own inference?

[5:44](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=344s) And then finally, like I said, any

[5:45](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=345s) model, any harness. And so really

[5:48](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=348s) preserve optionality and treat um treat

[5:51](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=351s) these software factories as something

[5:52](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=352s) that's very key to your business and

[5:55](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=355s) that you want to, you know, mostly take

[5:57](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=357s) ownership of the core ingredients of.

[6:01](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=361s) Um, another really important point here

[6:03](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=363s) is like as you think about building your

[6:06](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=366s) software factory, like if you were to

[6:08](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=368s) look on Twitter on any given day, you

[6:10](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=370s) would probably see different model like

[6:13](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=373s) someone would say like PI is the best or

[6:14](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=374s) someone say Grock or like there's all

[6:16](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=376s) these different things in the world

[6:18](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=378s) around like what is best and you might

[6:19](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=379s) even see benchmarks like Sweetbench or

[6:21](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=381s) Terminal Bench. But the world that you

[6:23](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=383s) want to get to um as you automate more

[6:26](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=386s) development is actually being

[6:29](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=389s) measurement first on your own data. And

[6:31](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=391s) so you should whatever factory model you

[6:33](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=393s) approach have the ability to score

[6:36](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=396s) meaning like eval how your coding agents

[6:38](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=398s) are doing. uh feed those scores into

[6:41](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=401s) agents that can improve your factory and

[6:44](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=404s) then also create your own benchmarks.

[6:46](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=406s) Meaning like be able to take all of your

[6:48](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=408s) real coding agent tasks and try them

[6:51](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=411s) against different models and measure how

[6:52](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=412s) it goes.

[6:55](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=415s) So I'll also talk a little bit about

[6:57](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=417s) like um how engineering changes in this

[7:01](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=421s) factory world.

[7:02](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=422s) [snorts] And so the way that I I conceive of this

[7:05](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=425s) and talk about it is you will still have

[7:07](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=427s) engineers who are very much responsible

[7:09](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=429s) for building useful things for your

[7:12](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=432s) customers. And so they'll build be

[7:13](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=433s) building the features, they'll be fixing

[7:15](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=435s) bugs, they'll be doing product

[7:16](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=436s) engineering and they'll be doing it like

[7:18](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=438s) through the factory. So they put work

[7:20](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=440s) into the factory. But there's a second

[7:22](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=442s) kind of engineering which I call factory

[7:25](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=445s) engineering which is uh either uh your

[7:29](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=449s) platform team or just your general

[7:31](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=451s) engineers will have to get good at like

[7:34](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=454s) trying to make the factory better. So

[7:37](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=457s) improving the skills,

[7:39](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=459s) setting the context, changing the model

[7:41](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=461s) mix, all of these things are going to

[7:43](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=463s) become more important as engineers are

[7:45](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=465s) like building and improving the system

[7:48](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=468s) that actually builds the software.

[7:51](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=471s) So, I'm going to I'm going to show what

[7:53](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=473s) some of this looks like in practice. And

[7:54](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=474s) again, this is going to be in the

[7:56](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=476s) context of our own product, Warp

[7:58](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=478s) Factories. But these general concepts, I

[8:00](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=480s) think, will translate to however you're

[8:03](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=483s) thinking about building your factories.

[8:06](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=486s) I'll try to leave a little bit of time

[8:07](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=487s) for questions at the end if I can.

[8:10](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=490s) So

[8:12](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=492s) I'm going to start with um the view of

[8:16](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=496s) the factory that the factory engineer or

[8:19](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=499s) the manager has. And so this is like

[8:23](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=503s) if you are

[8:25](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=505s) someone who is trying to get better uh

[8:28](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=508s) productivity across your team, you're

[8:30](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=510s) going to want some view of like what is

[8:34](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=514s) happening from a productivity standpoint

[8:35](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=515s) like how automated is my factory? Uh

[8:38](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=518s) you're going to want a view of um what's

[8:41](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=521s) it what's the cycle time. So like a

[8:43](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=523s) dorometric view. You're going to want a

[8:45](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=525s) view of uh spending. This might take a

[8:47](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=527s) second to load. Um but basically you

[8:50](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=530s) want all this data tracked. And the way

[8:53](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=533s) that you get these views is by having a

[8:57](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=537s) system where everything that your team

[9:01](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=541s) is doing, I don't know if you all can

[9:03](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=543s) see this, but everything that's

[9:05](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=545s) happening on our team, and this is

[9:07](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=547s) Warp's internal software development

[9:09](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=549s) that I'm leaking here, but it's like I

[9:12](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=552s) can see what I can see what every

[9:14](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=554s) engineer on our team is doing, and I can

[9:16](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=556s) see what all the automations that we're

[9:17](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=557s) running are doing. And not just can I

[9:20](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=560s) see them like I can see all the

[9:21](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=561s) artifacts that they're creating, but all

[9:23](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=563s) this is getting stored. And so all these

[9:26](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=566s) agent traces, you know, I I could

[9:28](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=568s) actually go in I could see what this

[9:30](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=570s) this one is doing. I could go in and

[9:32](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=572s) like see the actual conversation that

[9:34](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=574s) someone is doing. But the important

[9:36](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=576s) thing here is that um all this data is

[9:39](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=579s) getting stored and what your coding

[9:41](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=581s) agents are doing is the raw material for

[9:44](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=584s) improving them. And so, uh, you might be

[9:46](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=586s) wondering, well, how do how do engineers

[9:49](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=589s) actually do this work? Like, how are

[9:50](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=590s) they putting stuff in? Um, and the

[9:52](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=592s) answer on this is that they put it in

[9:55](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=595s) through the tools that they're already

[9:56](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=596s) using. So, on our team, they're using

[9:59](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=599s) Slack uh to do a lot of the work. Um,

[10:02](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=602s) they put in uh agentic work through our

[10:04](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=604s) task tracker, which we use linear, but

[10:06](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=606s) it could be Jira. They have the same

[10:08](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=608s) context whether they're uh responding to

[10:11](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=611s) a GitHub PR comment. And so the the idea

[10:14](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=614s) here is like the product engineers are

[10:16](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=616s) using the tools that they're already

[10:17](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=617s) using, but all the work goes into this

[10:19](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=619s) centralized location where you get a

[10:21](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=621s) centralized view and all this stuff gets

[10:23](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=623s) recorded over time by recording

[10:25](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=625s) everything. Uh just to go back to this

[10:28](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=628s) dashboard for a second. Um actually let

[10:32](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=632s) me let me not do that for one second.

[10:33](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=633s) The other thing that's important here

[10:34](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=634s) that I I talked about earlier is like

[10:36](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=636s) the factory workflow itself defining it

[10:40](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=640s) as code. And so this is um our kind of

[10:43](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=643s) like default factory workflow where

[10:46](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=646s) think of it as like workflows in we have

[10:49](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=649s) this orchestrator agent that will figure

[10:51](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=651s) out okay should this next go to a triage

[10:54](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=654s) step should it go to a spec step an

[10:55](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=655s) implementation step code review the

[10:58](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=658s) actual details of this need to be very

[11:00](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=660s) very flexible. Um if you click into one

[11:03](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=663s) of these you can see like essentially

[11:06](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=666s) you are providing an agent definition

[11:09](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=669s) there's uh which is like what are all

[11:11](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=671s) the secrets the MCPS the agent prompt

[11:14](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=674s) the model all this stuff the the cool

[11:17](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=677s) thing though that I would emphasize that

[11:20](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=680s) however you're approaching this you

[11:21](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=681s) probably want to do is that there's an

[11:24](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=684s) actual code repo that sits behind this

[11:27](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=687s) where if you go and like look in that

[11:29](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=689s) repo you can see like I that earlier, a

[11:32](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=692s) definition of the entire factory at any

[11:35](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=695s) given point in time. And so that

[11:36](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=696s) definition includes, think of it as like

[11:39](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=699s) everything that goes into the process of

[11:41](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=701s) automating development. Um, it's your

[11:44](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=704s) it's your uh agent definitions, it's

[11:47](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=707s) your automations, your benchmarks, it's

[11:49](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=709s) where these agents run, the skills, the

[11:50](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=710s) web hooks, everything. So everything

[11:52](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=712s) gets stored in in a repo that it allows

[11:55](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=715s) for automation. Um, and I'll I'll I'll

[11:58](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=718s) show a little more why that matters in a

[12:00](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=720s) second.

[12:01](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=721s) The other thing uh just to return to

[12:03](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=723s) this view for a second because I think

[12:04](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=724s) it's interesting. You get these metrics

[12:07](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=727s) around productivity and spend but you

[12:09](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=729s) also want to set up something where

[12:11](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=731s) there are like we call them scores. You

[12:14](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=734s) could think of it as like online evals.

[12:17](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=737s) And what what you're looking at here is

[12:20](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=740s) basically we have a set of agents that

[12:23](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=743s) are looking at all of these runs that

[12:25](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=745s) are going through our system. So

[12:27](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=747s) remember I go back here. It's like you

[12:28](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=748s) can see everything that everyone's doing

[12:30](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=750s) on our team. You then have a scoring

[12:32](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=752s) mechanism that goes and looks at those

[12:35](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=755s) runs and sort of figures out by using

[12:39](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=759s) LLM as a judge whether or not um those

[12:43](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=763s) runs are good or not. And so for

[12:46](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=766s) instance we have a scorer here that is

[12:49](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=769s) looking for agents that are writing

[12:50](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=770s) redundant tests which is a common

[12:52](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=772s) failure mode with these coding agents.

[12:54](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=774s) And so for the most part, our our agents

[12:56](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=776s) seem to be doing pretty well here. But

[12:58](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=778s) if you look back in time, you'll see

[12:59](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=779s) there's some set of them where an agent

[13:01](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=781s) is deciding, okay, this was this thing

[13:04](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=784s) wrote tests that didn't need to be

[13:05](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=785s) written. And so every one of these

[13:07](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=787s) failures is an opportunity for improving

[13:11](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=791s) how your system works. And so as you

[13:14](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=794s) gather this information, and you know,

[13:17](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=797s) you could go look at this as a person

[13:18](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=798s) and grade it, but it's much more

[13:20](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=800s) efficient to have agents do it. As you

[13:22](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=802s) gather this information, you can then

[13:24](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=804s) use it to feed what we call a

[13:26](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=806s) self-improvement loop. And so the idea

[13:29](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=809s) behind a self-improvement loop is that

[13:32](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=812s) uh agents will go and they'll look at

[13:34](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=814s) all these failure modes. They'll

[13:36](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=816s) aggregate them. They will look for

[13:38](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=818s) patterns and then they will come up with

[13:40](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=820s) ideas for how do you improve the

[13:43](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=823s) underlying factory definition so that

[13:46](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=826s) going forward you don't have this same

[13:48](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=828s) failure mode. And so you know for

[13:52](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=832s) instance um I don't know I'm going to

[13:53](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=833s) pick a random one here. Um this is like

[13:58](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=838s) read matching repository skills before

[14:00](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=840s) implementing. The key thing here is like

[14:03](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=843s) it's saying I saw this fail 37 times and

[14:07](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=847s) then it's saying okay we can actually um

[14:11](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=851s) oops we can actually fix this

[14:15](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=855s) um with a PR

[14:18](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=858s) to the factory definition.

[14:21](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=861s) So this is like closing the loop where

[14:24](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=864s) the agent has identified a problem then

[14:26](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=866s) it's suggesting a change to actually

[14:28](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=868s) drive a better result in the factory. So

[14:30](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=870s) this is like a a core concept I think as

[14:33](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=873s) you think of like how do you really

[14:34](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=874s) improve development it's it's about

[14:37](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=877s) creating these loops where agents can

[14:39](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=879s) identify problems and then fix them.

[14:42](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=882s) There's one more thing that I want to

[14:43](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=883s) show and then maybe I'll take a couple

[14:45](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=885s) questions. Um I mentioned early on like

[14:49](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=889s) you can't really trust the public

[14:52](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=892s) benchmarks [clears throat]

[14:53](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=893s) as like a way of understanding how good

[14:55](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=895s) a certain model or harness is going to

[14:57](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=897s) be on your own workflows. And so what

[14:59](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=899s) you want to get to a point where you can

[15:01](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=901s) do is basically take any past run. So we

[15:06](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=906s) can do this like with any anything that

[15:09](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=909s) we've done in our system. I could go

[15:10](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=910s) into it here and I have the um

[15:14](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=914s) sorry I picked a bad example. I um

[15:20](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=920s) uh maybe I'm picking a bad example. The

[15:22](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=922s) the idea here is like you can basically

[15:24](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=924s) take past runs. I don't know why I can't

[15:28](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=928s) do this here and say okay I want to add

[15:30](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=930s) this to a benchmark suite and what this

[15:33](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=933s) is doing is say okay this is like a

[15:35](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=935s) representative thing that I did in my

[15:36](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=936s) factory that could be useful to measure

[15:40](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=940s) by testing it with different model

[15:41](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=941s) configurations and so think of like

[15:43](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=943s) building up a a suite of like front-end

[15:45](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=945s) tasks and the question that you're

[15:47](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=947s) trying to answer is like what's the best

[15:50](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=950s) model for me to use for front-end tasks

[15:51](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=951s) and like you might look on Twitter but

[15:53](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=953s) you might also just be like let

[15:55](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=955s) actually test. And so what you can do or

[15:59](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=959s) should try to set up for your own system

[16:02](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=962s) and this is built into warp factory. So

[16:04](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=964s) I'll show it here is basically curate

[16:07](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=967s) one of these benchmarks and what what it

[16:11](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=971s) does is it will replay your real past

[16:15](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=975s) coding agent tasks and then generate for

[16:18](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=978s) you what looks like a basically a cost

[16:21](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=981s) versus quality trade-off graph. And so

[16:24](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=984s) this is something you'll see when um you

[16:26](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=986s) know the public benchmarks come out, but

[16:28](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=988s) this is on your own data. And so in in

[16:31](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=991s) this case, for example, like we could

[16:33](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=993s) see GLM53 flash uh was 93% correct on

[16:38](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=998s) these tasks at a cost of $2. GPT56 soul

[16:42](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1002s) was like 95% correct, but it cost three

[16:45](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1005s) times as much. And so you can take this

[16:48](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1008s) information and then build a model

[16:49](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1009s) routing strategy that you have real

[16:52](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1012s) confidence is not just a generic one but

[16:56](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1016s) for your actual code your stack is

[16:59](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1019s) actually providing value. And so um this

[17:02](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1022s) is like I think a good example of like

[17:06](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1026s) closing the loop. And then just to go

[17:08](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1028s) back I'm going to finish up here in a

[17:10](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1030s) second.

[17:12](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1032s) The

[17:15](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1035s) the sort of like way that we think about

[17:17](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1037s) this is like at the bottom of your

[17:19](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1039s) stack, it's defined in code. It's think

[17:22](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1042s) of a data layer. What's the data going

[17:24](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1044s) in and out? Where do these agents run?

[17:26](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1046s) You really want to sandbox that. You

[17:29](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1049s) want inference that's flexible. You want

[17:32](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1052s) a self-improvement layer that gives you

[17:34](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1054s) something like benchmark scoring. You

[17:36](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1056s) want orchestration so you can have

[17:38](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1058s) multi-step workflows. And then you want

[17:40](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1060s) access from all these existing tools.

[17:42](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1062s) And so that's like the infrastructure

[17:44](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1064s) that I think is going to it's going to

[17:46](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1066s) look something like this. Um no matter

[17:49](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1069s) how you approach this problem uh if

[17:52](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1072s) you're interested in our approach

[17:53](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1073s) obviously we're we're we would love to

[17:55](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1075s) talk to you. But let me pause there.

[17:58](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1078s) Hopefully this was interesting to folks.

[17:59](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1079s) And I might have time for one question.

[18:02](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1082s) I don't know. Or I might have to go.

[18:06](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1086s) Yes. What what uh what advice do you

[18:08](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1088s) have for testing and verification of

[18:11](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1091s) code at the factory?

[18:12](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1092s) Awesome question. So the you know one

[18:15](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1095s) thing that could happen if you do this

[18:17](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1097s) approach is you get like a pile up of

[18:19](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1099s) PRs right and so you want some way of

[18:21](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1101s) like getting them through the big things

[18:23](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1103s) that I advise are spec driven

[18:25](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1105s) development. So you can actually test um

[18:28](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1108s) make sure that the changes are adherent

[18:30](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1110s) to a spec. And the other thing is

[18:32](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1112s) computer use verification which I didn't

[18:34](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1114s) show but actually um you know if you can

[18:37](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1117s) have agents that verify or produce

[18:40](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1120s) evidence from the user standpoint that

[18:42](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1122s) the thing works that can go a long way

[18:44](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1124s) in terms of like lessening the code

[18:46](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1126s) review burden. So those are the those

[18:47](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1127s) are the two things that I recommend.

[18:50](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1130s) Yes.

[18:50](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1130s) So how do you plan um the budgets when

[18:53](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1133s) we are starting a project? How do you

[18:57](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1137s) plan these verification costs?

[18:59](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1139s) Yeah. So the the way that we do this is

[19:02](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1142s) like we give people the tools to measure

[19:04](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1144s) like what's your cost per PR and then

[19:07](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1147s) what percent of the cost is going

[19:08](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1148s) towards like scoring or benchmarking or

[19:10](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1150s) stuff like that. Uh and then you can

[19:12](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1152s) kind of extrapolate it out like are you

[19:14](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1154s) willing to spend x amount on a project

[19:16](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1156s) or whatever. But the the key the key

[19:19](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1159s) thing um that that I want people to

[19:21](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1161s) leave if there's one idea is like really

[19:24](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1164s) set up something that you can measure

[19:25](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1165s) and improve. And once you can truly

[19:27](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1167s) measure in a closed loop system, you can

[19:30](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1170s) you can do all of these things in a way

[19:31](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1171s) that's just never been possible before.

[19:36](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1176s) Any other questions?

[19:38](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1178s) Question in the back.

[19:38](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1178s) Ycess.

[19:48](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1188s) Yeah. So, so the question was around uh

[19:51](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1191s) how do we do cost per PR? So this is

[19:53](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1193s) data curation

[19:54](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1194s) the the data curation

[19:57](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1197s) um are you so I just want to make sure

[19:59](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1199s) we're talking about the same thing. So

[20:00](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1200s) you're talking about like when I

[20:03](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1203s) for the the benchmarking or for like our

[20:06](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1206s) general

[20:10](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1210s) Okay. Yeah. So the way that we we do

[20:12](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1212s) this is like you can either build up a

[20:14](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1214s) benchmark set by hand which a lot of

[20:16](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1216s) people will do. They'll you know and

[20:18](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1218s) that's kind of what we recommend to

[20:19](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1219s) start. uh we will pick like for ours we

[20:22](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1222s) pick like a set of sort of small medium

[20:24](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1224s) hard tasks whatever you want uh to be

[20:27](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1227s) representative or you can actually ask

[20:30](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1230s) the factory to do it because the factory

[20:33](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1233s) is aware of all the past sessions and so

[20:35](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1235s) and everything is stored in code so it's

[20:37](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1237s) possible to have the factory itself look

[20:40](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1240s) at prior sessions and identify

[20:42](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1242s) interesting ones for a benchmark set

[20:44](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1244s) which is pretty cool

[20:47](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1247s) y

[20:47](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1247s) how is correctness evaluated on chart.

[20:50](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1250s) Yeah. So everything in this chart is

[20:52](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1252s) driven by LLM as a judge which we we may

[20:56](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1256s) I think another way of doing it is human

[20:58](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1258s) grading which makes total sense. We just

[20:59](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1259s) haven't added that yet. So I would

[21:01](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1261s) suggest that as well. But basically

[21:03](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1263s) anything that is defined uh in the

[21:06](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1266s) system as a scorer. So you define these

[21:09](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1269s) scores which basically grade how well

[21:11](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1271s) something is doing can be used as a

[21:13](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1273s) benchmark evaluation criteria.

[21:17](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1277s) Yeah.

[21:18](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1278s) this work platform is this open source

[21:21](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1281s) or is this only for enterprise?

[21:23](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1283s) So it is is not open source. It's it's

[21:27](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1287s) like very flexible and like like the

[21:30](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1290s) idea is to make it open so you can bring

[21:31](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1291s) any existing tools into it. It's not

[21:34](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1294s) only for enterprise like this is for

[21:36](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1296s) small teams and you can try it out on

[21:38](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1298s) your personal project. It is best for

[21:41](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1301s) teams though I will say because like if

[21:43](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1303s) you're um if you're an it's just

[21:45](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1305s) overkill for an individual like you're

[21:47](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1307s) probably gonna want to use like your

[21:48](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1308s) cloud code sub or something but once you

[21:50](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1310s) have a team of like maybe 10 or more

[21:51](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1311s) engineers having a system like this is

[21:53](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=1313s) very valuable.
