# Pick Problems That Survive When AI Solves Coding - Ara Khan, Cline

Ara Khan, Cline

[Talk page](https://ai.engineer/talks/iK9gmxXyBR0-pick-problems-that-survive-when-ai-solves) · [Video](https://www.youtube.com/watch?v=iK9gmxXyBR0)

## Pick Problems That Survive When AI Solves Coding

At Cline, Ara Khan gave a coding agent 17 hours to improve the harness around a second model. The failure traces and a slow-talking hardware project point to a way to choose engineering work that grows more useful as models improve.

![A staircase plots the 69/89 baseline, harness changes, and the 79/89 confirmation run.](https://ai.engineer/api/talk-essay-media/e5d3bedaa22c601e4c5f7b65cfbff915bda4fe413058526b3b971ff5f445a819)

Khan’s Terminal-Bench chart tracks changes to one model’s harness from the baseline to the confirmation run.

### What still counts as proof of work?

[0:17](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=17s)

An engineer used to be able to put a weekend project on GitHub and let the working software speak for itself. What does the same repository tell a hiring manager when a coding agent can build it in an afternoon? Ara Khan, a founding engineer at Cline, takes that question seriously. A project can be useful to its users and still say very little about the judgment it took to build.

He starts with a piece of his own work that a stronger coding model made more interesting. Running **Kimi K3** inside Cline gave his team 69 passes on 89 Terminal-Bench 2.1 tasks. The model developer had reported a higher score. Cline could read the failures and change the software around K3. Could another coding agent do the investigating and the editing, and leave the team with a harness worth keeping?

### Test the work the agent leaves behind

[5:00](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=300s)

The jump from a chatbot to a coding colleague required tools that could change things outside a conversation. A system that can list a directory, read files, edit them and run commands can try its own fix. Khan remembers building early multi-agent conversations in which much of the apparent collaboration came from the developer wiring messages between classes. Giving agents more people to talk to still left the developer with the hard question of whether any work had been done.

Take a request to set up an MCP server. An agent might have to find the documentation, edit a configuration file, install a dependency and start a process. A confident final message tells you little about whether the client can now use the server. An independent test can open the client and exercise the tool. The transcript then becomes a way to explain a pass or a failure.

Cline used the same separation at benchmark scale. Each of the 89 terminal tasks got an isolated container and a grader. The pass count gave the team a starting point. Tool calls, token use, elapsed time and command output let it distinguish a broken file reader from a working session that exhausted its time allowance. The next decision depended on which explanation recurred.

### Let a second agent read the failures

[11:05](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=665s)

Khan put **GPT-5.6 Sol** in charge of the outer loop. K3 attempted the tasks through Cline. Sol read the traces, grouped similar failures, proposed a harness change and ran the affected cases again. With a full evaluation taking roughly an hour, a useful lead agent had to remember which hypotheses it had tried and when a change had earned another full run.

The loop detector gave it a particularly good clue. A shell command could appear unchanged across several turns because the agent was polling a background job. The job’s output kept changing. Killing the session for repeating a command threw away evidence of progress that the harness already had. Checking the output as well let healthy tasks continue. A different group of failures came from a provider rate limit ending otherwise viable sessions. Those failures called for waiting and retrying.

The results slide puts both runs on the same denominator. The stock harness passed 69 of 89 tasks (77.5%). The confirmation run passed 79 of 89 (88.8%). In Khan’s published account, that final run cost $49.80. The full 17-hour experiment, including the repeated evaluations and the agent doing the editing, cost roughly $680. Those numbers answer two different questions an engineering lead would ask about doing this again.

The team kept the grader separate, ruled out task-specific shortcuts and reviewed the resulting pull request. Khan also cautions against adapting the harness to every mistake as it happens. A tool change that helps on several unrelated tasks has a better claim to belong in the product.

![The rollout agent acts on its environment while a separate meta-agent changes the harness and evaluates the result.](https://ai.engineer/api/talk-essay-media/529eb69ad03efcca98894f44f458481b0a1a6dad54766f1271fe997540891167)

The inner agent works on a task. A second agent uses the traces to change and evaluate the harness.

### Turn a correction into a test

[14:47](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=887s)

Khan widens the lens from the harness to the rest of the stack. Training-data selection, post-training, inference kernels and evaluation all offer different ways to make the next run better. Faster inference matters twice in a fixed-budget experiment. It improves the experience of the person waiting for an answer and may buy the team another round of trials.

His database example starts with an ordinary request. Add a column, populate it and put a cache in front. The agent completes the migration but chooses the wrong cache lifetime. A developer tells it which TTL to use, watches it fix the problem and moves on. That short exchange identifies a boundary of the system’s competence much more precisely than a thumbs-down on the whole task.

An engineer can turn that observation into a reproducible case with a check for what happens when the cached value expires. Then it becomes possible to compare a better tool, different instructions or a different model on the behavior that actually failed. A collection of such cases also gives the engineer somewhere to look for regressions after the next change.

![An attempt leads to human correction and then to a verified improvement.](https://ai.engineer/api/talk-essay-media/c488239ed13e26dda3208a4dc0f1f83a5fb4ea75a7cb55650d0a2ebb678ed478)

The slide separates a human correction from the test that verifies an improvement.

### A hardware loop needs a measurement

[18:50](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1130s)

In a personal project, Khan had a small device with a camera, speaker and microphone. It responded to a question about what it could see, but spoke back too slowly. He connected it to his computer and had a coding agent send requests to the device and time the replies. The device supplied something a second chatbot could not: an end-to-end measurement taken on the hardware that would actually be used.

The overnight experiments could change the firmware, the tool calls, the inference model or the speech-to-text component. Once the computer could put a repeatable question to the device, a change in any one component could be judged by what happened at the speaker. Khan reports the device got faster. The transferable part of the experiment is how he made latency observable across the whole path.

### Choose a problem that benefits from a better model

[21:53](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1313s)

Khan’s career advice follows from the difference between the weekend repository and the hardware on his desk. A new model release might build the old demo in one shot. That same release gives the engineer with the device a better candidate to test. The apparatus, the latency measurement and the knowledge of what a good interaction feels like all carry forward.

He suggests asking two questions before committing to a long project. If the next model gets much better, which part of the work becomes more valuable? Which part could disappear entirely? Data pipelines that expose real failures, hardware that can test a change and networks that make experiments cheaper all give concrete answers to the first question. Sometimes the honest answer to the second is that it makes sense to wait one release.

The time horizon is a judgment call, and Khan invites disagreement about it. His practical test works on a shorter clock. When an agent can already finish most of an engineer’s daily tasks, look for the correction that still requires that engineer. Can someone capture the failure, tell when it is fixed and give the next attempt better conditions? That is a piece of work a more capable model can help compound.

## Resources

- [Cline’s experiment log and cost breakdown](https://cline.bot/blog/recursive-self-improvement-for-coding-agents) — Ara Khan documents the five harness experiments, failed runs and final confirmation.
- [Terminal-Bench 2.1](https://www.tbench.ai/news/terminal-bench-2-1) — The benchmark authors describe the 89-task suite and its evaluation changes.

## Slides

### Recursively Self-Improvement for Coding Agents

[0:00](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=0s) [Ara Khan introduces recursive improvement for coding agents and identifies himself as a founding engineer at Cline.](https://assets.ai.engineer/sites/aiecode2025/sha256/f49ded0ac94902550c0bc3b0350b9a4d788909f6b956f0c1a59491b1da1d820b.webp)

Recursively Self-Improvement for Coding Agents
Ara Khan · Founding Engineer, Cline
AI Engineer Summit

### Understand the intent of frontier labs

[0:36](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=36s) [The first goal is understanding what frontier labs want to build.](https://assets.ai.engineer/sites/aiecode2025/sha256/0e4736e78342775bad90022260d4ec7ad32df6b3b524bf8a0b25d703ea76dad5.webp)

Goal 01 of 03
Understand the intent
of frontier labs

### Career goals for the next few years

[1:07](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=67s) [The second goal asks engineers to think about their careers as coding models improve.](https://assets.ai.engineer/sites/aiecode2025/sha256/212681b1079e10509b57dddfa59a2a06113b6915f9db8b76957eaca0fa197bca.webp)

Goal 02 of 03
Think about your career goals
over the next few years

### Where does the value go?

[1:50](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=110s) [The third goal is headed “Escape the permanent underclass.”](https://assets.ai.engineer/sites/aiecode2025/sha256/0bdc4873791c8e5d0291a86498ce06e3741cafd66726a6aa3a6888212165103e.webp)

Goal 03 of 03
Escape the
permanent underclass

### Kimi K3 and Cline on Terminal-Bench 2.1

[3:07](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=187s) [A staircase plots the 69/89 baseline, harness changes, and the 79/89 confirmation run.](https://assets.ai.engineer/sites/aiecode2025/sha256/e5d3bedaa22c601e4c5f7b65cfbff915bda4fe413058526b3b971ff5f445a819.webp)

Kimi K3 + Cline score on Terminal-Bench 2.1
Eval Score
77.5% · Baseline 69/89
Bounded provider retries
Progress-aware loop detection
File-index request liveness
Safe background-process guidance
86.5% · Frozen full stack 77/89
88.8% · Best run 79/89
Unchanged confirmation
Intermediate step heights are schematic — individual 89-task contributions were not isolated.
Iterative Cline harness improvements

### Self-improving coding agents

[4:56](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=296s) [The example is a coding agent that improves itself.](https://assets.ai.engineer/sites/aiecode2025/sha256/3b3ff478fd8453e15cc22ab78fb898a5f90ec7f58bfa99affdcd4093fc888d04.webp)

Example:
Self Improving Coding Agents

### Anthropic’s July 2021 roadmap

[5:32](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=332s) [An excerpt describes a conversational AI assistant with the capabilities of a software engineering colleague.](https://assets.ai.engineer/sites/aiecode2025/sha256/7f7115352aeb0f8c764c651512c7cf39b9899467d763b352cf6764cb4f84afd6.webp)

Anthropic’s roadmap in July 2021
Last Updated: July 2021
Long Term Goal
Our top level goal is to work towards the creation of safe, beneficial AGI.
As a proof of concept, we are working to develop a general AI assistant that can interact conversationally with a user to create, edit, and explain software. The most ambitious and successful version of this agent could answer general questions, write functions, identify bugs, design an API, discuss style and best practices, etc — it would display all the capabilities of a software engineer colleague. We don't expect to reach this goal right away, but it’s the direction in which we're striving.
Jared Kaplan, Concise Research Roadmap / Original PDF, page 2

### ChatGPT in 2022

[6:10](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=370s) [The slide dates the ChatGPT release to November 2022.](https://assets.ai.engineer/sites/aiecode2025/sha256/2150cbf2e8dece112d3db3391a57e14a0033066b1279ed5f7d8414bc965468c3.webp)

The 2022
- Chat GPT got released in Nov 2022

### Function calling in 2023–24

[6:19](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=379s) [A question asks what changes when a chat model can call functions.](https://assets.ai.engineer/sites/aiecode2025/sha256/05f79ba5976a9239029a81a46bd3f5e301d20881fd29b9a6c0407a0242acee3d.webp)

The 2023-2024 intuition
- What is chats could call functions

### Sonnet 3.5 and agent tools

[6:35](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=395s) [The next slide connects the 2024 Sonnet 3.5 release with agents calling functions.](https://assets.ai.engineer/sites/aiecode2025/sha256/448f93dccbc4ee04d9ecfa5ec9a77031d58d4f6870322006d7444d7b4da1cc87.webp)

2024 Sonnet 3.5 Release
What if agents could call functions

### Chatbots talking to each other

[6:55](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=415s) [The slide asks what would happen if chatbots could talk to each other.](https://assets.ai.engineer/sites/aiecode2025/sha256/43165ba114bf22de19bdfd4e8c6b2a61420b53ab34979b6e51625ef28b6683e8.webp)

The 2024 intuition
“What if the chatbots
talked to each other?”

### Multi-agent conversations

[7:02](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=422s) [An agent customization diagram shows joint and hierarchical conversation patterns and an example chat.](https://assets.ai.engineer/sites/aiecode2025/sha256/9b9b36767c8d5d477ebcef88435fb7bf5522b6d58c6460915db6df99f691e995.webp)

Conversable agent
Agent Customization
Multi-Agent Conversations
Flexible Conversation Patterns
Joint chat
Hierarchical chat
Example Agent Chat
Output:

### Faster and cheaper agents

[7:34](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=454s) [Khan asks what happens if agents keep improving in speed and cost.](https://assets.ai.engineer/sites/aiecode2025/sha256/f7544aa42cb610473a64c7aa47bb22b67ac2f70a87083f6d8dd74ec62b2f7918.webp)

The 2025 intuition
What is agents just kept getting better
and a faster and cheaper

### An agent improving itself

[7:46](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=466s) [The slide asks whether an agent can improve itself.](https://assets.ai.engineer/sites/aiecode2025/sha256/02529e69a1b5b4ecd9eb798e435381ae41c8c7dcef9f8c10a3eb8a5f4402456d.webp)

The 2026 Intuition
What if the agent improved itself?

### Single-turn and agent evaluations

[7:58](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=478s) [Two diagrams contrast grading a model response with verifying the changes an agent makes to an environment.](https://assets.ai.engineer/sites/aiecode2025/sha256/371326af90715e8f24599f8adb57e1011ca669de8d8affa96f7013ae2108a5d7.webp)

Comparison: Single-Turn vs Agent Evaluations
Single-turn
Prompt: How many toes does {ANIMAL} have?
Data: ANIMAL = Cat
LLM
Response
Grading logic: Correct if response == 18
Agent
Tools
Environment
Task: Set up an MCP server to connect to my app
Grading logic: Run a bunch of tests to make sure the MCP server is actually working

### Terminal-Bench

[9:15](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=555s) [The benchmark’s home page describes assessments for AI agents in terminal environments.](https://assets.ai.engineer/sites/aiecode2025/sha256/bfaf9569596d109ceb3312208b3d2c356e3c74979170979dbc03f567e2d92f02.webp)

terminal-bench: benchmarks for ai agents in terminal environments
terminal-bench is a collection of harbor-native benchmarks to help agent makers quantify their agents’ terminal mastery

### The evaluation harness

[9:47](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=587s) [A diagram separates tasks, graders, tracked metrics, traces and the agent harness.](https://assets.ai.engineer/sites/aiecode2025/sha256/76c38aaa2f9532f7101f8d8d564f51742fc797ad72c98f2168534259fbb48b59.webp)

Components of Evaluations for Agents
Evaluation harness
Evaluation suite
Task
Graders
Tracked metrics
Trials
Trace: trajectory, messages, tool calls, reasoning
Agent harness
Outcome
Grader evaluate

### Running 89 tasks in sandboxes

[10:18](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=618s) [The Harbor run diagram divides benchmark tasks among isolated sandboxes.](https://assets.ai.engineer/sites/aiecode2025/sha256/2fc8d305f64d2616462b773d8d2fe6ccce59359190342d5548e4b404edf5ab34.webp)

User’s local machine
Harbor Dataset Download
Pull terminal bench dataset locally
List of Datasets: SWE Bench, SWE Bench pro, Terminal Bench, MMLU, Aider Polyglot
Harbor Run
Splits 89 sandboxes for 89 tasks

### From a score to a harness change

[10:43](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=643s) [A five-step process groups failures and changes one thing between evaluation runs.](https://assets.ai.engineer/sites/aiecode2025/sha256/f33bf2e54a8dcfad27ac9d517c50b2ff8e8742e2f1defa22672573d56d2e5222.webp)

The Process: Get a Score, Then Improve
1. Run the eval: get an original score
2. Evaluate all failures: is there a common theme?
Example: a read_file tool blows up on massive files
Example: inference is so slow that individual runs time out
Example: the agent gets distracted and goes completely off the rails
3. Bucket your failures: 10 of this kind, 20 of that kind, 30 of another
4. Make one tiny change: re-run the whole eval — repeat
5. The agent iteratively figures out its flaws and improves

### Hill climbing on public evaluations

[11:42](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=702s) [The 2026 intuition slide introduces iterative improvement on public evaluations.](https://assets.ai.engineer/sites/aiecode2025/sha256/3a95e17e678ec435c7faedc221fa585198ed9cf204fa05d2ac56282f07aad16a.webp)

The 2026 Intuition
Hill Climbing on Public Evals

### One agent improves another’s harness

[11:58](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=718s) [GPT-5.6 Sol led the experiment while Kimi K3 ran the tasks.](https://assets.ai.engineer/sites/aiecode2025/sha256/f4f3a6b654dabbb1ee891b52c7d96eb227af7d5e160cb04f45a553741cc56fde.webp)

Our Cline experiment
One agent improved the harness that another agent used
GPT-5.6 Sol led the work. Kimi K3 ran the tasks.

### The same 89 tasks

[12:57](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=777s) [The scorecard compares a baseline of 69/89 (77.5%) with confirmation at 79/89 (88.8%).](https://assets.ai.engineer/sites/aiecode2025/sha256/4842955759a16b687152e49d93b202150b24f355591a5d783d380e451666ac84.webp)

The same 89 tasks
69/89 · Baseline · 77.5%
79/89 · Confirmation · 88.8%

### The inner and outer learning loops

[13:47](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=827s) [The rollout agent acts on its environment while a separate meta-agent changes the harness and evaluates the result.](https://assets.ai.engineer/sites/aiecode2025/sha256/529eb69ad03efcca98894f44f458481b0a1a6dad54766f1271fe997540891167.webp)

Experience becomes a better harness.
A two-level learning loop, from individual tasks to reusable improvements.
INNER LOOP: Interact within each task
Rollout Agent · Harness · Model · Environment · feedback + reward
OUTER LOOP: Improve across tasks
Meta Agent · Diagnose & propose
Editable harness · Prompts · Tools · Skills · Context management
Seed tasks · Model
Evaluate & promote

### Model-assisted inference optimization

[14:49](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=889s) [OpenAI-reported figures on the slide distinguish 20% lower serving cost from more than 15% higher token-generation efficiency.](https://assets.ai.engineer/sites/aiecode2025/sha256/791c7f1f4da1a7c38319ac838bca4aa13d588952b1719d52e0abcf14039e4714.webp)

A model can improve its own inference
20% lower serving cost
>15% higher token-generation efficiency
OpenAI-reported results across distinct optimizations

### Rewriting production kernels

[14:59](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=899s) [A highlighted excerpt discusses kernel writing, kernel verification and end-to-end serving costs.](https://assets.ai.engineer/sites/aiecode2025/sha256/3d7826960c3f4f29c2136d820fe1d8461a98afa57d08d4cab3f12e7113d31bc7.webp)

A model can improve its own inference
With Codex, GPT-5.6 Sol autonomously rewrote and optimized our production kernels, the core code that executes the mathematical operations that make up the model. This worked in part because we’ve trained GPT-5.6 to be effective at writing and improving kernels in Triton and Gluon, two open-source GPU programming languages maintained by OpenAI. These efforts, combined with broader kernel advancements from GPT-5.6 Sol, reduced end-to-end serving costs by 20%. We’ve also heavily invested in verification tooling, such as the open-source tool FpSan (Floating-Point Sanitizer), to help validate the correctness of the kernels written by GPT-5.6 Sol.

### An agent-assisted GPU competition entry

[15:40](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=940s) [A quoted competitor describes writing GPU operators with language models and agents.](https://assets.ai.engineer/sites/aiecode2025/sha256/08a319dbf5bed19c7b054fbb769c7c48e4690593cb57e4f64ca3ffb080739b7d.webp)

A model can improve its own inference
Mark Saroufim
“I just learnt that one of the top submitters in the nvfp4 competition has never hand written GPU code before. I did not expect this.”
“That's me. Purely AI so far.”

### An improvement surface across the stack

[16:19](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=979s) [The slide names evaluation, harnesses, inference, training data, post-training and interaction with the world.](https://assets.ai.engineer/sites/aiecode2025/sha256/dcb2c26ff5e654ac069ac979fb99f4dc7e5315270c418399a176e427a5c6f6ba.webp)

The improvement surface is the whole stack
Evaluation · Training data
Harness · Post-training
Inference · World interaction

### The whole recursive improvement loop

[16:51](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1011s) [A five-column diagram links pre-training, post-training, inference, eval generation and evals.](https://assets.ai.engineer/sites/aiecode2025/sha256/d9d39c481d7c9aa04cd3c9ef6822480ba4593e5033fc56d22dff9796d85a5327.webp)

The whole RSI loop
AI can improve the machinery that builds and tests the next AI.
Pre-training: Nemotron-CC — AI-curated + rephrased data
Post-training: Constitutional AI — Self-critiques + AI preferences
Inference: GPT-5.6 Sol — Improves its own speculative decode
Eval generation: Bloom — Behavior description → new test suite
Evals: Self-Taught Evaluators — Synthetic preferences improve the judge
Validated gains feed better data, training, inference and tests into the next cycle.

### Human corrections as a training signal

[17:21](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1041s) [An attempt leads to human correction and then to a verified improvement.](https://assets.ai.engineer/sites/aiecode2025/sha256/c488239ed13e26dda3208a4dc0f1f83a5fb4ea75a7cb55650d0a2ebb678ed478.webp)

Human corrections become training signal
Attempt
Human correction
Verified improvement
A useful delta preserves what changed and why.

### Post-training closes another loop

[18:17](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1097s) [Experience, verified data, better experiments and a new checkpoint make up the proposed feedback cycle.](https://assets.ai.engineer/sites/aiecode2025/sha256/213392f6d7c05b32b51b8ff74492ae55224cd22bfaee0bee48c417f76b49b9b4.webp)

Post-training closes another loop
Experience · Verified data
Better experiments · New checkpoint
A proposed feedback loop, tested at each iteration

### Testing in the physical world

[18:53](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1133s) [A robot workspace illustrates a cycle of reset, act, verify and improve.](https://assets.ai.engineer/sites/aiecode2025/sha256/f23e4832046553daa3cce87afcdf6d6d2092efd5a2b80f53eb97b3424ffdcbe2.webp)

The physical world adds a harder test
Reset
Act
Verify
Improve

### Connecting the loops

[20:26](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1226s) [Cheaper inference buys more experiments per budget and creates opportunities for better data and models.](https://assets.ai.engineer/sites/aiecode2025/sha256/3ab8dae4fc649b8b2e1325d02ad5061930d48af758f7d9466ecd10306f078da5.webp)

What happens when the loops connect?
Cheaper inference
More experiments per budget
Better data + models

### Which parts already work?

[21:38](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1298s) [Khan asks whether this is the singularity and observes that parts of the loop already exist.](https://assets.ai.engineer/sites/aiecode2025/sha256/ff5c198dc72a678b2e430f7c469e63959440985b48a503e642a2ce9be031c9a3.webp)

Is this the singularity?
Parts of the loop are already real

### Move closer to the improvement loop

[21:53](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1313s) [Khan advises the audience to build where progress is verifiable, find bottlenecks and enable the next improvement.](https://assets.ai.engineer/sites/aiecode2025/sha256/14ee61211d8df8b7bb01b7daa5f7fdb565ec7dc70e300fd9da0683a1406e6505.webp)

Move closer to the RSI loop
Build where progress is verifiable.
Find the bottleneck.
Make the next improvement possible.
A career compass: pipelines, models, and measurable improvement.

### Choosing a project with a short time horizon

[23:58](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1438s) [The closing advice weighs ambitious collaborations, short project timescales and work that benefits from model scaling.](https://assets.ai.engineer/sites/aiecode2025/sha256/2025e0b956182e47316aff9b413c70a264eeba3b8f969469f1baec49bcc1f94c.webp)

I have 1–5 years to contribute intellectually to the world. What should I do?
Prefer ambitious, targeted, fast collaborations.
The timescale of the project needs to be shorter than the characteristic timescale of the exponential.
Otherwise, it's better to wait, and use a newer foundation model to do it better-faster.
This does not favor slow, independent, meandering exploration.
The bitter lesson [Sutton, 2019]
General methods that leverage computation are ultimately the most effective.
Work on projects that will benefit from scale.
Avoid projects that will be solved by scale alone, before you even finish the project.
2 years hard work

## Full transcript

[0:17](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=17s) Uh, hey guys, this is Ara. I'm a

[0:20](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=20s) founding engineer at Cline and today I'm

[0:22](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=22s) going to be talking about recursively

[0:24](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=24s) self-improving coding agents. So, this

[0:26](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=26s) is a very interesting conversation in

[0:30](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=30s) the sense that everyone wants to

[0:32](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=32s) understand what RSI is, how we

[0:34](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=34s) accomplish it, how we go through it, and

[0:35](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=35s) how it affects our career. So I'm going

[0:37](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=37s) to start with giving you just three

[0:38](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=38s) goals of this entire conversation. Just

[0:40](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=40s) three simple goals and that would help

[0:43](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=43s) you understand like what what we get out

[0:45](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=45s) of it and what the future uh for this

[0:47](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=47s) looks like. So the goal number first is

[0:49](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=49s) that understanding the intent of

[0:50](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=50s) frontier labs right everyone who's

[0:52](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=52s) outside of the frontier labs is always

[0:53](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=53s) trying to speculate like hey what what

[0:55](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=55s) is the topic in open or what is deep

[0:57](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=57s) mind going to come out with or what what

[0:58](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=58s) are they trying to make and what's the

[1:00](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=60s) future going to look like who has

[1:02](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=62s) cracked the RSI. So like this kind of

[1:04](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=64s) helps you get get a rough outline of

[1:06](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=66s) what their road map looks like. And the

[1:08](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=68s) second thing is that like if the models

[1:10](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=70s) are improving so fast like how do you

[1:12](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=72s) think about your career or the next few

[1:14](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=74s) years right that is really what we want

[1:16](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=76s) to understand. And we want to understand

[1:17](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=77s) like okay things are changing so fast

[1:20](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=80s) like the ground underneath us is

[1:22](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=82s) shifting so quickly right now like it's

[1:24](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=84s) like uh

[1:26](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=86s) there used to be a time when you could

[1:28](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=88s) like demonstrate your side projects on

[1:31](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=91s) your personal website or GitHub and now

[1:33](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=93s) it's just like you could literally

[1:34](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=94s) oneshot them using any Frontier coding

[1:37](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=97s) engine. And with that it's like it's

[1:39](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=99s) like oh how how do we demonstrate proof

[1:41](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=101s) of work? How do you demonstrate, you

[1:43](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=103s) know, that that that we know what we're

[1:45](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=105s) doing and how do we think of our career

[1:47](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=107s) for the next coming years, right? So,

[1:49](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=109s) it's a very interesting times that we're

[1:51](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=111s) living in. And then the third and the

[1:52](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=112s) most important goal of the conversation

[1:54](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=114s) is escaping the permanent underclass,

[1:56](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=116s) right? So, I know this seems like a

[1:58](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=118s) joke, but this really is a very valid

[2:01](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=121s) possibility that people are

[2:02](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=122s) contemplating. Everyone's like, "Oh,

[2:04](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=124s) well, if everything that we want to be

[2:06](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=126s) doing can potentially easily be done by

[2:08](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=128s) a frontier model, like what are we what

[2:09](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=129s) are we good for?" And like where does

[2:11](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=131s) software go? Where where where does the

[2:13](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=133s) future of technology going to go? And

[2:14](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=134s) what are we going to do as as things

[2:16](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=136s) evolve so fast, right? So I think that

[2:19](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=139s) there's like a lot of like

[2:20](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=140s) fear-mongering but also there's like

[2:22](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=142s) some very valid merge where people are

[2:25](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=145s) concerned that like if all the revenue

[2:27](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=147s) of the planet is going to go to um a few

[2:30](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=150s) hardware companies that make frontier

[2:31](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=151s) GPUs and then a few few uh AI companies

[2:35](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=155s) that that make the best models like how

[2:38](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=158s) are we how are the rest of us going to

[2:39](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=159s) thrive and what what are we going to do

[2:41](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=161s) here? So, I think I'll I'll walk you

[2:43](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=163s) through a few examples and a few ideas

[2:45](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=165s) that will like really really help you

[2:47](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=167s) reconsider the career choices and and

[2:49](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=169s) and the life choices you're making and

[2:51](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=171s) what your problems you're picking up and

[2:53](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=173s) then and then that will help you like

[2:55](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=175s) realign

[2:56](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=176s) what it is that we should be working on

[2:58](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=178s) and what it is that you you know would

[3:00](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=180s) would that you could work on that will

[3:02](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=182s) stand the test of time and help you both

[3:05](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=185s) just not just like on a psychological

[3:06](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=186s) but like financial and other levels as

[3:08](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=188s) well. So uh to give you a very precise

[3:10](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=190s) example, let's start with something very

[3:11](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=191s) precise that I had done. So a couple

[3:14](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=194s) months ago I uh came up with recursive

[3:18](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=198s) shampoo improvements on a coding

[3:21](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=201s) harness. So this was the first public

[3:23](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=203s) example of uh an automatic eval

[3:26](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=206s) improvement. So to give you an example

[3:28](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=208s) um we used to do this thing where we

[3:30](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=210s) would measure the quality of a coding

[3:32](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=212s) agent using evaluation scores. So on

[3:35](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=215s) Terminal-Bench 2.1 Kimi K3 with client

[3:38](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=218s) would score um usually like you know um

[3:42](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=222s) 60 70% and then depending on like how

[3:45](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=225s) good the model is like you get better

[3:46](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=226s) scores on on Terminal-Bench 2.1. So we

[3:49](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=229s) noticed that when Kimi K3 came out, our

[3:51](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=231s) score was 77.5%. Which is a good score,

[3:53](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=233s) but we were like, "Oh, is this is this

[3:55](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=235s) truly is it truly the best score that

[3:57](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=237s) this model can get on this benchmark?

[4:00](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=240s) And also like what does the model

[4:02](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=242s) release paper claim about what their

[4:04](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=244s) best score is?" So the model's release

[4:05](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=245s) paper said that they could score up to

[4:07](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=247s) 85% or even more. So we were just like,

[4:09](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=249s) okay, okay, so we are not getting the

[4:11](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=251s) sort of score. And often what we do with

[4:14](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=254s) model releases in client is that we try

[4:15](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=255s) to like optimize your score and try to

[4:17](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=257s) get the best out of the model. So when

[4:20](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=260s) we started hill climbing on the on the

[4:22](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=262s) model like we we realized this we

[4:25](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=265s) realized that like you know there's a

[4:27](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=267s) lot of steps that we do says that we do

[4:28](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=268s) it gracefully says that we pursue the

[4:30](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=270s) right kind of problems and do it the

[4:31](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=271s) right way. And we started thinking like

[4:35](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=275s) okay if we there's a lot of ground work

[4:37](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=277s) here how do we automate it and this is a

[4:39](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=279s) demonstration of like how we automated

[4:40](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=280s) it. So I'll I'll just leave that as a

[4:42](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=282s) placeholder for now and we'll get get

[4:43](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=283s) back to it later where we talk about in

[4:45](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=285s) detail of like how does like this

[4:47](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=287s) automatic improvement loop of a client

[4:49](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=289s) harness work. So this basically means

[4:52](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=292s) that like you have a coding agent

[4:53](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=293s) harness that could just like basically

[4:54](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=294s) improve itself and become better on on

[4:56](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=296s) the same set of problems. So let's start

[4:59](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=299s) with the ground terms. What is recursive

[5:00](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=300s) self improvement? Recursive

[5:01](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=301s) self-improvement is the idea that an AI

[5:03](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=303s) system that helps [snorts] build a more

[5:06](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=306s) capable successor which repeats the

[5:08](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=308s) process you know until you hit

[5:10](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=310s) artificial general intelligence right

[5:11](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=311s) you have better systems you have better

[5:12](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=312s) improvements you have a better system

[5:14](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=314s) and then you know you have singularity

[5:16](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=316s) so a classic example is like a

[5:18](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=318s) self-reing coding agent right

[5:19](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=319s) self-reving coding agent is just like a

[5:21](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=321s) coding agent that can improve itself so

[5:23](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=323s) you know this this is all like a pie in

[5:25](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=325s) the sky idea like it's like it's like

[5:27](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=327s) you know yeah sure this looks cool but

[5:28](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=328s) how how did this even come out to be

[5:30](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=330s) like how did we even get to here? So if

[5:33](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=333s) you notice in topics road map in 2021 in

[5:36](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=336s) 2021's road map they very specifically

[5:38](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=338s) said that uh uh they wanted to develop a

[5:42](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=342s) general AI assistant that can interact

[5:43](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=343s) conversationally with a user to create

[5:46](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=346s) edit and explain software. It would

[5:48](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=348s) display all the capabilities of a

[5:49](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=349s) software engineer colleague. It would

[5:50](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=350s) write functions. It would identify bugs.

[5:52](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=352s) It would design an API discuss style and

[5:54](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=354s) best practices. Right? So we don't

[5:56](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=356s) expect to reach this goal right away but

[5:58](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=358s) it's in the direction which we're

[5:59](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=359s) striving. It's interesting how this was

[6:01](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=361s) more than five years ago. More than five

[6:03](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=363s) years ago, people had predicted that

[6:05](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=365s) there exist a day where you will have

[6:07](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=367s) like a full-blown journal assistant that

[6:09](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=369s) can basically act like a colleague,

[6:10](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=370s) which we do have right now. Um, then in

[6:13](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=373s) 2022, Chad Deputy got released. We all

[6:15](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=375s) know how that happened. It was it was a

[6:17](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=377s) fun moment. It was like the first time

[6:19](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=379s) things got sparked for us. Um, then in

[6:22](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=382s) 2023, 2024, there was the first time

[6:24](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=384s) chat models could call functions. These

[6:27](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=387s) functions would be simple functions like

[6:29](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=389s) hey what's the weather or you know

[6:31](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=391s) what's the what's you know what's this

[6:33](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=393s) computation or what this computations

[6:35](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=395s) result in python um and then finally in

[6:38](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=398s) 2024 when s 345 was really easy that was

[6:40](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=400s) when the agents could like truly call

[6:42](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=402s) like actual tools the way we call like

[6:45](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=405s) think of tools so it's like reading a

[6:47](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=407s) file editing a file you know reading

[6:49](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=409s) searching something on web uh listing

[6:51](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=411s) listing a folder searching and grapping

[6:53](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=413s) through you know uh a file system things

[6:55](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=415s) like that. So, and in 2024, we had the

[6:59](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=419s) intuition that like chatbot, what if

[7:01](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=421s) chatbots could actually talk to each

[7:02](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=422s) other, you know? So, then that's was the

[7:04](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=424s) first time when we had like, you know,

[7:06](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=426s) uh multi-agent conversations. Now, at

[7:08](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=428s) the time, these things were very janky,

[7:10](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=430s) right? like they were just like taped

[7:11](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=431s) together in this like weird weird uh

[7:14](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=434s) bizarre ways where you have just like

[7:16](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=436s) these classes and these links and then

[7:18](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=438s) like these hierarchies of classes and it

[7:20](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=440s) was like it was all like kind of like a

[7:22](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=442s) very janky attempt to like make us think

[7:24](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=444s) that like these these models could like

[7:26](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=446s) have these pipeline of flows of

[7:27](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=447s) responses that can all communicate with

[7:29](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=449s) each other but a [snorts] lot of it was

[7:31](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=451s) like mostly deterministic and it was

[7:33](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=453s) like you know it was not as free flowing

[7:34](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=454s) as it is right now. So, and you know, we

[7:38](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=458s) started getting the intuition. So, this

[7:40](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=460s) was the autogen project, but like in

[7:41](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=461s) only in 2025, we we [snorts] started

[7:43](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=463s) getting the like what if the agents just

[7:45](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=465s) kept getting better and faster and

[7:46](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=466s) cheaper, right? What if that just kept

[7:48](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=468s) happening? And then in 2026, uh we we

[7:51](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=471s) had the intuition that like what if what

[7:54](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=474s) if the agent could improve itself, which

[7:56](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=476s) is like the final, you know, the final

[7:58](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=478s) part of the whole process. So, let me

[8:00](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=480s) give an example like what does it mean

[8:01](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=481s) for an agent to improve itself, right?

[8:03](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=483s) So like it's like a lot of times when we

[8:05](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=485s) think of like evaluating an agent, we

[8:07](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=487s) initially used to think of evaluating an

[8:09](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=489s) agent in the form of like just a single

[8:11](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=491s) turn conversation. So we would tell we

[8:13](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=493s) would tell like um um agents that like

[8:16](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=496s) ask a simple question like how many toes

[8:19](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=499s) does a dash animal has like it could be

[8:21](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=501s) let's say how many toes does a cat have

[8:23](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=503s) and then al would go through and the

[8:24](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=504s) element would like prepare response and

[8:26](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=506s) then we would get the response. So like

[8:27](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=507s) a cat has maybe 10 toes or 18 toes,

[8:30](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=510s) however many. I don't I actually don't

[8:31](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=511s) know how many toes the cat has. But

[8:33](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=513s) anyway, um yeah. So then we create the

[8:37](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=517s) response as like a yes or no. But like

[8:38](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=518s) when you work with an agent, there's a

[8:40](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=520s) ton of things that you're doing. So let

[8:42](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=522s) me give an example. Let me give an

[8:43](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=523s) example. So for example, you're going to

[8:45](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=525s) ask an agent like hey set up this MCP

[8:47](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=527s) for me. So now it's not just like it's

[8:49](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=529s) not just like you know a trivial oneable

[8:52](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=532s) like thing like it will like search

[8:53](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=533s) online and go oh what's the MCP? what is

[8:56](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=536s) the config? So, I'll like edit and

[8:58](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=538s) download some soft files. Like maybe I

[9:00](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=540s) need to add database update the MCB

[9:02](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=542s) tool. Maybe I need to read and edit some

[9:04](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=544s) files. Maybe I need to run some bash

[9:06](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=546s) commands. Right? So, you do all of that

[9:08](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=548s) and then the agent response like, hey, I

[9:09](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=549s) did it. And then you grab that with a

[9:11](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=551s) bunch of tests to gauge like, hey, did

[9:13](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=553s) the agent actually accomplish the thing

[9:15](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=555s) that it said it would do. So to do that

[9:18](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=558s) thing the second part which is like more

[9:20](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=560s) intricate where you're actually like

[9:22](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=562s) truly you know testing agents like uh to

[9:25](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=565s) their best extent like it's like you

[9:27](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=567s) know um we use this thing called

[9:29](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=569s) Terminal-Bench Terminal-Bench is a

[9:31](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=571s) benchmark for you know AI agents in

[9:33](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=573s) terminal environments and that is

[9:34](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=574s) something that we we have a lot of like

[9:37](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=577s) experience with at client and uh it's

[9:39](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=579s) like a it's like one of the best ways to

[9:41](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=581s) like really gauge like is your coding

[9:42](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=582s) agent accomplishing uh in general the

[9:45](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=585s) task that you're doing uh that you want

[9:47](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=587s) to be doing with a hudded model. So

[9:49](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=589s) there's a couple there's a many

[9:50](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=590s) different parts of a validation suit. So

[9:53](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=593s) ideally you want that to run in um

[9:56](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=596s) [snorts] you know you you want that to

[9:58](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=598s) run in um individual sandboxes. Then you

[10:01](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=601s) have like independent graders that like

[10:03](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=603s) check everything like how many turns it

[10:05](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=605s) took, how many tokens it took, how much

[10:06](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=606s) latency it took, how many how many how

[10:08](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=608s) many tool calls did it make, what was

[10:10](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=610s) the trajectory looking like and then

[10:12](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=612s) grading all the results for things like

[10:13](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=613s) hey did you not reward hack? Did you not

[10:15](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=615s) cheat through the solution? do not just

[10:16](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=616s) like look up the solution online things

[10:18](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=618s) like that. Um so we we basically like a

[10:22](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=622s) lot of these uh data sets of evaluation

[10:24](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=624s) problems have a set of tasks. So when

[10:27](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=627s) here's a Terminal-Bench 2.1 they have 89

[10:29](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=629s) tasks. So what we do is like we spin up

[10:31](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=631s) 89 parallel containers where they all

[10:34](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=634s) run online and they run in parallel and

[10:36](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=636s) they're sandbox and then they each try

[10:39](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=639s) to solve a problem. So then they're all

[10:41](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=641s) uh in parallel and independent of each

[10:43](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=643s) other and then at the end we get get a

[10:45](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=645s) score. So our score could like give us

[10:47](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=647s) measures like is is a is this is the

[10:50](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=650s) model performing well and what kind of

[10:52](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=652s) failures it has right? So like you run

[10:54](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=654s) the evaluation you get a score. So you

[10:56](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=656s) get an aggregate score and there

[10:58](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=658s) aggregate score could reveal a set of

[11:00](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=660s) failures. Okay failures could be of any

[11:01](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=661s) kind. Failures could be hey the read

[11:03](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=663s) file tool is not working. Sometimes the

[11:05](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=665s) entrance is so slow that like you know

[11:07](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=667s) the agent's working it's doing it's

[11:09](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=669s) making progress maybe it's running a

[11:10](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=670s) circle but it's so slow that it

[11:12](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=672s) literally like runs out of time it runs

[11:13](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=673s) out of like 45 minutes or where many

[11:15](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=675s) that were given for the task and just

[11:17](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=677s) like stop working. So once you do that

[11:19](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=679s) once you understand hey this is what we

[11:21](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=681s) ran this is what the failures we got we

[11:23](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=683s) bucket the failures in different

[11:24](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=684s) categories. So like 10 failures of you

[11:26](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=686s) know this kind of tool called 10 10 20

[11:28](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=688s) failures of like say inference not

[11:30](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=690s) working or something else right and then

[11:31](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=691s) we once we bucket the failures into

[11:33](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=693s) different categories where like we try

[11:34](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=694s) to make an improvement to the harness

[11:36](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=696s) because like what if there's like

[11:38](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=698s) something wrong with the harness that we

[11:39](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=699s) could change such that the model score

[11:41](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=701s) would improve right and on and on and on

[11:43](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=703s) and you do this iteratively it takes a

[11:44](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=704s) while it's just like you know hill

[11:46](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=706s) climbing is like a very groundworky kind

[11:48](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=708s) of process and you know usually usually

[11:51](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=711s) we we took we used to take a lot of time

[11:53](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=713s) trying to do this but it was a great way

[11:55](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=715s) for us to like learn how to improve

[11:56](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=716s) models. So [snorts]

[11:58](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=718s) one way to hill climb on public eval is

[12:01](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=721s) that like you know we we could have a

[12:04](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=724s) thing where you know we started doing

[12:06](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=726s) this thing where we could just like what

[12:09](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=729s) if we could have like a longunning agent

[12:11](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=731s) that runs for a while and automatically

[12:13](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=733s) improves the model score and that's what

[12:14](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=734s) we did. So we ran GPD 5.6 soul that led

[12:18](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=738s) the work and that was like the main

[12:19](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=739s) agent and then uh Ky K3 was just running

[12:22](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=742s) eval. So what it was doing is that like

[12:24](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=744s) GBD 5.6 was like hey run these events

[12:27](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=747s) get these scores let me look at the

[12:29](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=749s) failures let me look at where where it

[12:31](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=751s) went wrong and then I'll make changes to

[12:33](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=753s) client's harness and then I'll run these

[12:35](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=755s) eval again and then I'll keep trying. So

[12:37](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=757s) again, you can see how like very quickly

[12:39](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=759s) this can like go off the rails because

[12:41](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=761s) this is supposed to be like the running

[12:42](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=762s) and evaluation itself takes like an

[12:44](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=764s) hour. Then you have to read the

[12:45](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=765s) feedback. Then you have to like see if

[12:47](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=767s) you're properly interpreting the

[12:48](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=768s) feedback to make reasonable changes to

[12:50](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=770s) the harness that are like not reward

[12:52](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=772s) hacking that are just like actual

[12:53](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=773s) genuine improvements and then run the

[12:54](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=774s) eval again. So we ran that for 17 hours

[12:57](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=777s) and uh when we ran that for 17 hours

[13:00](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=780s) there was like a legitimate hell

[13:01](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=781s) climbing and what we found that was very

[13:03](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=783s) interesting. This happened with a very

[13:05](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=785s) detailed prompt that accomplished this

[13:07](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=787s) is that like we had baseline score of

[13:10](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=790s) 27% and then you know we we found issues

[13:14](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=794s) like loop detection. Loop detection is

[13:16](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=796s) when a harness just runs in circles

[13:17](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=797s) around a problem right uh we found like

[13:20](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=800s) background process grinding where

[13:21](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=801s) background processes would run too long.

[13:23](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=803s) Uh we found some file index issues and

[13:25](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=805s) other things as well. I mean when I say

[13:27](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=807s) we what I actually meant is GP 5.6 solve

[13:30](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=810s) right? So that found those issues that

[13:32](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=812s) found and bucketed those issues. It's

[13:34](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=814s) like hey how about I make those changes

[13:36](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=816s) in the harness and I run the eval again

[13:38](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=818s) and it iteratively built the hill

[13:39](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=819s) climbing process completely autonomously

[13:41](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=821s) on its own without my interference and

[13:43](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=823s) 13 hours later when I come back to it it

[13:45](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=825s) like it has improved the harness's

[13:47](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=827s) performance in so many ways. So see what

[13:50](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=830s) does that mean? What does that mean? It

[13:51](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=831s) just it just basically means that like

[13:54](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=834s) we're at a point where the experience

[13:56](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=836s) could make the harness better. That like

[13:58](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=838s) there's like a two-level learning loop

[14:00](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=840s) where the harness does some action,

[14:02](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=842s) interacts with the environment,

[14:04](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=844s) understands whether or not it did the

[14:05](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=845s) right thing or the wrong thing and gets

[14:07](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=847s) some reward and and like almost and then

[14:10](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=850s) has there's like a meta agent that's

[14:12](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=852s) observing the main harness and finding

[14:14](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=854s) the changes to it and just like

[14:16](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=856s) incorporating those changes and

[14:17](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=857s) improving. Now it's I think in my

[14:19](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=859s) personal opinion it's much harder to do

[14:21](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=861s) in real time because if you do it in

[14:23](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=863s) real time you overindex on solving

[14:25](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=865s) certain problems or you're just like too

[14:28](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=868s) cartel to a specific thing. But I do

[14:30](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=870s) think that this is something that works

[14:32](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=872s) like on a broader scale of like working

[14:33](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=873s) on a wide range of problems

[14:36](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=876s) and you want to be able to like you want

[14:38](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=878s) you want to be able to justify the

[14:39](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=879s) changes you're making on a broader act

[14:42](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=882s) applicability so that the changes you

[14:44](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=884s) make here will apply to a broader range

[14:46](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=886s) of problems as well. So what does that

[14:49](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=889s) mean? Like what am I trying to get at

[14:52](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=892s) here? The point I'm trying to get at is

[14:54](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=894s) that the models are so good right now

[14:55](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=895s) that they can if they can improve the

[14:57](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=897s) harness, they can improve a lot of other

[14:58](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=898s) things. So then what else can they

[15:00](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=900s) improve? So this is this is a this is a

[15:02](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=902s) post uh primarily from OpenI where GP2

[15:05](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=905s) 5.6 autonomously rewrote and optimized

[15:08](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=908s) its own production kernels and those

[15:10](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=910s) production kernel improvements remove

[15:13](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=913s) the end to-end serving costs by 20%. you

[15:16](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=916s) know it also had 15% higher to token

[15:19](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=919s) generation efficiency and this was like

[15:20](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=920s) completely done by 5.6 soul on itself,

[15:24](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=924s) right? So what that means is that like

[15:26](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=926s) inference is becoming much cheaper and

[15:28](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=928s) that means that like if on if if for a

[15:30](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=930s) company like OpenAI if you're having 20%

[15:33](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=933s) low lower serving cost for a frontier

[15:35](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=935s) model that could be a billions of

[15:37](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=937s) dollars of improvement in price and you

[15:39](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=939s) could you could do that as well for your

[15:40](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=940s) own models. Um another interesting thing

[15:44](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=944s) is that like when people have kernel

[15:45](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=945s) writing uh competitions so core auto is

[15:47](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=947s) a company which has a kernel writing

[15:49](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=949s) competitions and you'll notice that like

[15:52](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=952s) they they mentioned that like the person

[15:54](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=954s) who won the kernel writing competition

[15:56](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=956s) the explic said that I have never

[15:58](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=958s) handwritten a GPU operator it's all ALM

[16:01](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=961s) and agent driven so notice that like how

[16:03](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=963s) like the best uh inference improvements

[16:06](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=966s) kernel improvements are at this point

[16:08](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=968s) completely exclusively done by a swarm

[16:10](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=970s) of agents or generally like agents doing

[16:12](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=972s) some magic to you know magically improve

[16:15](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=975s) like the kernel kernel code and have

[16:17](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=977s) faster inference cheaper inference. Um

[16:20](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=980s) yeah so basically like the improvement

[16:23](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=983s) surface that you think of is not just

[16:25](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=985s) like the harness improvement it's

[16:27](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=987s) literally the whole surface stack right

[16:28](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=988s) so the whole stack from like evaluation

[16:31](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=991s) to harness improvements to training data

[16:33](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=993s) to post training to world interaction to

[16:35](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=995s) inference like literally the entire pip

[16:38](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=998s) part of the pipeline like all the way

[16:40](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1000s) from like electricity going through your

[16:43](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1003s) machine to like making a network request

[16:45](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1005s) going through the model models doing

[16:47](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1007s) inference um making a response and

[16:49](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1009s) getting back to you like every single

[16:51](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1011s) part of the pipeline we can optimize it.

[16:53](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1013s) So the whole RSI loop has like um

[16:55](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1015s) there's a couple different parts so it's

[16:56](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1016s) like pre-training um uh neatronic

[16:59](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1019s) examples of like you know AI curated uh

[17:01](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1021s) pre-training data that was good um

[17:03](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1023s) there's like post- training examples

[17:04](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1024s) which are you know where there's like

[17:07](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1027s) constitutional AI examples of like post-

[17:10](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1030s) training uh improvements then inference

[17:12](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1032s) I gave you the example of GP 5.6 soul

[17:14](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1034s) for eval and eval generation. I give you

[17:16](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1036s) our own example of how we improved

[17:18](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1038s) ourselves on our own eval. Um in general

[17:21](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1041s) what you want to think of is that like

[17:23](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1043s) anytime a model is trying to do

[17:24](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1044s) something and it's like not able to do

[17:26](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1046s) it or not able to accomplish it. So for

[17:28](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1048s) example like imagine you're having a

[17:30](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1050s) long conversation where you ask the

[17:31](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1051s) model something obvious. So like it's

[17:33](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1053s) like or something that you think is not

[17:36](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1056s) necessarily obvious but it's like it's

[17:38](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1058s) hard but you would assume that the model

[17:40](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1060s) could potentially do it. So to give an

[17:42](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1062s) example, let's say you want to make uh

[17:44](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1064s) add a new column to a database and then

[17:46](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1066s) fill that column, right? And then put a

[17:49](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1069s) cache in front of it. So what happens is

[17:51](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1071s) it does the whole thing, but for some

[17:53](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1073s) reason the way it pushes the cache in

[17:54](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1074s) front of the whole uh database like it

[17:56](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1076s) kind of doesn't do it right. So at that

[17:58](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1078s) point you're like, "Oh no, set this

[18:00](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1080s) cache differently. Set this as the TTL."

[18:02](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1082s) And then you give like a correction

[18:03](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1083s) signal and then that kind of like fixes

[18:05](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1085s) your problem and you kind of go about

[18:07](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1087s) your day.

[18:08](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1088s) The way I want to think of it is that I

[18:09](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1089s) want to think of like every single time

[18:11](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1091s) like a human correction to the digital

[18:14](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1094s) landscape is a part of a delta that

[18:18](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1098s) needs to be covered. And what that means

[18:21](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1101s) is that like every single time you're

[18:22](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1102s) finding like the edge of model

[18:24](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1104s) capabilities of like how far it can push

[18:26](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1106s) for infants, how much it can push for

[18:27](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1107s) this, how much it can push for, you

[18:29](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1109s) know, solving your problems. You want to

[18:32](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1112s) find that delta and we want to like push

[18:34](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1114s) that frontier a little bit further. a

[18:36](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1116s) little bit further and that's what post

[18:38](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1118s) training does, right? So post training

[18:39](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1119s) is is this is in that post training we

[18:42](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1122s) find these like really interesting and

[18:45](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1125s) really hard long horizon problems for uh

[18:47](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1127s) for coding agents or for other kinds of

[18:48](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1128s) problems and then just like slowly hill

[18:50](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1130s) climb their models on on those kind of

[18:52](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1132s) problems. So anything that has a

[18:54](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1134s) verifiable loop can be optimized but is

[18:58](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1138s) this is not just limited to software

[19:00](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1140s) like literally like the physical world

[19:02](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1142s) itself could have a verifiable loop. So

[19:04](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1144s) to give an example like let me let me

[19:06](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1146s) give you a very funny kind of bizarre

[19:08](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1148s) example where like it's like you you

[19:11](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1151s) look at this and u you um

[19:15](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1155s) um look at this uh like robot and it's

[19:18](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1158s) it's typing something and it's like

[19:20](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1160s) really really like interesting what the

[19:22](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1162s) robot is typing.

[19:32](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1172s) So notice how like notice how the robot

[19:34](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1174s) is itself typing like how it wants to

[19:37](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1177s) like solve robotics and imagine if the

[19:39](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1179s) robot itself could like interpret the

[19:41](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1181s) results of of like the answers and just

[19:44](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1184s) like keep improving then that means that

[19:46](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1186s) like this is you know basically like an

[19:48](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1188s) ASI robot. Another example that I could

[19:50](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1190s) give you from my uh personal life is

[19:53](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1193s) that like um I have this um chip that

[19:56](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1196s) has the ability to see and has a speaker

[19:59](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1199s) and has a microphone inside of it. So I

[20:00](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1200s) was trying to set up this uh with my

[20:03](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1203s) machine and it was um a gift that I had

[20:07](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1207s) for my girlfriend. And uh what I found

[20:09](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1209s) very interesting about this is as I was

[20:11](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1211s) trying to set it up, it was very slow.

[20:13](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1213s) It was slow to respond. It had higher

[20:14](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1214s) latency. It had an agent in and of

[20:16](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1216s) itself. Right? So what I did was I

[20:18](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1218s) connected the step to my um computer and

[20:22](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1222s) I let um I let um coding agents run on

[20:27](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1227s) this hardware and basically run on this

[20:29](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1229s) hardware automatically make tests to it.

[20:32](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1232s) So like my computer would talk to my

[20:35](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1235s) hardware give it some tests like hey do

[20:38](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1238s) this do that and just like what do you

[20:39](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1239s) see right now? So then it would notice

[20:42](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1242s) the end to end latency of like the

[20:44](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1244s) inference response of like how fast uh

[20:46](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1246s) my hardware is responding and then um

[20:49](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1249s) what I found is that like it was able to

[20:51](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1251s) do the breakdown of like every single

[20:53](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1253s) case and scenario where there was like

[20:55](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1255s) high latency um slower response or or

[20:59](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1259s) just the algorithm wasn't right and I

[21:01](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1261s) could then set up a recursive loop where

[21:05](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1265s) both the firmware could be updated the

[21:07](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1267s) um the tool calls the inference model

[21:10](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1270s) used um the speech to text model used

[21:12](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1272s) all of those could be improved

[21:13](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1273s) automatically with like an OpenRouter

[21:15](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1275s) key which I'd given to a coding agent

[21:17](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1277s) and then overnight with self-running

[21:19](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1279s) experiments with the hardware uh my um

[21:22](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1282s) this tiny little piece of beautiful

[21:24](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1284s) hardware that I have was like deeply

[21:26](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1286s) optimized so was this also like even

[21:28](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1288s) hardware could be a part of the RSI loop

[21:30](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1290s) so what happens when you connect these

[21:32](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1292s) loops right that's the next question

[21:33](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1293s) like you could have cheaper inference

[21:35](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1295s) you could have like more experiments

[21:36](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1296s) budget you have like you know better

[21:38](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1298s) data and better models. So, is it

[21:40](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1300s) singularity? It's a very question to ask

[21:43](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1303s) and I think that the part of the parts

[21:44](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1304s) of the loop are already there, right?

[21:46](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1306s) Like we're we we are really we really

[21:48](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1308s) are very close to parts of the RSI loop,

[21:51](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1311s) right? And what do we do like what do we

[21:54](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1314s) do about this and it's like how do we

[21:55](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1315s) think of that in terms of like our

[21:57](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1317s) career and how do you think of that in

[21:59](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1319s) terms of you know our lives and what do

[22:00](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1320s) we want to do next? So here's my advice

[22:02](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1322s) to everyone who is watching this like

[22:04](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1324s) please please please please please move

[22:06](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1326s) closer to the RS right so it's like

[22:09](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1329s) right now like people are working on all

[22:11](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1331s) these like different kinds of software

[22:12](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1332s) which is like you know a agents for you

[22:14](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1334s) know construction agents for this agents

[22:16](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1336s) for that and it's like I think that

[22:17](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1337s) you're you know what the way I see those

[22:21](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1341s) things is that like I think of those as

[22:22](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1342s) like you're missing the forest for the

[22:24](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1344s) for the trees you really are and it's

[22:25](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1345s) like you have this like very interesting

[22:29](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1349s) godlike technology that can change parts

[22:31](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1351s) of humanity. And then we're not

[22:34](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1354s) ambitious enough to pursue the best

[22:35](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1355s) problems. We really are not. And we're

[22:37](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1357s) just like I think that it's not just

[22:39](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1359s) like the problem isn't just that we're

[22:41](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1361s) not ambitious and we're also like what

[22:43](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1363s) we're pursuing. A lot of those things

[22:46](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1366s) are just like one model release away

[22:47](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1367s) from like being completely like wiped

[22:49](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1369s) out, right? So like it's also like it's

[22:51](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1371s) also like it may not be that

[22:53](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1373s) intellectually simulating because like

[22:55](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1375s) all you're essentially all we're

[22:56](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1376s) essentially doing is like sitting all

[22:57](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1377s) day. It's like hey cloud do this for me.

[22:58](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1378s) Hey cloud do this for me. And I don't

[23:00](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1380s) know if that's like I don't know if

[23:01](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1381s) that's like the most intellectual

[23:02](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1382s) stimulating or like or like um

[23:05](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1385s) thoughtful use of a person's time like

[23:07](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1387s) maybe maybe you know we're supposed to

[23:08](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1388s) be solving harder and more uh more

[23:10](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1390s) frontier problems and I think that like

[23:12](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1392s) it's like the ceiling is like our

[23:13](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1393s) ambition. So instead of like working on

[23:15](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1395s) these kind of things my suggestion to

[23:17](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1397s) people is to like you know do things

[23:20](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1400s) that of course like goes without saying

[23:22](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1402s) but you should say things you believe in

[23:23](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1403s) and you know I'm just a guy on the

[23:24](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1404s) internet but do things that help you

[23:27](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1407s) move closer to the RSI loop right? So

[23:29](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1409s) like anything that just like the end

[23:31](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1411s) to-end pipeline of like hey this is

[23:34](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1414s) where like the models are lacking like

[23:36](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1416s) it's like whether that's inference

[23:37](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1417s) whether that's post training whether

[23:39](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1419s) that's literally like physical

[23:40](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1420s) interaction whether that could be you

[23:42](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1422s) know send sending end to-end pipelines

[23:44](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1424s) of truly any kind you know whether

[23:46](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1426s) that's hardware problems whether that's

[23:48](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1428s) um networking problems whether that's um

[23:50](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1430s) building a sense of community things

[23:51](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1431s) that just you know um things that just

[23:54](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1434s) like help um you as an engineer get

[23:57](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1437s) closer to the benefit benefits of the

[23:59](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1439s) improvements um in uh in u um in in the

[24:04](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1444s) model. So this is a very interesting

[24:05](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1445s) slide from Sasha topic. So um um they

[24:10](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1450s) mentioned that like you want to have

[24:13](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1453s) like this you're in this space where

[24:15](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1455s) it's like you know you only you only

[24:17](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1457s) have like 1 to five years left to

[24:18](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1458s) contribute to luxury to the world what

[24:19](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1459s) do you do right so it's like um they

[24:21](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1461s) they give great suggestions where like

[24:23](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1463s) prefer ambitious targeted and past

[24:25](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1465s) collaborations right and you know in

[24:28](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1468s) general like the time scale like all the

[24:30](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1470s) timelines that you give to yourself like

[24:32](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1472s) they should be just like completely

[24:33](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1473s) squeezed out should be completely

[24:34](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1474s) squeezed out because it's like a lot of

[24:35](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1475s) times and you also have to think through

[24:37](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1477s) that like if if the model if the problem

[24:39](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1479s) that you're working on is too ambitious

[24:40](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1480s) like it's like is it probably is it

[24:43](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1483s) probably better to wait for just like

[24:44](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1484s) the next next frontier model to come out

[24:46](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1486s) or not like that's that's something you

[24:48](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1488s) also have to weigh through right and uh

[24:51](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1491s) you you ideally want to work on problems

[24:53](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1493s) that benefit from scale and you know you

[24:56](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1496s) also at the same time want to avoid

[24:57](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1497s) projects that can be solved by sale

[24:58](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1498s) alone because then you know if even if

[25:00](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1500s) you like it's like if you're working on

[25:01](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1501s) a problem and then the next model

[25:03](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1503s) release comes out it solves your problem

[25:04](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1504s) so that's not that's probably not a good

[25:05](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1505s) thing to do so it's very like you have

[25:07](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1507s) to like think philosophically of like

[25:09](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1509s) what is the endeavor that benefits uh

[25:12](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1512s) greatly from auto improvements but like

[25:14](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1514s) isn't also completely wiped out from it

[25:16](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1516s) and then ideally like it's like if the

[25:18](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1518s) if the endeavor could also uh be a part

[25:20](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1520s) of the the like the improvement pipeline

[25:24](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1524s) uh that could be amazing. So you see a

[25:26](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1526s) lot of companies raising on say um

[25:29](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1529s) making data pipelines and there's a lot

[25:31](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1531s) of money there. There's a a lot of

[25:34](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1534s) companies working on physical hardware

[25:35](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1535s) and stuff. So I think if you're a

[25:36](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1536s) software engineer watching these, it's

[25:38](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1538s) like I think that if you're if you feel

[25:40](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1540s) like you're like here the delta of your

[25:42](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1542s) intellect and ability in your

[25:44](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1544s) day-to-day, it's like not contributing

[25:46](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1546s) as in like you have the intellect, you

[25:49](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1549s) can make a difference. But like if cloud

[25:50](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1550s) is just like oneshotting all those

[25:52](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1552s) problems like in your day-to-day and

[25:54](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1554s) you're just like kind of like becoming a

[25:55](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1555s) me proxy, you should work to think

[25:56](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1556s) through. Take a step back and think

[25:57](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1557s) through. like okay where are the

[25:59](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1559s) problems where where you're actually

[26:01](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1561s) truly uh contributing and that they will

[26:03](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1563s) stand the test of time and then that

[26:05](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1565s) kind of reflection and realignment right

[26:07](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1567s) now would I think like have like very

[26:09](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1569s) interesting consequences for you um in

[26:12](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1572s) the long term um so yeah so thank you so

[26:15](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1575s) much you know thank you so much for your

[26:17](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1577s) time like I think that uh I think that

[26:19](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1579s) if you're watching this and if you've

[26:20](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1580s) gone this far I think that you're

[26:22](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1582s) definitely contemplating like one of the

[26:24](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1584s) most interesting improvements in the

[26:26](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1586s) course of humanity And uh you know

[26:28](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1588s) there's many philosophical arguments

[26:30](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1590s) here. Some people might say hey you know

[26:32](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1592s) I I think I'm maybe I'm off on the

[26:35](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1595s) timelines or maybe maybe I'm I'm just

[26:37](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1597s) being too pessimistic or maybe I'm being

[26:39](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1599s) too optimistic. But I would love to hear

[26:41](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1601s) all those thoughts. So this is you know

[26:42](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1602s) my QR code. Please feel free to scan it

[26:44](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1604s) and let me know all your thoughts. Uh

[26:46](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1606s) thank you so much. Very kind of you to

[26:47](https://www.youtube.com/watch?v=iK9gmxXyBR0&t=1607s) give me your time.
