# Improving Agents is a Data Mining Problem

Vivek Trivedy, LangChain

AI Engineer World's Fair 2026

[Talk page](https://ai.engineer/talks/CvRngaQZQ3Y-improving-agents-is-data-mining-problem) · [Video](https://www.youtube.com/watch?v=CvRngaQZQ3Y)

## Improving Agents Is a Data Mining Problem

Agent mistakes become useful when traces turn them into evidence for experiments, cheaper models, better evaluations, and updates to the agent’s own state.

Familiarity with LLM agents, tool calls, and basic evaluation workflows will help; no prior fine-tuning experience is required.

### The agent made mistakes. What changes next?

[0:00](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=0s)

You run an agent. It takes a series of actions and makes some mistakes. Now you have the execution data—but what should you actually change? That is the starting problem for Vivek Trivedy, who leads applied research at LangChain: turning evidence of agent behavior into a repeatable improvement process, drawing on work with customers’ large collections of traces.

The process starts with a working agent, not a perfect one:

1. **Ship and run it.** Real environments produce feedback that an agent definition alone cannot supply.
2. **Collect traces.** Store tool calls, output messages, API calls, and CLI interactions so the execution remains available for inspection.
3. **Mine and curate the data.** As deployments grow, the material may amount to gigabytes or terabytes. Identify the interactions worth learning from.
4. **Run experiments.** Use those examples to test whether a different prompt, tool, orchestration scheme, or agent loop actually improves behavior.

The experiment closes the loop: a proposed improvement must answer to the evidence collected from previous runs.

![Slide listing four numbered steps: build a v1 agent and start running it; collect lots of traces; curate the trace data; run experiments on your data.](https://ai.engineer/api/talk-essay-media/8a75e4a2a0e56e8f719f0e211e7e39228fdf9919b761478b864d7ba5f85746d8)

A four-step recipe for continuously improving agents: build, collect traces, curate data, and run experiments.

**Observability and continual learning depend on the same evidence.** An agent acts in an environment, its traces record what happened, and those observations can inform changes to its knowledge or instructions. Collecting traces makes that feedback available; integrating it back into the agent is the learning step. Trivedy punctuates the point with a Will Hunting meme, included on a promise to Hamel: look at the data before deciding how to improve the system.

### Autonomy makes execution evidence essential

[3:26](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=206s)

With a conventional Python program, you can read the functions, follow their calls, and often reason through the result. An agent adds prompts, tools, skills, hooks, middleware, and sometimes other agents arranged into a swarm. Reading that configuration does not reliably tell you how a prompt edit will change behavior across many runs. The task domain matters too: the same kind of edit can have different consequences in medicine and law.

This is the practical consequence of trading determinism for autonomy. The agent has more freedom to choose its actions, so understanding it requires tools that expose what it actually did. Traces provide an execution record against which expectations about its behavior can be checked.

![Slide contrasting a Python code block with an agent panel showing a task, execution indicator, and question mark, beneath the heading 'No one knows what their agent will do until they run it.'](https://ai.engineer/api/talk-essay-media/520b62a4fb1ffb6f6021fd06e4b8c98e0c25fe96267ce6d5fbead35ccc961d19)

Agents trade determinism for autonomy.

### Give the reviewing agent a concrete question

[4:41](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=281s)

At LangChain, traces are centralized in projects, either for an individual agent or across several agents. Other agents then read that material with particular questions in mind. A useful first investigation is to find good and bad interactions: where did a user become upset, and where did the interaction leave them happy? This connects internal execution details to the experience the system is supposed to deliver.

Long runs raise a different question. If an agent works through millions of tokens, does its behavior deteriorate after its first context compaction? What about its second? Or does it preserve the information it needs throughout? The trace is where those questions become inspectable, rather than assumptions about what compaction ought to do.

Traces also support comparisons between model choices. A task run with **GPT-5.5** can motivate a counterfactual experiment with **GLM-5.2**: how would the alternative behave on the same task? Aggregate metrics help compare outcomes, while the traces expose finer differences in the actions and responses users actually see. The model swap is a question to test, not an improvement established merely by selecting another model.

### Trace mining has two scaling limits

[6:00](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=360s)

Trivedy expects agent activity to produce a rapidly expanding volume of data. His forecast is that agents will eventually eclipse the data humans have produced over a lifetime on progressively shorter timescales: years, then six months, three months, and perhaps a day. Whether or when that crossover happens, the engineering problem already has two parts: paying to inspect the data and making it accessible to the inspecting agent.

The first limit is input cost. Reading millions of traces becomes expensive when each trace may itself contain millions of tokens. For a single full reading pass, the basic calculation is:

```math
C_{\text{input}} = N_{\text{traces}}\,\bar{T}_{\text{trace}}\,\frac{P_{\text{million}}}{10^6}
```

Here, `N` is the number of traces, `T̄` is the average tokens per trace, and `P` is the price per million input tokens. The same calculation can be made directly in Python:

```python
def trace_reading_cost(
    trace_count: int,
    average_tokens_per_trace: int,
    price_per_million_input_tokens: float,
) -> float:
    total_tokens = trace_count * average_tokens_per_trace
    return total_tokens / 1_000_000 * price_per_million_input_tokens
```

This estimates input-token charges; generating analyses or repeatedly revisiting traces adds work beyond that single pass.

The second limit is context capacity. A long Claude Code, Codex, or Deep Agents interaction may not fit inside the reviewing agent’s context window at all. **Treat the trace as an external object that the agent can query**, rather than requiring the whole execution to be loaded into context. Efficient mining therefore needs both an economical reader and a way for that reader to access relevant portions of large traces.

### Find the least expensive model that can do the job

[7:36](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=456s)

The practical model-selection sequence begins with capability. Trivedy’s team starts with a frontier model such as **Opus** or GPT-5.5 to determine whether a task is feasible. Once that establishes a working baseline, they inspect the traces and ask whether an open model can perform the same job. The target becomes the minimum capability sufficient for this particular task, rather than the strongest available model for every workload.

Trace judging provides one example. In work with Harvey on its [Legal Agent Benchmark (LAB)](https://www.langchain.com/blog/designing-efficient-verifiers-for-legal-agents), Trivedy reports roughly matching Opus’s trace-judging capability with open models at one or two orders of magnitude lower cost. The comparison concerns this legal judging task; the talk does not specify the exact model pair, agreement metric, or pricing basis for that range.

![Slide titled 'Open Models Are Cost-Efficient Trace Judges,' with colored points and connecting lines, a model legend, and a highlighted low-cost, low-drift region.](https://ai.engineer/api/talk-essay-media/9b7adfee72db9e667386a5318eda3af84c57055b580c19cfd69f6727a0b0524b)

Comparing trace judges by cost and label drift.

Reaching that result involves trying models and engineering their harnesses. Reading how Opus approaches a judgment can reveal guidance a cheaper model needs explicitly in its prompt. Some differences reflect model capability, but some can be addressed by making the expected reasoning process clearer. Trace inspection thus informs the harness changes used to make the less expensive judge useful.

### When harness changes stop helping

[8:58](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=538s)

**Harness engineering** offers a fast feedback loop: change the prompt or surrounding execution system, run evaluations, and inspect the result. Eventually, however, further prompt edits may yield little improvement. That is the point at which Trivedy considers domain-specific fine-tuning.

Customers usually care about a narrower set of tasks than a general-purpose model must cover. Trivedy reports that fine-tuning base models on narrow customer tasks can match or exceed frontier-model performance on those tasks. That is an experience-based claim about specialization, without a single measurement protocol supplied for all such workloads.

The economics can change along with the model. Per-token inference makes each additional reading pass a metered expense; running an inference cluster makes hardware capacity and utilization the relevant costs.

| Billing approach | Main cost basis | Operational consideration |
| --- | --- | --- |
| Per-token inference | Tokens processed | Cost grows with usage |
| Dedicated cluster | Hardware running time | Keep it utilized; stop it when idle |

For sufficiently high inference workloads, Trivedy finds the cluster approach cheaper. His description of unlimited inference means avoiding a separate charge for each token while the cluster runs—not unlimited throughput or cost-free computation. The cluster can be spun down when the work is finished.

### Turn traces into assets people and models can use

[10:24](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=624s)

[LangSmith Engine](https://www.langchain.com/blog/how-we-built-langsmith-engine-our-agent-for-improving-agents) is introduced as a product for automating this trace-mining workflow. Given trace data and something to investigate, it searches the executions, finds issues, and prepares datasets, evaluations, or feedback for human readers. The useful output is not simply an account of what happened: it is material that supports a subsequent improvement step.

One destination is **distillation through supervised fine-tuning**, or SFT. Suppose GLM-5.2 already performs a task well, but a smaller 9B or 13B model could make it cheaper to serve. Curate the good traces and examples from GLM-5.2, assemble them into a training dataset, and fine-tune the smaller model to imitate that behavior. This is the proposed distillation workflow; the talk does not present a measured outcome for those model sizes.

Another destination is **evaluations and environments**. Observed interactions supply tasks against which future versions can be tested. Trivedy argues that seeing an agent’s evaluation suite gives a rough picture of its intended behavior: developers repeatedly modify the system to make those tests pass. Choosing the examples is therefore part of choosing what the agent will be optimized to do.

A third destination is **human review**. Someone still needs to understand the agent’s actions and whether customers are happy, but that person rarely has time to read every trace. Preparing trace-derived content makes oversight manageable, especially in legal and medical work where human review remains necessary. The mining system must produce evidence people can process, as well as data models can learn from.

### Fit the model, harness, and task together

[12:46](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=766s)

Trivedy connects this process to scikit-learn and his PhD work on adding learning algorithms to it. At an abstract level, scikit-learn supplies helpers for fitting learning systems to data. Agents change the components and algorithms, but the underlying machine-learning discipline remains recognizable: assemble the data, select a learning procedure, and fit a system to the tasks that matter.

He calls the agent version **model–harness–task fit**. The model and harness must work together on the target data; choosing either in isolation misses their interaction. The engineering work has two connected responsibilities:

- **Find good fit functions.** Explore improvement procedures such as autoresearch, reinforcement learning, OPD, OPSD, and SFT.
- **Find good data.** Supply examples and feedback that represent what the system needs to do.

Those two choices define the practical research problem for a team improving its agents.

Autoresearch makes the feedback loop explicit: perform an action, inspect the result and its traces, then update the system. If there is a score to improve, agents can often search for changes that increase it. But an increasing score is not sufficient evidence of progress: agents may exploit the scoring procedure, so the loop needs checks on how the improvement was obtained.

### A failure score does not explain the failure

[14:59](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=899s)

The [Terminal-Bench](https://www.tbench.ai/) example turns this into a concrete experiment: have an agent read its own traces, propose experiments, and attempt fixes. The important ingredient is **dense feedback**. A final pass/fail result tells the agent whether it succeeded, but gives little guidance about what to change after a failure.

The execution trace supplies the missing detail. Instead of receiving only a terminal score, the improving agent can inspect the sequence that produced it and use that evidence to choose a subsequent experiment.

| Feedback available | What it helps answer |
| --- | --- |
| Final pass/fail | Did the run succeed? |
| Execution trace | What happened along the way? |
| Trace analysis | What should we investigate or change next? |

Densifying feedback means making more of the evidence behind the outcome available for diagnosis. The trace is the record that holds it.

### Start with the fastest useful feedback

[15:58](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=958s)

When should a team keep engineering the harness, and when should it fine-tune? Trivedy starts with feedback speed. Gather evidence as quickly as possible, whether through human labeling or additional agent runs. Trivedy estimates that a harness-engineering experiment can provide feedback in roughly two minutes. The estimate illustrates a fast iteration cycle, rather than a fixed runtime for every evaluation suite.

The resulting sequence is a sandwich:

1. **Engineer the harness first.** Use prompt and execution changes to address observed failures. Many teams satisfy their customer use case here.
2. **Fine-tune if progress plateaus.** Introduce model training when the remaining gap justifies it.
3. **Revisit the harness.** After tuning, make further harness changes if they help the updated model perform the task.

There is no requirement to advance through every stage. The sequence preserves a fast learning loop while leaving room to change the model when harness edits are no longer enough.

### Continual learning must update agent state

[17:06](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=1026s)

The longer-term question is how an agent incorporates its experience. After performing tasks, it might update its prompts to work more efficiently. If users repeatedly ask for a particular kind of search, it might notify its creator that this need keeps appearing. In both cases, actions in the environment produce information that should change the system’s future behavior. Trivedy leaves the precise form of continual learning open, while identifying three areas that need to evolve.

- **Training data.** Collect observations from agents taking actions so experience can inform later learning.
- **Harness updates.** Evolve the execution system with the model and its tasks. Codex, Claude Code, and LangChain’s harnesses reflect both the environments in which models are trained and the work they perform in practice.
- **Memory.** Maintain useful knowledge over time, rather than treating the agent’s history as an append-only log.

Memory becomes especially consequential when an agent works with someone for years, decades, or a lifetime. A continually growing file plus search does not by itself provide the revision and maintenance that long-lived memory needs.

**Sleep-time compute** and dreaming describe a possible way to do that maintenance: revisit traces across the agent’s life cycle, revise stored information, and update agent state. The purpose is to turn accumulated activity into more useful memory, not merely to preserve an ever-larger archive.

![Continual Learning diagram with an Agent circle on the left and three connected boxes labeled Training Data, Harness Updates, and Memory on the right; below is 'Scaling Sleep Time Compute & Dreaming.'](https://ai.engineer/api/talk-essay-media/053e437971863e7593884cf841431a07b538f5f8fe902d6ec02aa5373c366cdf)

Continual learning connects agent traces to training data, harness updates, and memory.

The immediate starting point is small: enable tracing and give an agent a concrete question to investigate in those traces. Trivedy also points to LangChain’s open-model fine-tuning assistance and model services as ways to support the improvement loop. The larger research challenge remains learning how to manage the volume of data agents produce—and how to integrate those observations back into agent state so the next run benefits from the last one.

## Resources

- [Improving Agents is a Data Mining Problem](https://www.langchain.com/blog/improving-agents-is-a-data-mining-problem) — Vivek Trivedy's companion essay on mining traces, fitting agent systems to tasks, and continual learning.
- [Designing Efficient Verifiers for Legal Agents](https://www.langchain.com/blog/designing-efficient-verifiers-for-legal-agents) — LangChain and Harvey's study of batching, open models, and prompt tuning for legal benchmark verifiers.
- [How We Built LangSmith Engine](https://www.langchain.com/blog/how-we-built-langsmith-engine-our-agent-for-improving-agents) — Technical account of turning agent traces into recurring issues, fixes, evaluators, and dataset examples.
- [LangSmith Engine documentation](https://docs.langchain.com/langsmith/engine) — Current instructions for configuring trace analysis and reviewing proposed agent improvements.
- [Improving Deep Agents with harness engineering](https://www.langchain.com/blog/improving-deep-agents-with-harness-engineering) — A fixed-model Terminal-Bench 2.0 experiment using trace analysis, verification, and harness changes.
- [Terminal-Bench](https://www.tbench.ai/) — Benchmark and project resources for evaluating agents on tasks in terminal environments.
- [Self-Distilled Reasoner](https://arxiv.org/abs/2601.18734) — Research on training a model through on-policy self-distillation with different teacher and student contexts.

## Slides

### Improving Agents Is a Data Mining Problem

Vivek Trivedy introduces the question of how to continuously improve agents using data.

[0:24](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=24s) [Title slide for Vivek Trivedy’s talk on improving agents through data mining.](https://assets.ai.engineer/sites/aiecode2025/sha256/9fd7806b53e7490b99671e747d8b5da45239e3c690cb4e2cb6e0c555c70f6d36.webp)

AI Engineer
LangChain

Improving Agents is a
Data Mining Problem.

Vivek Trivedy · Applied Research Lead @ LangChain

### What Do I Do?

Vivek Trivedy opens with a familiar problem: an agent has run and made mistakes, leaving its operator to figure out what to do with the data.

[0:36](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=36s) [An agent ran, did many things and made mistakes; the slide asks what to do next.](https://assets.ai.engineer/sites/aiecode2025/sha256/e7692ee643e6610048adb6f5e0f828f4c579008b5200f5ba116740c0fde6a63e.webp)

So I ran my agent...
It did a lot of things...
...it made some mistakes.

What do I do?

LangChain
2 / 22

### A Recipe for Continuously Improving Agents

Run an agent, collect and curate its traces, then test changes to prompts, tools, orchestration, or loops against those traces to see whether they improve the agent.

[1:59](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=119s) [Four steps: run an agent, collect traces, curate trace data, and run experiments.](https://assets.ai.engineer/sites/aiecode2025/sha256/a661d2e4d5f45428d2fcc1c6e2ad6e94b13b9232d2853c7fc281d6132d2e6f8b.webp)

A Recipe for Continuously Improving Agents

1. Build a v1 agent and start running it
2. Collect lots of Traces
3. Curate the Trace data → hello data mining
4. Run experiments on your data

LangChain
3 / 22

### Observability and Continual Learning

Observability captures agent traces; continual learning uses those traces to update knowledge in response to environmental feedback. The slide connects the two in a feedback loop.

[2:24](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=144s) [A two-way loop connects capturing every trace with learning from every trace.](https://assets.ai.engineer/sites/aiecode2025/sha256/1dff3b5899ad19e120b8340c56018e2401d051a4d8a7473b0ef83d5d3e377c42.webp)

Observability ↔ Continual Learning

OBSERVABILITY
capture every trace

CONTINUAL LEARNING
learn from every trace

Every Continual Learning Company Is an Observability Company.

LangChain
4 / 22

### Engineers Who Read the Traces

A Will Hunting meme casts the engineer who reads traces as someone for whom improvement becomes easier over time.

[3:07](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=187s) [A social media post pairs a movie still with a joke about engineers who read traces.](https://assets.ai.engineer/sites/aiecode2025/sha256/3a18f1e2f47072d8746cdb606ebb5af474910779f0c48ba292dca3b997fd615a.webp)

Hamel Husain
@HamelHusain

AI Engineers who look at the data

The engineer who read the traces

Interface text:
LangChain
5 / 22

### Agents Trade Determinism for Autonomy

The slide contrasts readable Python functions with an agent whose prompts, tools, skills, hooks and subagents make its behavior harder to predict before execution.

[3:24](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=204s) [Code and agent workflows are contrasted to illustrate the difficulty of predicting agent behavior.](https://assets.ai.engineer/sites/aiecode2025/sha256/551ee04b56cf4a81938c0392582d0b0052055e534f8a9aefb57b38217d1edeb2.webp)

No one knows what their agent will do until they run it.

READING CODE

def calculate_total(items):
    return sum(i.price * i.qty for i in items)

def apply_discount(total, code):
    if code == "SAVE10":
        return total * 0.9
    return total

def checkout(cart):
    total = calculate_total(cart.items)
    return apply_discount(total, cart.code)

Read it → you know exactly what happens.

VS

AN AGENT
Prompt  Tools  Skills  Hooks  Subagents

Task: “Refund a duplicate charge”
↓
Executing...
↓
?

Read it → you still don’t know what happens.

Agents trade determinism for autonomy.

Interface text:
LangChain
6 / 22

### Why Read Traces?

LangChain centralizes traces in a project and sends agents to mine them using instructions and rubrics. They can look for good and bad interactions, performance after compaction, or model costs.

[4:39](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=279s) [A tracing project feeds an agent that examines traces using instructions and rubrics.](https://assets.ai.engineer/sites/aiecode2025/sha256/5c01f8bc2c831c0e41e53963221d74da120147dd557c55d423a7ed6f1cfa7f8e.webp)

Why Read Traces At All?

Traces tell you what your agents did, where they messed up, what users ask for

Tracing Project
run_id | tool_calls | status
[illegible]... | search, edit | fail
[illegible]... | read, write | pass
...

Agent
reading traces
mines the project
at scale

Instructions: what to look for
• find examples of good/bad interactions
• can agents complete tasks after compaction?
• how much did using gpt-5.5 cost?

Rubrics help codify what to look for.

Interface text:
LangChain
7 / 22

### Long Horizon Agents Produce a Lot of Data

An illustrated run grows through two compaction points to 1.2M+ tokens, alongside a nested trace. Modern agent runs can be millions of tokens long.

[5:58](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=358s) [A growing sequence of agent steps with compaction markers appears beside a trace interface.](https://assets.ai.engineer/sites/aiecode2025/sha256/ae9f561ee96f0b9469ebd9f97f56848caa02110304d684d49f2cb6b96e1256fa.webp)

Long Horizon Agents Produce a Lot of Data

compaction
compaction
1.2M+ tok

Modern agent runs can be millions of tokens long.

LangChain
8 / 22

Interface text:
TRACE
Waterfall
RetrievalGraph
create_research_plan
RunnableSequence
ChatOpenAI
JsonOutputKeyTo...
conduct_research
ResearcherGraph
generate_queries
RunnableSequence
ChatOpenAI
JsonOutputKeyTo...
retrieve_in_parallel
retrieve_documents
VectorStoreRetriever
retrieve_documents

ChatOpenAI
Tools
Company
Input
SYSTEM
[illegible]
Output
HUMAN
[illegible]

### Reading Traces at Scale is Hard

Large trace volumes drive up input-token costs, while individual traces can exceed a context window. The speaker proposes treating long traces as external objects that agents can query.

[6:34](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=394s) [Two cards describe the cost of processing traces and traces exceeding context limits.](https://assets.ai.engineer/sites/aiecode2025/sha256/994071617f1cde525d6d71f0b3694036b7c738e3b11b0041b66a0ab6f7e283f1.webp)

Reading Traces at Scale is Hard

Problem 1
$
Reading traces at scale is expensive.
Large trace volumes mean lots of input tokens to process.

Problem 2
?
Traces don’t fit in context.
Agents need to learn what to look for and how to manage traces too large for any single context window.

Interface text:
LangChain
9 / 22

### Open Models Are Cost-Efficient Trace Judges

On Harvey’s LAB benchmark, DeepSeek batch shows 6.2% label drift against Opus per-criterion at $0.09 per 1,000 criteria. The speaker describes using trace-informed guidance to help cheaper models approach Opus’s judging capability.

[7:35](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=455s) [A chart compares cost per criterion and label drift for several trace judges and batching methods.](https://assets.ai.engineer/sites/aiecode2025/sha256/d86aacbffc74bbc3a3bda24c90f73468a1d32d7c5d31571bd0bf0e2c5baa0e7f.webp)

Open Models Are Cost-Efficient Trace Judges

Experiments on Harvey’s LAB benchmark show open models can judge traces well with good rubrics.

LAB
Verifiers on Harvey’s LAB Benchmark

LABEL DRIFT vs OPUS PER-CRITERION

Y-axis ticks: 0%, 3%, 6%, 9%, 12%, 15%
X-axis: Cost per 1,000 criteria (log scale)
X-axis ticks: $0.1, $1, $10, $100

DeepSeek batch
$0.09/1k | 6.2% drift

LOW COST / LOW DRIFT

arrows: per-criterion → batch (same model)

CONDITIONS
DeepSeek batch — $0.09/1k · 6.2% drift
Haiku batch — $0.86/1k · 9.2% drift
DeepSeek per-criterion — $1.62/1k · 4.3% drift
Sonnet batch — $2.49/1k · 5.8% drift
GPT-5.5 batch — $3.50/1k · 5.3% drift
Opus batch — $4.21/1k · 5.8% drift
Haiku per-criterion — $13.53/1k · 12.1% drift
Sonnet per-criterion — $39.18/1k · 4.9% drift
GPT-5.5 per-criterion — $58.66/1k · 4.3% drift

Open models + batching cut verification cost ~100×.

LangChain
10 / 22

### Fine-Tuning Open Models Reaches Frontier Performance

Fine-tuned Qwen gains 7.6 points in fleet holdout accuracy at ~$4/day. Opus scores 90.2% at $412/day on-demand; costs are shown for 10k traces/day.

[8:58](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=538s) [A chart compares accuracy and daily judging cost, highlighting improved accuracy for fine-tuned Qwen.](https://assets.ai.engineer/sites/aiecode2025/sha256/556e5194b42d8edb34d1d46c94262ecf15ec409d43a0e182271c91f02535a800.webp)

Fine-Tuning Open Models Reaches Frontier Performance

Custom models to target any signal in your trace data.

Fireworks AI

Fleet holdout accuracy
Y-axis ticks: 72%, 80%, 84%, 88%, 92%
X-axis ticks: $0, $100, $200, $300, $400
Daily on-demand judge cost at 10k traces/day

Fine-tuned Qwen
~$4/day

+7.6 pts

Base Qwen
~$4/day

Haiku 73.4%
$82/day on-demand

Sonnet 89.7%
$248/day on-demand

Opus 90.2%
$412/day on-demand

GPT-5.5 89.1%
$415/day on-demand

Stronger judge,
order-of-magnitude cheaper
Fine-tuning improves quality

Fine-tuning reaches a new Pareto frontier of cost & performance.

Interface text:
LangChain
11 / 22

### What Trace Mining Gives Us

Trace mining turns successful trajectories into training data for smaller models, reconstructs eval environments from errors and edge cases, and exposes patterns that guide product and harness decisions.

[11:01](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=661s) [Trace data branches into distillation and fine-tuning, evals and environment generation, and product analytics and harness targets.](https://assets.ai.engineer/sites/aiecode2025/sha256/7cbc548257cdd9b1b0eeabbb969226cdf94fbaf72d2aeee23fc40599454a7cda.webp)

What Trace Mining Gives Us

TRACE DATA

GLM
large model

Distillation & Fine-Tuning
good trajectories become SFT / RL data
big model → traces → cheap small model

evals

Evals & Environment Gen
errors & edge cases reconstruct real environments and rubrics to test against

$ cost / latency / failures

Product Analytics & Harness Targets
tool-call & failure patterns at scale show what to fix in the harness, and what users want

Interface text:
LangChain
13 / 22

### Model-Harness-Task Fit

The fit function brings a model, harness, and task data together to make tasks pass. The slide defines an agent as a model plus a harness.

[12:44](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=764s) [A fit function combines model, harness and task, with data pointing to the task; an agent is model plus harness.](https://assets.ai.engineer/sites/aiecode2025/sha256/9b9d2047d72eb78ecba89155675d8c8488820934b0ab20a2af5c2cab99d416b1.webp)

Model-Harness-Task Fit

Data
↓
fit(model, harness, task)

Agent = Model + Harness

LangChain
15 / 22

### Auto-Research over Terminal Bench Traces

An agent runs experiments, reads traces, and proposes fixes in a loop. Traces provide denser feedback than Terminal-Bench’s pass-or-fail result, helping the agent decide what to change next.

[15:01](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=901s) [A loop runs experiments, reads traces and proposes fixes beside a Terminal-Bench leaderboard.](https://assets.ai.engineer/sites/aiecode2025/sha256/3c6360c8420881b48b70801c60794ef5d2ac357fe28f08ff672bad47af32c4d4.webp)

Auto-Research over Terminal Bench Traces

Any verifiable metric can be hill-climbed: run experiment → read traces → propose fix → loop.

RUN
experiment
→ READ
traces
→ PROPOSE
fix

/goal loop

System Prompt
Tools
Middleware

Experiments produce traces which help guide the Harness Engineering loop.

Interface text:
terminal-bench
Run Terminal-Bench  Leaderboard  Tasks  Registry  Contributors  News  Terminus  Discord
Showing 102 entries
Clear filters
Search leaderboard  Select agents  Select models  Select organizations
Rank  Agent  Model  Date  Agent Org  Model Org  Accuracy
1  Simple Codex  GPT-5.3-Codex  2026-02-06  OpenAI  OpenAI  75.1% ± 2.4
2  CodeBrain-1  GPT-5.3-Codex  2026-02-10  Feeling AI  OpenAI  70.3% ± 2.6
3  Droid  Claude Opus 4.6  2026-02-05  Factory  Anthropic  69.9% ± 2.5
4  Mux  GPT-5.3-Codex  2026-02-09  Coder  OpenAI  68.5% ± 2.4
5  Deep Agents  GPT-5.2-Codex  2026-02-12  LangChain  OpenAI  66.5% ± 3.1
6  Droid  GPT-5.2  2025-12-24  Factory  OpenAI  64.9% ± 2.8
7  Ante  Gemini 3 Pro  2026-01-06  [illegible]  Google  64.7% ± 2.7
8  Terminus 2  GPT-5.3-Codex  2026-02-05  Terminal Bench  OpenAI  64.7% ± 2.7
9  Junie CLI  Gemini 3 Flash  2025-12-23  JetBrains  Google  64.3% ± 2.8
10  Droid  Claude Opus 4.5  2025-12-11  Factory  Anthropic  63.1% ± 2.7

LangChain
18 / 22

### Harness Engineering, Fine-Tuning, and Harness Engineering Again

Start with harness engineering for fast feedback. If it hits a ceiling, fine-tune, then revisit the harness for the fine-tuned model’s new intelligence landscape.

[15:57](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=957s) [A funnel diagram shows harness engineering, fine-tuning, and another round of harness engineering.](https://assets.ai.engineer/sites/aiecode2025/sha256/cf79e0c9267e5b6f93d6ad2ee26255197a54b9bc2a3defa94e7f5494a79aacbb.webp)

Harness Eng → Fine-Tune → Harness Eng

A funnel. Start with harness engineering — fast feedback, cheap to swap models. Hit the ceiling, then fine-tune. The fine-tuned model has a new intelligence landscape — so you harness-engineer again.

Harness Engineering
fast feedback · cheap to swap models

Fine-Tuning
when harness eng hits its ceiling

Harness Engineering, Again

Interface text:
LangChain
19 / 22

### Continual Learning

Agent traces can feed training data, harness updates, and memory. The speaker proposes processing traces over an agent’s lifetime to update its state, including through sleep-time compute.

[16:49](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=1009s) [Agent traces feed training data, harness updates, and memory in a continual-learning loop.](https://assets.ai.engineer/sites/aiecode2025/sha256/9c5e27cb33fafe5a85492888855b9740f3df06c668821436eb74cee9eced1ee1.webp)

Continual Learning

CL is integrating the information that agents produce through action back into the agentic system.

Agent
 takes actions

traces

Training Data
good & bad trajectories become
SFT / RL data

Harness Updates
learning skills, adding tools,
updating prompts

Memory
learning to store & organize experiences
for contextual retrieval

Scaling Sleep Time Compute & Dreaming

Interface text:
LangChain
20 / 22

### Takeaways

Mining traces reveals where agents can improve. Open-model fine-tuning and compound agents help process large-scale trace data; continual learning feeds agent data back into agents over long time horizons.

[18:47](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=1127s) [Four takeaways cover trace mining, open-model fine-tuning, continual learning, and processing agent data at scale.](https://assets.ai.engineer/sites/aiecode2025/sha256/f2da56bc7907c7392cbbde73f7b5401e1f65b1db622adfdfafdf26e99e538f5d.webp)

Takeaways

1. Mining traces gives you signals to hill climb on
2. Open model fine-tuning & compound agent systems help you process large scale trace data
3. Continual Learning is about processing & integrating agent data back into agents over long time horizons
4. Agents will produce more data than humans have in history. We need to update our tooling to process it.

LangChain
21 / 22

### AI Engineer World's Fair

[19:55](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=1195s) [AI Engineer World's Fair logo on a black background, with the event website below.](https://assets.ai.engineer/sites/aiecode2025/sha256/8ac64db99dbc3d3553da9cc455c704f6be72d9f8dd560f11b59ff8c68e0c70a5.webp)

AI Engineer
World’s Fair

HTTPS://AI.ENGINEER

## Full transcript

[0:00](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=0s) [upbeat music] Hey, everyone. Uh, I'm Vivek, and I lead applied research at LangChain, and I'm gonna talk about something that I think is sexy, which is data mining, but it's not as sexy as LLM, so we're

[0:25](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=25s) gonna try to, like, make it sexy together. And the problem that we're gonna talk about today is how do we continuously improve agents, but how do we do that via data?

[0:35](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=35s) So to start, I'm gonna tell a little story that I think maybe a lot of us have felt before. Like, I ran my agent,

[0:44](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=44s) it did a bunch of things. It made some mistakes.

[0:49](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=49s) Now I ask someone, like, "What do I actually do about that?" Like, I have all this data, made some mistakes. What now?

[0:56](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=56s) Basically, what we're gonna do today is we're gonna motivate a recipe for what we should do to continuously improve agents over time, and then I'm gonna talk from some lived experience and, like, some stuff that we help customers do to run this over large-scale trace data.

[1:11](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=71s) So the first step in building a successful agent is shipping it. So if you put it out into the real world, then it can operate in environments, and then you can get feedback from what it's doing.

[1:23](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=83s) The second step is collect a ton of traces. So agents operate in the environment. Every single time they operate, they do tool calls, they have output messages, they call APIs, they use CLIs.

[1:36](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=96s) All of that generates data, and we wanna store all of that so we can, like, do stuff with it.

[1:42](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=102s) The next thing is the data mining in this talk, which is once we have tons of trace data, maybe gigabytes, maybe terabytes, depending on, like, how many agents you're shipping, we're gonna do data mining over that.

[1:54](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=114s) And I promise I will tell you exactly what data mining we're gonna do, uh, but we're gonna do some over it.

[2:00](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=120s) And then the fun part, which is I collected that data, I read it, I curated it, and now we actually need to run the experiments in a data-driven way to see, hey, is this new prompt or is this new tool or is this new orchestration or is this new loop, is it actually improving things based on the

[2:18](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=138s) previous traces that I've seen? And this is maybe a bit of a hot take, but continual learning is super hot right now. I'm talking about it. This whole room is gonna hear about it for the next, like, five, six hours.

[2:30](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=150s) Um, but th-there's a very tight coupling between what observability is and what continual learning is. And the main reason for that is that agents that operate in environments, they produce trace data, and what continual learning for agents and continual learning for humans basically is, is I do a bunch of stuff in the world, I think about what

[2:51](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=171s) I did, and then I need to update my definition, like my knowledge, stuff I write down, in order to respond to the feedback from the environment. And if you're a continual learning company, you need traces, and if you have traces, then you can try to do continual learning over your agents.

[3:09](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=189s) I had to put in a meme because if you look at your data, then you can be like Will Hunting, if anyone's seen the movie, where, like, everything is super, super easy and you can, like, improve over time, and I promised Hamel I would put this in there, so putting it in there.

[3:26](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=206s) Cool. So why am I talking a bunch about traces anyway? So I'm sure a ton of us were software engineers before, we're software engineers now, and on the left, we have a code block, and we can sort of, like, read the code, and in my head, I can almost reason over what this code does.

[3:43](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=223s) I can see the functions. I can see, like, how they call each other. I can roughly understand the logic in Python. Um, that doesn't exactly exist in agent world because agents have prompts.

[3:55](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=235s) They have tools. They have skills. They have hooks. They have middlewares. Uh, some agents call other agents and I orchestrate them in swarms. It's really, really hard for humans to reason about how certain prompts that they change are actually going to affect agent behavior at scale.

[4:09](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=249s) And this also varies between the different domains that you're doing it on. So a prompt change and you're using it for the medical domain is gonna be, like, completely different than a prompt change that you wanna do for the law domain.

[4:19](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=259s) And in general, over the last four years since the ChatGPT moment, we've started trading determinism for autonomy. And in that shift, sort of what we need to do is create tools and create systems to still understand agents when they're autonomously operating in environments.

[4:41](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=281s) So I talked about traces. Um, why-- Like, why should you read them? And at LangChain, what do we actually do when we're reading traces? So we centralize a bunch of our data, so we put everything in a tracing project, and this is usually either, like, per agent or, like, centralized across all of our agents.

[4:59](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=299s) And then what we do is we send agents to read traces from other agents, right? And then we look for a bunch of different things. And we might ask for, hey, like, find a bunch of, like, good and bad interactions where, like, users got upset or, like, users are, like, really happy.

[5:15](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=315s) Um, another question I might ask is, uh, this is a technical question, like, agents now run for millions of tokens. Does the agent get really dumb after the first compaction?

[5:25](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=325s) After the second compaction? Does it never get dumb? Like, how do we actually answer those questions? We need to do it by actually looking at the traces. And then the, the other thing is, like, if I look at the traces, then I can try to prove some counterfactuals, which is, hey, like, I ran GPT-5.5 for this,

[5:41](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=341s) and I heard, like, GLM is really good. What happens if I run GLM-5.2 for this task, and how do I compare them? Metrics, awesome. The, the trace level captures the actual, like, behavior that users see, so that's also, like, very helpful for seeing behavior, like fine grain scales.

[5:59](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=359s) And-

[6:00](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=360s) The way that we sort of think about the data that's being generated by agents is that the data that we see today is going to be the smallest that humans have ever seen in their entire lives because we're in this massive exponential shift to agents are doing more and more work in the economy.

[6:18](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=378s) And what that means is, like, the amount of data that humans have produced in our entire lifetime will soon be eclipsed by agents running on, like, year scales, and then six-month scales, then three-month scales, and then maybe every day, right?

[6:32](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=392s) Um, and to understand a ton of that data, roughly what we need to do is contend with a couple problems. There's more, but these are the two that I'm gonna focus on.

[6:41](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=401s) So one, uh, reading traces at scale is super expensive, uh, especially if you have millions of traces and if you have millions of tokens per trace, right? Um, think of it as, like, an input token cost.

[6:55](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=415s) You can, like, literally multiply the input token cost, uh, times the number of traces times, like, how big each trace is on average, right? Um, the, the other thing is if I have a super long interaction with a coding agent like Claude Code or Codex or, like, Deep Agents, um, I can't even read that trace with another agent

[7:13](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=433s) because that, that context, like, doesn't fit in memory, right? So it's like we, we need to develop systems so I can sort of treat that context as, like, an external object, and then I can sort of query into it, right?

[7:24](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=444s) So we, we need to build agents to efficiently mine data from other agents, and it's, it's no longer as simple as just, like, feeding the data into context. And there's, like, tricks that we'll sort of talk about, uh, to, to do that well.

[7:36](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=456s) Great. So one of the things that I think is really, really cool in the last six months is that open models have basically hit an inflection point in intelligence that we at LangChain don't reach for the frontier models for every single use case.

[7:51](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=471s) We're quite conscious about what is the minimum level of intelligence that I need to do any given task. And, like, practically speaking, honestly, yes, we start with Opus, we start with 5.5 because we just wanna know if the task is even possible.

[8:06](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=486s) But then once we reach that sort of, like, waterline, then we, like, look back at those traces and we see, "Hey, can we use an open model to do the same thing?"

[8:15](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=495s) So this is a bunch of work that we did with, uh, Harvey and their lab Legal Benchmark. Basically, what we're looking at is, can I match the trace judging capability of Opus with an open, cheaper model?

[8:28](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=508s) And the answer is roughly yes, at like an order or like two orders of magnitude cheaper. And, like, the way we do that is we try a bunch of models.

[8:38](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=518s) We do a bunch of, like, harness engineering, and the harness engineering is informed by a bunch of the traces that we read. So it's like, hey, like, Opus reasons about things in this way.

[8:46](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=526s) Maybe that's because of the prompt. Uh, maybe Opus is just smarter, which it is, than a bunch of the open models. But that might mean I need to give it a little bit more guidance so it can reach the sort of same intelligence level at, like, a much, uh, much lower cost.

[8:58](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=538s) And the, the other thing that we sort of look at is, like, harness engineering is amazing. Um, you get instant feedback, and you can sort of, like, run on your evals.

[9:08](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=548s) But eventually, what we find is you hit a threshold of intelligence where it's like, if I keep tweaking this prompt, I'm not gonna get too much more out of it.

[9:17](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=557s) And once we reach that point, we sort of look at, okay, can I actually, like, fine-tune the model on my domain-specific task, and can I, like, make it better on those tasks?

[9:27](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=567s) And what we find is if we take, like, base models and we tune them on, like, very specific vertical tasks, which is what a lot of our customers do, um, they don't really care about the entire variance of tasks.

[9:37](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=577s) Like, they care about what their customers care about. So if we focus on that narrow set of tasks, then we can fine-tune base models to sort of, like, reach and then also go beyond frontier performance.

[9:47](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=587s) And I think one sort of, like, small thing I'll mention, uh, as a lot of people are getting into fine-tuning is that another sort of, like, economic decision is that you can move from token costs to hardware costs.

[10:01](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=601s) And, uh, this is, like, can be a really big change, right? 'Cause, like, you're very used to, hey, like, a million tokens cost this much, not as much like this cluster sort of costs this much.

[10:10](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=610s) But for, like, very high inference workloads, we find it to be way cheaper just to, like, run a cluster and I get, like, unlimited inference on that cluster. I don't have to worry about tokens, but I can just do the calculation of like, hey, um, this will end up being cheaper, and then I can spin it down

[10:24](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=624s) when I don't need it. Cool. And I said all of this, um, so we obviously, like, built a product to do that. Uh, I won't show it too much, but it's LangSmith Engine.

[10:36](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=636s) Uh, basically, this product is trying to automate this loop for you, which is if you have any volume of trace data and you're looking for something in that trace data, or you wanna generate evals from that trace data, or you want to, like, generate feedback for, like, humans to read from that trace data, it will go read

[10:53](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=653s) all of it. It will, like, find issues. It'll agentically search over it, and it can, like, prepare data sets for you to do something after. And a bit of a leader, um, what that something basically is, is the outputs of this trace mining exercise.

[11:08](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=668s) So there's like three things that I mention here, uh, which we see a bunch and we kind of put into the product. So one is distillation and fine-tuning, which is, let's say I'm running GLM-5.2.

[11:21](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=681s) It's doing great, but I think that I can run this task, like, way cheaper with like a 9B or a 13B model. Then what I'll do is, like, I'll take the good traces and the good examples from the GLM-5.2 runs.

[11:32](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=692s) I'll prepare them in a data set, and then I'll try to fine-tune a small model on that data set to, like, mimic behavior essentially, right? And this is like distillation SFT.

[11:41](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=701s) The, the other one is generating evals in environments. So maybe another slightly hot take. I think you can basically define agent behavior by showing the evals that you ran on it, right?

[11:53](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=713s) Like if someone showed me all the things that they're trying to test their agent on, I think I would have a rough idea about how that agent is gonna behave because it literally, like, hill climbs those evals and you, you alter the behavior of the agent to make the evals pass, right?

[12:08](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=728s) Like the purpose of evals is roughly to try to make them pass, right? So I update my agents so that they essentially pass. And then the, the other thing is, um, like humans are still in the loop.

[12:19](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=739s) Like I need to know that- Customers are happy. I also want to know what my agents are doing. I just don't have the bandwidth to read a bunch of traces.

[12:27](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=747s) So preparing content for humans is still, like, really, really valuable today, especially in, like, high-trust domains like legal and medical. Like, some human needs to review this, um, but they can't read it all, so we try to make it easy for them to process all that data.

[12:46](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=766s) Great. This is, um, maybe a bit of a throwback. Like, how many people here know what, like, scikit-learn is? Uh, maybe put your-- Oh, sick. This crowd is just awesome.

[12:54](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=774s) Um, cool. So, uh, when I was, like, first doing my PhD, uh, my PhD was, like, kind of trying to do this but, like, add new algorithms to scikit-learn.

[13:05](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=785s) And, like, what scikit-learn basically is at an abstract level, it's a bunch of helpers to fit learning systems to data, right? And, like, classical machine learning, I have, like, a data set, and I try to fit it to it.

[13:19](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=799s) But I think the same principles that we use in modern-- in-- I call it classical machine learning. It was, like, six years ago. Um, that we do in classical machine learning, uh, definitely still apply to this agent-first world.

[13:31](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=811s) Um, the way that they apply is what I like to call model harness task fit. So we still have this sort of like fit function that I'm going to try to, like, take my data, take a harness, take a model, and I'm gonna try to fit it all together to make sure that all of my tasks pass,

[13:51](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=831s) right? The algorithms look slightly different, uh, but the overall process of machine learning doesn't really look that different. And we'll talk about maybe roughly what our job becomes in this data-first, agent-first, fit-first world.

[14:07](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=847s) So a couple of our main jobs now are

[14:11](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=851s) find good fit functions. So these are like autoresearch. This is tons of great work that's being done in RL on different methods like OPD, OPSD, try SFT,

[14:22](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=862s) and also find good data, right? So if you put those two things together, then that is basically the applied or just overall research question that every team has to make their agents better.

[14:33](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=873s) And, like, some, some examples that we've seen that are very popular that we're pretty bullish on are just generally autoresearch. So if you have some sort of score that you can make number go up.

[14:44](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=884s) Uh, agents are pretty good at making that number go up. They might cheat a little bit. You need to, like, check them on some stuff. Um, but this sort of, like, general feedback loop of do something, read the results, read the traces, and then do an update ends up being pretty useful.

[14:59](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=899s) And then I talked about, like, model fine-tuning a bunch as well. Um, so we, we just, like, went and did this. Uh, this was, I think, even before the term autoresearch came out, but a lot of people were doing it, which is, hey, like, Terminal-Bench is, like, really hard.

[15:12](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=912s) Uh, what would happen if an agent just, like, read its traces, uh, proposed experiments, and then tried to do fixes? Um, I think one, like, really key thing here is, uh, giving agents dense feedback signals.

[15:26](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=926s) So, like, Terminal-Bench, the output is just a number, right? Like, did you pass or did you not pass? Uh, that's, like, kinda helpful. But if I gave you, like, a super random task and, like, you just did a bunch of stuff, and then I just said, like, "You failed" or, "You passed," uh, if you failed, like,

[15:39](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=939s) you wouldn't really have a good signal to figure out what you should do next, right? So densifying feedback is a really good way to improve agents. And, like, traces are the substrate that hold that feedback.

[15:51](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=951s) And then agents are very good at, like, reading those, uh, those traces and then figuring out, like, what to do next. Um,

[15:58](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=958s) and then this sort of question always comes up, which is when should I, like, harness eng? When should I fine-tune? Uh, should I do more harness eng after it?

[16:09](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=969s) I'm, like, pretty bullish on the idea of if you need to do something for improving your agent, the best thing that you can do is collect feedback as quickly as possible, like, either from humans labeling or just letting the agents run.

[16:23](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=983s) So, like, harness engineering gives you feedback in maybe two minutes. Um, once you sort of saturate the harness engineering ceiling, right, then you can maybe try to do, like, fine-tuning after that.

[16:34](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=994s) But we find a lot of teams are happy with harness engineering and, uh, it solves their customer use case. So, like, we always sort of recommend it. And then we have this, like, sort of sandwich, which is like try harness engineering, try to do fine-tuning to sort of like break through that ceiling, and then do more harness

[16:48](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=1008s) engineering again if you need to. And then I'll sort of end on the, the idea generally of continual learning is that there's an agent taking actions in the environment, and then it needs to use that information-- Sorry, guys.

[17:06](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=1026s) Needs to use that information to update information about itself, right? So it's like I did a bunch of these tasks and, like, I need to update my prompts to make sure I do them more efficiently.

[17:17](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=1037s) Or users are-- users keep asking to, uh, search for these types of things. I should maybe tell, like, tell my creator that, like, they're, they're doing this sort of stuff, right?

[17:28](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=1048s) It's like taking action in the environment kind of like humans do and updating ourselves. What that looks like today, slightly unclear, but we think that you're going to have to do it across all three axes, which is, one, collect a bunch of training data, which is like observational data from, from agents taking actions.

[17:46](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=1066s) The, the other one is, like, harness updates generally. Like, uh, you know, the, the Codex harness and the Claude Code harness and, like, our harness and everyone's harness, like, they look a certain way because, like, models are, are trained in them, and they look a certain way because of the tasks that they do in the real world.

[18:00](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=1080s) And we think, like, evolving those over time is gonna be super important in, in order to make them work. And the, the last thing is, like, memory. So, uh, we-- humans are, like, really good at, like, remembering stuff over time, but we are not append-only logs of information.

[18:18](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=1098s) And if agents are going to be working with us over, like, year, five-year, decade, lifetime timescales, we cannot just append everything to, like, a really big file and then search over it.

[18:28](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=1108s) There's a ton of stuff that needs to happen with, like, updating those files over time and then just making memory, like, really efficient. But we think a lot of that actually comes from this idea of scaling sleep time compute and, and dreaming generally.

[18:40](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=1120s) So it's like read all of the traces over the entire agent life cycle and then, like, do things to update agent state.

[18:49](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=1129s) Awesome. So, like, quick, quick takeaways. Uh, mining traces gives you signals to hill climb on. Uh, I would say, like, if you have an agent, just turn on tracing and point an agent at it, and that's, like, the easiest thing that you can do to see, like-- to basically understand what your agents are doing.

[19:06](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=1146s) Uh, we're very excited about open models. Uh, we wanna help you fine-tune open models. Um, we provide them as a service as well. So if you're interested in that, would, would love to chat how you can use open models to make everything smarter and cheaper.

[19:18](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=1158s) Um, continual learning is about operating in environments and then integrating that data back into agent state. And finally, I think this is so cool that, like, we have systems that's gonna pro-- that are gonna produce more data than we ever have before.

[19:31](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=1171s) Uh, we need to all come up with, like, interesting research directions to learn how to, like, manage that at scale and, like, make all of our agents better. And with that, thank you.

[19:41](https://www.youtube.com/watch?v=CvRngaQZQ3Y&t=1181s) Thank you all for coming. [audience applauding] [upbeat music]
