# Tokens Should Have Jobs — Katelyn Lesse & Angela Jiang, Anthropic

Katelyn Lesse, Anthropic · Angela Jiang, Anthropic

[Talk page](https://ai.engineer/talks/PXj0p_mW9nI-tokens-should-have-jobs) · [Video](https://www.youtube.com/watch?v=PXj0p_mW9nI)

## Tokens Should Have Jobs

Katelyn Lesse and Angela Jiang show why an agent’s token budget is also an allocation problem: advice, evaluation, and reflection can outperform spending the same allowance entirely on execution.

### A token budget hides an allocation decision

[0:19](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=19s)

Katelyn Lesse, who leads platform engineering at Anthropic, and Angela Jiang, who leads platform product, start with the familiar way teams improve an agent: give it more tokens or more expensive tokens. That treats budget as the main lever and assumes every token makes an interchangeable contribution to the result.

A conventional agent receives a task and spends its allowance continuing the work. “Tokens should have jobs” reframes that allowance as compute that can serve different functions. Some tokens can execute the task, while others check the approach, judge an attempt, or preserve a lesson for later. A particular allocation of those roles becomes a **strategy**.

This does not make individual tokens intrinsically different. The difference comes from the prompts, context, agent roles, and control flow that determine what the model does with them. The practical question changes from “How large is the budget?” to “What work should this budget buy?”

### Advice, grading, and dreaming improve different moments

[2:16](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=136s)

The three strategies intervene at different points in an agent’s lifecycle:

- **Advising — during execution:** An executor can ask a separate adviser whether its next step makes sense, then use that response to adjust its work. A sales agent, for example, might use advice while checking overdue follow-ups and stalled deals.
- **Grading — after an attempt:** A grader compares completed work with an explicit rubric. A passing attempt finishes; a failing attempt returns to the executor for another iteration.
- **Dreaming — between runs:** A dreamer inspects the executor’s work and transcript, extracts findings, and writes them to memory for the next run.

Grading works best when “good” can be stated clearly. Consider a customer asking a store for a refund. The executor drafts or selects a response; the rubric encodes the store’s refund criteria; the grader checks the proposed outcome against those criteria; and a failed check triggers another attempt. The loop makes policy the reference point, although its reliability still depends on whether the rubric captures the right rules and the grader applies them correctly.

Dreaming targets repeated work rather than the current answer. In the recruiting example, feedback about candidate fit accumulates across interactions. A dreamer turns findings from those interactions into persistent memory, and the next execution reads that memory before proceeding. The described improvement comes from changing future context, not from updating model weights. The talk does not specify how memories are selected, corrected, or retired, so those remain implementation problems.

### The first benchmark confounds strategy with spending

[4:28](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=268s)

The experiment moves these strategies into a benchmark of financial-analysis tasks intended to resemble work performed by an expert analyst. Execution alone serves as the control; advising, grading, and dreaming are alternative ways to organize the work.

The first one-shot comparison cannot isolate the value of those jobs. Execution scores 15% while using 39,000 tokens, whereas more complex strategies spend more and perform better. Dreaming reaches roughly 600,000 tokens. Strategy and compute have both changed, so the result cannot tell whether the improvement came from orchestration, additional spending, or both.

The next comparison fixes a common maximum allowance at roughly 600,000 tokens. More test-time compute improves every strategy: execution rises from 15 to 76, while advising and grading move from scores in the 60s toward the 90s. The revealing comparison is within the shared allowance: execution scores 76 and advising scores 89. The same budget produces a different outcome when some of it funds consultation rather than continuation.

That reported gap motivates strategy design, but its scope is uncertain: the presentation does not provide the number of tasks, model configurations, repeated-run variance, or statistical uncertainty needed to establish how broadly advising will outperform execution. The supported result is narrower—on this financial-analysis benchmark, allocation mattered under the fixed allowance.

### An 80% accurate P&L is still unfinished work

[6:44](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=404s)

The benchmark score still leaves a product question unanswered: can an analyst use the result? Suppose the agent produces a profit-and-loss statement that is 80% accurate. That sounds respectable as partial credit, but the analyst cannot safely accept invented or incorrect income and cost figures. They must recompute the statement or pay for another run. Observable progress on the benchmark has not produced a finished P&L.

The experiment is therefore rescored around usable completion. A perfectly scored task passes; anything below 100% fails. This converts the metric from average partial accuracy into the probability that one run returns a fully correct answer. The threshold fits the financial tasks being discussed and should not be assumed for applications where partial results retain value.

Under that criterion, execution passes about 42% of runs, while the more complex strategies reach as high as 75%. A failed run now has an explicit business consequence: its tokens count toward the cost of obtaining the answer, even though its output cannot be used.

The talk rounds execution’s pass rate to about 40% and budgets approximately three attempts. Assuming 600,000 tokens per attempt, that produces the headline estimate of 1.8 million tokens for a perfect answer: `3 × 600,000 = 1,800,000`. This is a practical three-run estimate, not a guarantee or a precise expected-value calculation.

Applying retry cost across the strategies changes the decision. Advising and grading are described as comparatively token efficient because extra work inside a run can reduce the need to repeat the whole task. If cost per usable result is the priority, the recommendation for this domain is advising. If single-run reliability matters more, grading or dreaming becomes more attractive. Token efficiency and per-run reliability are related, but they are not the same objective.

### A meta-harness composes the jobs into one workflow

[10:28](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=628s)

The implementation separates two layers. An individual agent harness supports execution. Above it, a **meta-harness** coordinates the executor, adviser, grader, dreamer, and any other participating roles. The lower layer makes one agent capable of doing work; the upper layer decides how multiple kinds of work fit together.

![Recording frame at 652 seconds](https://ai.engineer/api/talk-essay-media/d1690f014c0f9b81038b508ce02dcd3b6e85d688d906fa4bb1b2f38814015173)

Recording frame at 652 seconds

The concrete lower layer in the presentation is **Claude Managed Agents**, Anthropic’s managed-agent offering on the Claude platform. Jiang says its architecture supplies the individual-agent harness, while orchestration above it coordinates the roles in a strategy. She also says some capabilities—including dreaming—are available there out of the box. The talk does not detail the harness’s internal components or provide an implementation guide for those managed capabilities.

What does the composed control flow look like? A task enters the executor, which can consult an adviser while working. The result goes to a grader. A failure returns to execution; a pass proceeds to dreaming, where findings are written to memory for the next run.

The relationship visible in this workflow is temporal: advice changes the current attempt, grading governs whether that attempt may finish, and dreaming changes future attempts. Combining them can cover all three moments, but the talk presents this composition as an architectural construction rather than a separately benchmarked result.

These roles are primitives, not a closed taxonomy. Builders can introduce new jobs and more complex coordination patterns for their own tasks. The longer-term goal is for models and the platform to construct strategies dynamically as work proceeds. That automatic construction is presented as future direction; meanwhile, builders can combine the available primitives and evaluate each strategy against the outcome, reliability target, and cost that matter for the application.

How advising, grading, and dreaming compose
Advice affects work in progress, grading controls retries, and dreaming carries findings into the next run.
- Task — The requested outcome enters the strategy.
- Executor — Performs the task and can request advice.
- Adviser — Checks or informs the executor’s next step.
- Grader — Evaluates the result and decides whether it passes.
- Dreamer — Extracts findings after accepted work.
- Memory — Supplies retained findings to the next execution.
- Task → Executor (begin)
- Executor → Adviser (request advice)
- Adviser → Executor (inform next step)
- Executor → Grader (submit result)
- Grader → Executor (fail: iterate)
- Grader → Dreamer (pass)
- Dreamer → Memory (write findings)
- Memory → Executor (inform next run)

## Key takeaways

- Treat an agent budget as an allocation across jobs, not merely a quantity of execution tokens.
- Advice, grading, and dreaming intervene at different times: during an attempt, after an attempt, and between runs through memory.
- Control the token allowance when comparing strategies. On the reported benchmark, execution scored 76 and advising scored 89 under the same roughly 600,000-token maximum.
- Measure the outcome the user can actually use. For the P&L example, anything short of a perfectly scored answer still requires correction or another run.
- Choose a strategy for the objective: advising is favored here for token efficiency, while grading or dreaming is favored for single-run reliability.
- Claude Managed Agents supplies the concrete individual-agent layer described in the talk; a meta-harness composes roles above it, while automatic strategy construction remains a longer-term goal.

## Resources

- [Building an Ecosystem, not a Walled Garden](https://sequoiacap.com/podcast/anthropics-katelyn-lesse-angela-jiang-building-an-ecosystem-not-a-walled-garden) — A longer conversation with Lesse and Jiang that places strategies and meta-harnesses within Anthropic’s knowledge, execution, and coordination layers.
- [Evolving Claude APIs for Agents](https://ai.engineer/talks/aqW68Is_Kj4-evolving-claude-apis-agents) — Lesse explains the lower-level agent primitives—reasoning budgets, tools, memory, context editing, and sandboxed execution—on which higher-level strategies can be built.

## Related talks

- [Claude for long-horizon tasks](https://ai.engineer/talks/9QebvrrY3KY-claude-long-horizon-tasks) — A related topic for readers considering how these strategies might apply to longer tasks.
- [How We Build Effective Agents](https://ai.engineer/talks/D7_ipDqhtwk-effective-ai-agents) — A companion topic for readers exploring agent design beyond token allocation.
- [12-Factor Agents: Patterns of reliable LLM applications](https://ai.engineer/talks/8kMaTybvDUw-12-factor-agents) — A related discussion of reliability patterns, complementing this talk’s distinction between efficiency and single-run success.

## Slides

### Executor calls out to Advisor for help

The executor calls an advisor for advice, then uses it to assess whether its next step is correct.

[2:14](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=134s) [Execute and Advise boxes exchange arrows.](https://assets.ai.engineer/sites/aiecode2025/sha256/551b47fdfa2ed237139c54d8ff050d458c647ecfa49b5282207210b9f6447f1a.webp)

Advise   Grade   Dream

Executor calls out to Advisor for help

Execute
Advise

### Advise: Sales example

In a sales-agent example, Advise flags a quiet or stalling deal so the sales rep can follow up.

[2:29](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=149s) [A quiet deal is flagged by Advise, then the sales rep follows up.](https://assets.ai.engineer/sites/aiecode2025/sha256/119510f1fbe58ad9192d3f9efb4dac23b36f33fc64ebdb79476e8fb3271456fe.webp)

Advise   Grade   Dream

Sales

Flags when a follow-up is overdue or a deal is stalling.

Deal goes quiet → Advise flags it → Rep follows up

### Define a rubric and grade each attempt

A grader checks each execution against a rubric. A pass ends the loop; a failure sends the executor back for another attempt.

[2:47](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=167s) [Execute leads to Grade, which branches to Pass or Fail; failure loops back to Execute.](https://assets.ai.engineer/sites/aiecode2025/sha256/de291f4fa7cd88c6d35841183bcd9532ad0a58f4e98ed9e60b8e0de53e7e2106.webp)

Define a rubric and grade each attempt

Execute
Grade
Fail
Pass

Advise   Grade   Dream

### Grade: Support example

A grader checks a customer service agent’s drafted reply against a rubric based on refund criteria before the reply goes out.

[3:16](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=196s) [A drafted reply is checked against refund rules before it goes out.](https://assets.ai.engineer/sites/aiecode2025/sha256/2af3c95b780bfa216c9d2c873aaf621115999fc1cfad9e63bf2c597adb970a6b.webp)

Advise   Grade   Dream

Support

A drafted reply is checked against refund rules.

Reply drafted → Grade checks it → Reply goes out

### Dream writes what it learned to memory

The dreamer inspects the executor’s work and transcripts, then writes its findings to memory for the next execution.

[3:47](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=227s) [Execute leads to Dream and then another Execute step; Dream also writes to Memory, which feeds the later execution.](https://assets.ai.engineer/sites/aiecode2025/sha256/5cdc867300ac1dd03767244f70ffa381225e7e60d870608f48782fa79c4b02d6.webp)

Dream writes what it learned to memory

Execute → Dream → Execute
Dream → Memory → Execute

Interface text:
Advise   Grade   Dream

### Dream: Recruiting example

A recruiting agent could review feedback from a completed hiring round to sharpen the next candidate screen.

[4:07](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=247s) [Dream reviews a completed hiring round to sharpen the next screen.](https://assets.ai.engineer/sites/aiecode2025/sha256/1f81d82d1dca913002f337d2400ecbe439f7a9f6bc7841fe8fe1874008dbe0b2.webp)

Advise
Grade
Dream

Recruiting

Reviews each hiring round, sharpens the next screen.

Hiring round ends → Dream reviews it → Next screen sharper

### Performance on complex financial tasks

The speakers introduce a benchmark of financial analysis tasks modeled on expert human work, comparing an execute-only control with other strategies.

[4:27](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=267s) [Experiments section title: Performance on complex financial tasks.](https://assets.ai.engineer/sites/aiecode2025/sha256/06d73b449d1b2cd24828bec52eda36bce6bda1c9e0a0fd19c74bf42e869d58e6.webp)

Experiments

Performance on complex financial tasks

### Performance by strategy with open budget

Dream scores 0.92 versus Execute’s 0.15, but uses 613k tokens versus 39k. With each strategy choosing its own spend, the comparison doesn’t isolate the effect of strategy from token budget.

[4:59](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=299s) [Open-budget scores and token counts: Execute 0.15 and 39k, Grade 0.63 and 142k, Advise 0.65 and 152k, Dream 0.92 and 613k.](https://assets.ai.engineer/sites/aiecode2025/sha256/8330bcf99e5c69093e8ff31c6ce137ab7a4c7f5ac22d39c22f5423603047a129.webp)

Performance by strategy with open budget
Different strategies can meaningfully hill climb performance but also spend more total tokens

Execute: 0.15, 39k tok
Grade: 0.63, 142k tok
Advise: 0.65, 152k tok
Dream: 0.92, 613k tok

All runs use Opus 4.8

### Performance by strategy under fixed budget

With the same roughly 600,000-token maximum, Execute scores 0.76, Advise 0.89, and Grade and Dream 0.92. Strategy still matters when budget is held constant.

[5:41](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=341s) [Fixed versus open-budget scores: Execute 0.76 versus 0.15, Advise 0.89 versus 0.65, Grade 0.92 versus 0.63, and Dream 0.92 versus 0.92.](https://assets.ai.engineer/sites/aiecode2025/sha256/678b58724d987bd2e997ca9ff011b651a752d4cd486930421aca1de623e25cc5.webp)

Performance by strategy under fixed budget

When budget is held constant, varying strategy still results in performance gains

Fixed budget / Open budget:
Execute: 0.76 / 0.15
Advise: 0.89 / 0.65
Grade: 0.92 / 0.63
Dream: 0.92 / 0.92

### Pass and fail criteria

For these financial tasks, a run passes only at 100% accuracy. The speaker notes that an 80%-accurate P&L can leave an analyst recomputing the result or running the agent again.

[6:42](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=402s) [Pass requires 100%; less than 100% is a fail.](https://assets.ai.engineer/sites/aiecode2025/sha256/23c2943eb9c54f68f6c471fb00d320f114129bfe6cc272dc3300c237b0b3835d.webp)

Pass
100%

Fail
less than 100%

### Percent of runs passed under fixed budget

Under a fixed budget, Execute passed 42% of runs, while Grade and Dream each passed 75%. A pass required 100% accuracy.

[7:55](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=475s) [Pass rates: Execute 42%, Advise 67%, Grade 75%, and Dream 75%.](https://assets.ai.engineer/sites/aiecode2025/sha256/d1296cba7985a00e011ad2f082171e22434822e56fb349d3599a2f1ba6b87903.webp)

Percent of runs passed under fixed budget

Execute: 42%
Advise: 67%
Grade: 75%
Dream: 75%

### Cost of one perfect answer

For the Execute strategy, three expected runs averaging 591k tokens each come to roughly 1.8M tokens to reach a perfect answer.

[8:33](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=513s) [Execute strategy: 3 expected runs multiplied by 591k average tokens per run equals 1.8 million tokens for a usable answer.](https://assets.ai.engineer/sites/aiecode2025/sha256/1e8a56c3712ac8f91a1a71f24696addcf596e4ef5b0697a6e4128cd6fb011867.webp)

Cost of one perfect answer

Execute strategy

3 × 591k = 1.8M

3: expected runs to get a perfect answer
591k: average tokens per run
1.8M: real tokens to get to a usable answer

### Perfect answers and what they cost

Advise has the lowest estimated cost at 1.2M tokens, followed by Grade at 1.3M and Dream at 1.7M. Execute costs 1.8M, with three expected runs versus two for each other strategy.

[9:19](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=559s) [Bar chart of tokens to reach a perfect answer: Advise 1.2M, Grade 1.3M, Dream 1.7M, and Execute 1.8M.](https://assets.ai.engineer/sites/aiecode2025/sha256/a3fc76942b90045de4deec8a056ef85b2f69f480c068b59acd2981aed09e9b45.webp)

Perfect answers and what they cost

We determine total token cost as how many tokens spent to achieve a perfect answer

Advise: 1.2M, 2 runs
Grade: 1.3M, 2 runs
Dream: 1.7M, 2 runs
Execute: 1.8M, 3 runs

### Cost and reliability of perfect answers

Advise reaches a perfect answer at the lowest token cost, 1.2M. Grade and Dream each pass 75% of runs under a fixed budget, making them the reliability-focused choices.

[9:49](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=589s) [Two charts compare token cost and fixed-budget pass rates. Advise has the lowest cost; Grade and Dream have the highest pass rates at 75%.](https://assets.ai.engineer/sites/aiecode2025/sha256/d85a25f4a5c2c27ced57b4ebbce4a801627fb7f2f7fd39dee75dd5521a3f45f6.webp)

Perfect answers and what they cost

Tokens to get to a perfect answer
Advise: 1.2M, 2 runs
Grade: 1.3M, 2 runs
Dream: 1.7M, 2 runs
Execute: 1.8M, 3 runs

Percent of runs passed under fixed budget
Execute: 42%
Advise: 67%
Grade: 75%
Dream: 75%

### Tokens are not fungible

Assigning tokens different jobs and trying different strategies can improve task outcomes within a fixed budget.

[10:27](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=627s) [Centered text reads, “Tokens are not fungible.”](https://assets.ai.engineer/sites/aiecode2025/sha256/739f0707a3eba949d369e8e7bcd0dabf43e3d0af5b1d62397fc7eda15e61e2c3.webp)

Tokens are
not fungible.

### Coordination and execution harnesses

The coordination harness connects an executor and adviser. Beneath it, the execution harness connects an individual agent to tools, a session, a sandbox, and orchestration.

[10:50](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=650s) [A coordination harness connects Execute and Advise. An execution harness connects an agent to tools, session, sandbox, and orchestration.](https://assets.ai.engineer/sites/aiecode2025/sha256/4f5d2f902ea41a49d71e060d7ae240c93421378b355c900f9036717ef2091d36.webp)

Coordination Harness
Execute
Advise

Execution Harness
Tools
Session
Sandbox
Orchestration

### Execute, advise, grade, and dream

Sonnet executes with advice from Fable. Haiku grades the result, returning failures to the executor and sending passes to Opus for dreaming to improve the next run.

[11:33](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=693s) [Execute with Sonnet exchanges information with Advise using Fable, then Grade with Haiku loops failures back to Execute and sends passes to Dream with Opus.](https://assets.ai.engineer/sites/aiecode2025/sha256/b5c36003c4d399ace4b76a2250dd6aa5494766ec0fc3195a9ea2fbabbb670746.webp)

Execute
Sonnet

Advise
Fable

fails

Grade
Haiku

passes

Dream
Opus

### Segmented execution strategy

Opus segments work by difficulty: Sonnet executes hard tasks with Opus advising, while Haiku executes easy tasks. Haiku grades both; failures loop back, and passes go through Sonnet reconciliation to Opus Dream.

[12:11](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=731s) [Segment with Opus routes hard and easy work through separate execution and grading paths, then Reconcile with Sonnet and Dream with Opus.](https://assets.ai.engineer/sites/aiecode2025/sha256/2ef6ed993453e837e64b9fef25c5f85c4218cf85f23e493e62d2ad8aa0830eb7.webp)

Segment
Opus

hard
Execute
Sonnet
Advise
Opus
Grade
Haiku
fails
passes

easy
Execute
Haiku
Grade
Haiku
fails
passes

Reconcile
Sonnet
Dream
Opus

### Give tokens jobs

A chessboard illustrates the idea: give tokens jobs and combine primitives into strategies for each task. The speakers aim to improve dynamic strategy construction over time.

[12:35](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=755s) [Give tokens jobs and create novel strategies, above an illustrated chessboard.](https://assets.ai.engineer/sites/aiecode2025/sha256/5fccee7e174ac3bfd1856b6089a762732828c70f23cdeea24378d62e3e549d4e.webp)

Give tokens jobs
and create novel strategies

### Thank you

[12:58](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=778s) [Thank you on a dark background.](https://assets.ai.engineer/sites/aiecode2025/sha256/1649cbe34f37dbdd837ff2e145b925eb22027d9a24b0f589bd15aed87dc08d0d.webp)

Thank you.

### AI Engineer World’s Fair

[13:14](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=794s) [Black closing card with the AI Engineer World’s Fair logo and website address.](https://assets.ai.engineer/sites/aiecode2025/sha256/16fa90eea42ff4962f5a7b27c2853a2e06e920eeac8a45a486b6e2aa717b6123.webp)

AI Engineer
World’s Fair

HTTPS://AI.ENGINEER

## Full transcript

[0:19](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=19s) Good morning. We're super excited to be

[0:22](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=22s) here at AI Engineer with all of you. I'm

[0:25](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=25s) Caitlyn and I lead platform engineering

[0:26](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=26s) at Anthropic.

[0:28](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=28s) >> And I'm Angela. I lead platform product

[0:29](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=29s) at Anthropic. And today we want to talk

[0:31](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=31s) to you about a concept that we've been

[0:33](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=33s) spending a lot of time thinking about

[0:35](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=35s) and working on with our team, which is

[0:37](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=37s) this idea that we think that tokens

[0:39](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=39s) should have jobs.

[0:41](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=41s) So if you're building an agentic system

[0:43](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=43s) and you're trying to accomplish some

[0:45](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=45s) specific outcome, you're trying to get

[0:46](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=46s) something done with agents, there's one

[0:49](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=49s) lever that everybody pulls in order to

[0:50](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=50s) get a better outcome, and that's usually

[0:53](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=53s) increasing your budget, which means you

[0:54](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=54s) spend more tokens or you spend more

[0:56](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=56s) expensive tokens.

[0:59](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=59s) But we've been wondering is that all

[1:01](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=61s) there is underlying this assumption of

[1:05](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=65s) uh using the budget is this kind of

[1:06](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=66s) implicit perspective that every single

[1:09](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=69s) token is basically fungeable. And we've

[1:11](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=71s) been wondering is that actually true?

[1:13](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=73s) Are all these tokens actually fungeible?

[1:15](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=75s) And to test that, we've been thinking,

[1:17](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=77s) what if we gave tokens jobs?

[1:20](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=80s) So, if you think about the way that you

[1:21](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=81s) would normally set up an agent to go

[1:23](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=83s) accomplish a task, you give it that

[1:24](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=84s) task, you give it this token budget, and

[1:26](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=86s) then all the tokens that are being spent

[1:28](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=88s) are basically indiscriminate in the

[1:30](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=90s) sense that they're all doing one job.

[1:31](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=91s) They're just executing.

[1:34](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=94s) But what if you take some of those

[1:35](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=95s) tokens and they're not just executing,

[1:37](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=97s) they're doing some other job. So, for

[1:40](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=100s) example, maybe you take some of your

[1:42](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=102s) tokens and they're advising the tokens

[1:44](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=104s) that are executing. Or maybe the tokens

[1:47](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=107s) that are executing try to get something

[1:49](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=109s) done well and you take some other tokens

[1:51](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=111s) and you actually grade how well the

[1:52](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=112s) executor is doing so that it can iterate

[1:54](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=114s) and try again. Or maybe you have tokens

[1:58](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=118s) that are dreaming. They're reflecting

[1:59](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=119s) back on the job that other executors

[2:02](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=122s) have done and writing learnings to

[2:04](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=124s) memory so that they can do it again. And

[2:07](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=127s) what we call each of these if you take

[2:09](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=129s) some tokens that are executing and some

[2:10](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=130s) tokens that are doing some other job.

[2:12](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=132s) Let's call this a strategy.

[2:16](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=136s) So let's go take a look at the first

[2:17](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=137s) strategy, the advising strategy. Here

[2:19](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=139s) we're splitting up an executor and an

[2:21](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=141s) adviser. The executor obviously

[2:23](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=143s) executes, but crucially they can call

[2:25](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=145s) out to an adviser for advice. And then

[2:27](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=147s) they can take this advice and figure out

[2:28](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=148s) if they're doing the next step

[2:29](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=149s) correctly.

[2:31](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=151s) This is really helpful in use cases. For

[2:33](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=153s) example, if you're building a sales

[2:34](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=154s) agent, in an ideal world, you'd have

[2:36](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=156s) that sales agent be able to actually

[2:37](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=157s) help the sales rep flag when a follow-up

[2:39](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=159s) is overdue or deal is stalling. In this

[2:42](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=162s) construct, having an adviser to be able

[2:44](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=164s) to kind of make sure that all the

[2:45](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=165s) different pieces are actually working is

[2:47](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=167s) really helpful.

[2:49](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=169s) So another example is grading. Let's say

[2:51](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=171s) you're executing and you kind of know

[2:54](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=174s) exactly what good really does look like.

[2:56](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=176s) You can define this in a rubric and then

[2:58](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=178s) each time an executor tries to

[3:01](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=181s) accomplish that outcome, you can have a

[3:03](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=183s) grader provisioned that grades how well

[3:05](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=185s) the executor did while looking at that

[3:07](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=187s) rubric. And if the executor did a good

[3:10](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=190s) job, then great, it can be done. But if

[3:12](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=192s) it didn't do such a great job, you can

[3:13](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=193s) iterate again until you get that good

[3:15](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=195s) outcome.

[3:17](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=197s) So an example in practice of when you

[3:19](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=199s) might want to use this is let's say you

[3:20](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=200s) have a customer service agent and you're

[3:22](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=202s) running a store and your customers are

[3:24](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=204s) writing in and they're saying, "H, I

[3:26](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=206s) should get a refund for this thing." And

[3:27](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=207s) your customer service agent needs to be

[3:29](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=209s) able to respond. You probably have some

[3:31](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=211s) like pretty specific criteria on when

[3:33](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=213s) you would give somebody a refund. And so

[3:36](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=216s) what you can do is define a rubric that

[3:38](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=218s) uses that criteria. You can have a

[3:40](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=220s) grader that goes and looks at the work

[3:42](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=222s) that the customer service agent is doing

[3:44](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=224s) and decide is it getting it right and is

[3:46](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=226s) it coming to the right outcome.

[3:49](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=229s) And the last strategy we have is

[3:50](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=230s) dreaming. So in dreaming there's an

[3:52](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=232s) executor who naturally executes and then

[3:54](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=234s) there's a dreamer. The dreamer is

[3:56](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=236s) actually able to inspect the work and

[3:58](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=238s) the transcripts of the executor and then

[4:00](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=240s) it takes any of the findings that it has

[4:02](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=242s) and it writes them to memory. This

[4:03](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=243s) memory is repicked up by the executor

[4:05](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=245s) for the next round. So ideally would

[4:07](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=247s) have improved.

[4:09](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=249s) A great use case for this is if you're

[4:10](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=250s) building a recruiting agent. Now

[4:12](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=252s) recruiting requires a lot of interaction

[4:14](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=254s) with feedback on whether or not a

[4:15](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=255s) candidate does or doesn't make sense and

[4:17](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=257s) if it's a good fit between both parties.

[4:19](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=259s) And so by taking all this type of data,

[4:20](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=260s) if you build a dreaming type of strategy

[4:22](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=262s) on this agent, it's actually able to

[4:24](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=264s) kind of sharpen the next round so that

[4:25](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=265s) it's more and more increasingly useful.

[4:28](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=268s) So let's make this concrete with some

[4:30](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=270s) experiments. So what we did was we

[4:33](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=273s) created a bench of a bunch of tasks

[4:35](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=275s) related to financial an analysis. And

[4:37](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=277s) what we were doing with each of these

[4:38](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=278s) tasks is trying to replicate in the real

[4:40](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=280s) world a expert human financial analyst.

[4:43](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=283s) How well would they do on each of these

[4:45](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=285s) various tasks? And so what we did was we

[4:47](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=287s) start with a control that's just

[4:49](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=289s) executing. Let's try each of these tasks

[4:51](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=291s) and we'll eval them when we're literally

[4:52](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=292s) just executing. But then we can

[4:54](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=294s) experiment with each of our strategies

[4:56](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=296s) and see how well we perform.

[5:00](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=300s) So, we start with a super basic

[5:01](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=301s) experiment. Let's just oneshot it. Let's

[5:03](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=303s) take each of our strategies and we'll go

[5:05](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=305s) and just make an attempt to accomplish

[5:07](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=307s) these tasks and we'll see how accurate

[5:09](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=309s) we are. And so, you can see here with

[5:11](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=311s) executing um it didn't do so well. 15%

[5:14](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=314s) accuracy, but because it was just a

[5:16](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=316s) oneshot, the strategy got to choose how

[5:18](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=318s) many tokens it would actually spend on

[5:20](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=320s) its own. And so, you can actually see

[5:22](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=322s) that execute decided not to spend that

[5:24](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=324s) many tokens, only 39,000. And as we go

[5:27](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=327s) into our larger strategies, our more

[5:29](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=329s) complex strategies, we did choose to

[5:31](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=331s) spend more tokens, but we did a better

[5:32](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=332s) job. So, this isn't really telling us

[5:34](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=334s) much because sure, Drain did really,

[5:36](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=336s) really well, but it used a whopping

[5:38](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=338s) 600,000 tokens to get there. That's

[5:40](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=340s) right. So, in order to actually figure

[5:42](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=342s) out if varying the jobs produces any

[5:45](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=345s) alpha, what we need to do is hold the

[5:46](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=346s) budget constant. And to do this, we're

[5:48](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=348s) going to take Dreaming's budget, that

[5:49](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=349s) 600,000 or so, as the maximum budget

[5:51](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=351s) that is fixed across the board. And we

[5:53](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=353s) give every single strategy this budget

[5:55](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=355s) in order to analyze how well it's

[5:57](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=357s) performing. And as expected again, if

[6:00](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=360s) you give a lot of strategies more

[6:01](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=361s) budget, you are going to see performance

[6:03](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=363s) increase across the board. So execute

[6:04](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=364s) went from 0.15 to 76. Advise and grade

[6:08](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=368s) went from the 60s to closer to the 90s.

[6:10](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=370s) And that's again expected given the fact

[6:12](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=372s) that if you give things more test time

[6:14](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=374s) compute, they should generally perform

[6:16](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=376s) better. But if that was the only thing

[6:18](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=378s) that mattered, we should actually expect

[6:20](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=380s) to see execute, advise, grade, dream

[6:22](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=382s) actually all be at the exact same level

[6:24](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=384s) given the exact same token budget. But

[6:26](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=386s) what we're actually seeing is that there

[6:28](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=388s) is an alpha or there is a difference and

[6:30](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=390s) therefore an alpha for us to exploit. If

[6:32](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=392s) you look at execute at this exact same

[6:34](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=394s) budget level, it gets to 76 but advise

[6:37](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=397s) is at 89. So while a minimal, it does

[6:40](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=400s) exist and so there is alpha for us to

[6:41](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=401s) take a look at.

[6:44](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=404s) Now, we decided to take a look at this

[6:45](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=405s) analysis from a completely different

[6:47](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=407s) lens. And as Kayla mentioned, you know,

[6:49](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=409s) we're doing this bench for a very

[6:51](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=411s) complex set of financial tasks in the

[6:53](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=413s) real world. And we wanted to analyze the

[6:56](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=416s) usage of agents with actual experts. So,

[7:00](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=420s) if we look at a financial analyst

[7:02](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=422s) expert, right, the kind of task that

[7:03](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=423s) they need to do with an agent is that

[7:05](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=425s) they're giving it something very

[7:07](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=427s) concrete like let's say make a P&L and

[7:09](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=429s) then they're getting the result back.

[7:11](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=431s) Now if that result is 80% accurate on a

[7:14](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=434s) bench that sounds great but in reality

[7:16](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=436s) what that means for that expert is they

[7:18](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=438s) have to go back and recomputee that P&L

[7:20](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=440s) themselves or alter or alternatively

[7:22](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=442s) send it through another run and that's

[7:24](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=444s) because in this kind of domain for this

[7:25](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=445s) kind of task if you're not 100% accurate

[7:28](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=448s) it's actually not useful.

[7:31](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=451s) You cannot make up an income number or

[7:32](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=452s) you can't make up a cost number right

[7:34](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=454s) you have to make sure that it's 100%

[7:35](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=455s) accurate. So with this lens of the real

[7:37](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=457s) world consequence associated with this

[7:39](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=459s) domain, we needed to recomputee our

[7:40](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=460s) experiments and score them a bit

[7:42](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=462s) differently. Crucially, we needed to

[7:44](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=464s) make sure that our experiment had this

[7:46](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=466s) kind of construct where if it was scored

[7:48](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=468s) perfectly, we'd actually give it a pass.

[7:50](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=470s) And if it scored anything less than 100%

[7:52](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=472s) on that kind of task, we would actually

[7:53](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=473s) mark it as a failure.

[7:56](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=476s) So let's look at a different cut of our

[7:58](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=478s) data from our experiments with this lens

[8:00](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=480s) where we're looking for this perfect run

[8:02](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=482s) 100% accuracy pass. And let's look at

[8:05](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=485s) what percent of the time each of these

[8:07](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=487s) strategies was able to achieve a pass.

[8:09](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=489s) Um so we we've got executes um down at

[8:12](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=492s) 42% and we've got our more complex

[8:14](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=494s) strategies doing a bit better up to 75%

[8:17](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=497s) accuracy. Um and again this doesn't

[8:20](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=500s) necessarily tell us a ton because um you

[8:22](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=502s) know each of these strategies might um

[8:25](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=505s) choose to use different budgets over

[8:26](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=506s) time, right? So what we did here was we

[8:28](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=508s) fixed the budget and we said within a

[8:30](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=510s) fixed budget, how well do each of these

[8:32](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=512s) strategies perform?

[8:35](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=515s) And so what really matters to us

[8:37](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=517s) actually is if you're trying to get this

[8:39](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=519s) perfect answer and you're in the real

[8:41](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=521s) world, you're running a business, what

[8:42](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=522s) matters to you is the cost to you to get

[8:45](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=525s) to that perfect answer. And so one way

[8:47](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=527s) we can think about this is we had our

[8:49](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=529s) execute strategy for example. The

[8:50](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=530s) execute strategy around 40% of the time

[8:53](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=533s) will give you that perfect answer. So on

[8:55](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=535s) average, you can expect to have to run

[8:56](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=536s) it three times and you should hopefully

[8:58](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=538s) sometime in those three runs get a

[9:00](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=540s) perfect answer. And as we talked about

[9:02](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=542s) earlier, we fixed our budget to that

[9:04](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=544s) highest token budget strategy, which was

[9:06](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=546s) 600,000 tokens. So if you spend 600,000

[9:09](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=549s) tokens in each individual run, you have

[9:11](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=551s) to run approximately three times. You

[9:13](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=553s) can expect on average to have to spend

[9:15](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=555s) 1.8 million tokens with the execution

[9:17](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=557s) strategy to get to your perfect answer.

[9:21](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=561s) And so if we take this analysis and run

[9:23](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=563s) it across the board against all these

[9:24](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=564s) strategies, this is actually the true

[9:26](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=566s) cost it took in this domain for that

[9:29](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=569s) agent to be useful for that strategy. So

[9:31](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=571s) as Caitlyn mentioned for execute, which

[9:33](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=573s) is our baseline, this is going to be 1.8

[9:35](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=575s) million true total token cost for you.

[9:38](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=578s) But advise, grade, and dream are showing

[9:40](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=580s) us a bit of difference. Crucially,

[9:42](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=582s) advise and grade are actually quite

[9:44](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=584s) token efficient when you think about the

[9:46](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=586s) actual usage of the end output of each

[9:48](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=588s) of these agents.

[9:51](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=591s) So what does this mean for you as a

[9:52](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=592s) business? Well, it actually really

[9:54](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=594s) depends on what kind of thing you want

[9:56](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=596s) to optimize for and it's going to vary,

[9:58](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=598s) right? There's going to be businesses

[10:00](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=600s) who say, "Actually, for me, the most

[10:01](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=601s) important thing is to be really token

[10:03](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=603s) efficient. In that case, you should

[10:05](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=605s) probably pick the advised type of

[10:06](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=606s) strategy in order to solve for that

[10:08](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=608s) particular domain in which you want to

[10:09](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=609s) optimize that." There's going to be

[10:10](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=610s) other areas or other businesses where

[10:12](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=612s) you're going to say, I'm not going to

[10:14](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=614s) care so much about token efficiency

[10:15](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=615s) because what I really care about is

[10:16](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=616s) reliability of that answer and so I need

[10:19](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=619s) to maximize the percentage of runs in

[10:20](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=620s) which I get that perfect answer. In

[10:22](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=622s) which case, you would actually pick

[10:23](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=623s) completely different strategies. You

[10:24](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=624s) probably lean towards grade or dream.

[10:28](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=628s) So if you take away one thing, the thing

[10:30](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=630s) we want everyone to think about is this

[10:31](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=631s) idea that tokens are not fungeible. You

[10:34](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=634s) can use your tokens to execute. You can

[10:36](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=636s) brute force your way through your task

[10:37](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=637s) and you can throw more budget at it. But

[10:39](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=639s) if you get really smart about having

[10:41](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=641s) your tokens do these different jobs and

[10:43](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=643s) try these different strategies, you're

[10:45](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=645s) very very likely to be able to get a

[10:46](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=646s) better outcome for the task at hand

[10:49](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=649s) within a fixed budget.

[10:52](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=652s) And so let's talk a little bit about how

[10:53](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=653s) we actually build strategies and how we

[10:55](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=655s) bring this to life. Um so we've done a

[10:57](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=657s) lot of work to create a really excellent

[10:58](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=658s) harness for individual agents. Um and if

[11:01](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=661s) you see this uh picture at the bottom

[11:03](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=663s) here, this is actually the architecture

[11:05](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=665s) that we've used for cloud managed agents

[11:07](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=667s) um which is our Aentic solution that we

[11:09](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=669s) give to you within the cloud platform.

[11:12](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=672s) And what we do on top of this is we

[11:14](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=674s) start to get into the meta harness level

[11:15](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=675s) like the multi- aent orchestration and

[11:17](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=677s) execution level where this strategy can

[11:20](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=680s) go and be um coordinated between our

[11:23](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=683s) executor and our adviser or the other

[11:25](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=685s) agents within our strategy. And some of

[11:28](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=688s) these um like dreaming and outcomes we

[11:30](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=690s) actually give to you out of the box

[11:31](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=691s) within cloud manage agents.

[11:34](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=694s) So with those set of primitives it's

[11:36](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=696s) actually relatively trivial for us to

[11:38](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=698s) construct this kind of you know

[11:40](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=700s) architecture where we're able to combine

[11:41](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=701s) these different types of strategies and

[11:43](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=703s) figure out how to they should work

[11:45](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=705s) together. So for example it's relatively

[11:47](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=707s) trivial for us to say okay now with this

[11:49](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=709s) I can take a task and I should be able

[11:51](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=711s) to execute it but also allow it to

[11:53](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=713s) advise and fable is back online. So we

[11:55](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=715s) could actually say Fable is the one

[11:57](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=717s) that's actually advising uh the

[11:58](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=718s) executor. And then I can take all these

[12:00](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=720s) results and say send them to a greater

[12:02](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=722s) so that I can make sure that this is

[12:03](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=723s) verifying in a loop that makes sense.

[12:05](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=725s) And if it passes, that's awesome. I want

[12:07](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=727s) to send all of that stuff to Dreaming

[12:09](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=729s) and make sure that my next run is better

[12:10](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=730s) than ever.

[12:12](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=732s) And of course, you don't have to stop

[12:14](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=734s) there, right? If the right primitives

[12:15](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=735s) are there and the right coordination is

[12:17](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=737s) there, then you can actually construct

[12:19](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=739s) really complex setups that fit for all

[12:21](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=741s) the different types of dynamic problems

[12:23](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=743s) that you have. You can invent these

[12:25](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=745s) kinds of large-scale architectures,

[12:27](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=747s) again, very triv. And you could also

[12:29](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=749s) invent completely new jobs, not just the

[12:31](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=751s) ones of the pieces that Caitlyn and I

[12:33](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=753s) have presented in this conversation.

[12:36](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=756s) So, a big goal that we have over time is

[12:38](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=758s) to get our models better and better and

[12:40](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=760s) our platform better and better at

[12:42](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=762s) dynamically constructing these

[12:43](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=763s) strategies for you as you're doing work.

[12:45](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=765s) But in the meantime, as we're working

[12:47](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=767s) our way there, we would love for you to

[12:49](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=769s) continue to think about this idea that

[12:50](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=770s) you should give your tokens jobs and you

[12:52](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=772s) should use different novel strategies by

[12:54](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=774s) combining these primitives in order to

[12:56](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=776s) get the outcomes that you want for your

[12:58](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=778s) tasks.

[12:59](https://www.youtube.com/watch?v=PXj0p_mW9nI&t=779s) >> Thanks for joining us.
