[
  {
    "startMs": 0,
    "text": "We can kick it off. Thank you all for coming. I'm Lance. I'm a member of technical staff at"
  },
  {
    "startMs": 17000,
    "text": "Anthropic. And I actually want to share some very simple things with you today. A few simple ways to"
  },
  {
    "startMs": 23220,
    "text": "get the most out of frontier models. So we typically think of cost and quality as a trade-off. So in order to get better performance, you have to spend more money. And in order to kind of reduce cost, you have to sacrifice performance. And so you can kind of think about on any given evaluation, you have models that kind of create this Pareto frontier. And you can imagine in this case, Fable is kind of up on the far right of the curve. And then you have Opus. You have Sonnet. Then you have Haiku."
  },
  {
    "startMs": 53220,
    "text": "Now what I want to share is actually that many times you're not actually on this frontier. Your application is actually sitting inside this frontier that you could be on. And it's because of minor misconfigurations, which I'm going to show you today how to fix. And we've seen this repeatedly improve performance and reduce cost for many customers using some very simple things."
  },
  {
    "startMs": 80280,
    "text": "So first, I'm going to cover five themes. And then we can make an interactive. The slides won't take an hour, not even close. So question answering is actually much more fun."
  },
  {
    "startMs": 91380,
    "text": "First, audit your prompts. This is a kind of important point that prompts accumulate debt just like code."
  },
  {
    "startMs": 101060,
    "text": "So if you recall, several generations ago, we used to use things like think step by step. We used to use verification rituals, double check your work."
  },
  {
    "startMs": 109520,
    "text": "We used to use thoughtfulness boosters, be maximally thorough. Sometimes we would use kind of scratch pads, tell the agent how to think, give it places to think."
  },
  {
    "startMs": 118860,
    "text": "You don't need that anymore. Frontier models are very good at thinking, choosing when to think."
  },
  {
    "startMs": 127820,
    "text": "They don't need to be told what to do and like kind of how to verify their work. They naturally verify their work."
  },
  {
    "startMs": 133280,
    "text": "They don't need to be told to be thoughtful. They're already very thoughtful."
  },
  {
    "startMs": 138960,
    "text": "And worse, when you do these things, it actually increases cost and can reduce performance in some ways I'll talk about briefly."
  },
  {
    "startMs": 146640,
    "text": "Another interesting one, with earlier models, if your prompt had contradictory rules in them, earlier models that weren't as good at instruction following would kind of ignore one of the two rules."
  },
  {
    "startMs": 157980,
    "text": "So it kind of still worked. But with frontier models, they'll often get confused by contradictory rules."
  },
  {
    "startMs": 163440,
    "text": "So I'm going to give you one very simple thing to do right now, if you have any code that you want to check."
  },
  {
    "startMs": 170700,
    "text": "In Cloud Code, you can run this command, checkup prompt audit."
  },
  {
    "startMs": 174500,
    "text": "You can also run cloud API prompt audit."
  },
  {
    "startMs": 176780,
    "text": "These are two different skills in Cloud Code."
  },
  {
    "startMs": 178900,
    "text": "They link to the same code."
  },
  {
    "startMs": 180540,
    "text": "It's just checkup is typically how people do checkups on their Cloud Code configuration itself."
  },
  {
    "startMs": 185900,
    "text": "So your skills, your prompts within Cloud Code itself."
  },
  {
    "startMs": 189440,
    "text": "The Cloud API skill is a separate skill bundled in Cloud Code that's meant for a kind of code that uses our APIs."
  },
  {
    "startMs": 195940,
    "text": "But both of these commands use the same principle and they kind of link down to the same underlying skill itself."
  },
  {
    "startMs": 201680,
    "text": "It's just two different ways to call it."
  },
  {
    "startMs": 204640,
    "text": "These skills will audit your code, anything in the working directory of Cloud Code."
  },
  {
    "startMs": 210080,
    "text": "So that could be skills, that could be tools, that could be prompts for these anti-patterns and we'll remove them for you."
  },
  {
    "startMs": 214660,
    "text": "This has become kind of very popular."
  },
  {
    "startMs": 217360,
    "text": "Many people use it within Anthropic."
  },
  {
    "startMs": 219560,
    "text": "It's kind of become somewhat popular on Twitter as well."
  },
  {
    "startMs": 222940,
    "text": "It's very simple and often it can improve cost and performance."
  },
  {
    "startMs": 229300,
    "text": "So here's an example where this is an evaluation that I built."
  },
  {
    "startMs": 234280,
    "text": "It's a synthetic evaluation of customer support tickets."
  },
  {
    "startMs": 237740,
    "text": "There's like 50 tickets."
  },
  {
    "startMs": 238660,
    "text": "And what I do is I start with Opus 4.8 and I basically simulate a migration to Opus 5.5."
  },
  {
    "startMs": 248220,
    "text": "Okay."
  },
  {
    "startMs": 249440,
    "text": "And in this process I inject each anti-pattern individually and measure the result."
  },
  {
    "startMs": 255860,
    "text": "That's this middle point."
  },
  {
    "startMs": 257420,
    "text": "Okay."
  },
  {
    "startMs": 258120,
    "text": "So what happens is it's taking the average of all the anti-patterns across the benchmark basically."
  },
  {
    "startMs": 263440,
    "text": "The key point is this."
  },
  {
    "startMs": 264420,
    "text": "When I do the migration to Opus 5.5 from 4.8 initially with these kind of anti-patterns injected, we do see some cost savings and a little bit of performance savings."
  },
  {
    "startMs": 276900,
    "text": "You can see it goes to the left and it goes up a bit."
  },
  {
    "startMs": 279280,
    "text": "If I run prompt audit on top of that, I get a further bump both in performance and cost reduction for a bunch of different reasons."
  },
  {
    "startMs": 288660,
    "text": "Because this command is going through and it's removing verification rituals, boosters that are actually causing Opus 5.5 to expend excessive tokens."
  },
  {
    "startMs": 297120,
    "text": "So this little simple command often will basically reduce token use, saving cost, and it can improve performance in cases where things like contradictory prompts or confusing frontier models."
  },
  {
    "startMs": 311100,
    "text": "So it's a very simple command that I recommend and we've used it very extensively within Anthropic and also many customers have enjoyed this quite a lot."
  },
  {
    "startMs": 318400,
    "text": "It's very simple to run."
  },
  {
    "startMs": 319460,
    "text": "You can run it kind of in any code using the API or even on your cloud code configuration to make sure your prompts are kind of well-tuned for frontier models."
  },
  {
    "startMs": 328380,
    "text": "That's point one."
  },
  {
    "startMs": 330400,
    "text": "Now point two is make sure you're using the prompt cache."
  },
  {
    "startMs": 334340,
    "text": "This is probably pretty familiar to many users, but I do just want to make sure it's extremely clear."
  },
  {
    "startMs": 339100,
    "text": "So agents, for example, operate across many turns."
  },
  {
    "startMs": 344060,
    "text": "Every turn adds just a little bit."
  },
  {
    "startMs": 345980,
    "text": "For example, one tool call."
  },
  {
    "startMs": 348220,
    "text": "Much of kind of the context is repeated across turns."
  },
  {
    "startMs": 351180,
    "text": "You don't want to pay for it twice."
  },
  {
    "startMs": 352840,
    "text": "That's why you can use the prompt cache."
  },
  {
    "startMs": 354360,
    "text": "So prompt cache is billed at like a tenth the rate of base token."
  },
  {
    "startMs": 358180,
    "text": "So it's much cheaper."
  },
  {
    "startMs": 359840,
    "text": "Varies a bit per model."
  },
  {
    "startMs": 361060,
    "text": "I think a tenth is, I think, the current pricing, but you can check the pricing pages."
  },
  {
    "startMs": 365620,
    "text": "The point is it's much cheaper."
  },
  {
    "startMs": 367420,
    "text": "It's also much faster."
  },
  {
    "startMs": 369120,
    "text": "So it's very important in multi-turn applications to be making heavy use of the prompt cache."
  },
  {
    "startMs": 374800,
    "text": "Now the problem is prompt cache are a strict prefix match."
  },
  {
    "startMs": 380400,
    "text": "So any, even a byte change in your prefix will invalidate the cache."
  },
  {
    "startMs": 385400,
    "text": "So you have to be careful in building agents or multi-turn applications that you don't break the prompt cache."
  },
  {
    "startMs": 389960,
    "text": "That's the key point."
  },
  {
    "startMs": 392800,
    "text": "And Cloud Console, if you're using the API, has nice diagnostics for you."
  },
  {
    "startMs": 396880,
    "text": "So if you're building applications with a Cloud API, you can see for your application your prompt cache hit rate, which is very important to track."
  },
  {
    "startMs": 402440,
    "text": "For example, with Cloud Code, we track the prompt cache hit rate very, very religiously."
  },
  {
    "startMs": 408480,
    "text": "It's very important for economics of the application, obviously."
  },
  {
    "startMs": 411360,
    "text": "And it's also, in your cases, if you're building apps with the Cloud API, it's very, very important to make sure you're making heavy use of the prompt cache."
  },
  {
    "startMs": 418260,
    "text": "Now, in general, how to organize your prompts."
  },
  {
    "startMs": 423820,
    "text": "You can think about, basically, system instructions, tools, things that are globally cached across all of your sessions."
  },
  {
    "startMs": 430180,
    "text": "You can think about Cloud MD and memory."
  },
  {
    "startMs": 431940,
    "text": "That's kind of cached per project."
  },
  {
    "startMs": 434320,
    "text": "Session state is kind of per session."
  },
  {
    "startMs": 435980,
    "text": "And then messages kind of increment at each turn."
  },
  {
    "startMs": 437960,
    "text": "So that's kind of how you want to structure your prompts."
  },
  {
    "startMs": 439440,
    "text": "And the key point is this."
  },
  {
    "startMs": 442820,
    "text": "Run this in Cloud Code."
  },
  {
    "startMs": 443900,
    "text": "So if you have a project using the API, you want to make check and make sure that you're making proper use of the prompt cache, you can run this."
  },
  {
    "startMs": 450480,
    "text": "It'll run diagnostics to make sure that you're using the prompt cache effectively."
  },
  {
    "startMs": 454500,
    "text": "It'll find opportunities to reorganize your prompts to make sure that they're organized such that you get a high prompt cache hit rate."
  },
  {
    "startMs": 460220,
    "text": "That's really the key point here."
  },
  {
    "startMs": 461340,
    "text": "And this is kind of a fun exercise."
  },
  {
    "startMs": 465360,
    "text": "So myself and some folks on our applied AI team ran this command on a bunch of internal benchmarks that had kind of default harnesses."
  },
  {
    "startMs": 473780,
    "text": "We found opportunities to reduce costs by up to 72% with no meaningful change in performance just by running this command because some of the harnesses on these evals were not using the prompt cache properly."
  },
  {
    "startMs": 485040,
    "text": "Really simple command."
  },
  {
    "startMs": 486120,
    "text": "You can run it in your API code and it'll just look for opportunities to make sure you're saving as much cost as you should be, i.e. by using the prompt cache."
  },
  {
    "startMs": 493540,
    "text": "It'll also check for other things."
  },
  {
    "startMs": 494680,
    "text": "It'll look for, I'll talk about it a little bit later, but it'll also check your effort levels."
  },
  {
    "startMs": 499040,
    "text": "It'll check your prompts."
  },
  {
    "startMs": 500180,
    "text": "So it'll do a number of other things, but in particular, it'll look at your prompt cache to make sure you have a high prompt cache hit rate."
  },
  {
    "startMs": 506940,
    "text": "Again, this has, you know, optimized prompt cache has no change in performance."
  },
  {
    "startMs": 512100,
    "text": "It's like effectively just free cost savings."
  },
  {
    "startMs": 514120,
    "text": "So it's something to be very serious about when you're building applications with the API in particular."
  },
  {
    "startMs": 518900,
    "text": "It's something to very seriously make sure you're utilizing effectively."
  },
  {
    "startMs": 525280,
    "text": "So I want to get into some other fundamentals."
  },
  {
    "startMs": 529580,
    "text": "We talked about prompt caching."
  },
  {
    "startMs": 531300,
    "text": "We talked about making sure your prompts are kind of well, that are well tuned."
  },
  {
    "startMs": 535520,
    "text": "You can audit them using the prompt cache."
  },
  {
    "startMs": 537920,
    "text": "Now let's talk about a few other things that are very important for working with frontier models."
  },
  {
    "startMs": 542720,
    "text": "So Claude, frontier Claude in particular, is very good at self-correction."
  },
  {
    "startMs": 548340,
    "text": "So if you give Claude a well-structured evaluation, it can basically just hill climb in that eval, and I'll talk about that in a minute."
  },
  {
    "startMs": 556300,
    "text": "But important in that is good eval design."
  },
  {
    "startMs": 559920,
    "text": "And you have to give Claude an evaluation that is climbable."
  },
  {
    "startMs": 563300,
    "text": "And so actually, we spend a lot of time thinking about eval design and making sure Claude is set up for success when building an eval and setting up hill climbing."
  },
  {
    "startMs": 571880,
    "text": "So some general features of good eval design."
  },
  {
    "startMs": 576120,
    "text": "This is basics."
  },
  {
    "startMs": 577540,
    "text": "This is nothing related to Anthropics."
  },
  {
    "startMs": 579280,
    "text": "This is just kind of fundamentals."
  },
  {
    "startMs": 580420,
    "text": "But it's good to revisit briefly."
  },
  {
    "startMs": 583540,
    "text": "What you should see generally is that these are a number of different curves looking at basically tokens per attempt."
  },
  {
    "startMs": 589300,
    "text": "This is kind of like thinking on the X and then score on the Y."
  },
  {
    "startMs": 593380,
    "text": "Frontier model at the top, mid-sized model in the middle curve, small model at the bottom."
  },
  {
    "startMs": 598380,
    "text": "So think about that as like Opus, Sonnet, Haiku."
  },
  {
    "startMs": 600980,
    "text": "You should generally see on your evaluation as you step up model and performance goes up."
  },
  {
    "startMs": 606200,
    "text": "That's point one."
  },
  {
    "startMs": 607740,
    "text": "As you increase thinking performance should go up."
  },
  {
    "startMs": 610180,
    "text": "That's point two."
  },
  {
    "startMs": 611860,
    "text": "There should be headroom to improve at the frontier."
  },
  {
    "startMs": 614700,
    "text": "So that's point three."
  },
  {
    "startMs": 616040,
    "text": "So basically it shouldn't be saturated."
  },
  {
    "startMs": 618600,
    "text": "And there should be low run-run variance."
  },
  {
    "startMs": 621300,
    "text": "Very simple points, but actually very important to get right."
  },
  {
    "startMs": 624860,
    "text": "And actually even a lot of very popular evaluations that we see have many tasks that are ambiguous,"
  },
  {
    "startMs": 629440,
    "text": "effectively limiting that headroom at the frontier."
  },
  {
    "startMs": 633900,
    "text": "And this is a problem I see all the time."
  },
  {
    "startMs": 635880,
    "text": "You build an evaluation."
  },
  {
    "startMs": 636880,
    "text": "It turns out you have a bunch of ambiguous cases that can't be solved anyway."
  },
  {
    "startMs": 639820,
    "text": "So it looks like, oh, I have 20% more that I can get on this, but no model can actually achieve it."
  },
  {
    "startMs": 645200,
    "text": "So you could spend many tokens hill-climbing on an evaluation that's effectively saturated."
  },
  {
    "startMs": 649780,
    "text": "It's very important to get this right."
  },
  {
    "startMs": 653280,
    "text": "Try this command."
  },
  {
    "startMs": 654840,
    "text": "Cloud API build eval."
  },
  {
    "startMs": 657000,
    "text": "This is actually a very nice command because it's not opinionated at all about where your data lives."
  },
  {
    "startMs": 663480,
    "text": "It doesn't care."
  },
  {
    "startMs": 664200,
    "text": "You can point this at whatever tracing solution you have."
  },
  {
    "startMs": 667420,
    "text": "It's just a skill in cloud code."
  },
  {
    "startMs": 668780,
    "text": "It's more employing the principles I talked about to test and sanity check the evaluation for you."
  },
  {
    "startMs": 675960,
    "text": "So you can kind of give this skill, hey, here's a problem I want to solve."
  },
  {
    "startMs": 679980,
    "text": "Here's what I'm interested in."
  },
  {
    "startMs": 681480,
    "text": "Here is the raw data."
  },
  {
    "startMs": 682740,
    "text": "For example, here's the traces that I want to work with."
  },
  {
    "startMs": 685020,
    "text": "And this command can help you build an eval set."
  },
  {
    "startMs": 686860,
    "text": "It'll walk you through a process and step you through building it."
  },
  {
    "startMs": 691220,
    "text": "It'll show you the data."
  },
  {
    "startMs": 693060,
    "text": "It'll show you the evaluation."
  },
  {
    "startMs": 694160,
    "text": "It'll sanity check the greater the criteria with you."
  },
  {
    "startMs": 697800,
    "text": "So it'll kind of walk you through the process of building eval based upon some of our best practice."
  },
  {
    "startMs": 702260,
    "text": "And we have a nice blog post on this as well that I'll share."
  },
  {
    "startMs": 707720,
    "text": "And it pairs very well with hill climbing."
  },
  {
    "startMs": 710340,
    "text": "So once you've built a good evaluation that is headroom at the frontier, low variance, scales as expected with thinking and model, you can then climb on it."
  },
  {
    "startMs": 718820,
    "text": "Frontier cloud models are very, very effective at hill climbing."
  },
  {
    "startMs": 721720,
    "text": "So autonomously look at an evaluation and autonomously kind of improving on it without you in the loop."
  },
  {
    "startMs": 728380,
    "text": "So you build a good eval, clock and hill climb on it."
  },
  {
    "startMs": 732960,
    "text": "The problem is there's many ways that this can kind of go wrong."
  },
  {
    "startMs": 737240,
    "text": "And in particular, you can end up overfitting on the evaluation itself."
  },
  {
    "startMs": 741640,
    "text": "So like some classy examples would be like the task in your benchmark have particular needs that are bespoke or specific to those tasks."
  },
  {
    "startMs": 752380,
    "text": "Like maybe one task needs OCR."
  },
  {
    "startMs": 754640,
    "text": "So what happens is your harness to solve the eval adds like an OCR tool."
  },
  {
    "startMs": 758900,
    "text": "But in production, you don't see that problem."
  },
  {
    "startMs": 760920,
    "text": "So you end up overfitting on the eval distribution."
  },
  {
    "startMs": 763900,
    "text": "It doesn't apply to production."
  },
  {
    "startMs": 765040,
    "text": "This is very common."
  },
  {
    "startMs": 765980,
    "text": "Yeah, like distinctive phrasing specific to your eval examples."
  },
  {
    "startMs": 773080,
    "text": "For example, another classic is like it just kind of it like the prompt when you're basically hill climbing overfits the specifics of your eval distribution."
  },
  {
    "startMs": 782240,
    "text": "It doesn't apply to your to your test distribution."
  },
  {
    "startMs": 784540,
    "text": "So overfitting is the major challenge concern when you're hill climbing."
  },
  {
    "startMs": 788540,
    "text": "Imagine you spend a lot of tokens hill climbing in evaluation."
  },
  {
    "startMs": 791800,
    "text": "It looks really good after hill climbing."
  },
  {
    "startMs": 793480,
    "text": "You run it in production and performance is not better at all."
  },
  {
    "startMs": 796520,
    "text": "That's frustrating because you spend a lot of money."
  },
  {
    "startMs": 798920,
    "text": "You hill climbed and like, oh, it's not applicable to Russians because often you've just kind of overfit to the eval distribution."
  },
  {
    "startMs": 806140,
    "text": "So try this command in cloud code is cloud API hill climb."
  },
  {
    "startMs": 811980,
    "text": "So if you build an eval, it'll hill climb for you."
  },
  {
    "startMs": 814540,
    "text": "And it tries to combat this with a few tricks."
  },
  {
    "startMs": 818140,
    "text": "Now, it's, of course, not perfect yet to audit this carefully and indeed be in the loop and make sure you really understand what's happening."
  },
  {
    "startMs": 824620,
    "text": "But it can help you."
  },
  {
    "startMs": 825880,
    "text": "And in particular, what it does is a process like this."
  },
  {
    "startMs": 831160,
    "text": "It will look at your evaluation and it will split into test and train sets."
  },
  {
    "startMs": 836140,
    "text": "This is kind of very classic kind of classic principles from statistics."
  },
  {
    "startMs": 840040,
    "text": "Create a test and train split."
  },
  {
    "startMs": 842220,
    "text": "What happens is when hill climbing, this is just on cloud code, by the way."
  },
  {
    "startMs": 846120,
    "text": "So cloud code basically has this skill."
  },
  {
    "startMs": 848180,
    "text": "You run it on an eval."
  },
  {
    "startMs": 849360,
    "text": "It will only ever look at the train split and reflect on those failures when it's making decisions about what to change to improve on the eval."
  },
  {
    "startMs": 858160,
    "text": "So what's nice about that is at every round, it never sees a test split."
  },
  {
    "startMs": 864220,
    "text": "And it will basically check performance per round against test and train."
  },
  {
    "startMs": 868900,
    "text": "So what happens is it's a little early signal for overfitting."
  },
  {
    "startMs": 872560,
    "text": "If you see train improving but test flat, it's a signal, ooh, you might be overfitting."
  },
  {
    "startMs": 877380,
    "text": "So basically it applies this check and a number of other more subtle things to check to make sure while hill climbing is happening, you aren't just overfitting to your eval distribution."
  },
  {
    "startMs": 887560,
    "text": "You're actually hill climbing in a way that will apply to production."
  },
  {
    "startMs": 891480,
    "text": "Assuming you built your eval distribution well."
  },
  {
    "startMs": 893440,
    "text": "And that's always something that we can address in that build eval phase."
  },
  {
    "startMs": 897780,
    "text": "Now I'll share some concrete examples that I've used here."
  },
  {
    "startMs": 901120,
    "text": "So this is actually the Cloud API skill itself."
  },
  {
    "startMs": 904880,
    "text": "So if I back up a little bit, this is a skill in Cloud code."
  },
  {
    "startMs": 910060,
    "text": "And it is a skill that gives Cloud full awareness of our APIs."
  },
  {
    "startMs": 915840,
    "text": "So basically our SDKs, APIs."
  },
  {
    "startMs": 918220,
    "text": "And it has to be grounded in our documentation."
  },
  {
    "startMs": 922280,
    "text": "So I built an eval set based upon our documentation."
  },
  {
    "startMs": 925980,
    "text": "And I tested the skill."
  },
  {
    "startMs": 927180,
    "text": "The skill initially was not very good at the eval questions in my set, 66%."
  },
  {
    "startMs": 932460,
    "text": "So I set up a hill climbing loop with giving Cloud code access to the skill text and the documentation itself."
  },
  {
    "startMs": 939500,
    "text": "And let it basically just iteratively look at the failures in the eval set and try to figure out why the skill was misrepresenting what's in the documentation."
  },
  {
    "startMs": 947420,
    "text": "And there's all sorts of little tricks."
  },
  {
    "startMs": 949240,
    "text": "And I could talk about that in more detail."
  },
  {
    "startMs": 951940,
    "text": "But the point is, Cloud can autonomously find, in this particular case, the right skill text to improve significantly on the evaluation."
  },
  {
    "startMs": 960540,
    "text": "And I talk about this in the blog link below."
  },
  {
    "startMs": 962320,
    "text": "But again, this is a good example of a kind of a well-scoped evaluation, a simple surface on modifying a skill, and an autonomous hill climbing loop that results in a much better performance."
  },
  {
    "startMs": 973480,
    "text": "Just when I set the system up this way with a carefully scoped eval, a surface that's easy to modify, Cloud can just rip through it and significantly improve performance."
  },
  {
    "startMs": 982500,
    "text": "This works well for prompts."
  },
  {
    "startMs": 983960,
    "text": "It works well for skills."
  },
  {
    "startMs": 985680,
    "text": "Simple text-based surfaces are very, very good for this kind of hill climbing process."
  },
  {
    "startMs": 989080,
    "text": "Now, this is another kind of interesting trick that I've seen a lot of customers like."
  },
  {
    "startMs": 995960,
    "text": "You can also apply hill climbing to cost reduction."
  },
  {
    "startMs": 999760,
    "text": "So what's kind of nice about this is I've added guidance to this hill climb command."
  },
  {
    "startMs": 1004100,
    "text": "If you tell it to reduce cost, what it'll do is it will basically try to find a configuration that is at least at parity in performance and for sure reduces cost."
  },
  {
    "startMs": 1015300,
    "text": "And it'll search things like effort, model, and your prompts."
  },
  {
    "startMs": 1017960,
    "text": "Here's a simple example on that same customer support benchmark."
  },
  {
    "startMs": 1022380,
    "text": "I start with Opus 4.8 high and some baseline prompt."
  },
  {
    "startMs": 1026440,
    "text": "And I basically tell the hill climber, reduce cost."
  },
  {
    "startMs": 1029120,
    "text": "So what it does, it basically will search through model effort combinations and basically pick those that continuously reduces cost over the rounds without degrading performance."
  },
  {
    "startMs": 1040900,
    "text": "In this case, it moves to Opus 5 low."
  },
  {
    "startMs": 1043240,
    "text": "We see a big bump in performance, a minor reduction in cost."
  },
  {
    "startMs": 1047680,
    "text": "The hill climber then autonomously tries Flaunt 5 low."
  },
  {
    "startMs": 1050740,
    "text": "We see, again, degrades performance a bit, but big cost reduction."
  },
  {
    "startMs": 1055040,
    "text": "And then it then decides to tune the prompt and we kind of rescue the performance."
  },
  {
    "startMs": 1059300,
    "text": "So we basically got a 90% reduction in cost with notable improvement in performance autonomously using the hill climber."
  },
  {
    "startMs": 1066400,
    "text": "So this is another very nice way if you have an evaluation, you can use the hill climber to basically say, okay, keep performance fixed but reduce my cost materially by tuning things like effort, things like model selection, and my prompts."
  },
  {
    "startMs": 1079060,
    "text": "Another very nice use of the hill climber."
  },
  {
    "startMs": 1080640,
    "text": "And actually what's nice about this is you can do this if you have a saturated eval."
  },
  {
    "startMs": 1086020,
    "text": "So you can basically say, okay, say my eval is saturated, fine, it doesn't matter."
  },
  {
    "startMs": 1089240,
    "text": "I don't care about performance."
  },
  {
    "startMs": 1090380,
    "text": "I just want to reduce cost."
  },
  {
    "startMs": 1091880,
    "text": "So then you can just tell the hill climber, okay, keep performance fixed and just try to burn down cost."
  },
  {
    "startMs": 1096840,
    "text": "And it can basically search for configurations."
  },
  {
    "startMs": 1098880,
    "text": "And often you'll find very interesting things."
  },
  {
    "startMs": 1101140,
    "text": "Like in this particular case, there's generally an intuition that moving to higher models at lower effort is often cheaper and more performant than lower models at high effort."
  },
  {
    "startMs": 1110640,
    "text": "It's like giving someone who's smarter, can like try less to actually get a better objective or to kind of solve the problem more effectively."
  },
  {
    "startMs": 1119400,
    "text": "And so it ends up actually using fewer tokens and as a result ends up being cheaper."
  },
  {
    "startMs": 1123800,
    "text": "That's kind of a broad intuition here that we've seen quite a bit."
  },
  {
    "startMs": 1126440,
    "text": "And that's baked into this hill climber skill."
  },
  {
    "startMs": 1129440,
    "text": "So another nice trick."
  },
  {
    "startMs": 1133380,
    "text": "Now I'm going to open up to questions pretty soon."
  },
  {
    "startMs": 1135580,
    "text": "I'll just kind of close with some other comments."
  },
  {
    "startMs": 1137520,
    "text": "And this is a bit of a nod to the manage agent stuff you heard this morning."
  },
  {
    "startMs": 1141240,
    "text": "I just want to reemphasize this a little bit."
  },
  {
    "startMs": 1142860,
    "text": "I do think as part of unhobbling the frontier, it's very important to design harnesses that can take most advantage of the latest cloud models."
  },
  {
    "startMs": 1153160,
    "text": "And in particular, it's interesting."
  },
  {
    "startMs": 1155620,
    "text": "If you look back the last few years, time is on the X."
  },
  {
    "startMs": 1159020,
    "text": "And this is task duration for a human on the Y."
  },
  {
    "startMs": 1162440,
    "text": "You've seen these curves."
  },
  {
    "startMs": 1163180,
    "text": "These are just meter charts."
  },
  {
    "startMs": 1165640,
    "text": "It's interesting to think back."
  },
  {
    "startMs": 1167040,
    "text": "You know, Claude Opus 3 was capable of only like four minutes of autonomous work."
  },
  {
    "startMs": 1172340,
    "text": "And in that kind of regime, there are very limited products you could build."
  },
  {
    "startMs": 1176140,
    "text": "It was really auto-completion or chat."
  },
  {
    "startMs": 1177560,
    "text": "Now we saw over the course of 2025, models become more capable."
  },
  {
    "startMs": 1181320,
    "text": "They can do like hours of work."
  },
  {
    "startMs": 1182800,
    "text": "Like maybe an hour to two hours of human-oclevant work."
  },
  {
    "startMs": 1185420,
    "text": "Claude code obviously kind of gained significant popularity over the past, over 2025."
  },
  {
    "startMs": 1191760,
    "text": "And we're kind of entering a new regime now where these task durations for models are exceeding like a day to two days."
  },
  {
    "startMs": 1199680,
    "text": "I think this is kind of outside the scope of what Mater can even measure now."
  },
  {
    "startMs": 1202440,
    "text": "But in this regime where you have models that can do the equivalent of like, say, days of work, what harnesses do you need?"
  },
  {
    "startMs": 1210400,
    "text": "Like the considerations change quite a bit."
  },
  {
    "startMs": 1212320,
    "text": "Because you want models that can just do things autonomously for very, very long periods of time."
  },
  {
    "startMs": 1216580,
    "text": "Now how do you make sure they're secure and reliable?"
  },
  {
    "startMs": 1218760,
    "text": "The considerations change a lot."
  },
  {
    "startMs": 1221660,
    "text": "And you heard a bit about this morning."
  },
  {
    "startMs": 1223800,
    "text": "But this is actually the motivation behind one of our newer APIs called Cloud Managed Agents."
  },
  {
    "startMs": 1229020,
    "text": "And the principle here, I wrote this eng blog back in April on it."
  },
  {
    "startMs": 1233160,
    "text": "Is decoupling the brain from the hands."
  },
  {
    "startMs": 1235620,
    "text": "And the idea is pretty simple here."
  },
  {
    "startMs": 1237200,
    "text": "You have an agent harness, which is effectively the brain."
  },
  {
    "startMs": 1239180,
    "text": "You have hands, which are sandboxes."
  },
  {
    "startMs": 1240480,
    "text": "That's where the agent harness writes code and performs actions."
  },
  {
    "startMs": 1243940,
    "text": "And you want those to be separate things."
  },
  {
    "startMs": 1246340,
    "text": "In particular, you want that, you want it to be the case that if the harness dies or the sandbox dies, the system can recover."
  },
  {
    "startMs": 1253340,
    "text": "Imagine you have an agent that you spin up overnight to just run."
  },
  {
    "startMs": 1256260,
    "text": "And the sandbox dies midway through the night."
  },
  {
    "startMs": 1258680,
    "text": "It's very frustrating."
  },
  {
    "startMs": 1259680,
    "text": "You lose state."
  },
  {
    "startMs": 1260540,
    "text": "You lose your session log and everything."
  },
  {
    "startMs": 1262440,
    "text": "So when we designed this, we designed this for a world where Cloud will be doing work for very, very long durations."
  },
  {
    "startMs": 1267320,
    "text": "And so it's recoverable."
  },
  {
    "startMs": 1268920,
    "text": "So that means the session log is an independent object separate from the harness, separate from the sandbox."
  },
  {
    "startMs": 1274020,
    "text": "That is append only."
  },
  {
    "startMs": 1275500,
    "text": "That is recoverable."
  },
  {
    "startMs": 1276540,
    "text": "So if the harness dies, the sandbox dies, you can always recover from the session log."
  },
  {
    "startMs": 1281300,
    "text": "Sandboxes are also recoverable."
  },
  {
    "startMs": 1283020,
    "text": "So if a sandbox dies, you can boot up a new one."
  },
  {
    "startMs": 1285160,
    "text": "If the harness dies, you can boot up a new one through this orchestrator."
  },
  {
    "startMs": 1287540,
    "text": "Another point here is safety."
  },
  {
    "startMs": 1290780,
    "text": "So one of the important things that managed agents does is it actually never gives Cloud direct access to secrets."
  },
  {
    "startMs": 1296280,
    "text": "Secrets are held in vaults."
  },
  {
    "startMs": 1298500,
    "text": "And that's actually stored in that kind of tools box you see at the top."
  },
  {
    "startMs": 1301400,
    "text": "So vaults ensure that Cloud can reference API keys or secrets, but it can't actually see them."
  },
  {
    "startMs": 1306280,
    "text": "Because obviously if you have an agent that's working for, say, days on end, it's a little bit spooky to give it direct access to very sensitive secrets."
  },
  {
    "startMs": 1314200,
    "text": "And so we ensure that security is a very important consideration when we build manage agents."
  },
  {
    "startMs": 1320880,
    "text": "So reliability and security become much more important as agents can operate for hours, days, and maybe soon weeks."
  },
  {
    "startMs": 1329520,
    "text": "And this morning you probably heard more about this, but we have a nice recent blog that I put out with CJ that walks through an example of basically building an agent that can serve as like kind of an agent automation for regular context updates."
  },
  {
    "startMs": 1346420,
    "text": "It can reach into different sources that you specify."
  },
  {
    "startMs": 1348620,
    "text": "It can write updates to Slack."
  },
  {
    "startMs": 1350840,
    "text": "Very simple automation example."
  },
  {
    "startMs": 1352280,
    "text": "It's highly customizable."
  },
  {
    "startMs": 1353560,
    "text": "But it shows how all these different primitives come together."
  },
  {
    "startMs": 1356060,
    "text": "It brings together a scheduled deployment."
  },
  {
    "startMs": 1358700,
    "text": "This all runs on Anthropic infrastructure."
  },
  {
    "startMs": 1361680,
    "text": "It securely stores secrets."
  },
  {
    "startMs": 1363900,
    "text": "It has memory."
  },
  {
    "startMs": 1365540,
    "text": "And it's a very nice kind of end-to-end example."
  },
  {
    "startMs": 1369700,
    "text": "Now again, the final point is you can just run this in Cloud code."
  },
  {
    "startMs": 1373340,
    "text": "So run Cloud API, manage it's on board, and then give it the link to the blog."
  },
  {
    "startMs": 1377600,
    "text": "And it'll just boot up this tutorial for you."
  },
  {
    "startMs": 1379560,
    "text": "It'll spin up the example, and you can just play with it."
  },
  {
    "startMs": 1383840,
    "text": "So really the key point here is there's a number of ways you can kind of harness and make sure the frontier is effectively unhobbled."
  },
  {
    "startMs": 1392960,
    "text": "It is very important to audit your prompts."
  },
  {
    "startMs": 1397620,
    "text": "It's very important to make sure that you're effectively using prompt cache, especially if you're building agents for long duration, to make sure that cost is managed."
  },
  {
    "startMs": 1403980,
    "text": "It's important to think about harnesses that are designed for long duration in terms of reliability and security."
  },
  {
    "startMs": 1408660,
    "text": "And it's also important to think about evaluations and hill climbing."
  },
  {
    "startMs": 1412460,
    "text": "And when this all comes together, you're going to have agents that you can give a task, for example, an evaluation, and let it run for a long period of time and hill climb on that evaluation, come back, for example, in the morning, and it's significantly improved."
  },
  {
    "startMs": 1423920,
    "text": "And that's really the way that I see many of my colleagues working now."
  },
  {
    "startMs": 1426840,
    "text": "People are spending much more time on eval design and basically just specifying tasks that Cloud can effectively hill climb on."
  },
  {
    "startMs": 1432940,
    "text": "And doing this in a way such that Agent is highly secure when doing this and is highly reliable for long duration jobs."
  },
  {
    "startMs": 1442140,
    "text": "And let's see where we are in time."
  },
  {
    "startMs": 1444040,
    "text": "With that, I think I'll just take it to questions because it's more fun when it's interactive anyway."
  },
  {
    "startMs": 1448420,
    "text": "You can follow me on all the socials here."
  },
  {
    "startMs": 1451980,
    "text": "And thank you for your time and attention."
  },
  {
    "startMs": 1454100,
    "text": "Thank you."
  },
  {
    "startMs": 1462940,
    "text": "Sure."
  },
  {
    "startMs": 1465780,
    "text": "Start there."
  },
  {
    "startMs": 1467540,
    "text": "This is a very good question."
  },
  {
    "startMs": 1469240,
    "text": "The question was, how automatable is eval design?"
  },
  {
    "startMs": 1472140,
    "text": "So actually, I was engaging a bit with Hamill on Twitter."
  },
  {
    "startMs": 1476060,
    "text": "Extremely thoughtful with respect to evals."
  },
  {
    "startMs": 1478160,
    "text": "And some feedback he had on our skill was that it should indeed walk the user through the data first, which I actually agree with."
  },
  {
    "startMs": 1485000,
    "text": "I'm making some minor modifications to the skill to do that."
  },
  {
    "startMs": 1487780,
    "text": "So I think that eval design, I'll kind of go back to it, Claude can assist you when building evals significantly."
  },
  {
    "startMs": 1500340,
    "text": "So, for example, in the example he gave when his walkthrough, he was looking at like it was a customer support app or something."
  },
  {
    "startMs": 1508260,
    "text": "And the first question was like, okay, like it's a bunch of traces."
  },
  {
    "startMs": 1513320,
    "text": "What are the failure cases here?"
  },
  {
    "startMs": 1515240,
    "text": "What should I even like promote to an eval set?"
  },
  {
    "startMs": 1517180,
    "text": "So Claude's really good at kind of identifying, ooh, okay, here's cases where there might be errors."
  },
  {
    "startMs": 1521880,
    "text": "It's effectively like acting as like an online evaluator but looking at your offline logs, scanning them, saying, okay, here's some problematic examples."
  },
  {
    "startMs": 1529420,
    "text": "And Claude was very good at that in his example."
  },
  {
    "startMs": 1532240,
    "text": "Then it is actually important and quite useful if Claude then shows those raw, like the raw data to you in a viewer."
  },
  {
    "startMs": 1540520,
    "text": "So I'm going to add that to the skill."
  },
  {
    "startMs": 1542140,
    "text": "I have a PR up and I'll add it."
  },
  {
    "startMs": 1543560,
    "text": "Currently, Claude goes directly to analyzing the data and like building an eval and showing it to you."
  },
  {
    "startMs": 1549600,
    "text": "I do think it's important and helpful for there to be an intermediate stage where you look at the data first."
  },
  {
    "startMs": 1554620,
    "text": "But I think having data annotated by Claude or Claude's giving you some ideas about what to look at is very important."
  },
  {
    "startMs": 1560300,
    "text": "So kind of like Claude assisted evaluation and scanning of the data is like I think a nice marriage."
  },
  {
    "startMs": 1566600,
    "text": "And then once that's done, you can kind of promote like examples to your eval set."
  },
  {
    "startMs": 1570820,
    "text": "So I think there is still quite a role for the human in the loop there."
  },
  {
    "startMs": 1574920,
    "text": "But Claude can assist quite a bit."
  },
  {
    "startMs": 1577140,
    "text": "Hill climb with human feedback."
  },
  {
    "startMs": 1578620,
    "text": "Hill climbing only requires that you have an eval and you have some signal that Claude can use to judge good versus bad."
  },
  {
    "startMs": 1589480,
    "text": "So if you've taken human feedback and you've rolled that into an eval signal, per example, then it should be perfectly feasible to hill climb on an evaluation structure that way."
  },
  {
    "startMs": 1597620,
    "text": "Yeah, this is a good one."
  },
  {
    "startMs": 1599460,
    "text": "So for hill climbing, what's like the best surface to start with?"
  },
  {
    "startMs": 1602580,
    "text": "So I kind of have that here is like the thing we edit."
  },
  {
    "startMs": 1605100,
    "text": "Yeah, so I've actually played with this quite a bit."
  },
  {
    "startMs": 1607100,
    "text": "But text is nice."
  },
  {
    "startMs": 1609760,
    "text": "In particular, things like prompts and skills are very nice because it is a very clear place that we know significantly alters and affects model behavior."
  },
  {
    "startMs": 1620980,
    "text": "So it's like a very constrained surface to modify."
  },
  {
    "startMs": 1623360,
    "text": "And we know it's highly coupled to the output typically in terms of like what you're evaluating."
  },
  {
    "startMs": 1628980,
    "text": "Like here's a good example."
  },
  {
    "startMs": 1631020,
    "text": "Like when I'm doing skill hill climbing, I'm testing something that is directly reading from the skill."
  },
  {
    "startMs": 1635420,
    "text": "So I know there's clear coupling."
  },
  {
    "startMs": 1637120,
    "text": "Now I'm going to contrast that to also in hill climbing with agent harnesses."
  },
  {
    "startMs": 1640920,
    "text": "But what happens there is it can be a bit more hard to interpret."
  },
  {
    "startMs": 1643920,
    "text": "So you're modifying the harness code."
  },
  {
    "startMs": 1646120,
    "text": "And so the changes per round are much larger diffs."
  },
  {
    "startMs": 1650660,
    "text": "And there's just more surface area for confusion."
  },
  {
    "startMs": 1653960,
    "text": "You can be making changes that are actually decoupled from the output signature measuring."
  },
  {
    "startMs": 1658080,
    "text": "So it's certainly possible."
  },
  {
    "startMs": 1659300,
    "text": "And indeed, people have done quite a bit of work hill climbing on agent harnesses."
  },
  {
    "startMs": 1662040,
    "text": "I just think it's a little bit trickier."
  },
  {
    "startMs": 1663700,
    "text": "And text-based services are often a good starting point."
  },
  {
    "startMs": 1666600,
    "text": "Prompts, skills."
  },
  {
    "startMs": 1668180,
    "text": "Because they're so clearly coupled to model behavior."
  },
  {
    "startMs": 1671760,
    "text": "And they're easy to modify."
  },
  {
    "startMs": 1673440,
    "text": "The catch actually is that things like prompts are very easy to overfit."
  },
  {
    "startMs": 1677280,
    "text": "I've seen that in many of my hill climbing efforts as well."
  },
  {
    "startMs": 1680720,
    "text": "So like, I don't know if I have a good diagram."
  },
  {
    "startMs": 1683400,
    "text": "But you have some eval distribution you're hill climbing on."
  },
  {
    "startMs": 1688020,
    "text": "Even with these regularizations in place, like the eval test train split and so forth,"
  },
  {
    "startMs": 1693160,
    "text": "prompts are very easy to wiggle."
  },
  {
    "startMs": 1695880,
    "text": "Claude's very good at wiggling the prompt into a shape that does well in your eval."
  },
  {
    "startMs": 1699960,
    "text": "That isn't really representative of what's in production."
  },
  {
    "startMs": 1702220,
    "text": "So you have to be very careful with prompts."
  },
  {
    "startMs": 1703540,
    "text": "But they are very easy to modify."
  },
  {
    "startMs": 1706080,
    "text": "So it's a good starting point."
  },
  {
    "startMs": 1707120,
    "text": "You do it to be careful with overfitting."
  },
  {
    "startMs": 1710400,
    "text": "Let's see."
  },
  {
    "startMs": 1711520,
    "text": "Turning a prompt into a skill."
  },
  {
    "startMs": 1712980,
    "text": "So they're kind of complicated."
  },
  {
    "startMs": 1714180,
    "text": "Like, maybe do you mean if you're like going through some standard operating procedure,"
  },
  {
    "startMs": 1719520,
    "text": "when to like promote that to a skill?"
  },
  {
    "startMs": 1721700,
    "text": "Okay, yeah."
  },
  {
    "startMs": 1723940,
    "text": "That's actually kind of interesting."
  },
  {
    "startMs": 1725520,
    "text": "I think often, and Claude is quite good at judging that."
  },
  {
    "startMs": 1728540,
    "text": "So basically, you can basically go through a workflow."
  },
  {
    "startMs": 1730940,
    "text": "And then you can ask Claude, okay, turn that into a skill."
  },
  {
    "startMs": 1733460,
    "text": "I do that all the time."
  },
  {
    "startMs": 1734500,
    "text": "So Claude is pretty good at like catching, okay, you're doing some repetitive thing."
  },
  {
    "startMs": 1737540,
    "text": "Make it a skill."
  },
  {
    "startMs": 1738940,
    "text": "I actually think you should just have Claude do that."
  },
  {
    "startMs": 1742240,
    "text": "Just tell it."
  },
  {
    "startMs": 1742940,
    "text": "Turn this into a skill."
  },
  {
    "startMs": 1743920,
    "text": "Then it'll do it."
  },
  {
    "startMs": 1744860,
    "text": "So usually, in the case of evals, the setup will be, I'll have an existing skill that I've kind of defined."
  },
  {
    "startMs": 1753960,
    "text": "And maybe it has some boilerplate starting text."
  },
  {
    "startMs": 1755800,
    "text": "And I'll say, okay, I want you to make this skill improved."
  },
  {
    "startMs": 1758580,
    "text": "I want to improve this skill text to basically perform better on this eval."
  },
  {
    "startMs": 1763220,
    "text": "So that's usually the way I set it up."
  },
  {
    "startMs": 1764820,
    "text": "Like I have a skill boilerplate as a starting point."
  },
  {
    "startMs": 1768000,
    "text": "I know I want this thing to be a skill."
  },
  {
    "startMs": 1770460,
    "text": "Like in this case, right, like I had this skill for the Claude API."
  },
  {
    "startMs": 1775220,
    "text": "And it was basically, you can imagine it's basically distillation of our docs."
  },
  {
    "startMs": 1778120,
    "text": "It's like, go to this page if you want to learn about this."
  },
  {
    "startMs": 1780860,
    "text": "Go to this page if you want to learn about this."
  },
  {
    "startMs": 1782060,
    "text": "It had a bunch of little gotchas and issues."
  },
  {
    "startMs": 1784760,
    "text": "And so my eval performance was pretty low to start."
  },
  {
    "startMs": 1787520,
    "text": "But I had a starting point."
  },
  {
    "startMs": 1789920,
    "text": "And that's what Claude was able to then modify."
  },
  {
    "startMs": 1792100,
    "text": "So I think usually with skills, you're kind of starting with some baseline."
  },
  {
    "startMs": 1796060,
    "text": "And it could be like a very crude default."
  },
  {
    "startMs": 1799480,
    "text": "But then give Claude something to like start to climb from."
  },
  {
    "startMs": 1802260,
    "text": "That's usually the way I would do it."
  },
  {
    "startMs": 1803360,
    "text": "Right."
  },
  {
    "startMs": 1804200,
    "text": "So verification in domains that have kind of open-ended criteria."
  },
  {
    "startMs": 1811840,
    "text": "So like, yeah, I think in general, heuristic evaluators are preferable, of course."
  },
  {
    "startMs": 1815700,
    "text": "So if you can define like a deterministic check for good or bad, that's obviously preferable."
  },
  {
    "startMs": 1822340,
    "text": "So in domains, that doesn't work."
  },
  {
    "startMs": 1823540,
    "text": "So what I've found is well-structured rubrics that produce similar grades."
  },
  {
    "startMs": 1834000,
    "text": "So okay, a few different things."
  },
  {
    "startMs": 1836360,
    "text": "Build a rubric that you as a human can reason about first."
  },
  {
    "startMs": 1841620,
    "text": "So it's like, let's say we're grading a piece of like text."
  },
  {
    "startMs": 1845560,
    "text": "Is this good or bad?"
  },
  {
    "startMs": 1847700,
    "text": "Build a rubric that like I could look at as a human to like look at some text and say,"
  },
  {
    "startMs": 1851480,
    "text": "okay, I can grade that."
  },
  {
    "startMs": 1852380,
    "text": "That's point one."
  },
  {
    "startMs": 1853020,
    "text": "Point two, don't use arbitrary scales."
  },
  {
    "startMs": 1856880,
    "text": "I see this all the time."
  },
  {
    "startMs": 1857920,
    "text": "Like grade it from zero to five."
  },
  {
    "startMs": 1860080,
    "text": "It's very ambiguous."
  },
  {
    "startMs": 1861220,
    "text": "Make it crisp and simple."
  },
  {
    "startMs": 1862480,
    "text": "Like ideally a binary classification, good versus bad."
  },
  {
    "startMs": 1866480,
    "text": "If you want to evaluate multiple things, you can set up like multiple binary classifiers"
  },
  {
    "startMs": 1870580,
    "text": "rather than, well, the worst thing is give Claude like from one to 10 rate this."
  },
  {
    "startMs": 1875700,
    "text": "That's extremely noisy."
  },
  {
    "startMs": 1877360,
    "text": "Another sanity check, test across different models."
  },
  {
    "startMs": 1879700,
    "text": "Have sonnet, opus, and fable all grade using your rubric."
  },
  {
    "startMs": 1883020,
    "text": "If they all disagree, that's like a bad sign."
  },
  {
    "startMs": 1885700,
    "text": "Another test, run replicates."
  },
  {
    "startMs": 1888240,
    "text": "So basically, like I show here how you want low variance, run replicates with your rubric"
  },
  {
    "startMs": 1893440,
    "text": "and make sure it's tight."
  },
  {
    "startMs": 1895500,
    "text": "So if you do all those things, you can definitely make it work pretty well for, yeah, like text-based"
  },
  {
    "startMs": 1901480,
    "text": "answers that are squishy."
  },
  {
    "startMs": 1902600,
    "text": "Like there is no very definitive like way to heuristically check that."
  },
  {
    "startMs": 1907320,
    "text": "But elements judged with a good, well-structured rubric can be quite effective."
  },
  {
    "startMs": 1911840,
    "text": "Yeah, that's actually a good one."
  },
  {
    "startMs": 1913300,
    "text": "So actually, I should have mentioned that."
  },
  {
    "startMs": 1915680,
    "text": "Pair-wise comparisons can actually be quite good."
  },
  {
    "startMs": 1917760,
    "text": "It's much easier for humans, even models, to kind of compare to artifacts than it is to"
  },
  {
    "startMs": 1922080,
    "text": "grade an artifact."
  },
  {
    "startMs": 1923600,
    "text": "Like you can typically answer, oh, I like this web page better than this one versus I'm sure"
  },
  {
    "startMs": 1927740,
    "text": "this web page is graded between zero and five."
  },
  {
    "startMs": 1929940,
    "text": "It's much harder."
  },
  {
    "startMs": 1931220,
    "text": "So pair-wise comparisons are pretty effective."
  },
  {
    "startMs": 1932900,
    "text": "And actually, I think that is a good way."
  },
  {
    "startMs": 1935220,
    "text": "If your problem allows for that, that actually can be an effective way to deal with more ambiguous"
  },
  {
    "startMs": 1942740,
    "text": "gradings."
  },
  {
    "startMs": 1944040,
    "text": "Well, I think it goes a bit back to just looking at the data carefully, kind of related to the"
  },
  {
    "startMs": 1948520,
    "text": "question asked over here."
  },
  {
    "startMs": 1951360,
    "text": "When I say the distribution, I more mean ensuring that you've sampled some set of cases for your"
  },
  {
    "startMs": 1961060,
    "text": "eval set and making sure those cases represent what you actually care about in production."
  },
  {
    "startMs": 1964780,
    "text": "That's really what I mean by distribution."
  },
  {
    "startMs": 1967300,
    "text": "And so there are some interesting tricks here."
  },
  {
    "startMs": 1969900,
    "text": "Like one problem that happens is adversarial sampling, where you select, like this is maybe"
  },
  {
    "startMs": 1975980,
    "text": "a little bit confusing, but I'll try to explain it clearly."
  },
  {
    "startMs": 1978440,
    "text": "You select cases that are specific failures from a particular model class."
  },
  {
    "startMs": 1984720,
    "text": "So models are jagged."
  },
  {
    "startMs": 1986580,
    "text": "And so you just look at, okay, like Sonnet 5 can't do these examples."
  },
  {
    "startMs": 1989820,
    "text": "I promote them to my eval."
  },
  {
    "startMs": 1990920,
    "text": "Well, Opus 5 may be able to solve them all just because intelligence is jagged."
  },
  {
    "startMs": 1996140,
    "text": "And so like a particular pocket of one model might just get solved by another."
  },
  {
    "startMs": 2000800,
    "text": "So typically, a good recommendation is promote eval cases."
  },
  {
    "startMs": 2006200,
    "text": "Promote cases of an eval set that are hard or interesting from first principles, from your"
  },
  {
    "startMs": 2009800,
    "text": "own intuition, from looking at production data, not necessarily completely sampled from the"
  },
  {
    "startMs": 2014280,
    "text": "failures of any given model."
  },
  {
    "startMs": 2015380,
    "text": "Because that can result in kind of this odd distribution that's shaped to the valleys"
  },
  {
    "startMs": 2019600,
    "text": "of one particular like jagged models intelligence that don't apply when you upgrade models."
  },
  {
    "startMs": 2024380,
    "text": "So like, can you use bucket-based evals?"
  },
  {
    "startMs": 2026320,
    "text": "Do you have to use binary classification?"
  },
  {
    "startMs": 2028040,
    "text": "I'm being a little bit overly prescriptive."
  },
  {
    "startMs": 2030140,
    "text": "You obviously can."
  },
  {
    "startMs": 2031400,
    "text": "But just as a general rule, I see people going too far in that direction."
  },
  {
    "startMs": 2035100,
    "text": "So there'll be some very arbitrary scale between zero and 10 with unclear grading and rules"
  },
  {
    "startMs": 2039320,
    "text": "that even a very small model can't resolve."
  },
  {
    "startMs": 2041160,
    "text": "So you obviously, if you have a well thought through scale, you obviously can do that as"
  },
  {
    "startMs": 2046620,
    "text": "long as it satisfies what we talked about."
  },
  {
    "startMs": 2048340,
    "text": "It has like low variance."
  },
  {
    "startMs": 2049700,
    "text": "It's reproduced across models."
  },
  {
    "startMs": 2051520,
    "text": "A human grader can kind of distinguish between these two grades."
  },
  {
    "startMs": 2054540,
    "text": "That's totally fine."
  },
  {
    "startMs": 2055800,
    "text": "I'm just saying binary is like a shortcut of like if to simplify the problem usually."
  },
  {
    "startMs": 2060180,
    "text": "And also for grading, you often want to use smaller, cheaper models."
  },
  {
    "startMs": 2062940,
    "text": "So you really don't want Fable doing your grading and depending on your cost sensitivity."
  },
  {
    "startMs": 2067280,
    "text": "So those are just the recommendations."
  },
  {
    "startMs": 2069060,
    "text": "But obviously, if it's well thought out, that's totally fine."
  },
  {
    "startMs": 2071080,
    "text": "You can think about using, yeah, like a multi-category classification."
  },
  {
    "startMs": 2077540,
    "text": "Yeah, that's a good point."
  },
  {
    "startMs": 2078900,
    "text": "Like when to build evals for what types of problems?"
  },
  {
    "startMs": 2082340,
    "text": "Like is it worth building it for my like personal like use cases?"
  },
  {
    "startMs": 2086120,
    "text": "Are we only doing it for things that are kind of production grade?"
  },
  {
    "startMs": 2089160,
    "text": "I think it depends on, let's see."
  },
  {
    "startMs": 2096120,
    "text": "It probably depends a bit on the importance of the task."
  },
  {
    "startMs": 2099020,
    "text": "If it's even a personal task, you feel that that is quite important, then it would be perfectly"
  },
  {
    "startMs": 2102860,
    "text": "reasonable to build an eval for it."
  },
  {
    "startMs": 2103960,
    "text": "I think that's totally fine."
  },
  {
    "startMs": 2104800,
    "text": "I think people do that."
  },
  {
    "startMs": 2106100,
    "text": "It kind of varies by person."
  },
  {
    "startMs": 2107260,
    "text": "I personally haven't necessarily built many personal evals for like my personal automations."
  },
  {
    "startMs": 2112040,
    "text": "But I think if it's a task that's highly important, that's in particular, okay, I'll give a good example."
  },
  {
    "startMs": 2121720,
    "text": "Like if say if you work in finance and like you've automated parts of your personal workflow, but quality is extremely important."
  },
  {
    "startMs": 2128520,
    "text": "Like this is a work output that's going to be shared externally or something that would absolutely build evals, even though it's something you've built personally."
  },
  {
    "startMs": 2135120,
    "text": "If it's my personal assistant, that's like, you know, it's like a meeting."
  },
  {
    "startMs": 2139760,
    "text": "It's like an app that I built to help me manage my calendar."
  },
  {
    "startMs": 2142780,
    "text": "I may or may not build evals."
  },
  {
    "startMs": 2144160,
    "text": "It depends upon the context."
  },
  {
    "startMs": 2145560,
    "text": "But I do think obviously if it's something that's important for your work, it's like, you know, producing deliverables that are, you know, externally viewed or doing something very sensitive to ask."
  },
  {
    "startMs": 2153680,
    "text": "You obviously want evals, even if it's a personal agent."
  },
  {
    "startMs": 2156060,
    "text": "And I actually worked at Langchain before Anthropic, but I can't advocate for any one or the other."
  },
  {
    "startMs": 2164460,
    "text": "Yeah, I plead the fifth on that."
  },
  {
    "startMs": 2167000,
    "text": "I mean, look, I've used many of them."
  },
  {
    "startMs": 2169920,
    "text": "I think actually if you're working at relatively small scale, you obviously could roll your own."
  },
  {
    "startMs": 2174800,
    "text": "Like you don't necessarily need a fancy observability system."
  },
  {
    "startMs": 2176860,
    "text": "In fact, we're going to be adding to the skill the ability to just like a simple data viewer where we're actually added to the skill."
  },
  {
    "startMs": 2185020,
    "text": "So Cloud can kind of, Cloud's very good at kind of building your UIs and visualization surfaces."
  },
  {
    "startMs": 2190320,
    "text": "So it can kind of roll."
  },
  {
    "startMs": 2191660,
    "text": "If you have like a very bespoke problem you want to look at the data for, it can kind of roll you whatever app you want very easily."
  },
  {
    "startMs": 2197680,
    "text": "And not very expensively either."
  },
  {
    "startMs": 2199740,
    "text": "No, this is a key point."
  },
  {
    "startMs": 2201000,
    "text": "It's not guaranteed to converge at all."
  },
  {
    "startMs": 2202560,
    "text": "So I've seen many cases in which the Hill Climber gets stuck."
  },
  {
    "startMs": 2205640,
    "text": "And actually we built, you can read the blog, but we built a fallback in the Hill Climber where if it gets stuck for multiple rounds, it'll actually go and basically try to cluster the remaining failure cases."
  },
  {
    "startMs": 2215740,
    "text": "And it'll try to look for, so for example, here's the problem."
  },
  {
    "startMs": 2218780,
    "text": "Oftentimes you'll run the Hill Climber without having run like a build eval command or without having built a very like carefully constructed eval set."
  },
  {
    "startMs": 2227340,
    "text": "So the Hill Climber will go and look at your evals and try to find, ah, are these cases ambiguous?"
  },
  {
    "startMs": 2232300,
    "text": "Like we talked about here, maybe that last 20% are all faulty greater examples, ambiguous instructions."
  },
  {
    "startMs": 2241520,
    "text": "This happens all the time."
  },
  {
    "startMs": 2242540,
    "text": "This is like very common."
  },
  {
    "startMs": 2244140,
    "text": "So Hill Climber's just stuck, not because Cloud can't do better."
  },
  {
    "startMs": 2246500,
    "text": "It's because the eval is poorly structured."
  },
  {
    "startMs": 2247980,
    "text": "So the Hill Climbing skill will try to find that for you as we added that."
  },
  {
    "startMs": 2252520,
    "text": "In fact, when I did this Hill Climbing I talk about here, this second stage actually went, it kind of got stuck and it went and basically clustered the eval cases and looked at, okay, why are the failure cases?"
  },
  {
    "startMs": 2264080,
    "text": "Why are they failing?"
  },
  {
    "startMs": 2264880,
    "text": "What's going on?"
  },
  {
    "startMs": 2265620,
    "text": "It found a bunch of ambiguous cases."
  },
  {
    "startMs": 2267240,
    "text": "It rewrote them."
  },
  {
    "startMs": 2268580,
    "text": "And anyway, so it'll actually do that for you."
  },
  {
    "startMs": 2270960,
    "text": "But that happens all the time."
  },
  {
    "startMs": 2272120,
    "text": "It's absolutely not guaranteed to converge."
  },
  {
    "startMs": 2274060,
    "text": "And usually it's because of failures in your eval or like kind of misconfigurations in the eval."
  },
  {
    "startMs": 2277980,
    "text": "So that's really interesting."
  },
  {
    "startMs": 2280300,
    "text": "You have a lot of respect for the DSPy team and Omar and colleagues."
  },
  {
    "startMs": 2286160,
    "text": "This actually just is a skill built like bundled with Cloud Code."
  },
  {
    "startMs": 2289900,
    "text": "The skill is also open source."
  },
  {
    "startMs": 2291540,
    "text": "So you could consider taking that skill and like vendoring it in any other surface."
  },
  {
    "startMs": 2295380,
    "text": "Even if you want to, it could probably work with DSPy."
  },
  {
    "startMs": 2299080,
    "text": "It's just some very simple general instructions for setting up a hill climbing loop following basically these principles."
  },
  {
    "startMs": 2305520,
    "text": "So that could be compatible with DSPy."
  },
  {
    "startMs": 2312360,
    "text": "I haven't used DSPy."
  },
  {
    "startMs": 2314240,
    "text": "So I don't know like the details and how it's structured."
  },
  {
    "startMs": 2317120,
    "text": "But it's possible the skill guidance here could be compatible with it."
  },
  {
    "startMs": 2320660,
    "text": "If you want to like effectively implement this or run it in DSPy."
  },
  {
    "startMs": 2324520,
    "text": "The skill is open source."
  },
  {
    "startMs": 2325800,
    "text": "You said choosing effort level."
  },
  {
    "startMs": 2329080,
    "text": "Yeah, this is a classic one."
  },
  {
    "startMs": 2331080,
    "text": "Effort's confusing, isn't it?"
  },
  {
    "startMs": 2332740,
    "text": "Yeah."
  },
  {
    "startMs": 2333380,
    "text": "I'm also confused by effort."
  },
  {
    "startMs": 2335620,
    "text": "It's, yeah."
  },
  {
    "startMs": 2337600,
    "text": "It's non-obvious."
  },
  {
    "startMs": 2339200,
    "text": "So this is, so what's nice is if you have an eval, the hill climber can absolutely try to configure effort for you."
  },
  {
    "startMs": 2353220,
    "text": "Now I get it."
  },
  {
    "startMs": 2354180,
    "text": "It's like not that easy to have an eval."
  },
  {
    "startMs": 2356620,
    "text": "So like it's not applicable to every problem."
  },
  {
    "startMs": 2358740,
    "text": "But if you happen to have an eval, then this hill climber can search through your model effort configurations."
  },
  {
    "startMs": 2363000,
    "text": "And it follows a very simple principle."
  },
  {
    "startMs": 2364560,
    "text": "The principle is just simply that typically the stronger model at lower effort is like more effective."
  },
  {
    "startMs": 2371420,
    "text": "Or sorry, is more token efficient and therefore cheaper than like a weaker model at high effort."
  },
  {
    "startMs": 2376180,
    "text": "That's like the general principle."
  },
  {
    "startMs": 2377380,
    "text": "That actually turns out to be true a lot."
  },
  {
    "startMs": 2379140,
    "text": "So it'll kind of search through the different models at low effort first."
  },
  {
    "startMs": 2383080,
    "text": "And basically continue that search until you see degradation performance below the baseline."
  },
  {
    "startMs": 2387960,
    "text": "So for here, I don't show it."
  },
  {
    "startMs": 2389180,
    "text": "But like when I tried Haiku 4, this was 4.5."
  },
  {
    "startMs": 2392100,
    "text": "Haiku, we now have 5.5, which is quite a bit better."
  },
  {
    "startMs": 2395240,
    "text": "4.5 at low effort was a significant regression."
  },
  {
    "startMs": 2397620,
    "text": "So we admitted that data point."
  },
  {
    "startMs": 2399560,
    "text": "But basically, if you have an eval, hill climbing is a very good way to try to search for a good model effort combination."
  },
  {
    "startMs": 2405740,
    "text": "More general effort guidance, if you don't have an eval, is obviously tricky and subtle."
  },
  {
    "startMs": 2409320,
    "text": "And like I think people typically, I will say I think the guidance I recommend generally in effort is try stronger model low effort in general."
  },
  {
    "startMs": 2419800,
    "text": "Opus 5.5, low effort."
  },
  {
    "startMs": 2422160,
    "text": "Try that, basically."
  },
  {
    "startMs": 2423660,
    "text": "Better models, low effort are often more token efficient, therefore cheaper than weaker models at high effort."
  },
  {
    "startMs": 2429980,
    "text": "That's like the key point I'll leave you with."
  },
  {
    "startMs": 2432420,
    "text": "Low, not medium?"
  },
  {
    "startMs": 2434340,
    "text": "Try low to start."
  },
  {
    "startMs": 2436620,
    "text": "You can always dial back up."
  },
  {
    "startMs": 2438300,
    "text": "But basically, I like very, very smart model, not trying too hard."
  },
  {
    "startMs": 2443520,
    "text": "That's often a sweet spot for kind of cost performance."
  },
  {
    "startMs": 2449420,
    "text": "Obviously, you can dial up effort."
  },
  {
    "startMs": 2452680,
    "text": "But I think the counter is people often start with something like max effort."
  },
  {
    "startMs": 2456100,
    "text": "It's extremely token hungry."
  },
  {
    "startMs": 2458640,
    "text": "So it is actually wise to often start like smarter model, low effort, and kind of go from there if you need it."
  },
  {
    "startMs": 2466600,
    "text": "Yeah."
  },
  {
    "startMs": 2467720,
    "text": "Yeah."
  },
  {
    "startMs": 2468220,
    "text": "Actually, we talked about this in the blog here."
  },
  {
    "startMs": 2472940,
    "text": "I'll open that up."
  },
  {
    "startMs": 2474780,
    "text": "I don't have internet."
  },
  {
    "startMs": 2475640,
    "text": "That's fine."
  },
  {
    "startMs": 2476440,
    "text": "I'll just tell you then."
  },
  {
    "startMs": 2478300,
    "text": "Yeah."
  },
  {
    "startMs": 2478560,
    "text": "So it's basically telling Claude how hard to think."
  },
  {
    "startMs": 2480260,
    "text": "So low effort is saying like don't think as hard."
  },
  {
    "startMs": 2484580,
    "text": "As you scale it up, it's saying think harder."
  },
  {
    "startMs": 2487700,
    "text": "And that's kind of why a more fundamentally intelligent model at low effort can actually be more token efficient and therefore cheaper than a cheaper model in terms of base tokens at high effort."
  },
  {
    "startMs": 2499940,
    "text": "Because that, for example, Sonic 5 at high effort is spending lots and lots of tokens thinking where smarter model at low effort will spend fewer tokens thinking at a higher baseline of intelligence."
  },
  {
    "startMs": 2511060,
    "text": "That's kind of the intuition."
  },
  {
    "startMs": 2513080,
    "text": "Yeah."
  },
  {
    "startMs": 2513640,
    "text": "So the question is about setting budgets in hill climbing."
  },
  {
    "startMs": 2515780,
    "text": "So basically, when you run this skill and set this up, it'll ask you for a budget."
  },
  {
    "startMs": 2520640,
    "text": "So it will manage that for it."
  },
  {
    "startMs": 2523580,
    "text": "It'll track the budget you allocate for the hill climbing round."
  },
  {
    "startMs": 2526360,
    "text": "That can definitely happen, though."
  },
  {
    "startMs": 2527420,
    "text": "And that's actually why this is a classic example."
  },
  {
    "startMs": 2530260,
    "text": "That's why designing good evaluation is really important."
  },
  {
    "startMs": 2532760,
    "text": "If you have a poorly specified evaluation, imagine, for example, you have an eval where that final 20% is impossible."
  },
  {
    "startMs": 2539240,
    "text": "You burn a huge number of tokens trying to solve those final 20% of problems that are effectively not solvable."
  },
  {
    "startMs": 2545980,
    "text": "Kind of an eval building."
  },
  {
    "startMs": 2547780,
    "text": "The question is about kind of human loop, kind of human cloud collaboration."
  },
  {
    "startMs": 2551360,
    "text": "So I think that humans still have the role to kind of steer the goal."
  },
  {
    "startMs": 2562120,
    "text": "Like, what are we actually trying to evaluate?"
  },
  {
    "startMs": 2563700,
    "text": "And why is it important?"
  },
  {
    "startMs": 2564580,
    "text": "So it's kind of taste."
  },
  {
    "startMs": 2565420,
    "text": "It's important."
  },
  {
    "startMs": 2566000,
    "text": "So what are you actually trying to answer?"
  },
  {
    "startMs": 2567260,
    "text": "For example, I'll take an example from, like, Hamill's public dogfooding of our build eval skill."
  },
  {
    "startMs": 2577920,
    "text": "So it's like, okay, it's a customer support app."
  },
  {
    "startMs": 2580120,
    "text": "So say that's your application."
  },
  {
    "startMs": 2581400,
    "text": "What matters?"
  },
  {
    "startMs": 2582100,
    "text": "What's important?"
  },
  {
    "startMs": 2582680,
    "text": "What do you care about?"
  },
  {
    "startMs": 2583360,
    "text": "Those are things that a human can guide cloud."
  },
  {
    "startMs": 2585780,
    "text": "Now, once those things are clearly specified, cloud can look at, like, raw traces and determine effectively, like, okay, where is the app likely falling short of my goals?"
  },
  {
    "startMs": 2598320,
    "text": "So that's, I think, kind of the nice collaboration you can have in place."
  },
  {
    "startMs": 2602460,
    "text": "And that's actually when, like, kind of looking at the data is very helpful."
  },
  {
    "startMs": 2605800,
    "text": "So cloud can kind of help to steer you."
  },
  {
    "startMs": 2607340,
    "text": "Okay, I've looked at the data."
  },
  {
    "startMs": 2608680,
    "text": "Here's a bunch of cases that look problematic."
  },
  {
    "startMs": 2610060,
    "text": "And that's just like having an online evaluator."
  },
  {
    "startMs": 2612900,
    "text": "So you, like, imagine you have a tracing system."
  },
  {
    "startMs": 2614460,
    "text": "You have an online evaluator."
  },
  {
    "startMs": 2615840,
    "text": "It has some, like, grading for, like, customer satisfaction."
  },
  {
    "startMs": 2619920,
    "text": "And it's flagging, like, here's problematic traces for you."
  },
  {
    "startMs": 2622760,
    "text": "It's the same kind of thing."
  },
  {
    "startMs": 2623600,
    "text": "Like, you're telling cloud."
  },
  {
    "startMs": 2624660,
    "text": "You're telling the online evaluator."
  },
  {
    "startMs": 2626060,
    "text": "Here's what I care about."
  },
  {
    "startMs": 2626880,
    "text": "Here's what I want you to check."
  },
  {
    "startMs": 2628080,
    "text": "It can look at the data."
  },
  {
    "startMs": 2628960,
    "text": "It can elevate a bunch of cases."
  },
  {
    "startMs": 2630460,
    "text": "Like, oh, these look problematic."
  },
  {
    "startMs": 2631840,
    "text": "Then given those cases, you can, like, review them, understand them, and then say, okay, yes, build me an eval based upon these."
  },
  {
    "startMs": 2637920,
    "text": "That's typically a very nice marriage between, like, cloud and human when it comes to building evals."
  },
  {
    "startMs": 2643080,
    "text": "Yeah."
  },
  {
    "startMs": 2643620,
    "text": "The question is, like, who's responsible for building evals?"
  },
  {
    "startMs": 2645480,
    "text": "I mean, it depends on the problem."
  },
  {
    "startMs": 2646520,
    "text": "It depends upon, I mean, like, AnthropBank, everyone is empowered to build evals."
  },
  {
    "startMs": 2650760,
    "text": "So it's not really delineated per team."
  },
  {
    "startMs": 2653000,
    "text": "It's typically based on ownership, like, of the product."
  },
  {
    "startMs": 2655260,
    "text": "And I think in particular when you have skills like this and make it easier to build evals to hill climb, it really opens up the number of people that can feasibly kind of build and design evals, which is actually quite nice."
  },
  {
    "startMs": 2667920,
    "text": "So we've seen, like, AnthropBank, many, many different types of roles build evaluations because cloud's so involved and can help so much."
  },
  {
    "startMs": 2676440,
    "text": "Much of the tedium of, like, you know, building an eval runner and stuff, that's, like, cloud can handle all that."
  },
  {
    "startMs": 2683080,
    "text": "So really it's about what we were talking about before, the human taste, guiding the process, what do I care about?"
  },
  {
    "startMs": 2687980,
    "text": "And that's very, many different types of roles can kind of serve that guidance or provide that steering."
  },
  {
    "startMs": 2694860,
    "text": "Let's see if I have a good one."
  },
  {
    "startMs": 2696120,
    "text": "Right."
  },
  {
    "startMs": 2703140,
    "text": "Off the top of my head."
  },
  {
    "startMs": 2706540,
    "text": "I've seen..."
  },
  {
    "startMs": 2708220,
    "text": "Okay, I have a good one."
  },
  {
    "startMs": 2709940,
    "text": "I have seen evals for generative UIs."
  },
  {
    "startMs": 2716020,
    "text": "So that's kind of a good example."
  },
  {
    "startMs": 2718700,
    "text": "Actually, I think in that case, it was a pairwise judge back to the point that you were making."
  },
  {
    "startMs": 2723100,
    "text": "Like, so it was basically comparing, like, configuration A, B, what is better between, like, two different generative UIs, and, like, the greater would do kind of a pairwise comparison."
  },
  {
    "startMs": 2734740,
    "text": "I think then those results were rolled across many trials and scored with something like an ELO score."
  },
  {
    "startMs": 2739960,
    "text": "So that's an example of, like, generative UI with a pairwise evaluator I've seen before."
  },
  {
    "startMs": 2745620,
    "text": "There's many other examples I can't think of at the top of my head."
  },
  {
    "startMs": 2748100,
    "text": "But people use LMS judge all the time for, like, retrieval tasks or, like, writing tasks, categorizing traces, looking at traces and saying, oh, this is bad or good."
  },
  {
    "startMs": 2760980,
    "text": "So, like, it's extremely common."
  },
  {
    "startMs": 2763220,
    "text": "Yeah."
  },
  {
    "startMs": 2763920,
    "text": "So, okay, a lot of the motivation for the hill climbing work is kind of flowing from auto research."
  },
  {
    "startMs": 2771360,
    "text": "Auto research is motivated heavily by the fact that models are getting very, very good."
  },
  {
    "startMs": 2776240,
    "text": "They're very good at self-correction."
  },
  {
    "startMs": 2777740,
    "text": "If you give them a well-structured problem, they can hill climb on."
  },
  {
    "startMs": 2782040,
    "text": "So I think Karpathy's case is a good example of, I think, I can't remember the precise problem."
  },
  {
    "startMs": 2786180,
    "text": "I think that general set of work with many different formulations."
  },
  {
    "startMs": 2789060,
    "text": "Like, it could be, you know, like, drive down loss or something."
  },
  {
    "startMs": 2793140,
    "text": "So, again, there's, like, a very clear verifiable objective."
  },
  {
    "startMs": 2796360,
    "text": "And Claude can look at that objective and use it as a guide to hill climb."
  },
  {
    "startMs": 2798980,
    "text": "I have, yeah, I didn't show it here, but I have some other examples doing auto research in this kind of hill climbing process."
  },
  {
    "startMs": 2806880,
    "text": "And it would, this kind of, this command in particular is actually designed more for eval-based hill climbing where you have, like, an eval set with a split."
  },
  {
    "startMs": 2816860,
    "text": "Auto research was looking more like you build an artifact and, like, grade it, like, look at the loss or something."
  },
  {
    "startMs": 2821960,
    "text": "But the setup could probably be very easily kind of modified to work with auto research with the very same intuition."
  },
  {
    "startMs": 2828180,
    "text": "The point is that models are good at autonomous work over long horizons."
  },
  {
    "startMs": 2831980,
    "text": "When given a clear objective or verifiable, effectively, a reward that they can kind of measure and use to inform what their next move is,"
  },
  {
    "startMs": 2842200,
    "text": "they're very, very good at autonomously solving problems."
  },
  {
    "startMs": 2844100,
    "text": "So it's driven by the same intuition as auto research."
  },
  {
    "startMs": 2846820,
    "text": "So you start with hill climb and then extend it to auto research?"
  },
  {
    "startMs": 2851240,
    "text": "The methodology for hill climb would, so it's just a question of problem formulation, just to be clear."
  },
  {
    "startMs": 2858480,
    "text": "So, like, the auto research problem formulation was initially focused more on, like, designing, for example, like a network with, for example, solving some problem where you have, like, a verifiable objective, like, loss."
  },
  {
    "startMs": 2872980,
    "text": "That's really it."
  },
  {
    "startMs": 2873780,
    "text": "The hill climbing process, though, was, like, very similar where, like, Claude runs some experiments."
  },
  {
    "startMs": 2877620,
    "text": "It measures loss."
  },
  {
    "startMs": 2878960,
    "text": "It looks at the results."
  },
  {
    "startMs": 2880180,
    "text": "It then adapts."
  },
  {
    "startMs": 2881140,
    "text": "It does something else."
  },
  {
    "startMs": 2881880,
    "text": "So it's the same kind of hill climbing loop."
  },
  {
    "startMs": 2883740,
    "text": "But, like, in that particular case, it's just applied to, like, a slightly different domain."
  },
  {
    "startMs": 2887540,
    "text": "So it's, like, you could probably take this skill and basically say, okay, set this up for if you had, like, the NanoGDPT repo, you could use this skill with it, I'm sure."
  },
  {
    "startMs": 2896440,
    "text": "It would cloud it to figure out how to adapt things."
  },
  {
    "startMs": 2898240,
    "text": "If you're interested in that, just, like, try it."
  },
  {
    "startMs": 2900580,
    "text": "But the core process is the same."
  },
  {
    "startMs": 2902640,
    "text": "The core intuition is also the same, that, like, models are very, very effective at using, at self-correction and hill climbing."
  },
  {
    "startMs": 2910340,
    "text": "That's the key intuition."
  },
  {
    "startMs": 2912320,
    "text": "Yep."
  },
  {
    "startMs": 2913740,
    "text": "Yeah."
  },
  {
    "startMs": 2914240,
    "text": "I kind of try to do that in, like, this example where I'm looking at cost and performance."
  },
  {
    "startMs": 2919880,
    "text": "So, like, kind of jointly optimizing across those two."
  },
  {
    "startMs": 2922540,
    "text": "And, yeah, you can just tell Claude the multi-objective."
  },
  {
    "startMs": 2926440,
    "text": "And usually it'll just kind of measure multiple objectives and take that into account."
  },
  {
    "startMs": 2932100,
    "text": "That's certainly feasible."
  },
  {
    "startMs": 2933400,
    "text": "And this is a good example of that."
  },
  {
    "startMs": 2938640,
    "text": "I mean, yeah, I haven't pushed that too far."
  },
  {
    "startMs": 2941200,
    "text": "I think what I've typically looked at is cost and performance."
  },
  {
    "startMs": 2944340,
    "text": "And, like, speed's another dimension."
  },
  {
    "startMs": 2945880,
    "text": "So you can let cost, performance, speed."
  },
  {
    "startMs": 2947100,
    "text": "That feels like a pretty reasonable, like, trifecta of things that you would, like, kind of jointly measure."
  },
  {
    "startMs": 2950620,
    "text": "Yeah."
  },
  {
    "startMs": 2952460,
    "text": "Yeah."
  },
  {
    "startMs": 2952900,
    "text": "I mean, I think that as you kind of scale up, it just hobbles the process of hill climbing a lot because there's lots of different objectives that Claude's trying to balance when making these changes."
  },
  {
    "startMs": 2961720,
    "text": "So I think it'll just, it'll make the problem harder."
  },
  {
    "startMs": 2964260,
    "text": "But, like, it is doable."
  },
  {
    "startMs": 2966220,
    "text": "Maybe you could think about kind of changing one thing at once, doing them serially."
  },
  {
    "startMs": 2971540,
    "text": "Yeah."
  },
  {
    "startMs": 2972040,
    "text": "So that's a very good question."
  },
  {
    "startMs": 2973120,
    "text": "So the prompt artist stuff, which actually, of all the things I talked about, is the simplest thing to try and often is, like, extremely, it's, like, do the simple thing first."
  },
  {
    "startMs": 2981220,
    "text": "It's very simple and often very effective."
  },
  {
    "startMs": 2983140,
    "text": "This general, the general guidance in the skill applies to the model delineations a little bit on, like, it's not precise."
  },
  {
    "startMs": 2994060,
    "text": "These are well-known anti-patterns that definitively frontier models like the 5.5 series don't need anymore."
  },
  {
    "startMs": 3000360,
    "text": "How far back do you need to go until these patterns are still, like, appropriate?"
  },
  {
    "startMs": 3004320,
    "text": "It's, like, not quite clear."
  },
  {
    "startMs": 3005640,
    "text": "I've tried to do some testing."
  },
  {
    "startMs": 3006700,
    "text": "It's maybe, like, Opus, like, 4.5 is the point at which these things start to, like, become less relevant."
  },
  {
    "startMs": 3013240,
    "text": "So it's a little bit squishy."
  },
  {
    "startMs": 3014220,
    "text": "I don't know, like, precisely the model at which, like, you need to remove these."
  },
  {
    "startMs": 3017360,
    "text": "But my view is basically when you do a migration to the latest, to, like, to, like, Opus 5.5, just do this."
  },
  {
    "startMs": 3024220,
    "text": "Like, definitively, like, the latest frontier models don't need these patterns."
  },
  {
    "startMs": 3028480,
    "text": "Yeah, yeah."
  },
  {
    "startMs": 3029120,
    "text": "Okay, good point."
  },
  {
    "startMs": 3030140,
    "text": "So the skill does have some model-specific caveats."
  },
  {
    "startMs": 3033320,
    "text": "So the skill takes that into account a little bit."
  },
  {
    "startMs": 3035060,
    "text": "If you're, for example, going to, like, Haiku, I get it."
  },
  {
    "startMs": 3037560,
    "text": "Like, maybe Haiku, like, isn't quite as autonomous."
  },
  {
    "startMs": 3040880,
    "text": "Maybe it does need slightly more boosters."
  },
  {
    "startMs": 3042520,
    "text": "I don't know if that's true."
  },
  {
    "startMs": 3043440,
    "text": "I'm just kind of suggesting something."
  },
  {
    "startMs": 3048060,
    "text": "It'll take that into account."
  },
  {
    "startMs": 3049220,
    "text": "So it does have some model-specific guidance built in."
  },
  {
    "startMs": 3052180,
    "text": "It's, like, smart to figure out, like, okay, here's the model you're working with,"
  },
  {
    "startMs": 3054520,
    "text": "and here's the anti-patterns that we know are not needed anymore."
  },
  {
    "startMs": 3058240,
    "text": "But it's a good call-out."
  },
  {
    "startMs": 3060800,
    "text": "Yeah."
  },
  {
    "startMs": 3061160,
    "text": "Yeah."
  },
  {
    "startMs": 3061240,
    "text": "Yeah."
  },
  {
    "startMs": 3064800,
    "text": "Yeah."
  },
  {
    "startMs": 3066060,
    "text": "It is not as..."
  },
  {
    "startMs": 3067860,
    "text": "Yeah."
  },
  {
    "startMs": 3068220,
    "text": "I should get back to you to, like, to be really clear about this."
  },
  {
    "startMs": 3073760,
    "text": "It is not a hard cap."
  },
  {
    "startMs": 3076400,
    "text": "It is basically telling Claude, emphasizing how much thinking to do to solve the problem."
  },
  {
    "startMs": 3081620,
    "text": "So, like, for example, even if you're on very high effort, you give it a simple problem that won't just arbitrarily spend huge amounts of tokens verifying it."
  },
  {
    "startMs": 3089920,
    "text": "There's some..."
  },
  {
    "startMs": 3091420,
    "text": "Claude has some thoughtfulness about, like, what problems require, you know, all the extra thinking."
  },
  {
    "startMs": 3097000,
    "text": "So, like..."
  },
  {
    "startMs": 3098320,
    "text": "And I don't know if I have any evals that show that."
  },
  {
    "startMs": 3101480,
    "text": "Yeah, maybe not."
  },
  {
    "startMs": 3102320,
    "text": "But it is suggested thinking."
  },
  {
    "startMs": 3107100,
    "text": "So, that's kind of the way I think."
  },
  {
    "startMs": 3107920,
    "text": "It's basically telling Claude how hard to think about a problem."
  },
  {
    "startMs": 3110660,
    "text": "The problem comes when you have, like, a harder problem, like a frontier coding problem."
  },
  {
    "startMs": 3114740,
    "text": "That's where, at, for example, extra high effort or max effort, you can see huge differences in token usage from, like, medium or low without much difference in actual outcome."
  },
  {
    "startMs": 3123000,
    "text": "Because Claude will do more work to verify the result, for example."
  },
  {
    "startMs": 3125960,
    "text": "So, I think out in the frontier regime of problems, that's where these effort differences can really result in, like, big token spend differences."
  },
  {
    "startMs": 3133680,
    "text": "And that's where you need to be more careful."
  },
  {
    "startMs": 3135900,
    "text": "Yeah."
  },
  {
    "startMs": 3136260,
    "text": "I mean, this is..."
  },
  {
    "startMs": 3139760,
    "text": "So, if you kind of think about what's happening, it's not dissimilar to training."
  },
  {
    "startMs": 3145800,
    "text": "And basically, you're giving a model some reward or objective."
  },
  {
    "startMs": 3149100,
    "text": "And it is, indeed, trying to make modifications to the system to improve the objective."
  },
  {
    "startMs": 3154940,
    "text": "So, yeah."
  },
  {
    "startMs": 3155980,
    "text": "Philosophically, that is similar."
  },
  {
    "startMs": 3157560,
    "text": "And, in fact, even we're used..."
  },
  {
    "startMs": 3159300,
    "text": "You know, we employ some simple principles, like test, train, splits, that are often employed, like, at training time."
  },
  {
    "startMs": 3164260,
    "text": "But the nice thing is these are kind of runtime modifications you can make."
  },
  {
    "startMs": 3167540,
    "text": "So, this is basically a way to employ some of those principles."
  },
  {
    "startMs": 3170040,
    "text": "But, basically, like, once a model has been deployed and, like, you're modifying services like prompts or skills,"
  },
  {
    "startMs": 3175700,
    "text": "you can employ some of those same principles that we use in training, but, like, in your particular domain."
  },
  {
    "startMs": 3182720,
    "text": "That's kind of the big idea."
  },
  {
    "startMs": 3184740,
    "text": "Yeah."
  },
  {
    "startMs": 3185180,
    "text": "So, like, is Claude aware of this being evaled?"
  },
  {
    "startMs": 3187780,
    "text": "Right."
  },
  {
    "startMs": 3188280,
    "text": "Exactly."
  },
  {
    "startMs": 3189120,
    "text": "That..."
  },
  {
    "startMs": 3189560,
    "text": "I think that's something we've published on a fair amount that is a concern in the..."
  },
  {
    "startMs": 3196760,
    "text": "That is certainly a concern kind of in training, in different stages of training."
  },
  {
    "startMs": 3201760,
    "text": "Absolutely."
  },
  {
    "startMs": 3202160,
    "text": "For what we're talking about here, I do think it could be somewhat of a concern."
  },
  {
    "startMs": 3212860,
    "text": "I think the bigger concern than what we're talking about here is typically more overfitting."
  },
  {
    "startMs": 3216900,
    "text": "So, Claude may know it's kind of being evaluated."
  },
  {
    "startMs": 3219640,
    "text": "But it will find ways to..."
  },
  {
    "startMs": 3224100,
    "text": "He'll climb in the objective based upon what's been provided in kind of this eval distribution that it has."
  },
  {
    "startMs": 3229680,
    "text": "The classic example being it'll find kind of prompt tricks and tuning that aren't applicable to production that, indeed,"
  },
  {
    "startMs": 3238640,
    "text": "improve performance significantly on your eval."
  },
  {
    "startMs": 3240440,
    "text": "I've seen that happen."
  },
  {
    "startMs": 3241560,
    "text": "That happens all the time."
  },
  {
    "startMs": 3242540,
    "text": "And it's not necessarily that Claude's, like, aware it's being evaled and saying, like, it's kind of some kind of unexpected behavior."
  },
  {
    "startMs": 3251640,
    "text": "It's more that, like, oh, okay, you've given it a prompt and Claude can find ways to shape this prompt to, like, make number go up and it'll just do that."
  },
  {
    "startMs": 3258980,
    "text": "That's why we use things like test training splits to try to regularize."
  },
  {
    "startMs": 3261440,
    "text": "So, I think overfitting is more the practical concern that we're going to think about here."
  },
  {
    "startMs": 3265080,
    "text": "But it is true that eval awareness is, like, a broader phenomenon that we have discussed in some of research blogs."
  },
  {
    "startMs": 3272300,
    "text": "Yeah, that's a good point."
  },
  {
    "startMs": 3273880,
    "text": "Kind of what's, like, the minimum kind of data set that's useful?"
  },
  {
    "startMs": 3278280,
    "text": "What is, like, the split between test and train?"
  },
  {
    "startMs": 3280400,
    "text": "I haven't looked at that in depth."
  },
  {
    "startMs": 3283220,
    "text": "Frankly, just ask Claude."
  },
  {
    "startMs": 3284880,
    "text": "Like, when you set this up and you run this command, Claude has some heuristics to reason about the size of the eval set that's actually helpful."
  },
  {
    "startMs": 3291640,
    "text": "Given the signal or, like, what you're trying to measure, Claude can kind of reason about, like, how many examples you should have."
  },
  {
    "startMs": 3297120,
    "text": "And there's some guidance in the skill for this."
  },
  {
    "startMs": 3299520,
    "text": "So, it's something that you can kind of work with Claude to kind of tune."
  },
  {
    "startMs": 3303660,
    "text": "But I haven't looked at, like, super specifically, like, oh, different eval set sizes."
  },
  {
    "startMs": 3308400,
    "text": "And, like, I think the central concern I would have is more, is probably overfitting and, like, variance."
  },
  {
    "startMs": 3317920,
    "text": "So, if you have, like, a very, very small number of samples, like, okay, is that really representative of what you care about?"
  },
  {
    "startMs": 3324480,
    "text": "Is it easier to overfit?"
  },
  {
    "startMs": 3325980,
    "text": "So, like, they may be out of distribution for your production, like, what you actually care about."
  },
  {
    "startMs": 3330540,
    "text": "They may be easier to overfit."
  },
  {
    "startMs": 3332320,
    "text": "They may be noisier."
  },
  {
    "startMs": 3334080,
    "text": "So, anyway, like, I think those are the problems that arise when you're in, like, the very, very small sample regime."
  },
  {
    "startMs": 3340620,
    "text": "Yeah, okay."
  },
  {
    "startMs": 3341320,
    "text": "So, the question is about hill climbing on tasks that are, like, very, very long duration."
  },
  {
    "startMs": 3344680,
    "text": "And so, that's funny because that's actually why I mentioned I typically do, like, simple surfaces like prompts because they're, like, very, very quick to change."
  },
  {
    "startMs": 3353020,
    "text": "You are right, though."
  },
  {
    "startMs": 3354100,
    "text": "There are some cases you're in a regime where, like, your changes could be very, very long duration."
  },
  {
    "startMs": 3359340,
    "text": "That effectively makes hill climbing much harder because, like, it's a much longer duration task then."
  },
  {
    "startMs": 3363080,
    "text": "So, it's a lot longer time."
  },
  {
    "startMs": 3364240,
    "text": "That's actually where that gets into our final point about kind of harness design."
  },
  {
    "startMs": 3367300,
    "text": "If you're going to do that, if you're doing, like, very, very long duration hill climbing, you actually need harnesses that can, like, remain reliable for, like, many, many hours to days."
  },
  {
    "startMs": 3376080,
    "text": "And so, we have that, obviously, internally."
  },
  {
    "startMs": 3378080,
    "text": "We use things like managed agents."
  },
  {
    "startMs": 3380000,
    "text": "I think it just becomes a bigger infrastructure lift."
  },
  {
    "startMs": 3382700,
    "text": "When you're doing that, it can be done."
  },
  {
    "startMs": 3384700,
    "text": "It obviously is harder because of what I just mentioned, infrastructure issues, failures, and just a lot longer time to see your signal."
  },
  {
    "startMs": 3392740,
    "text": "Single turn versus multi-turn evals."
  },
  {
    "startMs": 3394420,
    "text": "I mean, yeah, when you're building agents, you do typically need multi-turn evals."
  },
  {
    "startMs": 3401800,
    "text": "Claude can help you do that, particularly, like, the build eval command."
  },
  {
    "startMs": 3405400,
    "text": "It can look at traces."
  },
  {
    "startMs": 3406620,
    "text": "It can, if you give it a bunch of traces, it can help you promote particular traces to your eval set."
  },
  {
    "startMs": 3414460,
    "text": "In terms of grading, I mean, yeah, there's a lot of different philosophies on this."
  },
  {
    "startMs": 3418920,
    "text": "You can have evaluators looking at the entire trace trajectory."
  },
  {
    "startMs": 3421820,
    "text": "You can have evaluators looking at, like, the final agent outcome."
  },
  {
    "startMs": 3424420,
    "text": "There's a lot of different philosophies."
  },
  {
    "startMs": 3425900,
    "text": "The build eval command has a lot of guidance."
  },
  {
    "startMs": 3427660,
    "text": "So, like, if you have a particular, like, agent you're interested in, if you basically just give this command access to the traces,"
  },
  {
    "startMs": 3433660,
    "text": "it should be pretty effective at kind of shaping and designing the eval for you."
  },
  {
    "startMs": 3438360,
    "text": "And you can tell it, I want multi-turn, and here's what I'm considering."
  },
  {
    "startMs": 3442240,
    "text": "Cool."
  },
  {
    "startMs": 3443320,
    "text": "Interesting."
  },
  {
    "startMs": 3443680,
    "text": "So, like, for context engineering, what promotes reliability a lot versus what's, like, overhyped as a technique?"
  },
  {
    "startMs": 3451600,
    "text": "Let's see."
  },
  {
    "startMs": 3454840,
    "text": "So, okay."
  },
  {
    "startMs": 3456120,
    "text": "The key thing, I think, and I've actually mentioned this in the past."
  },
  {
    "startMs": 3460260,
    "text": "We mentioned in the paper, too."
  },
  {
    "startMs": 3461180,
    "text": "I think recursive language model work and RLM idea is quite good."
  },
  {
    "startMs": 3465500,
    "text": "And we actually employ that idea here."
  },
  {
    "startMs": 3467620,
    "text": "I think a very important thing in terms of context engineering reliability is decoupling brain in hands."
  },
  {
    "startMs": 3472600,
    "text": "So, basically, designing your system such that it, the key point is this, the context object is independent from the harness."
  },
  {
    "startMs": 3482480,
    "text": "So, the harness is basically, or Claude, can effectively query this context object that lives independently."
  },
  {
    "startMs": 3488080,
    "text": "That's beneficial for all these different reasons."
  },
  {
    "startMs": 3490280,
    "text": "One, the context is always recoverable."
  },
  {
    "startMs": 3493240,
    "text": "So, typically, imagine, alternatively, you just have, like, you know, a message history, you do a compaction event, and everything you compacted is gone."
  },
  {
    "startMs": 3499840,
    "text": "You can never recover it if you haven't, like, backed it up in a way like this."
  },
  {
    "startMs": 3503560,
    "text": "That's why I actually like this architecture a lot for reliability in terms of context engineering."
  },
  {
    "startMs": 3507580,
    "text": "And that's exactly what the RLM work advocated."
  },
  {
    "startMs": 3509980,
    "text": "Like, it's basically splitting the context object from the model, and the context object becomes something that the model can interrogate through different mechanisms."
  },
  {
    "startMs": 3516780,
    "text": "And that's exactly what we do here."
  },
  {
    "startMs": 3518960,
    "text": "So, interrogating that session log is basically done by the harness through different tools."
  },
  {
    "startMs": 3524000,
    "text": "And that's a very nice architecture."
  },
  {
    "startMs": 3525380,
    "text": "So, I think that's, like, one of the tricks that I found to be very nice for reliability, recoverability."
  },
  {
    "startMs": 3529840,
    "text": "And, in particular, building, like, long-duration agents, it's hard to know what will matter in the future."
  },
  {
    "startMs": 3536940,
    "text": "So, when you do a compaction, you don't necessarily know if you need to, like, go back and interrogate that context that you're summarizing."
  },
  {
    "startMs": 3544700,
    "text": "And when you have it backed up in an object like this, you can always do that."
  },
  {
    "startMs": 3548380,
    "text": "So, I think that's a trick that I like a lot."
  },
  {
    "startMs": 3551060,
    "text": "And the RLM work was very nice in that direction."
  },
  {
    "startMs": 3555640,
    "text": "Yep."
  },
  {
    "startMs": 3556860,
    "text": "Yeah."
  },
  {
    "startMs": 3557700,
    "text": "Well, I think that's why, you know, we spend a lot of time on good eval design."
  },
  {
    "startMs": 3567600,
    "text": "Yeah."
  },
  {
    "startMs": 3568300,
    "text": "These models are non-deterministic."
  },
  {
    "startMs": 3571700,
    "text": "In many cases, they operate in regimes where outcomes are not easily verifiable."
  },
  {
    "startMs": 3577600,
    "text": "Like, not everything is a coding problem."
  },
  {
    "startMs": 3579700,
    "text": "In many cases, you want agents to do things that are more judgment-based, more free-from-open-end outputs."
  },
  {
    "startMs": 3585180,
    "text": "In those regimes, you have to be very smart about your eval design so that you're giving Claude an objective that it can climb."
  },
  {
    "startMs": 3590500,
    "text": "And I think all the things we talked about in terms of, like, kind of verifier design, like good objectives, clear grading scales, multiple models can validate the result, low variance, all are, like, super important when designing, like, the problem that you want Claude to go and solve."
  },
  {
    "startMs": 3606860,
    "text": "So, I think that's why I mentioned it in this talk."
  },
  {
    "startMs": 3609960,
    "text": "When it comes to on hobbling frontier models, I think spending a lot of time up front in a very, very good eval design is a way to make sure that you're getting the most out of the model because if you say, tell the model, okay, optimize performance on this, like, very poorly structured evaluation, it won't go anywhere."
  },
  {
    "startMs": 3626820,
    "text": "And it's not the model's fault because you haven't thought through the eval."
  },
  {
    "startMs": 3629480,
    "text": "It's because cases are impossible or ambiguous."
  },
  {
    "startMs": 3631240,
    "text": "So, I think part of the message here is to be thoughtful and spend a lot of time up front in the eval design such that you kind of set up frontier models for success and, in fact, unhobble them."
  },
  {
    "startMs": 3642520,
    "text": "That was kind of the namesake of the talk."
  },
  {
    "startMs": 3645600,
    "text": "Maybe we're probably at time and everyone's probably tired."
  },
  {
    "startMs": 3648900,
    "text": "I was at a wedding last night, so I'm very tired."
  },
  {
    "startMs": 3651720,
    "text": "It was a funny timing, but it's very nice to be here, of course."
  },
  {
    "startMs": 3654540,
    "text": "And, actually, it's been a lot of fun."
  },
  {
    "startMs": 3656240,
    "text": "Very good questions."
  },
  {
    "startMs": 3657520,
    "text": "Maybe one more question over here."
  },
  {
    "startMs": 3660000,
    "text": "Hmm."
  },
  {
    "startMs": 3660400,
    "text": "Hmm."
  },
  {
    "startMs": 3661020,
    "text": "Yeah."
  },
  {
    "startMs": 3661420,
    "text": "So, I think that's a good question."
  },
  {
    "startMs": 3662600,
    "text": "It's about, like, yeah, models are getting better all the time."
  },
  {
    "startMs": 3664760,
    "text": "So, like, how do I modify my evals to keep up?"
  },
  {
    "startMs": 3668780,
    "text": "And I think, simply, it should be the case that as models get better, you should see performance on your eval going up like we see in this diagram."
  },
  {
    "startMs": 3675080,
    "text": "If you don't see that, then I would go back and question, is the eval well structured?"
  },
  {
    "startMs": 3679360,
    "text": "It is true."
  },
  {
    "startMs": 3680840,
    "text": "Like, yeah, we see this all the time."
  },
  {
    "startMs": 3682380,
    "text": "Models are structuring evals very, very quickly."
  },
  {
    "startMs": 3684900,
    "text": "So, we do need the development of new and harder evals that, like, keep up with the frontier."
  },
  {
    "startMs": 3689960,
    "text": "And, actually, that's a good point."
  },
  {
    "startMs": 3691540,
    "text": "It's actually pretty hard."
  },
  {
    "startMs": 3692680,
    "text": "It's hard to come up with evals that really challenge, like, the best models today, like the Fables, Opus 5.5 even."
  },
  {
    "startMs": 3698660,
    "text": "So, I do think it's a recurring challenge to design good evals that test the capabilities of the best models."
  },
  {
    "startMs": 3710280,
    "text": "It's absolutely a challenge."
  },
  {
    "startMs": 3712560,
    "text": "But I do think, so, if you have evals that you built some time ago, you should be seeing this trend on them."
  },
  {
    "startMs": 3718380,
    "text": "So, as models get better, you should be seeing performance going up and up and up."
  },
  {
    "startMs": 3721280,
    "text": "If you don't, I would question things about the eval itself."
  },
  {
    "startMs": 3724500,
    "text": "That's kind of the key point I would make."
  },
  {
    "startMs": 3726820,
    "text": "I mean, I've actually seen a lot of interesting work on this from the various tracing companies."
  },
  {
    "startMs": 3731440,
    "text": "So, like, the workflow I've seen quite a bit of is, like, you have agents that are deployed."
  },
  {
    "startMs": 3737840,
    "text": "They're, like, producing traces."
  },
  {
    "startMs": 3739740,
    "text": "You have an online evaluator looking at those traces and scoring them."
  },
  {
    "startMs": 3743420,
    "text": "Traces that score poorly are, like, then pulled into some offline eval set."
  },
  {
    "startMs": 3747820,
    "text": "You kind of then hill climb on that."
  },
  {
    "startMs": 3750400,
    "text": "And then that loop continues."
  },
  {
    "startMs": 3752000,
    "text": "That's a good, that's, like, a very good pattern, I think."
  },
  {
    "startMs": 3754460,
    "text": "It's very reasonable."
  },
  {
    "startMs": 3755920,
    "text": "And so, what's happening there is you're typically, like, hill climbing on these cases that you're failing on in production."
  },
  {
    "startMs": 3761380,
    "text": "And then you're, like, you know, putting those in regression tests once they all pass."
  },
  {
    "startMs": 3765620,
    "text": "And that, like, continues until in production you're seeing very few failures."
  },
  {
    "startMs": 3768600,
    "text": "So, that actually is a good general way to structure your kind of eval loop."
  },
  {
    "startMs": 3775440,
    "text": "And many of the, you know, the Lang Chains Arises, Brain Trusts, all these companies kind of have product offerings that offer this kind of, this, like, outer loop, which does seem quite effective."
  },
  {
    "startMs": 3788560,
    "text": "You do."
  },
  {
    "startMs": 3789440,
    "text": "So, the challenge there becomes, okay, well, what are you on on evaluators?"
  },
  {
    "startMs": 3792820,
    "text": "So, like, okay, how do you know what good is in production?"
  },
  {
    "startMs": 3794720,
    "text": "And what are you promoting to your offline eval sets?"
  },
  {
    "startMs": 3796620,
    "text": "And, yeah, it's not easy at all."
  },
  {
    "startMs": 3802820,
    "text": "I think that is, yeah, that's, like, there is, like, a lot of product work in that particular domain now."
  },
  {
    "startMs": 3808380,
    "text": "As far as, I haven't followed that area too closely."
  },
  {
    "startMs": 3811940,
    "text": "Like, with all the latest kind of updates that the various, like, tracing observability companies are working on."
  },
  {
    "startMs": 3817480,
    "text": "But my understanding is that is, like, a very, very active area of, like, product development among that class of companies."
  },
  {
    "startMs": 3822360,
    "text": "So, I think it's pretty interesting."
  },
  {
    "startMs": 3826620,
    "text": "Okay, with that, Mabel, let everyone go."
  },
  {
    "startMs": 3829020,
    "text": "And very nice to see you all."
  },
  {
    "startMs": 3831540,
    "text": "So, feel free to reach out to me anywhere."
  },
  {
    "startMs": 3833560,
    "text": "And thanks for the time."
  }
]
