[
  {
    "startMs": 1309,
    "text": "[music]"
  },
  {
    "startMs": 12559,
    "text": "Uh, I'm Jason. I'm from Modal and today"
  },
  {
    "startMs": 15679,
    "text": "I'm talking about scaling agent RL"
  },
  {
    "startMs": 18400,
    "text": "across the globe. Um, but first I think"
  },
  {
    "startMs": 23039,
    "text": "a lot of AI engineers view models as"
  },
  {
    "startMs": 26160,
    "text": "kind of like static backends that um"
  },
  {
    "startMs": 29439,
    "text": "that they"
  },
  {
    "startMs": 31920,
    "text": "kind of swap out like as the new models"
  },
  {
    "startMs": 33760,
    "text": "come out they'll maybe uh test the"
  },
  {
    "startMs": 36960,
    "text": "models, run evals, uh do AV AB tests and"
  },
  {
    "startMs": 40879,
    "text": "kind of like rebuild their harness"
  },
  {
    "startMs": 42320,
    "text": "around whatever model is cheapest but"
  },
  {
    "startMs": 45440,
    "text": "also highest quality for for their use"
  },
  {
    "startMs": 47520,
    "text": "cases. Um, and the claim I want to make"
  },
  {
    "startMs": 51440,
    "text": "for this talk is that this is actually a"
  },
  {
    "startMs": 54320,
    "text": "losing strategy. Uh, when you end up"
  },
  {
    "startMs": 56879,
    "text": "having to compete on like not only the"
  },
  {
    "startMs": 59760,
    "text": "agent economics but also the experience"
  },
  {
    "startMs": 61760,
    "text": "of the agent and like these two"
  },
  {
    "startMs": 63840,
    "text": "competing priorities of like offering"
  },
  {
    "startMs": 66159,
    "text": "differentiated value. Uh, and I think"
  },
  {
    "startMs": 69600,
    "text": "this is what we hear from our customers"
  },
  {
    "startMs": 71600,
    "text": "who do fine-tuning. Um it's like the"
  },
  {
    "startMs": 74640,
    "text": "first the first area where I think um"
  },
  {
    "startMs": 77840,
    "text": "fine-tuning took off is is people who"
  },
  {
    "startMs": 80080,
    "text": "had agent deployments who um"
  },
  {
    "startMs": 84640,
    "text": "need reliable SLOs's like like they they"
  },
  {
    "startMs": 87040,
    "text": "need reliability SLOs's if they if they"
  },
  {
    "startMs": 89360,
    "text": "need those then they need to own their"
  },
  {
    "startMs": 90479,
    "text": "own inference and uh the open source"
  },
  {
    "startMs": 92720,
    "text": "models out of the box were not always"
  },
  {
    "startMs": 94640,
    "text": "capable of supporting their deployments"
  },
  {
    "startMs": 97600,
    "text": "uh without you know uh degrading in in"
  },
  {
    "startMs": 100159,
    "text": "quality and I think fine-tuning"
  },
  {
    "startMs": 102720,
    "text": "especially"
  },
  {
    "startMs": 104400,
    "text": "you know after these deployments became"
  },
  {
    "startMs": 106159,
    "text": "more mature I think was the only path to"
  },
  {
    "startMs": 108799,
    "text": "owning your own inference uh but more"
  },
  {
    "startMs": 111040,
    "text": "importantly kind of like pushing"
  },
  {
    "startMs": 112640,
    "text": "yourself up this curve in terms of"
  },
  {
    "startMs": 115040,
    "text": "latency quality"
  },
  {
    "startMs": 116591,
    "text": "[clears throat]"
  },
  {
    "startMs": 116720,
    "text": "uh and cost al together like if you if"
  },
  {
    "startMs": 119040,
    "text": "you do fine tuning you like own all"
  },
  {
    "startMs": 120560,
    "text": "these yourself kind of um there were"
  },
  {
    "startMs": 123600,
    "text": "other use cases though I think one we"
  },
  {
    "startMs": 125439,
    "text": "heard which was interesting was uh"
  },
  {
    "startMs": 128080,
    "text": "proprietary data sets that their agents"
  },
  {
    "startMs": 130879,
    "text": "actually needed to reason about and"
  },
  {
    "startMs": 132319,
    "text": "understand and take action on. But uh"
  },
  {
    "startMs": 135040,
    "text": "these these data sets were like not"
  },
  {
    "startMs": 136800,
    "text": "present in any pre-training set. And so"
  },
  {
    "startMs": 139840,
    "text": "there like out of the box sometimes they"
  },
  {
    "startMs": 142080,
    "text": "they would build agents on that had to"
  },
  {
    "startMs": 144480,
    "text": "deal with this type of data. And Claude"
  },
  {
    "startMs": 147680,
    "text": "Gbt just like would not work at all uh"
  },
  {
    "startMs": 150160,
    "text": "for these tasks. And so they actually"
  },
  {
    "startMs": 152800,
    "text": "just had no other alternative but to"
  },
  {
    "startMs": 155280,
    "text": "fine-tune open source models."
  },
  {
    "startMs": 158319,
    "text": "And lastly, I think like there's an"
  },
  {
    "startMs": 160319,
    "text": "emerging attitude around fine-tuning,"
  },
  {
    "startMs": 162400,
    "text": "which is um in order to build"
  },
  {
    "startMs": 165680,
    "text": "differentiated value with agents, uh"
  },
  {
    "startMs": 168400,
    "text": "like more and more people are deciding"
  },
  {
    "startMs": 170080,
    "text": "to vertically integrate across all the"
  },
  {
    "startMs": 172959,
    "text": "different levels of the stack. Um these"
  },
  {
    "startMs": 175280,
    "text": "are this is most common in industries"
  },
  {
    "startMs": 177360,
    "text": "where uh people are already competing on"
  },
  {
    "startMs": 180160,
    "text": "like some metric of task intelligence"
  },
  {
    "startMs": 183120,
    "text": "per dollar uh like coding. But I think"
  },
  {
    "startMs": 186640,
    "text": "um what we should see as the industry"
  },
  {
    "startMs": 189280,
    "text": "matures outside of just coding is like"
  },
  {
    "startMs": 192480,
    "text": "we should see this trend kind of"
  },
  {
    "startMs": 194080,
    "text": "continue elsewhere. Um so I think that's"
  },
  {
    "startMs": 197920,
    "text": "that's why I think a lot of engineers"
  },
  {
    "startMs": 199760,
    "text": "have not really been fully bought in"
  },
  {
    "startMs": 201360,
    "text": "because the the latest model comes out"
  },
  {
    "startMs": 203040,
    "text": "and it's better than any model they"
  },
  {
    "startMs": 204400,
    "text": "could fine-tune from the last"
  },
  {
    "startMs": 205680,
    "text": "generation. But I think there's like a"
  },
  {
    "startMs": 207360,
    "text": "lot of different reasons people do"
  },
  {
    "startMs": 208400,
    "text": "finetuning and in the long haul I think"
  },
  {
    "startMs": 211040,
    "text": "uh yeah this this is only going to"
  },
  {
    "startMs": 213120,
    "text": "become more common."
  },
  {
    "startMs": 215440,
    "text": "So you may have heard of fine-tuning in"
  },
  {
    "startMs": 217120,
    "text": "the context of RL or SFT. And the"
  },
  {
    "startMs": 220400,
    "text": "question is why is this talk about RL"
  },
  {
    "startMs": 222560,
    "text": "and not SFT?"
  },
  {
    "startMs": 224480,
    "text": "Um"
  },
  {
    "startMs": 226400,
    "text": "I think the idea that I want to"
  },
  {
    "startMs": 228239,
    "text": "communicate here is that uh if you're if"
  },
  {
    "startMs": 230159,
    "text": "you're doing SFT you're basically"
  },
  {
    "startMs": 231680,
    "text": "training on traces recorded uh during"
  },
  {
    "startMs": 234560,
    "text": "production. Um but the problem is that"
  },
  {
    "startMs": 237840,
    "text": "agents in particular have this like"
  },
  {
    "startMs": 240640,
    "text": "branching structure uh based on the"
  },
  {
    "startMs": 242879,
    "text": "number of turns that they run. Um"
  },
  {
    "startMs": 247360,
    "text": "so in production if you train on this"
  },
  {
    "startMs": 249280,
    "text": "particular trace uh later on when you"
  },
  {
    "startMs": 251680,
    "text": "deploy that model it may see like a"
  },
  {
    "startMs": 253599,
    "text": "completely different branch in the kind"
  },
  {
    "startMs": 256400,
    "text": "of like session like the the tree of"
  },
  {
    "startMs": 258959,
    "text": "possible sessions and uh this is quite"
  },
  {
    "startMs": 261919,
    "text": "bad. SFT is quite brittle in in this"
  },
  {
    "startMs": 265199,
    "text": "regard. So kind of like why RL has"
  },
  {
    "startMs": 268320,
    "text": "emerged as this like dominant paradigm"
  },
  {
    "startMs": 270160,
    "text": "of fine-tuning is that um in RL the"
  },
  {
    "startMs": 273280,
    "text": "model actually explores all these"
  },
  {
    "startMs": 274639,
    "text": "branches uh during training time. At"
  },
  {
    "startMs": 278000,
    "text": "least that's kind of the hope and when"
  },
  {
    "startMs": 279440,
    "text": "people talk about RL like generalizing"
  },
  {
    "startMs": 282000,
    "text": "better that's often what they mean."
  },
  {
    "startMs": 286880,
    "text": "I think one reason why this trade-off"
  },
  {
    "startMs": 290639,
    "text": "has been hard for people is that they"
  },
  {
    "startMs": 292320,
    "text": "view like post- training and fine-tuning"
  },
  {
    "startMs": 294479,
    "text": "in RL as difficult. Um, but I think my"
  },
  {
    "startMs": 297520,
    "text": "claim here is actually that if you have"
  },
  {
    "startMs": 300560,
    "text": "an agent deployment in production, you"
  },
  {
    "startMs": 302479,
    "text": "probably already have the ingredients"
  },
  {
    "startMs": 304320,
    "text": "you need. Um, so if you're in"
  },
  {
    "startMs": 307280,
    "text": "production, you have an agent harness"
  },
  {
    "startMs": 308560,
    "text": "that that's deployed somewhere and"
  },
  {
    "startMs": 310560,
    "text": "hopefully you have eval."
  },
  {
    "startMs": 312720,
    "text": "Maybe calling it eval is not necessarily"
  },
  {
    "startMs": 315440,
    "text": "required. It could be AB tests. It could"
  },
  {
    "startMs": 317280,
    "text": "be um basically any any way you do"
  },
  {
    "startMs": 319440,
    "text": "change management for this harness, for"
  },
  {
    "startMs": 320960,
    "text": "this deployment. Um but those two"
  },
  {
    "startMs": 323440,
    "text": "together give you the the tools you need"
  },
  {
    "startMs": 325280,
    "text": "to basically build out something that"
  },
  {
    "startMs": 327360,
    "text": "you can consider an environment, an RL"
  },
  {
    "startMs": 329039,
    "text": "environment. Um"
  },
  {
    "startMs": 332000,
    "text": "and why production traces are important"
  },
  {
    "startMs": 333680,
    "text": "for this is that you need to kind of get"
  },
  {
    "startMs": 336560,
    "text": "high quality tasks to train on in this"
  },
  {
    "startMs": 338960,
    "text": "environment from somewhere. And the best"
  },
  {
    "startMs": 340960,
    "text": "place often to do that is to"
  },
  {
    "startMs": 343199,
    "text": "synthetically generate these tasks from"
  },
  {
    "startMs": 345600,
    "text": "actual live production traces of things"
  },
  {
    "startMs": 347600,
    "text": "that your users are are doing um with"
  },
  {
    "startMs": 350800,
    "text": "with your product product and yeah kind"
  },
  {
    "startMs": 353759,
    "text": "of forming uh finding examples where"
  },
  {
    "startMs": 356160,
    "text": "it's failing, mining those and creating"
  },
  {
    "startMs": 358160,
    "text": "tasks around them so that fine-tuning"
  },
  {
    "startMs": 360479,
    "text": "actually gives you some chance of"
  },
  {
    "startMs": 361919,
    "text": "improving your product in those areas."
  },
  {
    "startMs": 365600,
    "text": "And this is basically all you need to do"
  },
  {
    "startMs": 367280,
    "text": "RL."
  },
  {
    "startMs": 370240,
    "text": "But then the next part part about it"
  },
  {
    "startMs": 372000,
    "text": "being hard is that like okay once just"
  },
  {
    "startMs": 373919,
    "text": "because you have an agent deployment"
  },
  {
    "startMs": 375120,
    "text": "doesn't mean that your team may be"
  },
  {
    "startMs": 376639,
    "text": "capable or maybe that's how you feel. Um"
  },
  {
    "startMs": 379600,
    "text": "and probably because RL has a reputation"
  },
  {
    "startMs": 382080,
    "text": "of just being difficult generally. Um,"
  },
  {
    "startMs": 384800,
    "text": "but I think over the last couple of at"
  },
  {
    "startMs": 387440,
    "text": "least over the last year, over the last"
  },
  {
    "startMs": 389120,
    "text": "several months, uh, like we have kind of"
  },
  {
    "startMs": 392960,
    "text": "figured out the science of of RL. And"
  },
  {
    "startMs": 396639,
    "text": "that's what I'm going to basically talk"
  },
  {
    "startMs": 398319,
    "text": "about today is that, uh, a particular RL"
  },
  {
    "startMs": 401039,
    "text": "recipe that we've had a lot of success"
  },
  {
    "startMs": 402800,
    "text": "with that I think simplifies and"
  },
  {
    "startMs": 405840,
    "text": "stabilizes a lot of what, you know, was"
  },
  {
    "startMs": 408800,
    "text": "difficult before."
  },
  {
    "startMs": 411600,
    "text": "But first, okay, just to kind of recap"
  },
  {
    "startMs": 413600,
    "text": "how how does reinforcement learning"
  },
  {
    "startMs": 415199,
    "text": "work. Uh it's it's this loop"
  },
  {
    "startMs": 417919,
    "text": "mechanically. This is this is what's"
  },
  {
    "startMs": 419840,
    "text": "happening. You have a bunch of uh"
  },
  {
    "startMs": 422240,
    "text": "inference GPUs that are running your"
  },
  {
    "startMs": 424240,
    "text": "sampler aka the model that you're"
  },
  {
    "startMs": 425840,
    "text": "training. Um"
  },
  {
    "startMs": 428319,
    "text": "and those rollouts you you basically"
  },
  {
    "startMs": 430400,
    "text": "generate rollouts in your environment uh"
  },
  {
    "startMs": 432400,
    "text": "with your harness. Those rollouts get"
  },
  {
    "startMs": 434400,
    "text": "collected and fed to a trainer which is"
  },
  {
    "startMs": 437280,
    "text": "consuming all this data and updating the"
  },
  {
    "startMs": 439919,
    "text": "model uh with you know gradient descent"
  },
  {
    "startMs": 442560,
    "text": "and then uh that updated model you you"
  },
  {
    "startMs": 445440,
    "text": "broadcast those new weights to all the"
  },
  {
    "startMs": 447680,
    "text": "samplers and um over time you the"
  },
  {
    "startMs": 451039,
    "text": "rollouts you're generating are from this"
  },
  {
    "startMs": 452560,
    "text": "like continuously improving model. Uh so"
  },
  {
    "startMs": 454720,
    "text": "you're you're you're kind of like going"
  },
  {
    "startMs": 456560,
    "text": "back and forth between them and"
  },
  {
    "startMs": 457599,
    "text": "iterating and and that's how the model"
  },
  {
    "startMs": 459599,
    "text": "improves."
  },
  {
    "startMs": 461759,
    "text": "Uh so it turns out that like the math of"
  },
  {
    "startMs": 464319,
    "text": "RL prefers this to be done sequentially."
  },
  {
    "startMs": 467039,
    "text": "So first you do your rollouts then you"
  },
  {
    "startMs": 469680,
    "text": "train and update your model then you you"
  },
  {
    "startMs": 471840,
    "text": "take that model update send it back to"
  },
  {
    "startMs": 473680,
    "text": "your rollout side so that the new"
  },
  {
    "startMs": 475199,
    "text": "rollouts you generate are from that new"
  },
  {
    "startMs": 476560,
    "text": "version of the model. Uh but the problem"
  },
  {
    "startMs": 478479,
    "text": "is that this wastes a ton of idle"
  },
  {
    "startMs": 480879,
    "text": "compute. Um there are ways to like share"
  },
  {
    "startMs": 484000,
    "text": "the GPUs between the two but I think"
  },
  {
    "startMs": 486400,
    "text": "especially with agents. Um what's much"
  },
  {
    "startMs": 489440,
    "text": "better is that if you actually are"
  },
  {
    "startMs": 490720,
    "text": "continuously doing both in parallel"
  },
  {
    "startMs": 492479,
    "text": "asynchronously."
  },
  {
    "startMs": 494000,
    "text": "Um this is better because you can like"
  },
  {
    "startMs": 496800,
    "text": "pack your GPUs, get high utilization"
  },
  {
    "startMs": 498639,
    "text": "from them and uh not waste money. But"
  },
  {
    "startMs": 501520,
    "text": "the downside is that um your rollouts"
  },
  {
    "startMs": 504240,
    "text": "that you generated were very likely from"
  },
  {
    "startMs": 506000,
    "text": "an old copy of the model. And this is"
  },
  {
    "startMs": 509360,
    "text": "what people refer to as staleness in"
  },
  {
    "startMs": 511199,
    "text": "asynchronous RL. Basically uh rollouts"
  },
  {
    "startMs": 514560,
    "text": "were gener generated by V1. uh the"
  },
  {
    "startMs": 517279,
    "text": "current model that you're training is"
  },
  {
    "startMs": 518399,
    "text": "V3. Are those rollouts still useful for"
  },
  {
    "startMs": 520640,
    "text": "V3 if the model's gotten better since"
  },
  {
    "startMs": 522880,
    "text": "then? Um and this is like one of the"
  },
  {
    "startMs": 525440,
    "text": "reasons why it was unstable. Another"
  },
  {
    "startMs": 527600,
    "text": "reason why it was unstable is actually"
  },
  {
    "startMs": 529279,
    "text": "purely at the systems level. There's a"
  },
  {
    "startMs": 530959,
    "text": "lot of details that can show up as"
  },
  {
    "startMs": 534560,
    "text": "differences in between what the trainer"
  },
  {
    "startMs": 537920,
    "text": "thinks it's training and what the"
  },
  {
    "startMs": 539600,
    "text": "samplers think we're training."
  },
  {
    "startMs": 541839,
    "text": "uh basically at all these different"
  },
  {
    "startMs": 544160,
    "text": "levels of the stack there can be these"
  },
  {
    "startMs": 545760,
    "text": "like small differences in behavior and"
  },
  {
    "startMs": 549279,
    "text": "that means that like in in the math of"
  },
  {
    "startMs": 551040,
    "text": "RL I think the the idea is that both"
  },
  {
    "startMs": 553040,
    "text": "both sides of this think they're doing"
  },
  {
    "startMs": 554560,
    "text": "the same thing they think they're"
  },
  {
    "startMs": 556000,
    "text": "learning from the same data training"
  },
  {
    "startMs": 557920,
    "text": "like the same distribution but actually"
  },
  {
    "startMs": 559920,
    "text": "it's like slightly different and so the"
  },
  {
    "startMs": 562560,
    "text": "the solution that I'll present here is"
  },
  {
    "startMs": 565040,
    "text": "accounting for these differences um"
  },
  {
    "startMs": 567600,
    "text": "broadly with like kind of the simplest"
  },
  {
    "startMs": 569360,
    "text": "set of fixes that will uh correct for"
  },
  {
    "startMs": 572640,
    "text": "this mismatch without drowning out the"
  },
  {
    "startMs": 574720,
    "text": "signal of the the training loop."
  },
  {
    "startMs": 578880,
    "text": "So yeah, what's the recipe? Here it is."
  },
  {
    "startMs": 580959,
    "text": "Uh we start from gRPO. This is uh what"
  },
  {
    "startMs": 583440,
    "text": "DeepS introduced for Deepseek R1 and"
  },
  {
    "startMs": 586320,
    "text": "since then has become kind of a"
  },
  {
    "startMs": 587519,
    "text": "cornerstone of of doing RL on language"
  },
  {
    "startMs": 590560,
    "text": "models. Uh we add something called mass"
  },
  {
    "startMs": 593360,
    "text": "importance sampling. This is correcting"
  },
  {
    "startMs": 595760,
    "text": "for token-wise mismatch. And this"
  },
  {
    "startMs": 598240,
    "text": "mismatch is not just the mismatch from"
  },
  {
    "startMs": 600880,
    "text": "all the systems level stuff that I"
  },
  {
    "startMs": 602320,
    "text": "pointed out. Uh it's also from the"
  },
  {
    "startMs": 603920,
    "text": "staleness as well. So if an old version"
  },
  {
    "startMs": 606880,
    "text": "of the model would have sampled a"
  },
  {
    "startMs": 608240,
    "text": "different token here, uh the ratio"
  },
  {
    "startMs": 609920,
    "text": "between the two can be can be very"
  },
  {
    "startMs": 612160,
    "text": "different. And then uh this is a"
  },
  {
    "startMs": 613760,
    "text": "mismatch that we want to correct for"
  },
  {
    "startMs": 615519,
    "text": "somehow and and uh I won't go too much"
  },
  {
    "startMs": 618160,
    "text": "into details here about how these"
  },
  {
    "startMs": 619920,
    "text": "mechanisms work, but uh I will you know"
  },
  {
    "startMs": 622399,
    "text": "reference them. Um and then the second"
  },
  {
    "startMs": 625360,
    "text": "is score centering. So even though we're"
  },
  {
    "startMs": 627920,
    "text": "correcting for each token-wise mismatch,"
  },
  {
    "startMs": 630800,
    "text": "what ends up happening is over the"
  },
  {
    "startMs": 632320,
    "text": "course of training, whatever mismatch is"
  },
  {
    "startMs": 634800,
    "text": "still remaining accumulates and it kind"
  },
  {
    "startMs": 636720,
    "text": "of accumulates in the the weights"
  },
  {
    "startMs": 638880,
    "text": "themselves like the latest version of"
  },
  {
    "startMs": 640240,
    "text": "the model is slightly different than uh"
  },
  {
    "startMs": 642959,
    "text": "what it should be even though we've like"
  },
  {
    "startMs": 644720,
    "text": "corrected for the most obvious"
  },
  {
    "startMs": 646320,
    "text": "differences. Um and so this score"
  },
  {
    "startMs": 648959,
    "text": "centering approach is it basically"
  },
  {
    "startMs": 650800,
    "text": "removes this drift over the course of"
  },
  {
    "startMs": 652320,
    "text": "training uh not just at each token or at"
  },
  {
    "startMs": 655200,
    "text": "each step."
  },
  {
    "startMs": 656320,
    "text": "um it kind of like builds up a term that"
  },
  {
    "startMs": 658399,
    "text": "accumulates with the drift and removes"
  },
  {
    "startMs": 660800,
    "text": "it. Um and then the last thing is top"
  },
  {
    "startMs": 663120,
    "text": "heat sampling uh with mass replay. So"
  },
  {
    "startMs": 666240,
    "text": "top p is is one of the parameters you"
  },
  {
    "startMs": 668000,
    "text": "use when you sample from a language"
  },
  {
    "startMs": 669120,
    "text": "model. Basically says only consider the"
  },
  {
    "startMs": 671360,
    "text": "top p of the of the distribution. P"
  },
  {
    "startMs": 674160,
    "text": "being like 0.98 0.95 basically only the"
  },
  {
    "startMs": 677279,
    "text": "tokens that have probability mass"
  },
  {
    "startMs": 679200,
    "text": "greater than 95 uh or greater than 5%."
  },
  {
    "startMs": 682480,
    "text": "Um, and so it just removes a bunch of"
  },
  {
    "startMs": 684640,
    "text": "very rare, likely bad tokens that can"
  },
  {
    "startMs": 688160,
    "text": "just never be sampled from. And so if"
  },
  {
    "startMs": 690160,
    "text": "your sampler decides to only sample from"
  },
  {
    "startMs": 692240,
    "text": "these tokens, but your trainer decides"
  },
  {
    "startMs": 694720,
    "text": "unluckily to sample one of these rare"
  },
  {
    "startMs": 696399,
    "text": "tokens, this mass mismatch can be"
  },
  {
    "startMs": 698480,
    "text": "basically catastrophic if you don't"
  },
  {
    "startMs": 699760,
    "text": "correct for it. So we correct for it. Um"
  },
  {
    "startMs": 702320,
    "text": "and it turns out actually in the math of"
  },
  {
    "startMs": 703920,
    "text": "score centering that if you're doing top"
  },
  {
    "startMs": 706000,
    "text": "sampling like this that actually the the"
  },
  {
    "startMs": 708160,
    "text": "technique we we use to do score"
  },
  {
    "startMs": 709839,
    "text": "centering uh is exact instead of"
  },
  {
    "startMs": 711839,
    "text": "approximate because if you don't do top"
  },
  {
    "startMs": 713839,
    "text": "it's actually only an approximate"
  },
  {
    "startMs": 715440,
    "text": "correction for drift. So all this is to"
  },
  {
    "startMs": 717440,
    "text": "say that uh with this recipe uh things"
  },
  {
    "startMs": 720160,
    "text": "get much much better much more stable."
  },
  {
    "startMs": 722560,
    "text": "Um"
  },
  {
    "startMs": 724160,
    "text": "and this is this is basically just an"
  },
  {
    "startMs": 726880,
    "text": "ablation that we did on quen 3.635b 6"
  },
  {
    "startMs": 729120,
    "text": "35B of thee. Um, it's trained on the"
  },
  {
    "startMs": 733519,
    "text": "MIMO code environments with evalu"
  },
  {
    "startMs": 738800,
    "text": "subset. Um, basically what this shows is"
  },
  {
    "startMs": 741440,
    "text": "an ablation of these different"
  },
  {
    "startMs": 742560,
    "text": "techniques uh putting them all together"
  },
  {
    "startMs": 744480,
    "text": "showing like much more stable training"
  },
  {
    "startMs": 746880,
    "text": "uh than any of uh the alternatives"
  },
  {
    "startMs": 750399,
    "text": "that we've tested here. Um, so yeah, I"
  },
  {
    "startMs": 754399,
    "text": "think the that's all well and good, but"
  },
  {
    "startMs": 756560,
    "text": "I guess the question is like how far can"
  },
  {
    "startMs": 757680,
    "text": "we push this recipe? Does it actually"
  },
  {
    "startMs": 758959,
    "text": "scale? Does it does it work for broad"
  },
  {
    "startMs": 760480,
    "text": "problems? Before I talk about that, I"
  },
  {
    "startMs": 763040,
    "text": "think I want to talk briefly about why"
  },
  {
    "startMs": 765279,
    "text": "we care about RL at modal and like some"
  },
  {
    "startMs": 767839,
    "text": "things about doing RL on modal are"
  },
  {
    "startMs": 770079,
    "text": "different. I kind of want to just like"
  },
  {
    "startMs": 771120,
    "text": "get into that a little bit first. Um, so"
  },
  {
    "startMs": 773839,
    "text": "yeah, first why modal for agent RL in"
  },
  {
    "startMs": 775920,
    "text": "particular? Well, we do training. We"
  },
  {
    "startMs": 778000,
    "text": "just released uh clusters as GA last"
  },
  {
    "startMs": 781600,
    "text": "week I think or week before. Um we also"
  },
  {
    "startMs": 784399,
    "text": "do inference really well. We have a"
  },
  {
    "startMs": 785600,
    "text": "bunch of endpoints. Uh we've we've done"
  },
  {
    "startMs": 787200,
    "text": "a a lot of the hard work around doing LM"
  },
  {
    "startMs": 789040,
    "text": "inference at scale. Um and the"
  },
  {
    "startMs": 791839,
    "text": "environments you need they run in modal"
  },
  {
    "startMs": 793200,
    "text": "sandboxes. So like just naively we have"
  },
  {
    "startMs": 795760,
    "text": "all of the kind of infrared layers like"
  },
  {
    "startMs": 798639,
    "text": "or infrop primitives that you'd want to"
  },
  {
    "startMs": 800560,
    "text": "be able to compose into like a full"
  },
  {
    "startMs": 802480,
    "text": "production agent RL uh training system."
  },
  {
    "startMs": 806880,
    "text": "The problem, the only problem actually"
  },
  {
    "startMs": 809760,
    "text": "uh because all that sounds great, but"
  },
  {
    "startMs": 811360,
    "text": "the only problem is that our capacity"
  },
  {
    "startMs": 813120,
    "text": "model is very different compared to"
  },
  {
    "startMs": 814720,
    "text": "other people. Um the reason for that is"
  },
  {
    "startMs": 817279,
    "text": "we aggregate tons of capacity across"
  },
  {
    "startMs": 819279,
    "text": "many different providers here. There's"
  },
  {
    "startMs": 821360,
    "text": "three pictured, but in reality it's like"
  },
  {
    "startMs": 823040,
    "text": "15 or 20 at this point. Um we we run all"
  },
  {
    "startMs": 826079,
    "text": "the hyperscalers, but we also run on a"
  },
  {
    "startMs": 827680,
    "text": "bunch of NeoClouds. Um and we handle all"
  },
  {
    "startMs": 830800,
    "text": "of the kind of like GPU health and"
  },
  {
    "startMs": 832880,
    "text": "reliability concerns. Uh but"
  },
  {
    "startMs": 837440,
    "text": "this is great but uh it's great for"
  },
  {
    "startMs": 839040,
    "text": "reliability. It's great for kind of like"
  },
  {
    "startMs": 840639,
    "text": "fast and flexible scheduling. It gets us"
  },
  {
    "startMs": 842399,
    "text": "around compute shortages. Uh but it also"
  },
  {
    "startMs": 845680,
    "text": "fragments our compute pool. So uh if you"
  },
  {
    "startMs": 848720,
    "text": "want to schedule large training"
  },
  {
    "startMs": 849920,
    "text": "clusters, it's actually quite difficult"
  },
  {
    "startMs": 851760,
    "text": "in this like multi-provider multi-reion"
  },
  {
    "startMs": 854560,
    "text": "fleet. Uh we've engineered it so that we"
  },
  {
    "startMs": 857360,
    "text": "can schedule clusters pretty quickly up"
  },
  {
    "startMs": 859120,
    "text": "to like four nodes or eight nodes. Uh"
  },
  {
    "startMs": 861120,
    "text": "but once you get past that to 16 nodes"
  },
  {
    "startMs": 863040,
    "text": "or 32 nodes uh it gets pretty painful."
  },
  {
    "startMs": 865920,
    "text": "It takes a really long time to find like"
  },
  {
    "startMs": 867600,
    "text": "a contiguous block of all these nodes."
  },
  {
    "startMs": 870320,
    "text": "Um and in RL this is particularly bad"
  },
  {
    "startMs": 873360,
    "text": "because in RL we generally have"
  },
  {
    "startMs": 875440,
    "text": "especially for agent RL we have like two"
  },
  {
    "startMs": 877760,
    "text": "two inference GPUs for every training"
  },
  {
    "startMs": 879440,
    "text": "GPU. So 2/3ish"
  },
  {
    "startMs": 881920,
    "text": "of your cluster is actually just stuck"
  },
  {
    "startMs": 883839,
    "text": "doing inference. Um, so if you're"
  },
  {
    "startMs": 886160,
    "text": "scheduling an eight node cluster max,"
  },
  {
    "startMs": 888160,
    "text": "you're probably only using two or three"
  },
  {
    "startMs": 889760,
    "text": "of those nodes for for the trainer side"
  },
  {
    "startMs": 891519,
    "text": "and most of it's just spent doing"
  },
  {
    "startMs": 892639,
    "text": "inference."
  },
  {
    "startMs": 894639,
    "text": "Um, it'd be really great if we could"
  },
  {
    "startMs": 896560,
    "text": "move these inference GPUs out of the"
  },
  {
    "startMs": 898639,
    "text": "cluster."
  },
  {
    "startMs": 901120,
    "text": "And one reason why we think we can do"
  },
  {
    "startMs": 903760,
    "text": "this is because uh during RL the weight"
  },
  {
    "startMs": 906560,
    "text": "updates, the actual like gradients that"
  },
  {
    "startMs": 908720,
    "text": "you're applying to your model end up"
  },
  {
    "startMs": 910560,
    "text": "being incredibly sparse. And there's uh"
  },
  {
    "startMs": 912800,
    "text": "two reasons for that. But um yeah,"
  },
  {
    "startMs": 916399,
    "text": "essentially if you if you take the model"
  },
  {
    "startMs": 917600,
    "text": "at step t and subtract the the new model"
  },
  {
    "startMs": 920560,
    "text": "t+1 from it or sorry take the model at t"
  },
  {
    "startMs": 923680,
    "text": "plus1 subtract t uh you get this delta."
  },
  {
    "startMs": 926079,
    "text": "And it turns out that even though the"
  },
  {
    "startMs": 927440,
    "text": "full model weights at each step are very"
  },
  {
    "startMs": 929760,
    "text": "dense and very large. Uh the updates"
  },
  {
    "startMs": 932240,
    "text": "themselves are incredibly sparse. 100x"
  },
  {
    "startMs": 934560,
    "text": "or a thousandx as sparse."
  },
  {
    "startMs": 938079,
    "text": "So you can actually just feed this to"
  },
  {
    "startMs": 939440,
    "text": "offtheshelf compression. And it turns"
  },
  {
    "startMs": 941279,
    "text": "out that the amount of data you need to"
  },
  {
    "startMs": 943279,
    "text": "communicate from the train uh yeah from"
  },
  {
    "startMs": 945839,
    "text": "the trainer to the rollout engines goes"
  },
  {
    "startMs": 948399,
    "text": "down by a factor of a thousand. This"
  },
  {
    "startMs": 950639,
    "text": "means that you can actually move your"
  },
  {
    "startMs": 952480,
    "text": "inference GPUs outside of the cluster."
  },
  {
    "startMs": 955680,
    "text": "And so actually for the for the run that"
  },
  {
    "startMs": 957759,
    "text": "I mentioned uh of coin 3.6 we ran the"
  },
  {
    "startMs": 961519,
    "text": "training cluster I think it it landed"
  },
  {
    "startMs": 962959,
    "text": "somewhere in GCP but the inference GPUs"
  },
  {
    "startMs": 965279,
    "text": "actually ran all across our fleet and"
  },
  {
    "startMs": 966720,
    "text": "all the different hyperscalers and"
  },
  {
    "startMs": 967839,
    "text": "neoclouds that we run across. Um and it"
  },
  {
    "startMs": 970720,
    "text": "turns out with this like sparse delta"
  },
  {
    "startMs": 972240,
    "text": "compression technique you can do this."
  },
  {
    "startMs": 974079,
    "text": "Uh it works well and it does not"
  },
  {
    "startMs": 976959,
    "text": "introduce like bad performance. Um"
  },
  {
    "startMs": 981279,
    "text": "not only that we we actually fragment"
  },
  {
    "startMs": 983120,
    "text": "our rollout fleet across different G GPU"
  },
  {
    "startMs": 985440,
    "text": "types. So again because the we have"
  },
  {
    "startMs": 988560,
    "text": "fragmentation due to all the different"
  },
  {
    "startMs": 990000,
    "text": "providers we run on. Uh but we also have"
  },
  {
    "startMs": 992480,
    "text": "fragmentation because we have a lot of"
  },
  {
    "startMs": 993839,
    "text": "different GPUs that different customers"
  },
  {
    "startMs": 995360,
    "text": "want to use. Uh it'd be really great if"
  },
  {
    "startMs": 997279,
    "text": "we could leverage as many of them as"
  },
  {
    "startMs": 998639,
    "text": "possible for RL runs and that's that's"
  },
  {
    "startMs": 1000560,
    "text": "what we've done. Um each of these GPU"
  },
  {
    "startMs": 1004240,
    "text": "types they they increase the mismatch"
  },
  {
    "startMs": 1006320,
    "text": "potentially. So if the train is running"
  },
  {
    "startMs": 1007600,
    "text": "in B200 uh but your rollouts are running"
  },
  {
    "startMs": 1010000,
    "text": "in H100 or A100 there are hardware level"
  },
  {
    "startMs": 1012639,
    "text": "differences in the the kernels the math"
  },
  {
    "startMs": 1014800,
    "text": "you can run that implement the model."
  },
  {
    "startMs": 1016560,
    "text": "And so this like increases more and more"
  },
  {
    "startMs": 1018480,
    "text": "mismatch which traditionally should"
  },
  {
    "startMs": 1020959,
    "text": "cause your training runs to destabilize."
  },
  {
    "startMs": 1023920,
    "text": "Um this is just kind of like a graph to"
  },
  {
    "startMs": 1026640,
    "text": "show uh how many requests are being"
  },
  {
    "startMs": 1028880,
    "text": "processed by each GPU type in the run we"
  },
  {
    "startMs": 1030880,
    "text": "did over the course of training."
  },
  {
    "startMs": 1035919,
    "text": "So actually the recipe that I mentioned"
  },
  {
    "startMs": 1038480,
    "text": "uh is capable of handling this amount of"
  },
  {
    "startMs": 1041760,
    "text": "mismatch that we throw at it which uh"
  },
  {
    "startMs": 1045360,
    "text": "yeah I think is like kind of"
  },
  {
    "startMs": 1046400,
    "text": "unprecedented. I think this just goes to"
  },
  {
    "startMs": 1048000,
    "text": "show how much we've figured out the"
  },
  {
    "startMs": 1050240,
    "text": "science of RL over the past year or two."
  },
  {
    "startMs": 1053039,
    "text": "Um basically this shows that you have up"
  },
  {
    "startMs": 1057679,
    "text": "to like four or five times as much"
  },
  {
    "startMs": 1059600,
    "text": "mismatch. Uh but the training reward and"
  },
  {
    "startMs": 1062559,
    "text": "gradient norm everything is is training"
  },
  {
    "startMs": 1065039,
    "text": "as stably as it was uh in the"
  },
  {
    "startMs": 1068559,
    "text": "homogeneous case which is basically"
  },
  {
    "startMs": 1070400,
    "text": "running training on B200s and inference"
  },
  {
    "startMs": 1072640,
    "text": "on B200s."
  },
  {
    "startMs": 1076240,
    "text": "And kind of a bonus, like a side effect"
  },
  {
    "startMs": 1077840,
    "text": "of this is that uh the rollouts that we"
  },
  {
    "startMs": 1080720,
    "text": "run for this full training run are"
  },
  {
    "startMs": 1082000,
    "text": "actually 15% cheaper. And that's because"
  },
  {
    "startMs": 1084640,
    "text": "the the demand for the the high-end GPUs"
  },
  {
    "startMs": 1087120,
    "text": "like B300s uh means that the price is"
  },
  {
    "startMs": 1089600,
    "text": "just generally higher. But if you can"
  },
  {
    "startMs": 1091520,
    "text": "actually run your rollouts cheaply on on"
  },
  {
    "startMs": 1093280,
    "text": "cheaper hardware that there's less"
  },
  {
    "startMs": 1094720,
    "text": "demand for like the overall cost of"
  },
  {
    "startMs": 1096559,
    "text": "doing this run is is significantly"
  },
  {
    "startMs": 1098240,
    "text": "lower. This is actually without any"
  },
  {
    "startMs": 1100240,
    "text": "inference tuning for each GPU type."
  },
  {
    "startMs": 1102400,
    "text": "we're just doing like the basic"
  },
  {
    "startMs": 1103840,
    "text": "off-the-shelf inference recipe for each"
  },
  {
    "startMs": 1105600,
    "text": "GPU. If you actually decided to uh you"
  },
  {
    "startMs": 1109360,
    "text": "know optimize each individual deployment"
  },
  {
    "startMs": 1112559,
    "text": "for each GPU type with for example"
  },
  {
    "startMs": 1114720,
    "text": "different levels of quantization in the"
  },
  {
    "startMs": 1116160,
    "text": "model um you could actually probably"
  },
  {
    "startMs": 1118000,
    "text": "make this significantly more cheap. We"
  },
  {
    "startMs": 1119840,
    "text": "just uh this is not the focus of this"
  },
  {
    "startMs": 1121600,
    "text": "work but uh but yeah oh I did also maybe"
  },
  {
    "startMs": 1125600,
    "text": "I forgot to mention but in this case"
  },
  {
    "startMs": 1128880,
    "text": "there are cases where like for example"
  },
  {
    "startMs": 1130640,
    "text": "H200 we have two different pools of"
  },
  {
    "startMs": 1132080,
    "text": "H200s that's because we run two"
  },
  {
    "startMs": 1134400,
    "text": "different uh quantizations of the model"
  },
  {
    "startMs": 1136799,
    "text": "uh on H200. So we have one H200 pool"
  },
  {
    "startMs": 1139360,
    "text": "doing it at BF16 which is like the the"
  },
  {
    "startMs": 1141679,
    "text": "safest way to do it but we have another"
  },
  {
    "startMs": 1143440,
    "text": "running at FP8. Uh and again like the"
  },
  {
    "startMs": 1146080,
    "text": "the correction that we do can like drown"
  },
  {
    "startMs": 1148400,
    "text": "out the mismatch that's introduced by"
  },
  {
    "startMs": 1150559,
    "text": "even excessive quantization of the model"
  },
  {
    "startMs": 1152720,
    "text": "and still learn from these quantized"
  },
  {
    "startMs": 1154160,
    "text": "rollouts. Um so yeah I think the the"
  },
  {
    "startMs": 1158559,
    "text": "point I'm trying to make here is that um"
  },
  {
    "startMs": 1162000,
    "text": "like we've released this there's a blog"
  },
  {
    "startMs": 1163280,
    "text": "post that where we go into like all the"
  },
  {
    "startMs": 1164799,
    "text": "technical detail. We'll be releasing the"
  },
  {
    "startMs": 1167039,
    "text": "source code for it very soon. uh it's"
  },
  {
    "startMs": 1169280,
    "text": "based on miles and silang the the"
  },
  {
    "startMs": 1171520,
    "text": "reinforcement learning framework in the"
  },
  {
    "startMs": 1172799,
    "text": "inference engine and um I think the the"
  },
  {
    "startMs": 1176640,
    "text": "takeaway is that um I do believe that if"
  },
  {
    "startMs": 1179840,
    "text": "you want to be competitive in a world"
  },
  {
    "startMs": 1181360,
    "text": "where a bunch of different competitors"
  },
  {
    "startMs": 1182720,
    "text": "are running agent stacks you do need to"
  },
  {
    "startMs": 1185280,
    "text": "some degree own the full stack yourself"
  },
  {
    "startMs": 1187120,
    "text": "or at least like build competency in"
  },
  {
    "startMs": 1189440,
    "text": "every area of the stack. Um I think a"
  },
  {
    "startMs": 1192960,
    "text": "lot of work traditionally ends at this"
  },
  {
    "startMs": 1194559,
    "text": "agent harness but actually um the"
  },
  {
    "startMs": 1197840,
    "text": "vertical integration across all of this"
  },
  {
    "startMs": 1199280,
    "text": "is just extremely important in a in a"
  },
  {
    "startMs": 1201679,
    "text": "hyperco competitive uh industry. We see"
  },
  {
    "startMs": 1204480,
    "text": "this again in certain industries like"
  },
  {
    "startMs": 1206000,
    "text": "coding. Um but I think it we we also see"
  },
  {
    "startMs": 1209440,
    "text": "it in other industries and it's only"
  },
  {
    "startMs": 1211280,
    "text": "going to keep growing I think. Um, and"
  },
  {
    "startMs": 1213520,
    "text": "of course like if you're building agent"
  },
  {
    "startMs": 1215840,
    "text": "deployments, you own everything up the"
  },
  {
    "startMs": 1217760,
    "text": "stack as well. And vertically"
  },
  {
    "startMs": 1219200,
    "text": "integrating across all of this is"
  },
  {
    "startMs": 1221200,
    "text": "extremely important uh for maintaining"
  },
  {
    "startMs": 1223679,
    "text": "kind of competency and and competitive"
  },
  {
    "startMs": 1225679,
    "text": "edge. Um, yeah, you should own your own"
  },
  {
    "startMs": 1228799,
    "text": "agent RL and it's actually possible now"
  },
  {
    "startMs": 1230799,
    "text": "because the science has been figured out"
  },
  {
    "startMs": 1232640,
    "text": "and there are very good tools and very"
  },
  {
    "startMs": 1234159,
    "text": "good infrastructure that lets you do"
  },
  {
    "startMs": 1236000,
    "text": "that. Um, that's all I got. So thanks."
  },
  {
    "startMs": 1240240,
    "text": "We're we're working on hard problems."
  },
  {
    "startMs": 1242240,
    "text": "Come visit our booth and and uh talk to"
  },
  {
    "startMs": 1244240,
    "text": "us."
  }
]
