[
  {
    "startMs": 12720,
    "text": "All right. I think we'll go ahead and"
  },
  {
    "startMs": 14800,
    "text": "get started with the with the"
  },
  {
    "startMs": 16600,
    "text": "presentation. So my name is James So. I"
  },
  {
    "startMs": 19200,
    "text": "am uh"
  },
  {
    "startMs": 20480,
    "text": "going to explain some of the work we're"
  },
  {
    "startMs": 22000,
    "text": "doing with Together AI and it's also in"
  },
  {
    "startMs": 24680,
    "text": "collaboration with Stanford around"
  },
  {
    "startMs": 26920,
    "text": "designing and optimizing environments"
  },
  {
    "startMs": 29520,
    "text": "for AI agents to enable these agents to"
  },
  {
    "startMs": 32119,
    "text": "make new kinds of scientific"
  },
  {
    "startMs": 33840,
    "text": "discoveries."
  },
  {
    "startMs": 35520,
    "text": "All right."
  },
  {
    "startMs": 39240,
    "text": "So so that I guess the current paradigm"
  },
  {
    "startMs": 41080,
    "text": "of how people often are using or"
  },
  {
    "startMs": 42680,
    "text": "deploying AI agents is often involves"
  },
  {
    "startMs": 44880,
    "text": "designing workflows that sort of tells"
  },
  {
    "startMs": 48080,
    "text": "the agents you know what to do, right?"
  },
  {
    "startMs": 50960,
    "text": "Or how the agent should work. And it's"
  },
  {
    "startMs": 53440,
    "text": "typically done through a series of steps"
  },
  {
    "startMs": 55360,
    "text": "or prompts, tools, and instructions."
  },
  {
    "startMs": 58840,
    "text": "In contrast, the way we imagine the"
  },
  {
    "startMs": 60920,
    "text": "environment is that the environment"
  },
  {
    "startMs": 62400,
    "text": "should really specify"
  },
  {
    "startMs": 64120,
    "text": "not how the agent should work, but"
  },
  {
    "startMs": 65960,
    "text": "really where the agent should work,"
  },
  {
    "startMs": 68400,
    "text": "right? And the environment then should"
  },
  {
    "startMs": 69680,
    "text": "provide a set of incentives and"
  },
  {
    "startMs": 71800,
    "text": "infrastructure for the agents and"
  },
  {
    "startMs": 74680,
    "text": "guardrails and resources so that agent"
  },
  {
    "startMs": 77400,
    "text": "can then flexibly work within that"
  },
  {
    "startMs": 79040,
    "text": "environment."
  },
  {
    "startMs": 80800,
    "text": "Right. And our thesis here is that as"
  },
  {
    "startMs": 82720,
    "text": "agents become more and more powerful,"
  },
  {
    "startMs": 84920,
    "text": "right? If we try to design workflows"
  },
  {
    "startMs": 86760,
    "text": "that often can limit the capabilities"
  },
  {
    "startMs": 88800,
    "text": "and creativity of the agents. Whereas if"
  },
  {
    "startMs": 91080,
    "text": "we properly design the environment, this"
  },
  {
    "startMs": 92960,
    "text": "can enables a lot more creativity and"
  },
  {
    "startMs": 96080,
    "text": "capabilities and intelligence for the"
  },
  {
    "startMs": 97480,
    "text": "agents to naturally emerge. This why I"
  },
  {
    "startMs": 100000,
    "text": "think we're trying to shift away from"
  },
  {
    "startMs": 101320,
    "text": "designing workflows and harnesses"
  },
  {
    "startMs": 103400,
    "text": "towards designing environments."
  },
  {
    "startMs": 106640,
    "text": "So what I want to do today is to give a"
  },
  {
    "startMs": 108560,
    "text": "few examples of the how we design"
  },
  {
    "startMs": 110520,
    "text": "environments for agents."
  },
  {
    "startMs": 112360,
    "text": "And in particular also show how they're"
  },
  {
    "startMs": 114120,
    "text": "able to then with with the right"
  },
  {
    "startMs": 115480,
    "text": "environment able to actually solve some"
  },
  {
    "startMs": 117160,
    "text": "really interesting and innovative"
  },
  {
    "startMs": 118480,
    "text": "problems."
  },
  {
    "startMs": 122040,
    "text": "So, the first example I want to share is"
  },
  {
    "startMs": 123720,
    "text": "the system that we environment that we"
  },
  {
    "startMs": 125479,
    "text": "created called the Einstein Arena."
  },
  {
    "startMs": 127640,
    "text": "It's sort of like the one of the first"
  },
  {
    "startMs": 129160,
    "text": "environments that enables AI agents to"
  },
  {
    "startMs": 131800,
    "text": "be able to collaborate in the wild and"
  },
  {
    "startMs": 134000,
    "text": "to compete to really solve open-ended"
  },
  {
    "startMs": 136560,
    "text": "scientific problems."
  },
  {
    "startMs": 138760,
    "text": "So, we designed this Einstein Arena to"
  },
  {
    "startMs": 140360,
    "text": "be really agent native. So, I So, that"
  },
  {
    "startMs": 142800,
    "text": "means that"
  },
  {
    "startMs": 143959,
    "text": "it's very easy for agents to just read"
  },
  {
    "startMs": 146040,
    "text": "the skills talk on our on our arena and"
  },
  {
    "startMs": 148920,
    "text": "be able to access the arena."
  },
  {
    "startMs": 151200,
    "text": "And it's actually also designed so that"
  },
  {
    "startMs": 153640,
    "text": "it's intentionally very hard for humans"
  },
  {
    "startMs": 155640,
    "text": "to enter the arena, right? So, you"
  },
  {
    "startMs": 157200,
    "text": "actually have to solve a little puzzle"
  },
  {
    "startMs": 159040,
    "text": "to prove that you're an AI agent in"
  },
  {
    "startMs": 160840,
    "text": "order to participate in this arena. But,"
  },
  {
    "startMs": 163320,
    "text": "any agent in the world can openly and"
  },
  {
    "startMs": 165280,
    "text": "freely participate on the arena."
  },
  {
    "startMs": 167720,
    "text": "And once the agent actually enters into"
  },
  {
    "startMs": 169320,
    "text": "the Einstein Arena, this is what they'll"
  },
  {
    "startMs": 171200,
    "text": "see, right? They'll see actually see a"
  },
  {
    "startMs": 172520,
    "text": "list of curated problems. Each of these"
  },
  {
    "startMs": 175120,
    "text": "problems is actually a problem that we"
  },
  {
    "startMs": 176640,
    "text": "curated, so it's a scientifically"
  },
  {
    "startMs": 178440,
    "text": "interesting problem. And we curated"
  },
  {
    "startMs": 180400,
    "text": "these problems so that first, there's"
  },
  {
    "startMs": 181880,
    "text": "actually an existing community of human"
  },
  {
    "startMs": 184280,
    "text": "researchers that are interested in these"
  },
  {
    "startMs": 185840,
    "text": "problems. So, these are important"
  },
  {
    "startMs": 187600,
    "text": "problems for human scientists. And"
  },
  {
    "startMs": 189800,
    "text": "second is that for each of these"
  },
  {
    "startMs": 190959,
    "text": "problems, we can actually create a"
  },
  {
    "startMs": 192920,
    "text": "well-defined and deterministic"
  },
  {
    "startMs": 195080,
    "text": "deterministic verifier to assess the"
  },
  {
    "startMs": 198080,
    "text": "quality of the solutions to each of"
  },
  {
    "startMs": 199519,
    "text": "these problems. And I'll give some"
  },
  {
    "startMs": 201320,
    "text": "examples in a couple of slides."
  },
  {
    "startMs": 205280,
    "text": "So, So, the agents can actually decide"
  },
  {
    "startMs": 207320,
    "text": "which of these problems they're"
  },
  {
    "startMs": 208239,
    "text": "interested in once they log onto the"
  },
  {
    "startMs": 209680,
    "text": "arena, right? So, if they enter into a"
  },
  {
    "startMs": 211560,
    "text": "particular problem space, this is what"
  },
  {
    "startMs": 214000,
    "text": "they'll see, right? They'll see some"
  },
  {
    "startMs": 215239,
    "text": "description that precisely explains what"
  },
  {
    "startMs": 217480,
    "text": "is the problem. We have a discussion"
  },
  {
    "startMs": 219720,
    "text": "forum where the agents can communicate."
  },
  {
    "startMs": 222640,
    "text": "It's almost like a social network where"
  },
  {
    "startMs": 224000,
    "text": "the agents can actually communicate and"
  },
  {
    "startMs": 225280,
    "text": "talk to each other and ask for help or"
  },
  {
    "startMs": 227480,
    "text": "give recommendations."
  },
  {
    "startMs": 229280,
    "text": "Um and we also have a leaderboard. This"
  },
  {
    "startMs": 232480,
    "text": "is where the agent can actually see each"
  },
  {
    "startMs": 234080,
    "text": "other's solutions. Right? So in any in"
  },
  {
    "startMs": 237280,
    "text": "at any time they want, the agent can"
  },
  {
    "startMs": 238680,
    "text": "actually submit a solution to one of"
  },
  {
    "startMs": 240520,
    "text": "these problems. And because we have this"
  },
  {
    "startMs": 242920,
    "text": "verifier, we can actually then determine"
  },
  {
    "startMs": 245040,
    "text": "what is the quality of that solution and"
  },
  {
    "startMs": 246600,
    "text": "provide a score in real time. So this"
  },
  {
    "startMs": 249280,
    "text": "leaderboard is being constantly updated"
  },
  {
    "startMs": 251360,
    "text": "in real time. And the agents can also"
  },
  {
    "startMs": 253480,
    "text": "see how other agents are doing on this"
  },
  {
    "startMs": 255840,
    "text": "problem. And they can also see other"
  },
  {
    "startMs": 257920,
    "text": "agents' solutions and download those"
  },
  {
    "startMs": 259359,
    "text": "solutions."
  },
  {
    "startMs": 260680,
    "text": "So there's both a collaboration dynamics"
  },
  {
    "startMs": 262520,
    "text": "and also a competition dynamics in this"
  },
  {
    "startMs": 264520,
    "text": "arena, right? They can collaborate and"
  },
  {
    "startMs": 266480,
    "text": "ask each other questions and help in the"
  },
  {
    "startMs": 268520,
    "text": "discussion forum. But agents are also"
  },
  {
    "startMs": 270000,
    "text": "competing with each other. And that's"
  },
  {
    "startMs": 271880,
    "text": "why I think this also sort of simulates"
  },
  {
    "startMs": 273320,
    "text": "how human researchers can compete and"
  },
  {
    "startMs": 274960,
    "text": "also collaborate to solve interesting"
  },
  {
    "startMs": 276840,
    "text": "problems."
  },
  {
    "startMs": 279840,
    "text": "So we launched this AI instant arena"
  },
  {
    "startMs": 282040,
    "text": "environment"
  },
  {
    "startMs": 283600,
    "text": "earlier this year, I think in March. And"
  },
  {
    "startMs": 286040,
    "text": "within a few weeks, it's already"
  },
  {
    "startMs": 287640,
    "text": "actually we're very impressed and very"
  },
  {
    "startMs": 289480,
    "text": "surprised that the agents were actually"
  },
  {
    "startMs": 290760,
    "text": "able to already discover new solutions"
  },
  {
    "startMs": 294760,
    "text": "to 11 problems that are of the best"
  },
  {
    "startMs": 298200,
    "text": "solutions that have ever been found."
  },
  {
    "startMs": 299680,
    "text": "Right? So that means that the solutions"
  },
  {
    "startMs": 301080,
    "text": "that they discovered by the agents on AI"
  },
  {
    "startMs": 303000,
    "text": "instant arena were better than any"
  },
  {
    "startMs": 305000,
    "text": "previous human solutions or any"
  },
  {
    "startMs": 306680,
    "text": "solutions that we acquired using more"
  },
  {
    "startMs": 309160,
    "text": "specialized AI tools."
  },
  {
    "startMs": 312480,
    "text": "So I'll just give you example of one"
  },
  {
    "startMs": 314160,
    "text": "such solution or one such problem"
  },
  {
    "startMs": 316680,
    "text": "which is called the kissing number"
  },
  {
    "startMs": 318240,
    "text": "problem."
  },
  {
    "startMs": 319440,
    "text": "So this is actually a very famous"
  },
  {
    "startMs": 320720,
    "text": "problem. It's been around for hundreds"
  },
  {
    "startMs": 322080,
    "text": "of years. So for example, Isaac Newton"
  },
  {
    "startMs": 324560,
    "text": "was already working on some version of"
  },
  {
    "startMs": 326040,
    "text": "this kissing number problem. And it's"
  },
  {
    "startMs": 327320,
    "text": "actually relatively easy to state."
  },
  {
    "startMs": 329440,
    "text": "Right? So the kissing number problem"
  },
  {
    "startMs": 330600,
    "text": "basically asks that what is the maximum"
  },
  {
    "startMs": 333520,
    "text": "number of spheres that you can place"
  },
  {
    "startMs": 335520,
    "text": "around the central sphere so that these"
  },
  {
    "startMs": 337520,
    "text": "additional spheres do not overlap each"
  },
  {
    "startMs": 339200,
    "text": "other?"
  },
  {
    "startMs": 340480,
    "text": "So for example, in one dimensions,"
  },
  {
    "startMs": 342200,
    "text": "right? So around the central sphere I"
  },
  {
    "startMs": 343440,
    "text": "can place one sphere to the left and one"
  },
  {
    "startMs": 344960,
    "text": "sphere to the right without overlap. So"
  },
  {
    "startMs": 347000,
    "text": "the kissing number in one dimension is"
  },
  {
    "startMs": 348840,
    "text": "easy to compute. This is two."
  },
  {
    "startMs": 350800,
    "text": "In two dimensions, it's also easy to"
  },
  {
    "startMs": 352240,
    "text": "show that you can at most place six"
  },
  {
    "startMs": 354280,
    "text": "spheres. So, that's the kissing number"
  },
  {
    "startMs": 355880,
    "text": "in two dimensions is six."
  },
  {
    "startMs": 358120,
    "text": "But, it turns out that in higher"
  },
  {
    "startMs": 359280,
    "text": "dimensions, it actually becomes really"
  },
  {
    "startMs": 361240,
    "text": "hard to compute what's the maximum"
  },
  {
    "startMs": 362640,
    "text": "number of over non-overlapping spheres."
  },
  {
    "startMs": 365360,
    "text": "And the kissing number problem in higher"
  },
  {
    "startMs": 366600,
    "text": "dimensions is actually open, right? It's"
  },
  {
    "startMs": 368600,
    "text": "not been It's not clear what is the"
  },
  {
    "startMs": 371120,
    "text": "optimal number."
  },
  {
    "startMs": 372760,
    "text": "And so, scientists have been trying to"
  },
  {
    "startMs": 374960,
    "text": "work on this problem for the last"
  },
  {
    "startMs": 376760,
    "text": "several centuries."
  },
  {
    "startMs": 379240,
    "text": "And in particular, right, so the kissing"
  },
  {
    "startMs": 381360,
    "text": "number problem in 11 dimensions has"
  },
  {
    "startMs": 382960,
    "text": "attracted a lot of interest for various"
  },
  {
    "startMs": 385120,
    "text": "reasons."
  },
  {
    "startMs": 386800,
    "text": "So, this is actually sort of a"
  },
  {
    "startMs": 387680,
    "text": "progression of the solutions in 11"
  },
  {
    "startMs": 390120,
    "text": "dimensions."
  },
  {
    "startMs": 391600,
    "text": "So, in the 1980s, right, so it's best"
  },
  {
    "startMs": 394600,
    "text": "known that there you can place 440"
  },
  {
    "startMs": 396960,
    "text": "spheres, right, in 11 dimensions without"
  },
  {
    "startMs": 399760,
    "text": "overlap."
  },
  {
    "startMs": 401520,
    "text": "And in"
  },
  {
    "startMs": 403680,
    "text": "I think 19"
  },
  {
    "startMs": 405120,
    "text": "uh"
  },
  {
    "startMs": 406560,
    "text": "So, yeah, so so in in 1980, there was a"
  },
  {
    "startMs": 408800,
    "text": "big advance that the first for the first"
  },
  {
    "startMs": 410960,
    "text": "time showed that you can actually just"
  },
  {
    "startMs": 412080,
    "text": "construct with 582 spheres in 11"
  },
  {
    "startMs": 415200,
    "text": "dimensions without overlap."
  },
  {
    "startMs": 417520,
    "text": "Uh and then that sort of stuck there for"
  },
  {
    "startMs": 420040,
    "text": "about 40 years, right, until 2022, where"
  },
  {
    "startMs": 423960,
    "text": "a mathematician is able to publish a new"
  },
  {
    "startMs": 426280,
    "text": "advance, right,"
  },
  {
    "startMs": 427880,
    "text": "a breakthrough that's able to improve"
  },
  {
    "startMs": 429040,
    "text": "that to 592 spheres."
  },
  {
    "startMs": 431680,
    "text": "And then there's another breakthrough"
  },
  {
    "startMs": 433080,
    "text": "from DeepMind the following year that"
  },
  {
    "startMs": 435320,
    "text": "advances that to 593 spheres."
  },
  {
    "startMs": 438600,
    "text": "But, with Alpha Zero, we know by having"
  },
  {
    "startMs": 440200,
    "text": "these agents able to collaborate"
  },
  {
    "startMs": 442480,
    "text": "actively, right, in the wild, within a"
  },
  {
    "startMs": 445120,
    "text": "few days they were actually able to"
  },
  {
    "startMs": 446400,
    "text": "construct a new solution that shows that"
  },
  {
    "startMs": 448680,
    "text": "for the first time you can create 604"
  },
  {
    "startMs": 451080,
    "text": "spheres in 11 dimensions that do not"
  },
  {
    "startMs": 452840,
    "text": "overlap."
  },
  {
    "startMs": 454680,
    "text": "And this is not just a problem that's of"
  },
  {
    "startMs": 456120,
    "text": "mathematical interest, because it turns"
  },
  {
    "startMs": 457800,
    "text": "out that"
  },
  {
    "startMs": 458800,
    "text": "the more of these sort of spheres you"
  },
  {
    "startMs": 460240,
    "text": "can place in higher dimensions without"
  },
  {
    "startMs": 461880,
    "text": "overlap that actually creates the better"
  },
  {
    "startMs": 464000,
    "text": "coding systems including ways of like"
  },
  {
    "startMs": 466200,
    "text": "doing error correction codes for"
  },
  {
    "startMs": 468400,
    "text": "information transfer. Right, so this"
  },
  {
    "startMs": 470320,
    "text": "actually is by creating this better"
  },
  {
    "startMs": 472040,
    "text": "constructions that also leads to this"
  },
  {
    "startMs": 474000,
    "text": "better engineering algorithms."
  },
  {
    "startMs": 477000,
    "text": "And in this case actually the"
  },
  {
    "startMs": 478160,
    "text": "collaborations among these agents is"
  },
  {
    "startMs": 480160,
    "text": "really critical for making these"
  },
  {
    "startMs": 481560,
    "text": "advances, right? So this is a problem"
  },
  {
    "startMs": 483920,
    "text": "where not a single agent is able to"
  },
  {
    "startMs": 485600,
    "text": "solve by itself, right? Not you know,"
  },
  {
    "startMs": 487800,
    "text": "GPT 5.5 or a cloud models that can't"
  },
  {
    "startMs": 490560,
    "text": "really solve the problem by itself. So"
  },
  {
    "startMs": 492280,
    "text": "the collaboration among multiple agents"
  },
  {
    "startMs": 493840,
    "text": "is really critical."
  },
  {
    "startMs": 495440,
    "text": "And here we're actually able to show"
  },
  {
    "startMs": 496720,
    "text": "that there's like this"
  },
  {
    "startMs": 498200,
    "text": "sort of a lineage trace of how the"
  },
  {
    "startMs": 499760,
    "text": "agents are able to collaborate and then"
  },
  {
    "startMs": 501760,
    "text": "basically take each other's solutions"
  },
  {
    "startMs": 503480,
    "text": "and refine that and further optimize it"
  },
  {
    "startMs": 505800,
    "text": "to arrive at this breakthrough."
  },
  {
    "startMs": 508320,
    "text": "And you can also see some of these"
  },
  {
    "startMs": 509520,
    "text": "interactions and discussions on Einstein"
  },
  {
    "startMs": 511360,
    "text": "Arena, right? Where here's an example"
  },
  {
    "startMs": 513880,
    "text": "where the one agent actually was asking"
  },
  {
    "startMs": 516000,
    "text": "other agents, \"Have you tried"
  },
  {
    "startMs": 518200,
    "text": "you know, some of these approaches?\" Um,"
  },
  {
    "startMs": 520800,
    "text": "with uh, these STP approaches and then"
  },
  {
    "startMs": 523120,
    "text": "the other agents showed that yes, we"
  },
  {
    "startMs": 524280,
    "text": "have tried these approaches and here are"
  },
  {
    "startMs": 525440,
    "text": "some of the things that we found. Right,"
  },
  {
    "startMs": 527200,
    "text": "so the information sharing on the forums"
  },
  {
    "startMs": 530000,
    "text": "on the arena is actually really"
  },
  {
    "startMs": 531040,
    "text": "important to help the agents to arrive"
  },
  {
    "startMs": 533320,
    "text": "at this solution together."
  },
  {
    "startMs": 537360,
    "text": "So in addition to solving these"
  },
  {
    "startMs": 538720,
    "text": "interesting scientific problems, but"
  },
  {
    "startMs": 540839,
    "text": "we've also been using platforms like the"
  },
  {
    "startMs": 542760,
    "text": "right Einstein Arena uh, to help to"
  },
  {
    "startMs": 545160,
    "text": "improve uh, you know, machine learning"
  },
  {
    "startMs": 547720,
    "text": "and AI itself."
  },
  {
    "startMs": 549520,
    "text": "Right, so here's one example where we"
  },
  {
    "startMs": 550880,
    "text": "actually use these agents to basically"
  },
  {
    "startMs": 552440,
    "text": "help us to create better kernels"
  },
  {
    "startMs": 555320,
    "text": "for and to speed up those kernels."
  },
  {
    "startMs": 557960,
    "text": "Right, and here we use the same"
  },
  {
    "startMs": 559560,
    "text": "environment, right? Where the agents can"
  },
  {
    "startMs": 561080,
    "text": "compete and they also can collaborate"
  },
  {
    "startMs": 563200,
    "text": "and they see these leaderboards. And we"
  },
  {
    "startMs": 565160,
    "text": "basically change the back end instead of"
  },
  {
    "startMs": 566680,
    "text": "trying to verify the solutions to this"
  },
  {
    "startMs": 568440,
    "text": "mathematics problem, here we're"
  },
  {
    "startMs": 570040,
    "text": "basically trying to"
  },
  {
    "startMs": 571320,
    "text": "you know, we will compile and benchmark"
  },
  {
    "startMs": 573360,
    "text": "and test and verify the quality and the"
  },
  {
    "startMs": 575120,
    "text": "speed of the individual kernels, right?"
  },
  {
    "startMs": 577320,
    "text": "And then we'll provide a feedback to the"
  },
  {
    "startMs": 578680,
    "text": "agents in real time in the form of these"
  },
  {
    "startMs": 580760,
    "text": "leaderboards."
  },
  {
    "startMs": 583520,
    "text": "In these kernel settings, we also found"
  },
  {
    "startMs": 585280,
    "text": "it to be quite useful to have different"
  },
  {
    "startMs": 586800,
    "text": "agents with different personas,"
  },
  {
    "startMs": 588800,
    "text": "right? And these different personas"
  },
  {
    "startMs": 589960,
    "text": "actually corresponds to a different uh"
  },
  {
    "startMs": 591680,
    "text": "roles and priors that agents can"
  },
  {
    "startMs": 593680,
    "text": "actually have. So, for example, we have"
  },
  {
    "startMs": 595640,
    "text": "one agent that looks at tends to look at"
  },
  {
    "startMs": 597480,
    "text": "more of the profiling, another agent"
  },
  {
    "startMs": 599200,
    "text": "that tends to look at more of the memory"
  },
  {
    "startMs": 600680,
    "text": "consumptions, a third agent that looks"
  },
  {
    "startMs": 602880,
    "text": "at, you know, the precisions, the tensor"
  },
  {
    "startMs": 604640,
    "text": "computations. And these agents can and"
  },
  {
    "startMs": 607200,
    "text": "then across different personas, they can"
  },
  {
    "startMs": 608920,
    "text": "able to collaborate and a compete on the"
  },
  {
    "startMs": 611040,
    "text": "arena to speed up the kernels."
  },
  {
    "startMs": 614680,
    "text": "And in this case, right here, the agents"
  },
  {
    "startMs": 616680,
    "text": "were also able to collaborate and lead"
  },
  {
    "startMs": 618520,
    "text": "to really quite substantial speed ups,"
  },
  {
    "startMs": 620720,
    "text": "uh including sometimes over two two x"
  },
  {
    "startMs": 623560,
    "text": "two-fold speed ups in some of these"
  },
  {
    "startMs": 625160,
    "text": "production kernels. So, here I'm just"
  },
  {
    "startMs": 627200,
    "text": "showing you a few examples where for"
  },
  {
    "startMs": 628840,
    "text": "things like page attention, uh and these"
  },
  {
    "startMs": 631000,
    "text": "are sort of for specific shapes, but we"
  },
  {
    "startMs": 632840,
    "text": "also have generalized this to many"
  },
  {
    "startMs": 634240,
    "text": "different shapes and different uh"
  },
  {
    "startMs": 636240,
    "text": "hardware types, right? Where we're"
  },
  {
    "startMs": 637920,
    "text": "actually seeing that we're getting up to"
  },
  {
    "startMs": 639839,
    "text": "sometimes over two x speed up in these"
  },
  {
    "startMs": 641440,
    "text": "kernels, and they uh compared to the"
  },
  {
    "startMs": 644120,
    "text": "previous state-of-the-art kernels for"
  },
  {
    "startMs": 645680,
    "text": "these problems."
  },
  {
    "startMs": 647240,
    "text": "And these improved kernels created"
  },
  {
    "startMs": 649880,
    "text": "designed by the agents are actually"
  },
  {
    "startMs": 651200,
    "text": "already used in in production at"
  },
  {
    "startMs": 653040,
    "text": "Together AI."
  },
  {
    "startMs": 657200,
    "text": "So, in the last few minutes, I want to"
  },
  {
    "startMs": 658440,
    "text": "show like a second example of a kind of"
  },
  {
    "startMs": 660680,
    "text": "environment that we created as a way to"
  },
  {
    "startMs": 663440,
    "text": "uh train and to create better data"
  },
  {
    "startMs": 665480,
    "text": "scientist agents,"
  },
  {
    "startMs": 667120,
    "text": "right? So, we call this DS Gym, which"
  },
  {
    "startMs": 668839,
    "text": "stands for data science gym, which is"
  },
  {
    "startMs": 671000,
    "text": "sort of like a unified environment that"
  },
  {
    "startMs": 672440,
    "text": "we created for both for evaluating and"
  },
  {
    "startMs": 674640,
    "text": "for training data science agents to"
  },
  {
    "startMs": 677160,
    "text": "solve complex data science problems."
  },
  {
    "startMs": 681120,
    "text": "So, here in this DS Gym environment, we"
  },
  {
    "startMs": 683760,
    "text": "also curated and created a unified list"
  },
  {
    "startMs": 686880,
    "text": "of different data sets and tasks,"
  },
  {
    "startMs": 689120,
    "text": "right? So, these data sets can combine"
  },
  {
    "startMs": 691040,
    "text": "uh spans across many different settings."
  },
  {
    "startMs": 693800,
    "text": "And the agents are then able to interact"
  },
  {
    "startMs": 695560,
    "text": "with these different data sets that we"
  },
  {
    "startMs": 696920,
    "text": "have through a unified uh interface and"
  },
  {
    "startMs": 700280,
    "text": "through code execution."
  },
  {
    "startMs": 702400,
    "text": "In the DSGM environment, we also provide"
  },
  {
    "startMs": 704520,
    "text": "a unified infrastructure for the agents."
  },
  {
    "startMs": 707000,
    "text": "So, for example, the agents can actually"
  },
  {
    "startMs": 708200,
    "text": "spin up many different Docker containers"
  },
  {
    "startMs": 710800,
    "text": "to test their data science algorithms"
  },
  {
    "startMs": 712880,
    "text": "and actually run them in parallel."
  },
  {
    "startMs": 718080,
    "text": "So, in the process of actually creating"
  },
  {
    "startMs": 719680,
    "text": "the data sets and tasks for the data"
  },
  {
    "startMs": 721200,
    "text": "DSGM environment, so we initially"
  },
  {
    "startMs": 723680,
    "text": "actually wanted to incorporate some of"
  },
  {
    "startMs": 725920,
    "text": "the existing data science benchmarks"
  },
  {
    "startMs": 727560,
    "text": "that have been used to evaluate agents."
  },
  {
    "startMs": 730360,
    "text": "But we actually quickly realized that"
  },
  {
    "startMs": 731720,
    "text": "many of the existing widely-used"
  },
  {
    "startMs": 733800,
    "text": "benchmarks actually have many problems."
  },
  {
    "startMs": 736280,
    "text": "And one big problem is that they're"
  },
  {
    "startMs": 737760,
    "text": "actually very vulnerable to shortcuts."
  },
  {
    "startMs": 739760,
    "text": "By shortcut, I mean here is that"
  },
  {
    "startMs": 741920,
    "text": "uh down here what I'm showing are three"
  },
  {
    "startMs": 743320,
    "text": "different common popular data science"
  },
  {
    "startMs": 745320,
    "text": "benchmarks."
  },
  {
    "startMs": 746640,
    "text": "Right? And the in green here we see"
  },
  {
    "startMs": 748040,
    "text": "shows like the performance of the agents"
  },
  {
    "startMs": 750280,
    "text": "on these benchmarks."
  },
  {
    "startMs": 752200,
    "text": "Uh but the red bar also shows how well"
  },
  {
    "startMs": 754080,
    "text": "they're able to the what fraction of the"
  },
  {
    "startMs": 755800,
    "text": "benchmark the agents can actually solve"
  },
  {
    "startMs": 757680,
    "text": "without actually using the data sets"
  },
  {
    "startMs": 759240,
    "text": "themselves. Right? So, just by reasoning"
  },
  {
    "startMs": 761400,
    "text": "or by, you know,"
  },
  {
    "startMs": 762600,
    "text": "uh doing other shortcuts without"
  },
  {
    "startMs": 763839,
    "text": "actually actually do working with the"
  },
  {
    "startMs": 765320,
    "text": "underlying data sets."
  },
  {
    "startMs": 767320,
    "text": "And across many of these different"
  },
  {
    "startMs": 768720,
    "text": "benchmarks, right, sometimes up to 20 to"
  },
  {
    "startMs": 771880,
    "text": "50% of the tasks can be solved without"
  },
  {
    "startMs": 774680,
    "text": "actually looking at any of the"
  },
  {
    "startMs": 776040,
    "text": "underlying data."
  },
  {
    "startMs": 777680,
    "text": "Which I think is uh really a significant"
  },
  {
    "startMs": 779200,
    "text": "problem with many of the existing"
  },
  {
    "startMs": 780600,
    "text": "benchmarks."
  },
  {
    "startMs": 783120,
    "text": "So, to address that, we actually"
  },
  {
    "startMs": 784560,
    "text": "carefully curated at our own our own"
  },
  {
    "startMs": 786880,
    "text": "benchmarks, right, for both for"
  },
  {
    "startMs": 788720,
    "text": "scientific analysis and also for"
  },
  {
    "startMs": 790640,
    "text": "predictive modeling."
  },
  {
    "startMs": 792440,
    "text": "So, for scientific analysis and"
  },
  {
    "startMs": 793680,
    "text": "discovery, the way we did this is that"
  },
  {
    "startMs": 795600,
    "text": "we actually went through recently"
  },
  {
    "startMs": 797000,
    "text": "published papers and then carefully"
  },
  {
    "startMs": 799200,
    "text": "curated data and then also tasks from"
  },
  {
    "startMs": 800880,
    "text": "those papers. And then we also had human"
  },
  {
    "startMs": 802600,
    "text": "scientists and experts to review each of"
  },
  {
    "startMs": 804320,
    "text": "those tasks."
  },
  {
    "startMs": 806040,
    "text": "And for predictive modeling, the way we"
  },
  {
    "startMs": 807760,
    "text": "did this is go through all the different"
  },
  {
    "startMs": 809720,
    "text": "Kaggle competitions to look for some of"
  },
  {
    "startMs": 811920,
    "text": "the recent Kaggle competitions that are"
  },
  {
    "startMs": 813600,
    "text": "still open and and where also you have"
  },
  {
    "startMs": 815800,
    "text": "high quality data sets and also high"
  },
  {
    "startMs": 817360,
    "text": "quality"
  },
  {
    "startMs": 818560,
    "text": "evaluations. Then we curated those into"
  },
  {
    "startMs": 821360,
    "text": "the DS Gym as a kind of task for"
  },
  {
    "startMs": 823640,
    "text": "evaluating how well models agents can"
  },
  {
    "startMs": 825800,
    "text": "actually build predictive models."
  },
  {
    "startMs": 828440,
    "text": "So all together in the DS Gym, we"
  },
  {
    "startMs": 830200,
    "text": "actually have created over a dozen"
  },
  {
    "startMs": 832320,
    "text": "different tasks. They span across"
  },
  {
    "startMs": 835320,
    "text": "dozens of different scientific domains"
  },
  {
    "startMs": 838280,
    "text": "ranging from biology to physics to"
  },
  {
    "startMs": 839960,
    "text": "economics. It also involves many"
  },
  {
    "startMs": 841840,
    "text": "different data types and data"
  },
  {
    "startMs": 842920,
    "text": "modalities."
  },
  {
    "startMs": 846600,
    "text": "So this actually makes it very easy for"
  },
  {
    "startMs": 848280,
    "text": "us to evaluate different models, both"
  },
  {
    "startMs": 850360,
    "text": "open and closed source models. And one"
  },
  {
    "startMs": 852640,
    "text": "thing we found is that the existing"
  },
  {
    "startMs": 854200,
    "text": "models, even the frontier models, often"
  },
  {
    "startMs": 856600,
    "text": "are only still achieves like less than"
  },
  {
    "startMs": 858640,
    "text": "50% accuracy performance on the DS Gym"
  },
  {
    "startMs": 861760,
    "text": "tasks. Right? So these are definitely"
  },
  {
    "startMs": 863320,
    "text": "not saturated benchmarks."
  },
  {
    "startMs": 866280,
    "text": "We can also use a DS Gym as sort of like"
  },
  {
    "startMs": 868000,
    "text": "a training factory to improve these open"
  },
  {
    "startMs": 870120,
    "text": "source models."
  },
  {
    "startMs": 871680,
    "text": "Right? So one thing we did here is"
  },
  {
    "startMs": 872680,
    "text": "actually generate in the DS Gym actually"
  },
  {
    "startMs": 874839,
    "text": "the gym itself will actually create all"
  },
  {
    "startMs": 876560,
    "text": "these execution verified trajectories,"
  },
  {
    "startMs": 878800,
    "text": "which means that these are trajectories"
  },
  {
    "startMs": 880440,
    "text": "generated by the agents that have been"
  },
  {
    "startMs": 881600,
    "text": "verified through the through"
  },
  {
    "startMs": 884520,
    "text": "through actually executing the code from"
  },
  {
    "startMs": 886440,
    "text": "the agents."
  },
  {
    "startMs": 887680,
    "text": "Right? So by generating these execution"
  },
  {
    "startMs": 889360,
    "text": "verified trajectories, then we are able"
  },
  {
    "startMs": 891280,
    "text": "to like fine-tune sort of small open"
  },
  {
    "startMs": 893320,
    "text": "source models"
  },
  {
    "startMs": 894760,
    "text": "that actually now achieve sort of the"
  },
  {
    "startMs": 896240,
    "text": "they're sort of the best in class open"
  },
  {
    "startMs": 898400,
    "text": "source models in terms of solving these"
  },
  {
    "startMs": 900280,
    "text": "kind of data science tasks. Right? And"
  },
  {
    "startMs": 901720,
    "text": "these models are small enough that you"
  },
  {
    "startMs": 902960,
    "text": "can actually run them locally on your"
  },
  {
    "startMs": 904839,
    "text": "laptops and your computers."
  },
  {
    "startMs": 908440,
    "text": "So just to summarize the this part was"
  },
  {
    "startMs": 910600,
    "text": "the data science gym. Right? So we with"
  },
  {
    "startMs": 912560,
    "text": "DS Gym, we created this unified"
  },
  {
    "startMs": 914160,
    "text": "execution layer so people can actually"
  },
  {
    "startMs": 916080,
    "text": "run and all these different tasks across"
  },
  {
    "startMs": 918560,
    "text": "dozens of different tasks across many"
  },
  {
    "startMs": 920120,
    "text": "different domains. We have carefully"
  },
  {
    "startMs": 922160,
    "text": "verified that there are no shortcuts in"
  },
  {
    "startMs": 924000,
    "text": "these tasks, which has been sort of a"
  },
  {
    "startMs": 925640,
    "text": "common challenge with existing data"
  },
  {
    "startMs": 927440,
    "text": "science benchmarks."
  },
  {
    "startMs": 929040,
    "text": "And we also enable in the DSG and a way"
  },
  {
    "startMs": 931200,
    "text": "to generate synthetic data, so you can"
  },
  {
    "startMs": 932920,
    "text": "easily use that to improve and to train"
  },
  {
    "startMs": 936440,
    "text": "your own data science agents."
  },
  {
    "startMs": 939680,
    "text": "So, just to summarize the presentation,"
  },
  {
    "startMs": 942480,
    "text": "um I think the main takeaway here is"
  },
  {
    "startMs": 944360,
    "text": "that I think we're in seeing this"
  },
  {
    "startMs": 945440,
    "text": "interesting progression as in terms of"
  },
  {
    "startMs": 947520,
    "text": "how we build different AI systems."
  },
  {
    "startMs": 950080,
    "text": "Right. So, then in the past, people have"
  },
  {
    "startMs": 951280,
    "text": "been building these AI systems mostly by"
  },
  {
    "startMs": 952880,
    "text": "designing individual models or"
  },
  {
    "startMs": 954640,
    "text": "individual tools."
  },
  {
    "startMs": 956320,
    "text": "And currently, there's a lot of focus on"
  },
  {
    "startMs": 957880,
    "text": "creating designing agents or harnesses"
  },
  {
    "startMs": 960080,
    "text": "and workflows around agents."
  },
  {
    "startMs": 962320,
    "text": "But what our research shows is that I"
  },
  {
    "startMs": 963760,
    "text": "think we're already moving towards the"
  },
  {
    "startMs": 964840,
    "text": "next stage, where you're not then trying"
  },
  {
    "startMs": 966920,
    "text": "to design workflows or specific or"
  },
  {
    "startMs": 969040,
    "text": "specific agents, what we really want to"
  },
  {
    "startMs": 971120,
    "text": "do is to design environments, which is a"
  },
  {
    "startMs": 973480,
    "text": "set of infrastructure and incentives"
  },
  {
    "startMs": 975840,
    "text": "that in that motivates the agents that"
  },
  {
    "startMs": 977960,
    "text": "you solve more and more challenging"
  },
  {
    "startMs": 979760,
    "text": "problems."
  },
  {
    "startMs": 980960,
    "text": "And with appropriate designs these"
  },
  {
    "startMs": 982280,
    "text": "environments can actually unlock much"
  },
  {
    "startMs": 984640,
    "text": "more creativity and collective"
  },
  {
    "startMs": 986520,
    "text": "intelligence from the agents that's"
  },
  {
    "startMs": 988839,
    "text": "that's limited by the existing"
  },
  {
    "startMs": 990400,
    "text": "workflows."
  },
  {
    "startMs": 991839,
    "text": "And here are some of the references for"
  },
  {
    "startMs": 993760,
    "text": "the papers that we published that"
  },
  {
    "startMs": 994960,
    "text": "describes these in more detail. So,"
  },
  {
    "startMs": 996920,
    "text": "thank you very much."
  },
  {
    "startMs": 998886,
    "text": ">> [applause]"
  },
  {
    "startMs": 1011986,
    "text": "[music]"
  }
]
