[
  {
    "startMs": 12560,
    "text": "Hey everyone, I'm Akshat, CTO of Modal, and I'm gonna talk about what do agents really need from their infrastructure."
  },
  {
    "startMs": 24760,
    "text": "So I, I'm told this audience is, uh, very finance adjacent, so I thought I'd talk about what if every cell in a spreadsheet could be backed by an agent. And, uh, imagine you're a financial analyst at a fund. You wanna run a complex analysis on every company in your portfolio. Uh, normally in the past, this would take days. You'd have to"
  },
  {
    "startMs": 54760,
    "text": "connect every portfolio company, look at their data sources, uh, run some Python for each of them. Uh, and now obviously with agents, this should be much easier. Well, actually, one of our customers, Hebia, built something that's exactly this. You can go in right now in, in their product. You can, uh, you have a spreadsheet, and you can, uh, run a query where literally every cell is backed by a different"
  },
  {
    "startMs": 84580,
    "text": "agent. And I wanted to talk about what does building this really look like. Well, uh, first, every agent is backed by a modal sandbox. Uh, so a sandbox is a, uh, secure environment for running untrusted code. And, uh, it used to be that every sandbox, uh, was one-to-one with a container or"
  },
  {
    "startMs": 114080,
    "text": "microVM. That's actually no longer the case. Uh, today actually, uh, Modal has this, uh, feature where you can run multiple containers or microVMs inside a sandbox. And you actually need to do that for doing something like this, where, um, in this architecture, you run the harness. Uh, the harness is the agent loop that does things like running the, calling the model API for inference, uh, running the tool calls, and then looping over"
  },
  {
    "startMs": 143880,
    "text": "it. Uh, you wanna separate that from the place where the tool calls actually run, uh, for security and isolation and a bun- bunch of other things. Uh, so you have the sandbox that's running multiple containers, including a sidecar. Uh, you also want shared storage, uh, because you want your agents to be able to talk to each other, and it's up to you how you, uh, stripe it. Like, maybe you have every row, uh, gets the same volume, and agents in that row in the spreadsheet"
  },
  {
    "startMs": 173840,
    "text": "can collaborate. Uh, and you also wanna share storage between your main container and the sidecar for some cases. A- and finally, you need all of this to spin up really, really fast because your spreadsheet can have lots of cells. If someone runs a query, you wanna launch thousands of agents, and then you wanna spin them back down when it's over."
  },
  {
    "startMs": 195280,
    "text": "So before I talk more about, uh, that, I just wanted to take a quick aside to talk about what is Modal. Uh, Modal is a platform of primitives that is built for AI applications. Uh, so we've basically built this platform for, uh, from scratch. We've built our own file systems, the scheduling, and, uh, the runtime, uh, to give you really great performance and DevX for use cases like inference, including LM inference, sandboxes, training,"
  },
  {
    "startMs": 224940,
    "text": "and, uh, functions, running a lot of batch processing and whatnot. Uh, and today in particular, I'm gonna talk about sandboxes, what this primitive is, and how it's evolving. Uh, so if you wanna build something like this, uh, or even further, and we'll talk more about what that means, uh, what are the primitives you need for that? So the first thing is if you're building an agent, you need a workspace, and I have some examples from some of our current customers. Uh, you might have heard of Ramp Inspect. Uh, Ramp Inspect is"
  },
  {
    "startMs": 255320,
    "text": "Ramp's internal coding agent that, uh, I think at this point pushes almost every single PR at Ramp. Uh, and if you're building a coding agent, you need your, uh, repo, uh, your GitHub repo, you need your dependencies, you need your dev services all in the same workspace, uh, for each agent. Uh, another example is Basis. They're an, uh, accounting agent, and if you're building that, you need your, uh,"
  },
  {
    "startMs": 285360,
    "text": "documents, you need your accounting work papers, you need scripts you've written, uh, all in one work- workspace. Uh, and we have another, uh, very interesting company. They're a game dev company, and if you're doing game dev with agents, you need your game engine, you need your assets, uh, all in one place."
  },
  {
    "startMs": 304080,
    "text": "The next step is you need these workspaces to persist. Uh, so going back to the accounting example with Basis, what it looks like is your documents might arrive, your, uh, agent is then triggered to work on them, but then it's waiting for a response from the accountant, and you don't know how long this could take. Could take many days. Uh, when the accountant responds, you want the agent to start up again and continue from that last place where things were left"
  },
  {
    "startMs": 333880,
    "text": "off. Uh, so you want to persist the state over time, uh, potentially indefinitely. But you don't wanna run the compute the entire time because compute is expensive, so, um, you only want compute to run when the agent is actually working. Uh, this is something we support at Modal, and we've realized that there's a bunch of different flavors of persistence that people need. Uh, sometimes you might want to snapshot your whole root file system. Uh, sometimes you might want to"
  },
  {
    "startMs": 363680,
    "text": "snapshot just a single directory. And, uh, maybe it's like your node modules folder, uh, but then you want to upgrade your base image over time. Uh, and sometimes you might need something that looks more like a shared file system, like a volume."
  },
  {
    "startMs": 380960,
    "text": "Um, a very interesting thing, uh, that's been showing up is agents actually need a lot of elasticity and resource consumption. Uh, so we got permission from Ramp to share some of the data from Ramp Inspect. And, um, if you look at random sandboxes running in their, um, workload, they actually all have a huge amount of variance, uh, both, uh, versus each other, but also over time within the same sandbox for how much CPU and memory the agent is"
  },
  {
    "startMs": 410920,
    "text": "using. Uh, and it makes sense. I mean, what if you're compiling, um, something that's very resource-intensive or doing some analysis that needs a lot of memory? Your usage is gonna change. And in the second chart, you can see peak usage for each sandbox, and, uh, there's definitely some consistent trends, but it's all over the place. So if you were provisioning this for the peak, you would end up provisioning, uh, like fifteen CPU cores and fifteen gigs of RAM for every"
  },
  {
    "startMs": 440900,
    "text": "sandbox, which is just, uh, doesn't make sense economically. Uh, and something we've worked on a lot at Modal is h- um, letting users basically not have to request these resources upfront and then reliably get them when the sandbox needs them, uh, and only pay for them for, for that duration."
  },
  {
    "startMs": 461400,
    "text": "The other interesting emerging thing is agents need a lot more, um, interesting compute. Like, they need GPUs. Over time, they might need TPUs, AMD, who knows? And a great example of this is, uh, Cloud Science, where, um, Cloud Science actually ships with a modal integration out of the box. And, uh, the way it's used is when Cloud needs to do some intensive compute, like preprocess some data on CPUs, uh, or"
  },
  {
    "startMs": 491500,
    "text": "spin up GPUs for running a scientific model, uh, it will offload that to a modal sandbox. And we're seeing this show up in things like game design, video editing, uh, three D dev, like CAD. All those things need GPUs, uh, for your agent. And of course, the, the thing that everyone's really excited about right now is RSI or recursive, uh, self-improvement, and, um, th-that's something that GPU sandboxes are really good at and agents are really good at using"
  },
  {
    "startMs": 521440,
    "text": "them. Uh, so we have this blog post about what happens when you give an agent, uh, sandboxes to hill climb over training a model, and it turns out they're very good. They can swap GPUs. They can run ablations in parallel, run lots of experiments, and then really get, uh, to the target very quickly."
  },
  {
    "startMs": 543020,
    "text": "The next emerging thing is agents are collaborating a lot more, and, um, that means that you share a file system with your sub-agent. You share file systems and files with your, uh, human, where, uh, you're all collaborating with the same thing. And this needs file systems that are fast, have consistency and, uh, DevX you can reason about. Uh, it's something we're working on with Modal Volumes. Related thing is"
  },
  {
    "startMs": 573320,
    "text": "we're seeing a lot more self-improvement. Uh, so agents will, uh, save and update memory. They'll run background tasks, uh, which is now called dreaming, to update this memory over time. And actually, this also just looks like files in a file system. But again, you want really great performance from this file system."
  },
  {
    "startMs": 597400,
    "text": "And the scary thing about agents is they can mess up, and, uh, they don't really know when they might leak your sensitive data to a German forum or something. So you need to have a lot of control over what they can and cannot do, uh, and you need things like controlling what IPs can even talk to the agent, how do you authenticate it, uh, what domains can it talk to. Uh, and also if you're using"
  },
  {
    "startMs": 627340,
    "text": "credentials for talking to services, um, basically you want to inject them in a way that the agent cannot actually see them and then exfiltrate them somewhere later."
  },
  {
    "startMs": 639160,
    "text": "And the interesting evolution of this is we've realized that domain, uh, related allow lists are actually, um, not enough. U-- Uh, a lot of our users actually want very specific control over, uh, even, like, what paths can I talk to, uh, on, let's say, Datadog or GitHub. Like, maybe I want to, uh, reason about specific operations that the agent's doing. And going back to sidecars, we've seen this emerging as a pattern where users actually write their own lightweight"
  },
  {
    "startMs": 668920,
    "text": "proxy that has custom rules about what exactly it can do. And Ramp's another example of a company that's been using this successfully."
  },
  {
    "startMs": 680540,
    "text": "Uh, and the final underrated thing is it's really important to have really great observability. Uh, so going back to the example of auto research and an agent doing a lot of training, uh, what happens when the training run fails? Uh, your... You, you want your agent to be able to easily look at the logs of the failure, uh, reason about what happened, and do another iter-iteration with a different set of parameters. And while training is running, you want it to be able to look at the metrics. You want to look at GPU"
  },
  {
    "startMs": 710500,
    "text": "utilization and make sure it's not stalled. Uh, maybe you want to look at the budget and make sure you're not spending too much in your sandbox. Uh, and obviously, all of this is really important for humans. Uh, but as agents are driving a lot more compute, uh, it's... we're spending a lot of time exposing all of this to the agents themselves so they can reason about all of this."
  },
  {
    "startMs": 732640,
    "text": "All right. I think, uh, we know what we need. Uh, why don't we have Astra build us an agent platform? Well, I, I don't think it's a good idea. I don't think you should vibe code a sandbox platform. Uh, and, and there's three main reasons for that. Uh, number one is reliability. I think we're all painfully aware how hard it is to get right. Uh, certain websites we use every day keep going down. Uh, and the reason for that is even with AI, it's actually really hard"
  },
  {
    "startMs": 762620,
    "text": "to reason about the feedback loop of, like, what happens in the case of this tail event when the system breaks. Uh, and this is why you need to be very careful in building systems. Uh, security is another one. Uh, you could vibe code a sandbox platform, but who is gonna do all the work to, like, continually reason about the security boundaries and make sure it's, um, defensible in, in an age when everything is evolving very fast? Uh, and finally, scale. Uh, we have a blog"
  },
  {
    "startMs": 792640,
    "text": "post about how we scale to a million sandboxes, which involves a lot of very interesting engineering design and reasoning about systems. And this is, again, something that you kind of don't wanna leave up to a vibe coded platform because when you get to scale, you want it to work. Uh, and on that point, we're, we're actually seeing massive scale for, for a lot of things I mentioned. So, uh, for example, for the snapshots I mentioned, um, it's growing quite fast. We're, we're seeing twenty million"
  },
  {
    "startMs": 822420,
    "text": "snapshots a week. Uh, we're processing, uh, about eight petabytes of data, uh, a day, uh, for, for each of these snapshots. And, um, our users are creating, uh, almost two point six million volumes a week, uh, and all of this is growing super fast. And just to take a customer workload for personal assistant, uh, you need to create ten thousand sandboxes a second. You need to run, um, almost a million sandboxes,"
  },
  {
    "startMs": 852620,
    "text": "and they're not gonna last very long. They'll just last for, like, two or three minutes. Uh, but when they come up, you want them to be interactive in, in less than a second. That's, that's what this workload really demands."
  },
  {
    "startMs": 866180,
    "text": "And the other interesting thing on scale is you actually cannot get all of the scale from a single cloud platform. The, the world is actually getting crunched on CPUs as well, not just GPUs. Uh, so one thing we spent a lot of time doing is actually aggregating cloud capacity from a bunch of hyperscalers and neo clouds. Uh, so our users have access to CPU capacity across the world when they need it, uh, but we cannot rely on one cloud provider to do that."
  },
  {
    "startMs": 894700,
    "text": "Okay, so where is all this going? I, I hinted at this earlier, but the harnesses are really good at making use of dynamic environments. And what this is leading to is a lot of separation of the brains, which is the harness, uh, from the hands, which is the sandbox or the runtime where stuff is running. And, um, what this means, I have an example here, is the harnesses are going to be able to spin up environments based on what they"
  },
  {
    "startMs": 924700,
    "text": "need right now. They're, they, they're good at snapshotting them and keeping them around for, uh, a-as artifacts that they need later, uh, and again, adjusting all of this as necessary. So I, I think this points at an interesting evolution of the pets to cattle, uh, analogy we're all familiar with. Uh, to recap, i-in the olden days, uh, people thought of services like pets. Uh, each one was bespoke. You had to manage"
  },
  {
    "startMs": 954560,
    "text": "it. Uh, you had to reason about this individual deployment, uh, specifically. Uh, and then over time, it, um, with containerization and Kubernetes, uh, people started to think about it as, um, homogenous, uh, templates, uh, of, uh, these servers are all the same. You can scale them up and down. If one of them fails, it's not an issue because our system can heal from that. And I think this is where a lot of people deploying agents are today. They're kind of using"
  },
  {
    "startMs": 984560,
    "text": "these homogenous environments and scaling up these agents. But, but I think that's really limiting because the models are so good at reasoning about dynamic environments, snapshotting, uh, creating lots of environments when needed, and working across them. Uh, so what, what does the new world look like? One analogy here is it kind of looks like an ocean of microbes, of all these organisms that are self-evolving and working with each other. Uh, and credit Suspix for the"
  },
  {
    "startMs": 1014360,
    "text": "analogy here. Um, but one thing I'm really excited about is as you give agents these really rock-solid primitives for compute and storage and networking, they will be able to actually do a lot more, uh, by doing the stuff I talked about, like spinning up lots of environments, working with each other, and being very dynamic. So yeah, I and us at Modal really think that great primitives let your agents do more."
  },
  {
    "startMs": 1044400,
    "text": "And, uh, yeah, if you're interested in talking about this more, reach out to us. Thank you."
  }
]
