Machine transcription with reviewed terminology and names. Some recognition errors may remain. Timestamps refer to the livestream; subtitle downloads are timed to the session clips.
4:42:11Great! Sorry, I'm good. Hey everyone, I'm Akshat, CTO of Modal, and I'm going to talk
4:42:20about really need from their infrastructure.
4:42:25I'm told this audience is very finance adjacent, so I thought I'd talk about what
4:42:33if every cell in a spreadsheet could be backed by an agent?
4:42:39And imagine you're a financial analyst at a fund.
4:42:44you want to run a complex analysis on every company in your portfolio.
4:42:52Normally in the past, this would take days.
4:42:54You'd have to connect every portfolio company,
4:42:58look at their data sources, run some Python for each of them.
4:43:03now obviously with agents, this should be much easier.
4:43:06Well, actually, one of our customers, Hebbia,
4:43:09built something that's exactly this.
4:43:11you can go in right now in their product,
4:43:15you have a spreadsheet,
4:43:18and you can run a query
4:43:20where literally every cell is
4:43:23backed by a different agent.
4:43:26And I wanted to talk about what does building this really look like.
4:43:34Well, first, every agent is backed by a modal sandbox.
4:43:41Sandbox is a secure environment for running untrusted code.
4:43:49It used to be that every sandbox was one-to-one with a container or micro VM.
4:43:56That's actually no longer the case.
4:43:58Today actually, modal has this feature where you can run multiple containers or micro VMs
4:44:05inside a sandbox.
4:44:06You actually need to do that for doing something like this, where in this architecture you
4:44:13run the harness.
4:44:15The harness is the agent loop that does things like running the, calling the model API for
4:44:21inference, running the tool calls, and then looping over it.
4:44:26want to separate that from the place where the tool calls actually run for security and
4:44:31isolation and a bunch of other things.
4:44:34So you have the sandbox that's running multiple containers, including a sidecar.
4:44:40You also want shared storage, because you want your agents to be able to talk to each
4:44:44other.
4:44:45And it's up to you how you stripe it.
4:44:47Like maybe you have every row gets the same volume, and agents in that row in the spreadsheet
4:44:54can collaborate.
4:44:56And you also want to share storage between your main container and the sidecar for some
4:45:01cases.
4:45:02And finally, you need all of this to spin up really, really fast, because your spreadsheet
4:45:07can have lots of cells.
4:45:08If someone wants to query, you want to launch thousands of agents, and then you want to
4:45:12spin them back down when it's over.
4:45:15So before I talk more about that, I just wanted to take a quick aside to talk about what is
4:45:22modal.
4:45:23Modal is a platform of primitives that is built for AI applications.
4:45:29We have built this platform from scratch, built our own file systems, scheduling and
4:45:35the runtime to give you great performance and devx for use cases like inference, including
4:45:42LLM inference, sandboxes, training and functions, running a lot of batch processing and what
4:45:48not.
4:45:49Today in particular I'm going to talk about sandboxes, what this primitive is and how
4:45:54it's evolving.
4:45:54So if you want to build something like this, or even further, and we'll talk more about
4:45:59what that means, what are the primitives you need for that?
4:46:04The first thing is, if you're building an agent, you need a workspace.
4:46:07And I have some examples from some of our current customers.
4:46:11You might have heard of Ramp Inspect.
4:46:13Ramp Inspect is Ramp's internal coding agent that I think at this point pushes almost every
4:46:21single PR at Ramp.
4:46:22If you're building a coding agent, you need your repo, your GitHub repo, your dependencies,
4:46:31you need your dev services, all in the same workspace for each agent.
4:46:38Another example is basis.
4:46:42it looks like is your documents might arrive, your agent is then triggered to work
4:46:48on them, but then it's waiting for a response from the accountant.
4:46:52You don't know how long this could take.
4:46:54It could take many days.
4:46:56When you want to persist the state over time, your usage is going to change.
4:47:02In the second chart, you can see usage for each sandbox.
4:47:07There's definitely some consistent trends, but it's all over the place.
4:47:11If you were provisioning this, it doesn't make sense.
4:47:14and then reliably get them when the sandbox needs them
4:47:17and only pay for them for that duration.
4:47:22The other interesting emerging thing is
4:47:25agents need a lot more cloud science
4:47:28where pre-processed some data on CPUs
4:47:32or spin-up GPUs for running pre-processed some data on CPUs
4:47:37or spin-up GPUs for running a scientific model,
4:47:41it will offload that to a modal sandbox.
4:47:43is a scientific model,
4:47:45it will offload that to a modal sandbox.
4:47:48And we're seeing this show up in RSI
4:47:50about what happens when you give an agent
4:47:53sandboxes to Hillclient.
4:47:57The next emerging thing is
4:47:59agents are collaborating a lot more,
4:48:03and that means that you share a file system
4:48:07with your sub-agent,
4:48:09you share file system.
4:48:11The next emerging thing is agents are collaborating a lot more, and that means that you share
4:48:20a file and files with your human where you're all collaborating with the same thing.
4:48:26This means file systems that are fast, have consistency, and devx you can reason about.
4:48:33Something we're working on with modal volumes.
4:48:36The related thing is we're seeing a lot more self-improvement.
4:48:41will save and update memory, they'll run background tasks, which is now called
4:48:47dreaming, to update this memory over time.
4:48:50Actually this also just looks like files in a file system.
4:48:54But again, you want really great performance from this file system.
4:49:02and the scary thing about agents is they can mess up and they don't really know when they might
4:49:11leak your sensitive data to a german forum or something so you need to have a lot of control
4:49:18over what they can and cannot do and you need things like controlling what ips can even talk
4:49:25to the agent, how do you authenticate it, what domains can it talk to, and also if you're
4:49:32using credentials for talking to services, basically you want to inject them in a way
4:49:37that the agent cannot actually see them and then exfiltrate them somewhere later.
4:49:44And the interesting evolution of this is we've realized that domain-related allow lists are
4:49:50actually not enough.
4:49:52a lot of our users actually want very specific control
4:49:56over even like what paths can I talk to
4:50:00on let's say Datadog or GitHub.
4:50:02Like maybe I want to reason about specific operations
4:50:06that the agent is doing.
4:50:07And going back to sidecars,
4:50:09we've seen this emerging as a pattern
4:50:10where users actually write their own lightweight proxy
4:50:14that has custom rules about what exactly you can do.
4:50:17And Ramp's another example of a company
4:50:21that's been using this successfully.
4:50:25And the final underrated thing is it's really important to have really great observability.
4:50:32Going back to the example of auto research and an agent doing a lot of training, what
4:50:37happens when the training run fails?
4:50:40You want your agent to be able to easily look at the logs of the failure, reason about what
4:50:45happened and do another iteration with a different set of parameters.
4:50:50While training is running, you want to look at the metrics, GPU utilization and make sure
4:50:57it's not stalled.
4:50:58Maybe you want to look at the budget and make sure you're not spending too much on your
4:51:02sandbox.
4:51:04And obviously all of this is really important for humans, but as agents are driving a lot
4:51:09more compute, we're spending a lot of time exposing all of this to the agents themselves
4:51:14so they can reason about all of this.
4:51:17All right.
4:51:18I think we know what we need.
4:51:20Why don't we have Astra build us an agent platform?
4:51:25Well, I don't think it's a good idea.
4:51:27I don't think you should vibe code a sandbox platform.
4:51:30And there's three main reasons for that.
4:51:32Number one is reliability.
4:51:34I think we're all painfully aware how hard it is to get right.
4:51:39Certain websites we use every day keep going down.
4:51:43and the reason for that is even with AI,
4:51:47it's actually really hard to reason about the feedback loop
4:51:50of what happens in the case of this tail event
4:51:53when the system breaks
4:51:54and this is why you need to be very careful in building systems.
4:51:59Security is another one.
4:52:00You could vibe code a sandbox platform
4:52:02but who is going to do all the work
4:52:04to continually reason about the security boundaries
4:52:07and make sure it's defensible
4:52:10in an age when everything is evolving very fast.
4:52:15And finally, scale.
4:52:17We have a blog post about how we scale to a million sandboxes,
4:52:21which involves a lot of very interesting engineering design
4:52:24and reasoning about systems.
4:52:26And this is, again, something that you kind of don't want to leave up
4:52:31to a Vibe-coded platform,
4:52:32because when you get to scale, you want it to work.
4:52:36And on that point, we're actually seeing massive scale
4:52:39for a lot of things I mentioned.
4:52:41So, for example, for the snapshots I mentioned,
4:52:45it's growing quite fast.
4:52:46We're seeing 20 million snapshots a week.
4:52:49We're processing about eight petabytes of data a day
4:52:54for each of these snapshots.
4:52:57And our users are creating almost 2.6 million volumes a week,
4:53:02and all of this is growing super fast.
4:53:05And just to take a customer workload for personal assistant,
4:53:10You need to create 10,000 sandboxes a second.
4:53:14You need to run almost a million sandboxes.
4:53:18And they're not going to last very long, they'll just last for like two or three minutes.
4:53:22But when they come up,
4:53:23you want them to be interactive in less than a second. That's what this workload really demands.
4:53:32And the other interesting thing on scale is you actually cannot get all of the scale from a single cloud platform. The world is actually getting crunched on CPUs as well, not just GPUs.
4:53:43So one thing we spend a lot of time doing is actually aggregating cloud capacity from
4:53:48a bunch of hyperscalers and neoclouds.
4:53:50So our users have access to CPU capacity across the world when they need it, but we cannot
4:53:56rely on one cloud provider to do that.
4:53:59Okay, so where is all this going?
4:54:02I hinted at this earlier, but the harnesses are really good at making use of dynamic environments.
4:54:10and what this is leading to is a lot of separation of the brains,
4:54:14which is the harness, from the hands,
4:54:17which is the sandbox or the runtime where stuff is running.
4:54:22And what this means, I have an example here,
4:54:25is the harnesses are going to be able to spin up environments
4:54:29based on what they need right now.
4:54:32They're good at snapshotting them and keeping them around
4:54:36as artifacts that they need later.
4:54:38And again, adjusting all of this as necessary.
4:54:42So I think this points at an interesting evolution
4:54:45of the pets to cattle analogy we're all familiar with.
4:54:51To recap, in the olden days,
4:54:55people thought of services like pets.
4:54:57Each one was bespoke.
4:54:59You had to manage it.
4:55:00You had to reason about this individual deployment specifically.
4:55:05And then over time, with containerization and Kubernetes, people started to think about
4:55:12it as homogenous templates.
4:55:16These servers are all the same, you can scale them up and down.
4:55:19If one of them fails, it's not an issue because our system can heal from that.
4:55:24And I think this is where a lot of people deploying agents are today.
4:55:29They're kind of using these homogenous environments and scaling up these agents.
4:55:32But I think that's really limiting because the models are so good at reasoning about
4:55:38dynamic environments, snapshotting, creating lots of environments when needed and working
4:55:43across them.
4:55:44So what does the new world look like?
4:55:48analogy here is it kind of looks like an ocean of microbes, of all these organisms
4:55:54that are self-evolving and working with each other.
4:55:58credits this fix for the analogy here.
4:56:01but one thing I'm really excited about
4:56:04is as you give agents these really rock-solid primitives
4:56:08for compute and storage and networking,
4:56:12they will be able to actually do a lot more
4:56:14by doing the stuff I talked about,
4:56:16like spinning up lots of environments,
4:56:18working with each other, and being very dynamic.
4:56:22So yeah, I and us at Modal really think
4:56:25that great primitives let your agents do more,
4:56:30and yeah if you're interested in talking about this more
4:56:32reach out to us
4:56:33thank you
