1,000 Agent Tasks in a Sandbox: What Breaks When LLMs Run Code — Kevin Orellana, AWS

Hello, everyone. I think we can get started.

So a few months ago, I did something that was, um, a bit slightly absurd. I took a coding agent, I pointed it at my team's, uh, cloud sandbox hosting environment, and I said, "Try to break it." And I let that run overnight, and this talk is a bit about what we found.

Uh, my name is Kevin. I work at Amazon Bedrock AgentCore. It's a sandbox hosting environment or platform, and we do more than that. We provide agents identity, memory, um, runtime environment, sandboxes, gateways, anything you wanna use to build and host your agents on, agents on the cloud.

So the way this started was basically with me taking a hard look at how, um, customers use our platform. And increasingly, I think a lot of us in the industry see that, uh, a lot of platform components on the internet are just agents. And increasingly, they're also sandbox hosted. I think today's talks will maybe walk you through all the architectures that can exist in cloud-hosted, uh, sandbox environments. And so this kinda got me thinking, how can I perhaps

emulate, simulate the customer, and also test the service the same way agents use our platform?

So the first thing I think we, we set out to do was create an attraction or interface that allows coding agents as well as agent applications to interact with our service, uh, in a way that's pretty native to them. And today, we know that frontier models know the MCP protocol pretty well. So we spent a day or two just creating an MCP server to expose access to our environment and platform, um, just so it can get started pretty quickly. The second piece was

polishing that. Uh, we spent some time refining the tools that can be made accessible and, uh, smoke testing some runs and, and things like that. So some of the tests we ran just by hand was, uh, for example, write a Python code file and execute it, or take a Python file and, uh, convert it into Java. And then we realized we wanted to just expand the test domains that we, we were gonna set the coding agents to do. So we asked to just plainly write a thousand tests across ten domains, and a

thousand tests here was pretty arbitrary. It was just to cast a wide net and see what we caught back. This took a bit of polishing because we knew that a thousand tests would take a, a few hours and was gonna be a little token heavy, so we, um, did things like credential management, credential management injection, um, what happens when a test fails? Is it a service issue? Is it an MCP layer issue? Uh, those are the kinda things we, we let the agent, um, figure out.

So the architecture is honestly very simple, but it's also very powerful, and I think a lot of us have probably seen this in our day-to-day, uh, development. So we have our coding agent on top left. We have an MCP server that basically wraps our cloud sandbox environment, and all the findings get sent back to us. We store them in a SQLite database just because, as some of you might have seen, um, agents are pretty prone to write markdown files, but that doesn't really scale when you have a thousand tests that you wanna run.

And this allowed us to tap into concurrency. We can have multiple sub-agents running locally and running and executing their tests and storing those results, uh, locally.

Now, a thousand tests, I think, is kind of easy to just say out loud, but really every single dot on the screen kind of represents a, a workflow that a customer would actually run. So we did things like testing the sandbox, uh, life cycle. How reliable is it when you put it under stress? Um, things like file system IO. What happens when you upload a file and download it from the sandbox? Uh, tested performance. We tested some software development life cycles like, uh, for example,

writing Python packages, Java packages, um, and the like.

So to give you a perspective of what's one of these dots could represent, it could be an end-to-end machine learning training, uh, program. It could be generating a PDF report. Uh, it could also just be, um, as simple as opening a webpage. And we re-- quickly realized that our platform, which at the time was hosting two, uh, major applications, one was a code sandbox, the other one was a cloud browser, would have different tests that mattered differently to the sandboxes that they were hosted in. So for example, here in

the browser, we c- ask the agent to create a bunch of, uh, tabs and try to scrape them all and summarize them, maybe take some screenshots, download them, maybe fill out a form, automate a login workflow. But the point here is these sandbox hosted applications, uh, tend to go through different workflows that demand different requirements from the sandbox.

So we ran through the tests, and I wanna talk about just three failures that I think, um, showcase some of the patterns we saw, uh, from these, uh, sandbox hosted applications. So a simple one was we uploaded an image, we downloaded it, and we wanted to see if the same thing that we put up, uh, we got back when we asked to download it. And lo and behold, that was not the case. Uh, we deep dived a bit into it, and we found that the encoding layer at the MCP layer was incorrect. The second one was a bit interesting too, which was

a null byte. It just threw off, off our tests. Um, a null byte can be a zero. It could also be a marker in some coding languages that just mean that-- means that the string is, is, uh, complete. Now, an interesting one that was particular to the sandbox layer is one where we actually wrote files to a specific directory in the sandbox, and when we tried to get it back, it just wouldn't be available. So, so now we started to segregate the patterns, uh, and failures into

two layers. One was at the interface, and the other one was at the sandbox layer. And it's important, I think, to draw a distinction because, uh, when we start to host applications in sandboxes, we, um, they deviate a little bit from the typical API service pattern where you have a long-running service that has-- exposes an API, uh, and you, you know, send a request, you get an output that's reasonably, um, shaped, and you, you kind of know what your backend service does. But obviously in today's world, we, th- we no longer have as many long-running

applications. Rather, we have specialized agents that are, uh, sandboxed in an environment. We expose access to them through a gateway, and the demands that they have on their, let's say, compute environment are gonna differ greatly. We know that-- And, and this is an experience I think a lot of us can relate to, which is you are developing something locally, and you just start to see your agent, uh, either a hero, cloud, Codex, whatever it might be, do funny things. Like, it might, uh, download a, a package you might not have expected, or it might

translate a CSV file to a JSON to PDF for whatever reason. And all of these workflows that your agents want to do are gonna demand, uh, different things from your environment. So things, for example, like how much memory or CPU your sandbox has, the permissions that are available to your processes and file systems. I think in a lot of the talks we'll see today, there are a lot of different architectures that your sandboxes can have, and the wider point here, I think, is all of these constraints have pretty massive implications for your

agent's ability to solve these problems.

And just to kind of visualize that, these are some of the faults that we saw when we compared two different applications hosted in sandboxes. So at the top, we have the code sandbox, and the bottom, we have the c- uh, cloud browser. At the bottom right, we have the failure modes that are highlighted in red, which signify how many times the sandbox actually, uh, perished. In this case, because cloud browsers, we know that whenever we open up tabs, they're pretty memory heavy, and so a lot of our tests failed just due to memory constraints. And that wasn't really a

fault we saw in the, uh, code interpreter environment. And when we map them out a little bit differently, we start to see that the failure modes coalesce around sandbox primitives. So we know some of the components around file systems tend to be file system permissions, uh, mounted file systems, uh, permissions to read, write, execute files, um, whether or not the file system, uh, is persistent or, uh, ephemeral. Um, and the tests, uh, were pretty unforgiving. Um,

for example, in the bottom right, we have, uh, the browser tests just starving the, the, the sessions and sandboxes out of memory and signaling that as a complete failure. And I think the wider point here is, as we host these applications in sandboxes, and we expect our customers to be able to build, uh, pretty arbitrary workloads in these sandboxes, we have to be pretty careful that the sandbox constraints have massive implications as to those agents' ability to complete those problems.

Now, if we look a little bit closely, and this slide is pretty heavy, on the second column, we have a lot of the primitives used to compose what a sandbox container, um, might look like. So we can go as deep as kernel capabilities, things like kernel interceptors, um, these things called extended VP- VPF programs, which allow you to run kernel modules and intercept programs at the, at the Python layer. Um, those might, uh, control how your agent actually can make some

egress calls, maybe make a internal API call to, um, your internal company services.

If we diagram this out, this is kind of what a, a very high-level overview of how the components in the sandbox might, uh, affect your, um, workload's ability to do its job. So here, for example, I think earlier in our talks we spoke about, uh, security, secure compute, which controls the system control programs that your applications might want to run. So for example, say I have a machine learning package that needs to spawn a process. Does my process or container even have the ability to start a new process?

Um, perhaps I have a code interpreter, uh, task that needs to download a Python repor-- a Python package into a very specific module for, uh, the Python kernel to be able to pick it up. Or are the Rust binaries in the right location for when the coding agent is able to, um, run or execute a program? All these very, very small things tend to add up into your agent's ability to, uh, execute th-the business problem.

Now, just taking a step back, I think this was kind of interesting, uh, which was the idea of, like, using an LLM to generate basically unit tests. I think we've all done it. We-- I don't know if anyone writes unit tests by hand anymore. Uh, and similarly, if we go up a s- up a layer, integration tests, uh, we, we, we can anticipate agents to be pretty good at understanding the problem space and writing tests for them. But as we move towards hosting more and more applications, which are just agents, uh, in

sandboxes, we start to see the cracks in the model, which is, um, these very low-level primitives and controls, um, might actually hamper your agent's ability to do its stated goal. Um, and what we were hoping to relay here is a, a very strong way to actually surface these failure modes that are m- that are pretty innocuous to us is just through fuzzy testing your agent sandbox, asking it to complete, perhaps, in this case, a thousand workflows and seeing what

results you get.

And just to also make it a little bit more tangible, um, a lot of us here, developers, we are very used to running coding agents locally. You might have your, you know, your, your Codex or your Claude running locally, and- As we expand and create more workspaces, you quickly put a constraint on your laptop and now, you know, your teams wanna put these coding agents in remote sandboxes so you can scale how many GitHub PRs you, you work through and the like. But as you've probably seen, the tools that you have in your local workstation, your

MCP tools, your locally running applications might not match what you have in a remote sandbox.

And just a small shout-out to, uh, AgentCore. This is actually one of the problems that we're very keen on because we recognize that customers want to, uh, scale their coding agents, um, and how customers are actually running these, uh, software development life cycles. So one of the releases we recently did was grant the ability to customers to SSH into their, um, sandbox just to see what's going on. And of course, it's not really a human who is SSH-ing into your sandbox anymore, it's a coding agent who you're developing with, uh, as well. But

the idea is there. Uh, bridging into the sandbox and poking around and seeing what problems are causing your coding agent or the application to not run as smoothly is a pretty, um, stated business problem for us, and one that we're trying to solve for customers.

And if you look a little bit, a bit, uh, if you look a little bit at academia, uh, one interesting example that I think a lot of us probably care about is when- whenever frontier models come out, the first thing we look at is the SWE benchmark report, which is, okay, this model got XYZ percentage on this SWE benchmark versus another. And there was a recent study made by some researchers at UC Berkeley where they found that given the right environment controls and an exploit agent, they actually beat the SWE benchmarks, like reaching up to a hundred percent. And

in- inversely, uh, there was another study where, um, researchers were trying to give a wide set of business problems to an agent with broken tools. Uh, and of course, the effectiveness of that agent, uh, went down greatly.

So I think this is, uh, mostly where I want to aim to leave today, which is as these applications that we build today are more and more agentic and they're not constrained to a, a REST API service that you, you build for customers, but rather, uh, an arbitrary agent running inside a sandbox, it's really, really, um, uh, challenging to find all the sharp edges where the sandbox primitives, uh, lead to, you know, a less efficient, uh, ability for your agent to complete that

problem. So my challenge here for you today is, uh, if you're looking to host your agents on sandboxes, sure, test it locally, but also fuzzy test it with a wide set of, uh, domain tests, um, remotely.

Yeah. Thank you.