AI Engineer World's Fair 2024
How to add secure code interpreting in your AI app
Loading slides…
Read the talk
How to add secure code interpreting in your AI app
Connect model tool calls to a persistent Linux sandbox, return notebook outputs to a Next.js app, and understand the isolation and lifecycle behind the preview.
From a talk by Vasek Mlejnsky
Before you start: Familiarity with TypeScript, React and Next.js request handlers will help; following the code-along also requires E2B and Anthropic API credentials.
A conversation that produces a working preview
How do you turn a model’s generated code into a result the user can see beside the conversation? Claude Artifacts provides the opening reference: chat on the left, a preview on the right. Vasek Mlejnsky describes its newly released web previews in terms of HTML, JavaScript and CSS, then sets a different execution goal for this application: run generated Python. That comparison concerns executable previews, rather than the full range of content an Artifact can contain.
Mlejnsky, an E2B co-founder, uses E2B to supply the execution environment. Claude 3.5 Sonnet generates code through a tool call; the application runs that code in a remote sandbox with the capabilities of a Linux or Ubuntu machine. The opening demonstration shows a Monte Carlo plot beside the chat. Multiple sandboxes can support separate execution sessions, and the underlying runtime can accommodate languages beyond Python.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Start with the scaffold
The original starting point is the E2B Cookbook, checked out at workshop-start. Running the example requires an E2B API key, obtained through the E2B documentation, and an Anthropic API key. Mlejnsky also distributes a temporary Anthropic credential for attendees, explicitly planning to revoke it after the workshop.
The first obstacle is downloading the repository. Mlejnsky suspects its generated images account for the size. While attendees wait, he clarifies a crucial boundary: the application runs locally, but the generated Python does not. E2B supplies the remote runtime. Self-hosting across major cloud providers is discussed as forthcoming, with the security explanation deferred until the application works.
The intended example is Power Artifacts under the cookbook’s examples directory. To recover the code-along, Mlejnsky creates a smaller personal workshop-repo containing just the relevant application. It initially contains only a README, then receives the source files; it does not carry the original repository’s checkpoint branches.
With the source available, the local setup is straightforward:
- Install the project dependencies with
npm install. - Create
.env.localwithE2B_API_KEYandANTHROPIC_API_KEY. - Run
npm run devand open the application on port3000.
The workshop’s temporary credential is shared through 1Password, but a reader needs their own credentials. The initial Next.js page contains a chat input. Asking it to generate a random chart produces HTTP 405: the frontend exists, but the POST endpoint has not been implemented.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make conversation work before adding execution
Two SDKs divide the work. Vercel AI SDK connects the application to the model and streams responses from backend to frontend. The E2B Code Interpreter SDK supplies a configurable execution environment: a Jupyter server inside a small cloud VM, returning standard output, error output and rich results such as charts. Mlejnsky mentions Python, JavaScript and R, with Java in beta. Notebook-style Python is the default because it matches the data-analysis code users commonly received from GPT-4 and GPT-4 Turbo.
The Code Interpreter SDK is a specialized wrapper around E2B’s general sandbox SDK. Creating an interpreter creates a sandbox; lower-level access supports more customized uses of the Linux environment. Mlejnsky initially describes startup broadly as a few hundred milliseconds, then gives a more specific estimate when implementing creation. The examples below retain the workshop’s historical API family: CodeInterpreter, notebook.execCell, and the AI SDK streaming helpers shown in the recording. Current E2B examples instead use Sandbox.create and runCode; these interfaces should not be mixed.
The first implementation belongs in route.ts. Its POST handler parses messages from the request, passes them to streamText with the Anthropic model, and includes a system prompt imported from lib/prompt.ts. The prompt describes a skilled Python developer capable of producing data visualizations. Returning result.toAIStreamResponse() completes this first path:
typescript
import { anthropic } from '@ai-sdk/anthropic';
import { streamText } from 'ai';
import { prompt } from '@/lib/prompt';
export async function POST(request: Request) {
const { messages } = await request.json();
const result = await streamText({
model: anthropic('claude-3-5-sonnet-20240620'),
system: prompt,
messages,
});
return result.toAIStreamResponse();
}
A greeting now receives an assistant response. This verifies conversation through the endpoint; Python execution has not been added yet.
The original cookbook has intermediate workshop checkpoints and a workshop-final branch. On the frontend, this is a Next.js 13 application with a layout and a main page.tsx. Its existing useChat hook targets /api/chat, sends messages, receives the stream and passes the resulting message list into the chat-bubble component. The first checkpoint, described as workshop-one, completes that connection.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Give the model a Python execution tool
A tool call turns generated code into a structured request the application can handle. The workshop uses runPython, but the same approach could expose tools for JavaScript, Bash commands or file creation. Mlejnsky also describes users who skip tool calling and parse code from model-generated Markdown because they prefer its streaming and orchestration behavior. For this application, a tool is the simplest connection.
At the workshop-two stage, the AI SDK’s tool helper declares a Zod schema with three strings:
| Argument | Purpose |
|---|---|
code | Python source to execute |
title | Short label for the artifact card |
description | Longer explanation for the UI |
The description tells the model what the tool does; the parameter schema describes the shape of its arguments. Mlejnsky mentions Instructor as a Python-side analogy for structured output. Declaring the tool makes execution requestable; it does not itself run the code.
The tool’s asynchronous execute callback supplies the actual behavior. The SDK calls it when the model chooses runPython, passing the declared arguments. A separate helper in lib/sandbox imports CodeInterpreter and divides execution into two operations: acquire a sandbox, then run Python inside it. The runPython key in the tools object and the imported runPython function happen to share a name; the key does not invoke the function automatically. The callback must explicitly call the helper.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Reconnect to preserve notebook state
Creating a sandbox starts the remote VM. Mlejnsky reports approximately 800–900 ms to create a cloud sandbox at the time of the workshop. He corrects an initial Python-style construction to the JavaScript call await CodeInterpreter.create(). A typical isolation unit is one sandbox per user session. Creating a new sandbox for every POST request would discard the execution context of earlier snippets.
The helper therefore follows a create-or-connect procedure:
- List the running sandboxes with
CodeInterpreter.list(). - Find one whose optional
metadata.userIdmatches the session’s identity. - If none exists, create a sandbox with that metadata.
- Otherwise, reconnect using its
sandboxId.
typescript
import { CodeInterpreter } from '@e2b/code-interpreter';
export async function createOrConnect(userId: string) {
const sandboxes = await CodeInterpreter.list();
const existing = sandboxes.find(
sandbox => sandbox.metadata?.userId === userId,
);
if (!existing) {
return CodeInterpreter.create({ metadata: { userId } });
}
return CodeInterpreter.reconnect(existing.sandboxId);
}
The metadata is a lookup label for the session. The workshop has one user; a real application needs to connect that lookup to an authenticated identity.
With acquisition separated out, execution is small:
typescript
export async function runPython(code: string, userId: string) {
const sandbox = await createOrConnect(userId);
const execution = await sandbox.notebook.execCell(code);
return execution;
}
execCell defaults to Python and accepts generated, handwritten or predefined source. Because the notebook session remains alive, later cells can refer to variables and functions defined in earlier cells. Mlejnsky also mentions work on connecting directly to the hosted notebook. The returned execution object contains logs, structured runtime errors with tracebacks, and rich outputs such as charts, PDFs and images. A traceback can be sent back to the model for a repair attempt.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keep logs, errors and rich results separate
The POST handler can now call the imported helper from execute, passing both generated code and the user ID. In the workshop, that ID arrives from the frontend; the real-app expectation is an authenticated session. The value returned by execute becomes the tool result delivered to the browser alongside the assistant message.
The response preserves four different channels:
| Channel | What it contains |
|---|---|
| Standard output | Printed output from the program |
| Error output | Text written to standard error |
| Runtime error | Structured execution failure and traceback |
| Cell results | Evaluated values and rich display output |
The distinction matters because a printed string is not a chart, and text written to standard error is not necessarily a structured runtime exception. cellResults is an array: it can include the cell’s final evaluated expression and explicit display calls, with representations such as PNG, JPEG, PDF, HTML, JSON or text. These channels are returned together from the tool callback.
The demonstrated environment includes Python 3.10 with NumPy, pandas and seaborn already installed. A custom Dockerfile can define different packages through the more general sandbox interface. The default is chosen to reduce the mismatch between code a model generates and the environment where that code runs: a notebook with familiar data-science dependencies resembles what GPT-4 expects from a code interpreter.
The notebook is also only one possible interface to the VM. Mlejnsky describes exposing it directly through then-undocumented, experimental work, with setup guidance available from the team. Other experiments provide a full Linux graphical interface that can launch applications and games. These support graphical evaluations and workflows where a person and an agent operate in the same environment.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Verify execution before building the preview
In page.tsx, the workshop supplies a dummy user ID through the body option of useChat and logs messages in Chrome DevTools. Asking the assistant to print hello world produces an assistant message with a toolInvocations array. Repeated console entries reflect the message state updating as the response streams. Within the invocation, args contains the model’s tool arguments and result contains the object returned by execute.
The hello-world result makes the channel distinction concrete: standard output contains the greeting, while cellResults is empty. Printing has produced a side effect, not a rich value to display. The next request asks the model to calculate π using Monte Carlo sampling and visualize it. In the AI SDK setup used here, Mlejnsky says the tool’s generated code is not streamed while its arguments are being produced.
When execution finishes, the Monte Carlo invocation contains standard output plus a cell result with png and text fields. The PNG is a base64-encoded visualization; the text is its accompanying representation. An attendee interprets the absent chart as continued processing, but the output is already in the browser’s message object. Execution completion and visible presentation are separate states. The missing work is now frontend rendering, not another sandbox call.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make tool progress visible
The preview begins by finding the most recent message with a nonempty toolInvocations array. Mlejnsky deliberately keeps selection simple: reverse the messages, find the matching message, then extract an invocation. A copy avoids mutating the original message array:
typescript
const latestToolMessage = [...messages]
.reverse()
.find(message => (message.toolInvocations?.length ?? 0) > 0);
const toolInvocation = latestToolMessage?.toolInvocations?.[0];
Both values can be absent while generation starts. The existing SideView component receives the invocation and is placed beside the chat. On the first attempt it still does not appear.
The missing input is data. SideView is built to return null without that extra information, which is intended to tell the frontend whether the tool is running and whether to show a loader. The server therefore creates a StreamData object and appends a running status before execution and a complete status afterward. Those labels are application-defined, not model-generated explanations.
The additional stream must be included in the response and closed when generation finishes. In the workshop API, the end of the handler has this shape:
typescript
const stream = result.toAIStream({
onFinal: async () => {
await data.close();
},
});
return new StreamingTextResponse(stream, {}, data);
Here data is the StreamData instance created earlier. StreamingTextResponse carries the model stream and the extra data together, while onFinal closes the additional stream. This completes the status transport at the workshop-four checkpoint.
On the client, useChat exposes the additional stream as data; passing it into SideView makes the existing component appear. The side panel now displays generated Python under its Code tab. The preview still needs its image and log rendering. Asked why this application uses Claude, Mlejnsky says the model can be replaced: he chose Sonnet for its ability to produce relatively advanced code examples in one attempt, while explicitly stopping short of claiming it can generate full projects that way.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A Dockerfile defines the environment; a VM runs it
Before finishing the preview, the discussion turns to why the runtime is Linux. E2B’s sandbox is a VM, and both its host and guest use Linux. The underlying virtual machine monitor is Firecracker, the AWS open source project also used for Lambda. Its Linux and kernel requirements determine the runtime constraints described here.
Custom dependencies do not change that isolation boundary. A Dockerfile can install Python packages with pip, install Node packages with npm, or set up another language; Mlejnsky cites a customer running Fortran. E2B uses the container build as a convenient way to define the environment, then extracts its filesystem and converts it for a Firecracker VM. The resulting sandbox is not a nested Docker container providing the security boundary.
Mlejnsky reports approximately 900 ms for both custom and premade sandbox creation, with approximately 400 ms presented as a future target. These are workshop-era estimates; hardware, region, network inclusion and latency percentiles are not specified.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Render the chart, then separate data from presentation
At workshop-five, the final implementation step is to enable the existing ArtifactView inside SideView. It receives the execution result and extracts cell results, standard output, error output and the runtime error. The workshop focuses on the image path: when cell results exist, it assumes a PNG is available, an assumption Mlejnsky says needs more thorough checking in a real application.
The PNG’s base64 content becomes the image source for a Next.js Image component. The demonstration uses an arbitrary width of 600 and height of 400, centers it in a padded container, and retains the prepared log output. A request for a random or 3D chart now produces a visible image beside the conversation, directly from Jupyter running in the sandbox. It is a static image; interactive charts are outside the implemented workshop.
For an interactive chart, change what crosses the backend/frontend boundary. Upload a CSV or connect storage containing the dataset, tell the model what columns exist, and let generated Python extract or transform the data in the sandbox. Return that data to the frontend and render it with Chart.js, Plotly or another charting library. The sandbox then performs computation while the browser owns interaction.
Table previews follow the same output contract. A dataframe can appear in cellResults; whether the browser receives an image, HTML or another representation depends on the generated code and its libraries. Mlejnsky excludes interactive Jupyter widgets from the output support he describes. A rich notebook result is therefore a representation the frontend must handle, not automatically a complete interactive application.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Separate guest freedom from host protection
The security discussion starts with a permissive guest: Mlejnsky says users receive root access inside the sandbox. He describes certain exit or system-restore actions as causing restart or replacement, while also acknowledging possible network holes. That account should not be treated as a general VM-escape defense. The architectural boundary is virtualization plus host-side containment and network configuration, not a promise that every escape attempt simply restarts the guest.
Firecracker uses Linux KVM for virtualization. Mlejnsky describes additional confinement through a jailer and syscall restrictions, and a separate VM for each user session. This introduces more overhead than treating each sandbox as a container process, but supports the sandbox-as-tenant model. The Firecracker design also makes the host’s role explicit: network isolation depends on host-managed networking rather than following automatically from guest separation.
The lifecycle boundary matters just as much as the isolation boundary. In Mlejnsky’s account of the service at the time, killing or automatically closing a sandbox destroys its contents. He says E2B logs operations such as file creation, but does not retain file contents. For persistence beyond the sandbox lifetime, the application can attach its own cloud storage. Reconnecting to a running notebook preserves a live session; it is not the same as durable storage after that session ends.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Build the filesystem your agent needs
The custom-environment workflow becomes more concrete in the closing questions. The workshop’s Dockerfile must be Ubuntu-based and can use most ordinary directives to install packages, define environment variables and preload files. One example is a scaffolded Next.js application: the model generates components, and the application running inside the VM renders them. The template supplies a ready workspace instead of requiring the agent to reconstruct it on every session.
The conversion is principally filesystem preparation:
- Build and run the container that defines the environment.
- Export its filesystem into a file outside the container.
- Convert that file to a root-filesystem format Firecracker can use.
- Start sandboxes from the resulting environment.
Mlejnsky also describes extracting an image’s TAR layers directly as a theoretical alternative to running the container. He points to the open source E2B Infrastructure repository for conversion scripts, while warning that self-hosting is not yet easy. For private modules, he says the build uses the developer’s Docker instance, allowing access to private packages and images.
Deployment support is still a roadmap in this recording. An attendee finds GCP support in the repository; Mlejnsky says E2B’s hosted service is moving toward AWS. His planned sequence is AWS support, a proper GCP tutorial, then support for arbitrary Linux machines and on-premises clouds.
The commercial direction is open core: most functionality is open source, with self-managed infrastructure or E2B’s hosted service as alternatives. Enterprise needs include loading very large datasets and observing an agent’s increasingly persistent workspace. Mlejnsky proposes programmatic visibility into file creation and network requests, plus the ability to stop unwanted actions before they happen. Some capabilities may be licensed; his delivery estimate is summer 2024 (‘this summer’ at the workshop), rather than a feature demonstrated in the application.
GPU sandboxes are outside the service’s scope at this point. Mlejnsky frames that as an intentional choice not to become a GPU provider and suggests offloading substantial GPU work to specialized providers. Future support is possible, but it is not the current focus.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From a running session to a restorable agent workspace
For external storage, Mlejnsky describes support for S3-compatible interfaces, naming Amazon S3, Google Cloud Storage and Cloudflare R2. Setup is more involved than a single SDK call in the workshop; simpler endpoint mounting is proposed for a future version. The mechanism uses FUSE to make remote objects appear within the sandbox filesystem. File reads and writes become network requests. His privacy framing is that the storage remains under the user’s control, but this does not mean bytes never travel into the sandbox for computation.
Why use VMs rather than containers for all of this? Mlejnsky estimates that 90% of customers arriving at E2B have an in-house solution based on serverless functions or container fleets. This is his customer-experience estimate, without a stated sample or methodology. He acknowledges that containers can be secured with more work, then adds an ergonomics argument: tasks such as running Docker inside Docker create additional friction. A quickly available VM gives the agent a general Linux computer.
The final reason reaches beyond the implemented code interpreter: snapshots can make an agent’s execution state revisitable. Mlejnsky explicitly says this is not yet implemented in E2B. The proposed snapshot captures memory as well as the filesystem, preserving more than saved source files. Mlejnsky tentatively estimates approximately 80 ms for VM snapshot creation, without specifying memory size, storage conditions or timing boundaries.
That capability suggests a tree-search workflow. Each node stores a VM snapshot; an agent explores one branch, restores an earlier state, and tries another. Restoring the environment does not make the model deterministic, but it gives subsequent attempts a controlled starting point and a way to return before an unwanted action. Mlejnsky closes by naming CRIU as a container-checkpointing alternative, which he characterizes as incompletely working at the time. The larger design goal is a workspace whose state can be saved, inspected and restored as the agent explores.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Example applications and guides for building with E2B.
Python and JavaScript/TypeScript SDKs for executing generated code in E2B sandboxes.
Infrastructure source and self-hosting guidance for the E2B platform.
The KVM-based microVM monitor underlying the workshop's isolation architecture.
Anthropic's announcement introduces Sonnet 3.5 and the Artifacts preview beside the conversation.
Further reading
An earlier E2B tutorial illustrating the CodeInterpreter and notebook.execCell API used in the workshop.
- Firecracker architecture and host integrationDocumentation
Explains Linux guests, the jailer, host networking and virtual block storage.
Updates since the talk
Current setup and sandbox-management documentation, with SDK examples and template concepts.
Full transcript
Loading transcript…