# Stop Giving Your Agent Tools. Give It a Sandbox - Mikyo King, Arize AI

Mikyo King, Arize AI

[Talk page](https://ai.engineer/talks/dAoOYFx7ZYw-stop-giving-your-agent-tools-give-it) · [Video](https://www.youtube.com/watch?v=dAoOYFx7ZYw)

## Stop Giving Your Agent Tools. Give It a Sandbox

Mikyo King explains how Arize Phoenix uses code mode to discover tools, compose operations, and keep intermediate data out of model context—and why the savings come with new security, reliability, and evaluation work.

### An inbox task reveals the value of writing software on demand

[0:12](https://www.youtube.com/watch?v=dAoOYFx7ZYw&t=12s)

Getting every invoice out of an email inbox is a small task with two very different workflows. A programmer can write a script that checks messages for attachments, downloads them, and puts them in a folder—perhaps fifty lines of code in Mikyo King's opening example. Someone restricted to the application's interface has to open messages and download the files manually. The difference is the ability to create a missing operation rather than repeat the operations the interface already offers.

King, who heads the open-source group at Arize AI, uses this example to introduce the team's work on agents inside its observability and evaluation platform. Coding agents such as Claude Code make that scripting capability available through a request: the agent can build a small piece of software for the task. His architectural question follows from that change. If a model can write the operation it needs, why design an agent that can only select from a fixed menu?

### The classic tool loop makes context carry the work

[1:42](https://www.youtube.com/watch?v=dAoOYFx7ZYw&t=102s)

A conventional tool-calling agent starts with a catalog of available tools. A user asks a question; the model chooses a tool; its output enters the context window; the model reasons over that output and chooses another tool. King explicitly acknowledges that this works well for a bounded set of capabilities. The strain appears when the catalog grows and tasks require more data and longer sequences of actions.

Two kinds of material accumulate: descriptions of tools that may be irrelevant to the current task, and results from operations the agent has already performed. Pagination makes the latter especially costly because retrieving a dataset can require repeated calls and repeated additions to context. More turns and more accumulated material eventually force compaction. The model's working context becomes both a place to reason and a transport for intermediate data.

![Repeated context-full diagram with a red-outlined context window filled by tool definitions, a user question, tool calls and results, beside a model and six MCP servers.](https://ai.engineer/api/talk-essay-media/62cb85b8a8be8a4cf118f8d874a0f36d75f914f5c4480ea3641dccfdb302a524)

The CRM, meetings, email, calendar, docs and tasks servers all contribute tool definitions. Each subsequent tool call and result takes up more of the same context window.

### Code mode moves composition and filtering into an execution environment

[3:12](https://www.youtube.com/watch?v=dAoOYFx7ZYw&t=192s)

**Code mode** gives the agent two capabilities: finding the tools relevant to its task and executing them inside a programming environment. The program can call operations in sequence, run independent calls in parallel, and put conditions around subsequent work. Multiple tool operations can therefore happen within one model turn. Before returning, the program can select the part of the result the model actually needs.

Where do intermediate results go? In the classic loop, tool outputs return to model context for the next reasoning step. In code mode, the execution environment can receive and process those outputs first, then return a smaller result. Tools still do work, but each intermediate result no longer has to become model-visible text. Fewer model turns and less returned text are the intended savings, rather than a promise that tool execution itself becomes free.

![A model sends a program to a sandbox that calls meeting, transcript, deal lookup and attachment tools; the attachment line is highlighted.](https://ai.engineer/api/talk-essay-media/038578579b00bfdb08a5e08b4cdb80a3da7bf6af2b66460cc7e07619bcf18993)

The program finds the Acme meeting, gets its transcript and attaches it to the deal. The only value it sends back to the model is “attached.”

King separates three mechanisms that make this architecture useful:

- **Progressive disclosure:** Search for a capability, learn how to use it, and load only the tools needed for the task instead of carrying the whole catalog upfront.
- **Composition:** Use ordinary program control flow to chain dependent operations or fan out independent ones. The model does not have to mediate every individual call.
- **Selective return:** Search, trim, or otherwise process tool outputs in the execution environment, then send only the useful portion back into context.

The fourth mechanism is the one Arize particularly values: creating a missing tool. Consider a request to remove personally identifiable information from a set of records. If the fixed catalog has no redaction operation, the agent either lacks the capability or tries to transform the records in its own context—bringing sensitive values such as credit card numbers into that context along the way. A coding environment lets it write a transformation that runs on the records before returning the needed output. This is a capability the architecture enables, not a guarantee that generated redaction code is correct or that sensitive values cannot escape.

Where intermediate tool results go
The classic loop returns tool output directly to model context. Code mode inserts a programmable execution environment that composes calls and selects what returns to the model. Arrows distinguish execution requests from returned data.
- Model context — Contains tool descriptions and accumulated tool outputs; the model chooses subsequent calls.
- Individual tool calls — Each returned output enters context before the model reasons about another call.
- Model context — Uses relevant tool information and receives the selected result of a program.
- Programming environment — Runs sequences, parallel calls, and conditions; searches or trims intermediate outputs.
- Discovered tools — Perform the operations composed by the program.
- Model context → Individual tool calls (Requests an individual operation)
- Individual tool calls → Model context (Returns tool output into context)
- Model context → Programming environment (Supplies code that composes operations)
- Programming environment → Discovered tools (Executes calls using program control flow)
- Discovered tools → Programming environment (Returns outputs for processing)
- Programming environment → Model context (Returns only the needed result)

### Phoenix adds Bash, Python, SQL, and a programmable UI

[6:11](https://www.youtube.com/watch?v=dAoOYFx7ZYw&t=371s)

The agent and harness inside [Phoenix](https://github.com/Arize-ai/phoenix) began with Bash. Shell utilities such as `sed` and `grep` could be composed with Arize's own CLI, and intermediate outputs could be written to files for later use. Files give the program somewhere to keep working material without requiring every byte to travel through the conversation.

A Python sandbox expanded this approach by exposing Phoenix's REST create, read, update, and delete operations inside the coding environment. Getting traces and manipulating datasets became operations a program could combine. The change was in how the existing capabilities could be used: instead of asking the model to coordinate a separate turn for every operation, code could compose them.

SQL addressed a different limitation. A catalog of record-level operations is an awkward interface for an aggregate question about LLM spans and P95 latency. Retrieving page after page of traces just to compute an aggregate moves too much data through the agent. Allowing the model to write a query lets the database perform the relevant aggregation and selection. King's example uses placeholder latency conditions rather than a concrete query, but the distinction is clear: query the population where it lives instead of fetching the population to answer the question.

The next example asks a more unusual question: how can generated code operate the interface the user is already viewing? Phoenix provides a UI-bound programming interface that runs in a Web Worker with two-way data binding to the browser UI. The agent can change the time range and read from the front-end data store. This connection allows code to produce visible UI changes and access UI data, yielding what King describes as computer-use-like behavior through a programming interface. His delight is part of the point: it is fun to watch it “cook.”

### The sandbox shifts complexity into security and operations

[8:12](https://www.youtube.com/watch?v=dAoOYFx7ZYw&t=492s)

“Code mode is not … for free” is King's transition from capability to responsibility. Executing model-written programs means managing CPU, memory, and how long hosted environments remain available. Network access adds another decision: an agent may need the web, but outbound access exposes the environment to destinations and content the application does not control. Choosing a sandbox does not settle its resource limits or network policy.

Two security questions remain distinct:

- **Secrets:** Code may need credentials to use a service, but those values must not return to model context. Vaulting and the return path both matter; enabling access is only half the problem.
- **Permissions across a program:** An authorization can fit the first five operations and be wrong for the sixth. Composing individually available operations can also create a permission problem. Guardrails therefore have to account for what the composed program does, rather than treating access to the environment as sufficient approval.

The operational failure modes also change:

- **Startup failures:** Sandboxes do not always boot successfully, so the harness must cope with environment failures before useful execution begins.
- **Program failures:** Debugging now means understanding generated code, not just inspecting one tool call. The tool's parameter is effectively a program, and it can fail halfway through its work.
- **Discovery failures:** Removing the upfront catalog creates a need to discover and understand available capabilities. That can require an additional handshake before the agent knows what it can call.

A less obvious cost appears when both an MCP integration and the surrounding agent harness implement code mode. Results from one sandbox may be serialized as JSON, parsed, and then brought into the other coding environment. King calls this the double-sandboxing problem: two layers each provide execution machinery, while data still has to cross between them. Adding code mode independently at every layer can recreate transfer and integration work that the architecture was meant to reduce.

### Longer programs need recovery, artifact handoff, and capable models

[10:43](https://www.youtube.com/watch?v=dAoOYFx7ZYw&t=643s)

Batching more work into a program makes recovery more consequential. If a large generated program breaks halfway through, a sandbox alone does not provide the ability to resume from the failed step. King identifies durability as an open problem rather than a blocker. He also flags determinism and mathematical requirements as reasons to consider the execution language: a language a model writes fluently may still be a poor fit for a particular numerical requirement. The talk does not develop a specific numerical failure or language comparison.

Images expose another return-path problem. Code can manipulate an image or create binary data inside the environment, but serializing the entire image into model context defeats the selective-return design. King suggests some form of signed image handoff so the artifact can leave the sandbox without turning its bytes into conversation content. This is a proposed approach, not a completed handoff protocol described in the talk.

Model choice constrains the economics too. King reports strong code-mode capabilities from frontier models, while warning that a smaller replacement may not be capable enough. Reducing tokens is one source of savings; substituting a cheaper model is another, and the first does not establish that the second will work. Code mode still depends on the model being able to write and manage the required program.

### Harbor trials measure task success alongside token and cost savings

[11:44](https://www.youtube.com/watch?v=dAoOYFx7ZYw&t=704s)

Arize evaluates its own agents using Phoenix and the Harbor Framework. In the described setup, Harbor places agents in sandbox environments, gives them clear task instructions, and turns network access off. Multiple trials and different configurations help distinguish repeatable behavior from a run that happened to work.

Harbor verifiers check whether the task was performed and assign a binary result: one or zero. The team tracks `pass@k` for individual tasks across repeated attempts. That matters because an agent can take different trajectories on the same task; a single successful run says little about how consistently the system reaches the required outcome.

What changed in the pre/post code-mode comparison? King reports **43% fewer tokens** and **31% lower cost**, while latency stayed approximately constant despite sandbox overhead. He describes the work as spanning Claude Code, MCP, and CLI interfaces. The figures apply to Arize's reported Harbor run. In his spoken comparison, King gives no task count, model configurations, or numerical before/after success rate.

![Dashboard comparison showing 43% fewer tokens, 31% lower cost, unchanged reward, and a 0.6% latency increase with code mode.](https://ai.engineer/api/talk-essay-media/3a3d7f7ca74333f102316da5fd4fdcb8c635b36384d510cd2d46d003ad0f4d60)

King reports that PXI used 43% fewer tokens at 31% lower cost in the Harbor comparison. Reward stayed the same and latency rose 0.6%.

### Use trajectories to explain why a new tool makes things worse

[13:14](https://www.youtube.com/watch?v=dAoOYFx7ZYw&t=794s)

The useful output of evaluation is more than a score. Capturing the agent's trajectory alongside the verifier result lets the team inspect how it reached that result. King gives a counterintuitive example: introducing a new tool can initially reduce efficiency. The additional capability changes the agent's behavior, and its instructions may steer it toward a worse path in particular scenarios.

Those scenarios provide something concrete to fix. The team can show a coding agent which parts of the instructions guided behavior in the wrong direction and use the evaluations to guide subsequent changes. The verifier checks the outcome; the trajectory helps explain the route. Together they make it possible to improve a harness based on observed behavior rather than assume that more tools must make it better.

King closes with Arize's reasons for adopting code mode: fewer turns, lower cost, and room for agents to create capabilities and perform longer-horizon tasks. Supporting a user's choice of model remains an ambition alongside the capability limits he has just described. His practical invitation has two parts: give the agent a sandbox, then evaluate what it actually does there.

## Key takeaways

- Code mode combines tool discovery with a programming environment. Programs can compose multiple calls and filter intermediate results before anything returns to model context.
- Phoenix applies the idea through Bash and files, Python-accessible REST operations, SQL aggregation, and a Web Worker programming interface bound to front-end state.
- A sandbox does not settle secrets, permissions, network access, recovery, or artifact transfer. Composed programs and nested execution environments introduce their own failure modes.
- Arize's reported Harbor comparison used 43% fewer tokens and cost 31% less, with approximately unchanged latency. Repeated task verifiers and captured trajectories support evaluating behavior rather than relying on one successful run.
- A new tool can initially make an agent less efficient. Inspect the affected trajectories, identify instructions that steer it poorly, and use task evaluations to guide changes.

## Resources

- [Arize Phoenix repository](https://github.com/Arize-ai/phoenix) — The open-source observability and evaluation platform in which Arize builds the agent and harness discussed in the talk.
- [Phoenix documentation](https://arize.com/docs/phoenix/) — A practical starting point for tracing agent behavior and running evaluations with the platform used in the talk.

## Related talks

- [Decoupling the Brain from the Hands: Claude Managed Agents (Part 1 of 4) — Anthropic](https://ai.engineer/talks/Uv0nAf_7tyU-decoupling-brain-from-hands-claude-managed-agents) — Anthropic’s distinction between the agent’s “brain” and “hands” parallels King’s distinction between model reasoning and sandbox execution.
- [How Many Credentials Should Your AI Agent Have? Zero. — Jim Clark, Docker](https://ai.engineer/talks/ZUZVNKFSmTM-many-credentials-should-your-ai-agent-have) — King’s discussion of secrets raises the question in Jim Clark’s title: how many credentials should an agent have?

## Slides

### Code mode: Why your agent should code

Mikyo King introduces why agents should write code, drawing on Arize AI’s experience building its own agents and embracing code mode.

[0:12](https://www.youtube.com/watch?v=dAoOYFx7ZYw&t=12s) [Arize Phoenix title slide reading Code mode and Why your agent should code, with presenter attribution and Arize branding.](https://assets.ai.engineer/sites/aiecode2025/sha256/c466af88aa5058ff5cce21a539619f759c57a0e1bc18f6c263c28126172be8d5.webp)

Arize Phoenix

Code mode
Why your agent should code

Mikyo King · @mikeldking

arize

Interface text:
OCTOBER 14, 2026
BREAKOUT SESSION
Turing Room

### The rise of the coding agents

An overwhelmed LLM juggles tool blocks; a relaxed one writes code on a laptop. King argues that agents should build the software a task needs dynamically, rather than stay restricted to existing interfaces.

[0:35](https://www.youtube.com/watch?v=dAoOYFx7ZYw&t=35s) [Two illustrated characters contrast LLM plus tools, shown juggling tool blocks, with LLM plus code mode, shown using a laptop.](https://assets.ai.engineer/sites/aiecode2025/sha256/438cb2758aa9b6954e9a5f2e94e698ba631cd18de4f05da815198be7d989d33a.webp)

The rise of the coding agents

LLM + tools
LLM + code mode

OCTOBER 14, 2026
BREAKOUT SESSION
Turing Room

### Too many tools

This section introduces the problem of giving an agent a huge catalog of tools, setting up the discussion of code mode.

[1:51](https://www.youtube.com/watch?v=dAoOYFx7ZYw&t=111s) [Minimal white slide with the large heading Too many tools.](https://assets.ai.engineer/sites/aiecode2025/sha256/7cfcf1aaa9208fda479af02ed6fa42c4a0cf714c35d1ecca4960d6a778790f8b.webp)

Too many tools

OCTOBER 14, 2026
BREAKOUT SESSION
Turing Room

### The too many tools problem — tool call 2

Tool definitions already occupy much of the context window before the model's second tool call. The model connects to six MCP servers; each call and result adds more content to the same window.

[2:01](https://www.youtube.com/watch?v=dAoOYFx7ZYw&t=121s) [Diagram connecting a context window to a model and six MCP server categories. Tool definitions occupy much of the window, followed by a user question, tool call and tool result; the model is labeled tool call 2.](https://assets.ai.engineer/sites/aiecode2025/sha256/cf8decf3bd96921d76b24f3d792162772f1fcf5e4051d6cc2d183d27024ec010.webp)

The too many tools problem

CONTEXT WINDOW
Tool definitions
User question
Tool call
Tool result

Model
tool call 2

MCP SERVERS
CRM
Meetings
Email
Calendar
Docs
Tasks

Interface text:
OCTOBER 14, 2026
BREAKOUT SESSION
Turing Room

### The too many tools problem

Tool definitions already occupy much of the context window. Repeated MCP tool calls and results fill the remaining space: the diagram moves from a Docs call to a red-outlined, full context window.

[2:07](https://www.youtube.com/watch?v=dAoOYFx7ZYw&t=127s) [The context-window diagram contains more tool-call and result blocks. The model is labeled tool call 4, and a dotted connection highlights the Docs MCP server.](https://assets.ai.engineer/sites/aiecode2025/sha256/2909f2752a9edbc458943fe1efda220d5ee842360bc55b90238e1b31e9bdc616.webp)

The too many tools problem

CONTEXT WINDOW
Tool definitions
User question
Tool call
Tool result

Model
tool call 4

MCP SERVERS
CRM
Meetings
Email
Calendar
Docs
Tasks

AI Engineer

[2:18](https://www.youtube.com/watch?v=dAoOYFx7ZYw&t=138s) [Tool definitions, a user question, tool calls and results fill a red-outlined context window beside a model labeled context full and six MCP servers.](https://assets.ai.engineer/sites/aiecode2025/sha256/3f914f60aff483e5e2890f450ee0d803eca78dfe38aa02a18a0917528e537e26.webp)

The too many tools problem

CONTEXT WINDOW
Tool definitions
User question
Tool call
Tool result

Model
context full

MCP SERVERS
CRM
Meetings
Email
Calendar
Docs
Tasks

AI Engineer

### The too many tools problem — context full

Tool definitions occupy much of the context window. Repeated tool calls and results fill the remaining space, leaving the model at capacity. The speaker connects this buildup to context rot, extra turns, and compaction.

[2:44](https://www.youtube.com/watch?v=dAoOYFx7ZYw&t=164s) [Repeated context-full diagram with a red-outlined context window filled by tool definitions, a user question, tool calls and results, beside a model and six MCP servers.](https://assets.ai.engineer/sites/aiecode2025/sha256/62cb85b8a8be8a4cf118f8d874a0f36d75f914f5c4480ea3641dccfdb302a524.webp)

The too many tools problem

CONTEXT WINDOW
Tool definitions
User question
Tool call
Tool result

Model
context full

MCP SERVERS
CRM
Meetings
Email
Calendar
Docs
Tasks

OCTOBER 14, 2026
BREAKOUT SESSION
Turing Room

### The model programs the tools

The model writes a sandbox program that finds today's Acme call, retrieves its transcript, and attaches it to the matching CRM deal. Multiple tools run in one turn, returning only what's needed to the model's context.

[3:09](https://www.youtube.com/watch?v=dAoOYFx7ZYw&t=189s) [A model sends a program to a sandbox that calls meeting, transcript, deal lookup and attachment tools; the attachment line is highlighted.](https://assets.ai.engineer/sites/aiecode2025/sha256/038578579b00bfdb08a5e08b4cdb80a3da7bf6af2b66460cc7e07619bcf18993.webp)

The model programs the tools

USER Attach today's Acme call to the Acme deal in our CRM.

Model
program

SANDBOX
call = find_meeting("Acme", "today")
text = get_transcript(call.id)
deal = find_deal("Acme")
attach(deal.id, text)
return "attached"

TOOLS
find_meeting()
get_transcript()
find_deal()
attach()
search_email()

OCTOBER 14, 2026
BREAKOUT SESSION
Turing Room

### Tool calling vs. code mode — 9-turn illustration

In this hypothetical, code mode runs tools in one program and returns only needed results: 1 turn, 2% context used and $0.02, versus tool calling’s 9 turns, 64% and $0.64.

[3:50](https://www.youtube.com/watch?v=dAoOYFx7ZYw&t=230s) [Illustrative comparison shows tool calling at 9 turns, 64% context used and $0.64, versus code mode at 1 turn, 2% context used and $0.02.](https://assets.ai.engineer/sites/aiecode2025/sha256/dd4f4a66be88c1ec2e4cbd9b55c6ed3555c67904c84ea73ab55afc53e4533402.webp)

Tool calling vs. code mode

Tool calling
Model
Tools
one call at a time
context
9 TURNS
64% CONTEXT USED
$0.64 COST

Code mode
Model
Sandbox
one program
context
1 TURNS
2% CONTEXT USED
$0.02 COST

OCTOBER 14, 2026
BREAKOUT SESSION
Turing Room

### Tool calling vs. code mode — 11-turn illustration

In this hypothetical comparison, one sandbox program runs multiple tools and returns only needed results: 1 turn, 2% context used and $0.02, versus 11 turns, 79% and $0.79 for one-at-a-time tool calling.

[3:56](https://www.youtube.com/watch?v=dAoOYFx7ZYw&t=236s) [Illustrative comparison shows tool calling at 11 turns, 79% context used and $0.79, versus code mode at 1 turn, 2% context used and $0.02.](https://assets.ai.engineer/sites/aiecode2025/sha256/e3201eb1ddf88b67dbd5eb3b7ead3de3b02b534ed0a7d57e32f4a6f137d1ea79.webp)

Tool calling vs. code mode

Tool calling
Model
Tools
one call at a time
context

11 TURNS
79% CONTEXT USED
$0.79 COST

Code mode
Model
Sandbox
one program
context

1 TURNS
2% CONTEXT USED
$0.02 COST

OCTOBER 14, 2026
BREAKOUT SESSION
Turing Room

### Why code mode works

Code mode discovers tools as needed, composes calls with loops and branches, and keeps bulky outputs in a sandbox. The model can also write helpers, such as a PII redactor, instead of bringing sensitive records into context.

[4:34](https://www.youtube.com/watch?v=dAoOYFx7ZYw&t=274s) [Four panels illustrate selective tool discovery, composable control flow, keeping transcripts in a sandbox, and a model-written redaction helper.](https://assets.ai.engineer/sites/aiecode2025/sha256/a2e95ae0491465e28be564da068fc2dfd4e07507fafd9d3efae47cf732f28a36.webp)

Why code mode works

Progressive disclosure
every tool → context

Composability
for m in meetings:                         ← loop
  t = get_transcript(m)                    ← call
  if "pricing" in t:                       ← branch
    attach_to_lead(m, summarize(t))        ← chain

Token efficiency
sandbox · transcripts → context
answer

Emergent behavior
# written by the model
def redact_pii(t):
  return CARD.sub("•", t)

get_transcript
summarize
+ redact_pii

Interface text:
OCTOBER 14, 2026
BREAKOUT SESSION
Turing Room

[5:52](https://www.youtube.com/watch?v=dAoOYFx7ZYw&t=352s) [Four panels illustrate progressive disclosure, composability, token efficiency, and emergent behavior through model-written tools.](https://assets.ai.engineer/sites/aiecode2025/sha256/f76e33b25cad69c651ead71f38c3a53f7a2252ab03acf34d49eb6a328f95d924.webp)

Why code mode works

Progressive disclosure
every tool → context

Composability
for m in meetings:                         ← loop
  t = get_transcript(m)                    ← call
  if "pricing" in t:                       ← branch
    attach_to_lead(m, summarize(t))        ← chain

Token efficiency
sandbox · transcripts → context
answer

Emergent behavior
# written by the model
def redact_pii(t):
  return CARD.sub("•", t)

get_transcript
summarize
+ redact_pii

Interface text:
Engineering the future of AI

### Code mode in PXI

PXI sends code to Phoenix and gets answers back. Bash queries GraphQL and saves results to files; Python composes REST operations; SQL runs database aggregations without paginating through traces.

[6:14](https://www.youtube.com/watch?v=dAoOYFx7ZYw&t=374s) [PXI sends code to a Phoenix server with bash, execute, and executeSql interfaces connected respectively to a GraphQL API, REST API, and database.](https://assets.ai.engineer/sites/aiecode2025/sha256/ab684c9502c720f5a40d40a3e15e6f44e6a1904c29a3d0574f75faff43a6057c.webp)

Code mode in PXI

PXI
code →
← answer

PHOENIX SERVER

bash
gql … > spans.json; grep -c ERROR spans.json
GraphQL API

execute
bad = [r for r in runs if r["error"]][:3]
REST API

executeSql
SELECT name, count(*) FROM spans GROUP BY 1
Database

Interface text:
OCTOBER 14, 2026
BREAKOUT SESSION
Turing Room

### Code mode in the browser — initial state

PXI has searched for browser actions, but yesterday’s failed tool calls are not yet displayed: Phoenix still shows 82 spans over the last 7 days. The demo sets up a ui.* script running in a Web Worker to drive the app.

[7:38](https://www.youtube.com/watch?v=dAoOYFx7ZYw&t=458s) [An initial browser span list appears beside a request for yesterday's failed tool calls and a completed browser-action search. Captions describe a script running in a Web Worker and driving the app.](https://assets.ai.engineer/sites/aiecode2025/sha256/c74c3ef7d9c244acf908bc0a5dd853dcf802c1e0dcbd6e1bfbfcdeda76f96a56.webp)

Code mode in the browser

1 Writes one ui.* script
2 Runs in a Web Worker
3 Drives the real app

OCTOBER 14, 2026
BREAKOUT SESSION
Turing Room

Interface text:
Projects › claude-code
1 Last 7 days
Spans    Traces    Sessions
82 spans
2 parent_id is None

✓ chain    Turn 14    CI is green…
✓ chain    Turn 13    push up the changes
✓ chain    Turn 12    make the user…
✓ chain    Turn 11    what's the difference…
✓ chain    Turn 10    attach screenshots…

PXI in Phoenix
Show me yesterday's failed tool calls
search_browser_actions ✓

### Code mode in the browser — time-range operation

PXI runs a UI-bound script in a Web Worker to find yesterday’s failed tool calls. The demo highlights the time-range command and its matching control; the browser still shows Last 7 days and 82 spans.

[7:41](https://www.youtube.com/watch?v=dAoOYFx7ZYw&t=461s) [The browser panel is outlined as driven by PXI. A Web Worker script highlights setting the time range, while the interface still displays its original range and span list.](https://assets.ai.engineer/sites/aiecode2025/sha256/1ff85301ccea38390cbc3acc62f233e80d3e2e560fd4b40950713d34e55ed2b6.webp)

Code mode in the browser

PXI in Phoenix
Show me yesterday's failed tool calls
search_browser_actions ✓

execute_browser_action
WEB WORKER
1 await ui.timeRange.set(yesterday)
2 await ui.spansFilter.set(
    "span_kind == 'TOOL' and
     status_code == 'ERROR'" )
  return { ok: true }

1 Writes one ui.* script
2 Runs in a Web Worker
3 Drives the real app

OCTOBER 14, 2026
BREAKOUT SESSION
Turing Room

Interface text:
driven by PXI
Projects › claude-code
1 Last 7 days
Spans    Traces    Sessions
82 spans
2 parent_id is None
✓ chain    Turn 14    CI is green…
✓ chain    Turn 13    push up the changes
✓ chain    Turn 12    make the user…
✓ chain    Turn 11    what's the difference…
✓ chain    Turn 10    attach screenshots…

### Code mode in the browser — filter update in progress

PXI runs a UI-bound script in a Web Worker to change Phoenix’s time range and span filter. The highlighted code sets the tool-and-error filter; the interface still shows only the tool condition and 82 spans.

[7:45](https://www.youtube.com/watch?v=dAoOYFx7ZYw&t=465s) [The browser shows an updated date and a highlighted tool-kind filter. The script highlights setting the combined tool-and-error filter, while the list still shows 82 spans.](https://assets.ai.engineer/sites/aiecode2025/sha256/4643db593b4f81482fa5b4e78dab2519d93eb6c79bbb25f6b195f9451508ab47.webp)

Code mode in the browser

driven by PXI

1 Sep 22
82 spans
2 span_kind == 'TOOL'

✓ chain  Turn 14  CI is green…
✓ chain  Turn 13  push up the changes
✓ chain  Turn 12  make the user…
✓ chain  Turn 11  what's the difference…
✓ chain  Turn 10  attach screenshots…

PXI in Phoenix
Show me yesterday's failed tool calls
search_browser_actions ✓

execute_browser_action
WEB WORKER
1 await ui.timeRange.set(yesterday)
2 await ui.spansFilter.set(
    "span_kind == 'TOOL' and
     status_code == 'ERROR'" )
  return { ok: true }

1 Writes one ui.* script
2 Runs in a Web Worker
3 Drives the real app

Interface text:
Projects › claude-code
Spans  Traces  Sessions
OCTOBER 14, 2026
BREAKOUT SESSION
Turing Room

### Code mode in the browser

PXI runs a ui.* script in a Web Worker to change Phoenix’s time range and filter for failed tool calls. The demo ends with 12 matching spans and a confirmation that they are from yesterday.

[7:47](https://www.youtube.com/watch?v=dAoOYFx7ZYw&t=467s) [The browser displays 12 spans matching the tool-and-error filter, with failed tool-call examples visible. Beside it, the Web Worker script highlights its successful return.](https://assets.ai.engineer/sites/aiecode2025/sha256/2890a847fb12e04d66ca04c537b19eabd518307f22ed61eb8dde1c61b972f461.webp)

Code mode in the browser

driven by PXI

span_kind == 'TOOL' and status_code == 'ERROR'

×  tool  Bash      git status → exit 128
×  tool  Bash      gh pr checks → blocked
×  tool  Bash      grep "theme" → exit 1
×  tool  Bash      export PATH → exit 1
×  tool  navigate  localhost → timed out

PXI in Phoenix
Show me yesterday's failed tool calls
search_browser_actions ✓

execute_browser_action
WEB WORKER

① await ui.timeRange.set(yesterday)
② await ui.spansFilter.set(
    "span_kind == 'TOOL' and
     status_code == 'ERROR'" )
  return { ok: true }

1 Writes one ui.* script
2 Runs in a Web Worker
3 Drives the real app

OCTOBER 14, 2026
BREAKOUT SESSION
Turing Room

Interface text:
Projects › claude-code
① Sep 22
Spans  Traces  Sessions
12 spans
②

[7:49](https://www.youtube.com/watch?v=dAoOYFx7ZYw&t=469s) [A browser tracing interface beside a generated script that changes the time range and filters failed tool calls, with a completion message and three workflow captions.](https://assets.ai.engineer/sites/aiecode2025/sha256/39766353e222923e4d27a0d63e99e262a5eaf958d7e43ba11836c0aad8bfee26.webp)

Code mode in the browser

span_kind == 'TOOL' and status_code == 'ERROR'

×  tool  Bash      git status → exit 128
×  tool  Bash      gh pr checks → blocked
×  tool  Bash      grep "theme" → exit 1
×  tool  Bash      export PATH → exit 1
×  tool  navigate  localhost → timed out

PXI in Phoenix
Show me yesterday's failed tool calls
search_browser_actions ✓

execute_browser_action
WEB WORKER

① await ui.timeRange.set(yesterday)
② await ui.spansFilter.set(
    "span_kind == 'TOOL' and
     status_code == 'ERROR'" )
  return { ok: true }

Showing 12 failed tool calls from yesterday.

1 Writes one ui.* script
2 Runs in a Web Worker
3 Drives the real app

OCTOBER 14, 2026
BREAKOUT SESSION
Turing Room

Interface text:
Projects › claude-code
① Sep 22
Spans  Traces  Sessions
12 spans
②

### LLM-generated code

A sandbox still needs resource caps, controlled network egress, secret handling and scoped permissions. The diagrams show an injected README sending data out, keys leaking into logs, and an admin token allowing production deletion.

[8:21](https://www.youtube.com/watch?v=dAoOYFx7ZYw&t=501s) [Four diagrams illustrate resource limits, data leaving a sandbox, secret leakage into logs, and excessive permissions.](https://assets.ai.engineer/sites/aiecode2025/sha256/85fbc86d2e726523014c31d15985566432dd1562d551411e2bb42e5312a96637.webp)

LLM-generated code

Resource limits
cpu
memory
time
killed

Network egress
sandbox
injected README
data
evil.com
data

Secrets in the sandbox
sandbox
API_KEY=[redacted]
print(env)
logs
[redacted]

Over-broad permissions
sandbox
admin token
read logs ✓
delete prod ✓

Interface text:
OCTOBER 14, 2026
BREAKOUT SESSION
Turing Room

### Operational complexity

Sandboxes add startup delay and make failures harder to trace inside code. Agents must discover tool signatures before calling them, while an MCP server can duplicate tool search and code execution already in the harness.

[9:33](https://www.youtube.com/watch?v=dAoOYFx7ZYw&t=573s) [Four panels compare sandbox startup costs, tool-call versus code debugging, tool discovery steps, and duplicated tool search and code execution in a harness and MCP server.](https://assets.ai.engineer/sites/aiecode2025/sha256/501753577af1720941d65aee37b275f235d1ce4c11346f05264ff075963f05b3.webp)

Operational complexity

Startup cost
isolate    ms
microVM    100s of ms
full VM    seconds

Harder to debug
 tool calling
get_spans() ✓
filter() ✓
summarize() ✓

code mode
run_code()
summary: empty

Tool discovery
search    search("traces")
learn     get_traces(project, limit)
call      get_traces("prod")

Duplicating the harness
harness
 tool search
 code exec

MCP server
 tool search
 code exec

Interface text:
Engineering the future of AI

### Open problems

Code mode still has rough edges: failed runs cannot resume midstream, language choice affects determinism, binary outputs are awkward to hand off, and smaller models may struggle where frontier models succeed.

[10:41](https://www.youtube.com/watch?v=dAoOYFx7ZYw&t=641s) [Four panels illustrate durability, determinism, handling images and binary data, and limitations of smaller models.](https://assets.ai.engineer/sites/aiecode2025/sha256/0cafe4715841e96653cd4dafe0ae67199c467898b3d4e3df533dcab2b269f13e.webp)

Open problems

Durability
 today
 snapshot

Determinism
JavaScript    a3f ≠ 9c1
Starlark      a3f = a3f

Images and binary data
 tool returns    what the code gets
 { json }        → r.traces ✓
 image           → "iVBORw0KGgoAAAA…"
 2 GB file       → × too large to pass

Smaller models
large model ✓
small model ×

Interface text:
OCTOBER 14, 2026
BREAKOUT SESSION
Turing Room

### Evaluating code mode

Code-mode agents and baselines run repeated trials in fresh sandboxes. Harbor verifiers score task success as 0 or 1; Phoenix records rewards, token cost, turns, tool calls and full traces.

[11:44](https://www.youtube.com/watch?v=dAoOYFx7ZYw&t=704s) [Code-mode configurations and baselines feed repeated sandboxed agent trials, with a verifier producing rewards and Phoenix recording run metrics and traces.](https://assets.ai.engineer/sites/aiecode2025/sha256/05223234f79064ec424d6b9783cf02bbccf5b47e36185e8f22696171cc02199f.webp)

Evaluating code mode

CODE MODE
PXI
inside Phoenix

Claude Code
MCP · code mode

Codex
MCP · code mode

BASELINES
Claude Code
MCP · one tool each

Claude Code
CLI + skill

Oracle
reference solution

Task
instruction   environment   tests

Trial × k trials
Environment   fresh sandbox
Agent
the surface under test

the run

Verifier
runs tests

Reward
0 to 1

Phoenix
Reward
Tokens · cost
Turns
Tool calls
Full trace

OCTOBER 14, 2026
BREAKOUT SESSION
Turing Room

### Every surface, many trials

Each task gets three trials in PXI code mode and three in Claude Code + MCP. max-llm-calls passes 6/6; root-cause and delegation each pass 5/6. llm-call-failure passes just 3/6, showing why a single successful run is not enough.

[12:37](https://www.youtube.com/watch?v=dAoOYFx7ZYw&t=757s) [Completed trial matrix shows pass rates of 5/6 for root-cause analysis, 6/6 for maximum LLM calls, 5/6 for subagent delegation and 3/6 for LLM-call failure.](https://assets.ai.engineer/sites/aiecode2025/sha256/2362597e56208f9de7e6e6ddeb1effc46461166b6119922399be91c9bdbe8e07.webp)

Every surface, many trials

PXI · code mode
Claude Code + MCP
pass rate

pagedown–root–cause
PXI · code mode: ✓ ✓ ✓
Claude Code + MCP: ✓ × ✓
pass rate: 5/6

max–llm–calls
PXI · code mode: ✓ ✓ ✓
Claude Code + MCP: ✓ ✓ ✓
pass rate: 6/6

subagent–delegation
PXI · code mode: ✓ ✓ ×
Claude Code + MCP: ✓ ✓ ✓
pass rate: 5/6

llm–call–failure
PXI · code mode: ✓ × ✓
Claude Code + MCP: × ✓ ×
pass rate: 3/6

Interface text:
AI Engineer

### Every surface, many trials — later return

Three trials each for PXI code mode and Claude Code + MCP give combined pass rates of 5/6, 6/6 and 5/6 on the first three tasks. LLM-call-failure remains unfinished: three passes, two failures and one pending trial.

[12:49](https://www.youtube.com/watch?v=dAoOYFx7ZYw&t=769s) [Later return to the trial matrix shows mostly resolved outcomes, pass rates of 5/6 and 6/6, a faded 5/6 rate, and one pending trial in the LLM-call-failure row.](https://assets.ai.engineer/sites/aiecode2025/sha256/e1b7d179e0750f3da368230699a537610a80ab3b4ac84790973931763719c729.webp)

Every surface, many trials

PXI · code mode
Claude Code + MCP
pass rate

pagedown-root-cause
PXI · code mode: ✓ ✓ ✓
Claude Code + MCP: ✓ × ✓
pass rate: 5/6

max-llm-calls
PXI · code mode: ✓ ✓ ✓
Claude Code + MCP: ✓ ✓ ✓
pass rate: 6/6

subagent-delegation
PXI · code mode: ✓ ✓ ×
Claude Code + MCP: ✓ ✓ ✓
pass rate: 5/6

llm-call-failure
PXI · code mode: ✓ × ✓
Claude Code + MCP: × ✓ •••

Interface text:
AI Engineer

### What code mode saves PXI

In this Harbor run, PXI’s code mode used 43% fewer tokens and cost 31% less. The dashboard compares runs with and without code mode: reward stays the same, while latency rises 0.6%.

[12:52](https://www.youtube.com/watch?v=dAoOYFx7ZYw&t=772s) [Dashboard comparison showing 43% fewer tokens, 31% lower cost, unchanged reward, and a 0.6% latency increase with code mode.](https://assets.ai.engineer/sites/aiecode2025/sha256/3a3d7f7ca74333f102316da5fd4fdcb8c635b36384d510cd2d46d003ad0f4d60.webp)

What code mode saves PXI

Tokens
−43%

Cost
−31%

Reward
Same

Latency
+0.6%

Engineering the future of AI

Interface text:
experiment
PXI · no code mode
comparisons
1 experiment
PXI · code mode
Grid
List
Metrics

turn_count
Y-axis ticks: 0.00, 1.00, 2.00, 3.00, 4.00, 5.00
X-axis labels: #17, #19
Legend: mean score, base

tool_count
Y-axis ticks: 0.00, 1.00, 2.00, 3.00, 4.00, 5.00
X-axis labels: #17, #19
Legend: mean score, base

Cost
Estimated cost in USD per experiment
Y-axis ticks: $0.00, $2.00, $4.00, $6.00, [illegible]
X-axis labels: #17, #19
Legend: completion, prompt, base

Run latency
Average run latency per experiment
Y-axis ticks: 0ms, 25s, 50s, 1m 15s, 1m 40s
X-axis labels: #17, #19
Legend: average latency, base

reward
Y-axis ticks: 0.00, 0.25, 0.50, 0.75, 1.00
X-axis labels: #17, #19
Legend: mean score, base

filter condition (e.g., evals["Hallucination"].label == 'hallucinated')
Charts

input
reference output
PXI · no code mode
AVG [illegible] 1m 25s [illegible] 130k · $0.17
PXI · code mode
AVG [illegible] 1m 26s ↑0.6% [illegible] 76k ↓43% · $0.11 ↓31%

example pagedown-root-[illegible]
reference output

"task_id": "pagedown-root-cause",
"task_name": "arize/trait-benchmark-pagedown-root-cause",
"instruction": "In the research-assistant project, the recorded spans named PageDownTool are erroring on almost every call. Looking at those spans, what is actually going wrong when the agent calls it?\n"

"reference": "Every failing call raises TypeError: PageDownTool.forward() got an unexpected keyword argument. The agent passes forward() a keyword argument that it does not accept; the argument name varies (most often an empty name [illegible], also page_down, arguments, url, query).",
"notes": "A match gives an unexpected keyword argument to forward() as the cause. Which argument name the reply cites, or whether it cites one at all, is [illegible]"

PXI · no code mode
2m 8s [illegible] 590,386 · $1.48
What's happening
110 PageDownTool spans in research-assistant, 109 of them ERROR — a 99% failure rate. Every one is the same Python TypeError:
TypeError: PageDownTool.forward() got an unexpected keyword argument '[illegible]'
The root cause is an argument-schema mismatch, not a browser or network problem. page_down is a zero-argument tool — its span attributes show tool.parameters: '{}' and the description is just "Scroll the viewport DOWN one [illegible]"
Expand
infra_ok  ok  1.00
reward  1.00
 tool_count  8.00
turn_count  6.00
1.00 AVG
1.00 AVG
6.50 AVG
5.50 AVG

PXI · code mode
2m 44s ↑27% [illegible] 301,350 ↓49% · $0.45 ↓69%
What's happening
PageDownTool in research-assistant: 109 of 110 calls error (99%), spread across 22 traces. Every failure is the same Python exception:
TypeError: PageDownTool.forward() got an unexpected keyword argument '[illegible]'
Root cause: the model is inventing arguments for a zero-argument tool
Expand
infra_ok  ok  1.00
reward  1.00
tool_count  7.00
turn_count  8.00
1.00 AVG  no change
1.00 AVG  no change
7.00 AVG  ↑0.50
8.00 AVG  ↑2.50

example max-llm-calls
reference output

"task_id": "max-llm-calls",
"task_name": "arize/trait-benchmark-max-llm-calls",
"instruction": "What is the largest number of LLM calls any single trace in research-[illegible]"

"reference": "78 LLM calls",
"notes": "An LLM call and an LLM span are the same thing, so '78 LLM spans' matches. The count must be 78.",
"source": "solution/solve.sh against [illegible]"

PXI · no code mode
1m 3s [illegible] 151,574 · $0.13
78 LLM spans — that's the maximum in research-assistant.
Top traces by LLM call count:
Trace | LLM calls
[illegible] | 78

PXI · code mode
1m 4s ↑2.3% [illegible] 80,136 ↓47% · $0.10 ↓22%
78 LLM spans — that's the maximum in research-assistant.
Top traces by LLM call count:
Trace | LLM calls
[illegible] | 78

### The case for code mode

The diagrams frame code mode as fewer turns, lower cost, and room for new behavior. King connects these aims to long-horizon tasks and a practical sequence: sandbox the agent, trace its work, then improve it.

[13:58](https://www.youtube.com/watch?v=dAoOYFx7ZYw&t=838s) [Three illustrated benefits labeled Fewer turns, Cheaper, and Smarter, followed by Sandbox it. Trace it. Improve it.](https://assets.ai.engineer/sites/aiecode2025/sha256/5dc80098faf25445c2126de7be775cd84e127acfea40ae5770c763c616b2c06b.webp)

The case for code mode

Fewer turns
Cheaper
Smarter

Sandbox it. Trace it. Improve it.

OCTOBER 14, 2026
BREAKOUT SESSION
Turing Room

### Give your agent a sandbox. Then evaluate it.

The closing slide pairs agent sandboxing with evaluation as the next step, and directs viewers to the Arize Phoenix repository.

[14:21](https://www.youtube.com/watch?v=dAoOYFx7ZYw&t=861s) [Arize Phoenix closing slide inviting viewers to give their agent a sandbox and then evaluate it, with a public repository reference.](https://assets.ai.engineer/sites/aiecode2025/sha256/bcfa183e373ca9247bf59e9f55ebc8c9a4a04cfc58258e21e9686511cc487356.webp)

Arize Phoenix

Give your agent a sandbox.
Then evaluate it.

github.com/Arize-ai/phoenix

arize

Interface text:
OCTOBER 14, 2026
BREAKOUT SESSION
Turing Room

## Full transcript

[0:12](https://www.youtube.com/watch?v=dAoOYFx7ZYw&t=12s) Thanks for everyone for coming. So my name is Mikyo. Uh, I head up, uh, the open source group at Arize AI, and we work on a fully open source, uh, observability and evaluation, uh, framework, and we also work on our own agents. So today we want-- I wanna kind of walk you through what code mode is and our journey, uh, embracing code mode. So if you look back, uh, in the last thirty years of software development, you could kind of segment

[0:42](https://www.youtube.com/watch?v=dAoOYFx7ZYw&t=42s) software into two different categories. You could think of there's software built for software engineers and software for non-engineers. Take, for example, you have, um, you know, you're asked to get all the invoices out of your email inbox. If you're a programmer, you could write a script, you can go in there, you can dynamically, like see if there's an attachment, download it, put it into a folder. It might be a, you know, fifty lines of coding. You get this tool that you can use, versus if you were a non-technical, you'd have to go into these

[1:11](https://www.youtube.com/watch?v=dAoOYFx7ZYw&t=71s) inboxes, you have to manually download it. You're restricted by the interface of the software, and you had to kind of buy that software. And the thing that's really interesting about, uh, things like Claude Code is that that boundary is completely dissolving, right? Basically, anything you need, the coding agent can actually build that software for you dynamically. And the labs and the foundation model providers know this, right? If you look at all the benchmarks like SWE-bench, that is what these models are being trained to do. And so

[1:42](https://www.youtube.com/watch?v=dAoOYFx7ZYw&t=102s) doesn't it make sense basically to align the agents you're building to be able to kind of build the software you need dynamically to dissolve that boundary? So before I get to, uh, started in code mode, let's look at kind of what an agent looks like. So hopefully you've built this, uh, uh, kind of agent before. You give an agent kind of a huge catalog of tools, and then iteratively a user question comes in, it calls the tools

[2:12](https://www.youtube.com/watch?v=dAoOYFx7ZYw&t=132s) that get pushed in the context window, it reasons about it, and then it calls another tool that get pushed in the context window. And, uh, hopefully most of you have built an agent like this. Uh, certainly if you have a certain set of tools, this can work pretty well. But we're now in that age where we want a lot more. We want, um, much more emergent behavior. We wanna put a lot more context into the agent. We wanna have longer horizons. And so this is where you kind of get this bloat inside of the context. You'll see there's a lot of

[2:42](https://www.youtube.com/watch?v=dAoOYFx7ZYw&t=162s) tools that get added into the context window that might not be relevant to the task. Um, maybe the tool call is not responding with a lot of junk, or maybe it needs to paginate through data. And so you deal with this kind of context rot. You also deal with multi-turn, a lot of turns as a result, and you're gonna have to deal with compaction. So luckily, code mode solves a lot of these things. It's kind of an alternative architecture. Um, instead of, let's say,

[3:12](https://www.youtube.com/watch?v=dAoOYFx7ZYw&t=192s) just giving it a huge index, uh, or, um, a menu of tools, you give it the ability to search for the specific tool, and then you also give it the ability to execute the tool. But you don't give it just the ability to execute the tool, you actually give it a programming environment to execute those tools in different types of flow. So they can do it sequentially, they can do them in parallel, um, can put conditionals around it. Um, and the nice thing about this is you can call multiple tools in one turn, and also

[3:42](https://www.youtube.com/watch?v=dAoOYFx7ZYw&t=222s) return only what's necessary to push back into the context window of the model. And so to kind of summarize it, put tho-- these two things hypotheticals side by side, traditional tool calling, you know, you accumulate context window bloat every time you call a tool. That's obviously gonna, you know, add to latency and cost, and it's also gonna increase, uh, increase turns. So the time your user takes to get to value increases, right? So has a lot of overhead.

[4:12](https://www.youtube.com/watch?v=dAoOYFx7ZYw&t=252s) Um, on, on the flip side, code mode potentially when done well can unlock a lot of abilities because it can call as many tools as it wants in a single turn, and it only has to put in exactly what's necessary into the context window to solve its problem, and you can also get a reduction in turn. So that's sort of the, the goal of, of code mode. And so let's just kind of go over some of the reasons why code mode works. Code mode works because, one,

[4:43](https://www.youtube.com/watch?v=dAoOYFx7ZYw&t=283s) you don't have to give it the giant catalog of tools up front. It can just pick and choose the tools that it needs and understand those tools, and then execute just those. So there's a progressive disclosure of tools. Uh, certainly there's things like tool search now also built into the foundation model providers. There's also the composability. There's a huge amount of benefit to not being able to-- being able to call tools in different sequences, either in sequence or parallel fan-out, chaining them together can create much more complicated and

[5:12](https://www.youtube.com/watch?v=dAoOYFx7ZYw&t=312s) efficient things. If you've heard of obviously Claude's dynamic workflows is built on this composability, but composability of sub-agents. I already mentioned sort of token efficiency. You don't have to spit out all the tools outputs and worry about the tools outputs being shoved into the context window. The model can dynamically kind of trim and search for certain areas and only put what it needs back into its context window. And the fourth one, which is the kind of the-- one of the major reasons we've embraced code mode,

[5:42](https://www.youtube.com/watch?v=dAoOYFx7ZYw&t=342s) is this idea of emergent behavior, being able to build its own tools, right? So for example, traditional-- with traditional tool calling, if I said, for example, like, "Take all these records and remove personally identifiable information," it won't be able to do that because it doesn't have a tool, or it might try to do it inside of its, its context window, but then you're leaking credit card numbers in the context window, and that's not great. So the emergent behavior of being able to create its own tools has a huge amount of, uh, benefit.

[6:11](https://www.youtube.com/watch?v=dAoOYFx7ZYw&t=371s) So here are some ways in which, uh, we've embraced code mode. Uh, so we build our own agent, uh, and harness inside of Phoenix, our open source, uh, our observability platform. And we started with Bash. You know, Bash is a great tool of being able to compose together things like, uh, you know, sed and, uh, grep to be able to, uh, compose together, you know, built-in Linux primitives with also our own CLI and be able to offload those into files for later use.

[6:42](https://www.youtube.com/watch?v=dAoOYFx7ZYw&t=402s) We also added, um, you know, a Python sandbox, and we hoisted all our REST CRUD endpoints. Those are like perform-- getting traces or, um, manipulating data sets, and we put that into this coding environment so that it can compose those create, read, update, delete operations, which unlocks like a whole category of things. And we also added one additional one, which is the ability to write SQL and, you know, text-to-SQL. You've obviously, like, heard,

[7:12](https://www.youtube.com/watch?v=dAoOYFx7ZYw&t=432s) like, that probably many times before, but this also unlocks a lot of capabilities. The reason why is not everything fits into the tool category of just creating, updating, and deleting. Maybe you need to be able to do aggregate queries like, "Tell me all the LLM spans that have P95 latency of X, Y, and Z." You don't wanna have to paginate through all your traces to do that. You want the LLM to be able to pinpoint and find exactly what you're looking for. This is kind of a, a neat and creative way we've also

[7:42](https://www.youtube.com/watch?v=dAoOYFx7ZYw&t=462s) used code mode. So we've actually also created a sort of a programming paradigm in our UI that's bound to our UI. So we've been able to kind of be able to give it the ability to change the time range or be able to read data from the data store inside of the front end. And what's really cool about this is you kind of almost get this computer use, like emergent behavior, because we're able to put that programming paradigm in a Web Worker that has a two-way data binding directly into our browser, and it can kind of do complex

[8:12](https://www.youtube.com/watch?v=dAoOYFx7ZYw&t=492s) UI interactions for you, and it's really pretty fun to, to watch it, uh, cook. So code mode is not, uh, obviously for free. Um, you're dealing with LLM-generated code. If you've been at-- to the, the Anthropic talks, you've certainly probably seen some of these. You're dealing with resources. So if you're working with some hosted sandboxes, you have to worry about CPU, memory, how long those sandboxes stay around. Networking is a huge one. Maybe you want it to be able

[8:42](https://www.youtube.com/watch?v=dAoOYFx7ZYw&t=522s) to go out to the web, but you never know what's out in the web, and so you have to carefully think about network egress. Uh, secrets is a hot topic. I think every sandbox provider is trying to come up with a different vaulting strategy. But let's say you want to use Jev inside of your sandbox, like Py, uh, Durable just shipped. You know, that's a crazy cool idea, uh, to be able to make fast decisions inside of a sandbox, but you have to make sure that those secrets don't come back into the context window.

[9:12](https://www.youtube.com/watch?v=dAoOYFx7ZYw&t=552s) And then the other one is permissions. Uh, there's some very interesting research around this as well, but typically, permissions are overbroad, and so maybe in the first five steps, the permissions are correct, but in the sixth operation it is not. Or maybe in composition, uh, the permissions are not, and so you have to kind of think of guardrailing some of these things if that is necessary. There's also operational complexity. Uh, certainly sandboxes don't boot, uh, all the time perfectly,

[9:42](https://www.youtube.com/watch?v=dAoOYFx7ZYw&t=582s) and so you have to, you know, be able to deal with failures. They're harder to debug because now you're debugging code, and you're not debug-debugging individual tools. And tool parameter accuracy becomes a much bigger issue because the tool parameter now is code. Uh, so what if something fails in the middle? Uh, tool discovery is also still, uh, an issue. Previously, if you did have a tool, the LLM would upfront know it, but now it might have to go through a handshake to be able to understand which tools

[10:12](https://www.youtube.com/watch?v=dAoOYFx7ZYw&t=612s) are in its catalog of things to be able to do. And then this is sort of a, a big one. You know, you, you see code mode being injected into MCP, but then you see a lot more code mode being pushed also into harnesses, and now you have a double sandboxing, uh, problem. So sandboxing, you know, is gonna return stuff out as serialized JSON, and then it needs to be parsed and then put into another code mode, and so there's this, like, double duty that ends up happening that also can be, uh, an operational complexity.

[10:43](https://www.youtube.com/watch?v=dAoOYFx7ZYw&t=643s) These are sort of open problems. Uh, they're not like blockers, but durability is a problem. So let's say you build a giant piece of code that needs to execute, and it breaks in the middle. You can't resume from that step. Um, if determinism and mathematics are important to you, for example, it might actually force you to think about which programming language you're using. So JavaScript is something LLMs are really good at doing, but maybe not great for math. Um, for example,

[11:13](https://www.youtube.com/watch?v=dAoOYFx7ZYw&t=673s) uh, code mode can be a great way to do multimodal and manipulating images or creating bytes, but getting that out of the sandbox is quite tricky, right? Because you don't wanna serialize the whole binary of the whole image that's gonna end up in the context window. So you might have to do some sort of signed image handoff. And then the other one, which is like the frontier models are amazing, uh, at code mode, but if you're trying to do cost-saving, you wanna swap out a smaller model, its capabilities might not just quite, um, get there.

[11:44](https://www.youtube.com/watch?v=dAoOYFx7ZYw&t=704s) So how do we at Arize, uh, evaluate code mode? Um, we use our own-- We dogfood our own platform, which is called Arize Phoenix. Um, and we run our agents through various permutations through something called the Harbor Framework, which basically puts the agents into, uh, sandbox environments and runs them on clear instructions with network access turned off and, uh- We run this multiple times in trials, so we eliminate a lot of the flake, and we build

[12:14](https://www.youtube.com/watch?v=dAoOYFx7ZYw&t=734s) evals in the form of Harbor verifiers to make sure that the agent's doing the correct thing. And then Harbor is sort of rooted in this RL framework. We give it a, a one or a zero to indicate whether or not it performs the task, and we test it across many different permutations. So this is kind of why you wanna test like this, because as I mentioned, you can get a lot of flake. Um, sometimes it might work where you're dealing with an indeterminate system. It might take different trajectories,

[12:44](https://www.youtube.com/watch?v=dAoOYFx7ZYw&t=764s) uh, depending how you do it, and we, uh, track pass@k of each individual task that we think it should pass. So, you know, how does this work? Uh, this is kind of our results from, uh, running this Harbor run. You can see pre, uh, code mode and post code mode. We, you know, decreased the amount of tokens by forty-three percent. The cost went down by thirty-one percent. And you know, there was some overhead with the sandbox, but the latency stayed pretty constant. So we knew that across these

[13:14](https://www.youtube.com/watch?v=dAoOYFx7ZYw&t=794s) multiple surfaces, whether or not it was Claude Code or MCP or the CLI, we had accomplished something that, uh, was definitely better than it was before. How do we use this data? It's really, uh, a huge unlock if you, you both capture the trajectory and the evals, as you've seen with a lot of the other talks. This gives you a very great environment to reinforce and look at the behavior of these agents and be able to tell it how to hill climb. So maybe, you know, first time we introduce a new tool, sometimes actually the

[13:44](https://www.youtube.com/watch?v=dAoOYFx7ZYw&t=824s) efficiency goes down, and so we can tell a coding agent why in these specific scenarios, what parts of the instruction are, are guiding it in the wrong direction, and we're able to hill climb these evals to be able to make our agents better.

[13:59](https://www.youtube.com/watch?v=dAoOYFx7ZYw&t=839s) So if you don't take anything other than this away, hopefully, uh, this will sum it up. We really like code mode at Arize because of the fewer turns you get out of code mode. Uh, we want you to be able to bring any model for us, uh, so we want it to be cheaper, and we're really excited about the emergent behavior and the long-horizon tasks that our agents can do. And so I hope this talk convinces you to give your agent a sandbox and then use one of our tools to evaluate it.

[14:29](https://www.youtube.com/watch?v=dAoOYFx7ZYw&t=869s) And that's it. Thank you.
