How to Test Your Agents on Real Tasks With Harbor - Xander Song, Arize AI

Loading slides…

Read the talk

How to Test Your Agents on Real Tasks With Harbor

Xander Song explains how isolated environments, outcome-based verifiers and reference solutions make agent changes testable, then follows a Phoenix prompt change that reduced tool calls while preserving passing results.

From a talk by Xander Song

At a glance

Ideas worth remembering

  • A Harbor task combines instructions, a seeded environment and verifier scripts. Independent trials start fresh; a job collects their outcomes and optional trajectories.

  • Run an oracle solution before spending tokens on agents to check that the environment, starting data and verifiers work together.

  • Outcome-based verifiers tolerate implementation changes. Trajectory-aware checks are useful for tool selection and skill triggering, but create additional maintenance coupling.

  • The SQL prompt experiment increased aggregate SQL use and reduced tool calls and turns while preserving passing rewards. Task-level comparisons and traces explained where behavior changed; larger-data benefits were not measured.

  • Choose the evaluation unit explicitly. Phoenix tests agent-plus-product integrations against seeded data, not the customer application or its production ingestion path; realistic fixtures determine which failures those tests can expose.

Hold the task constant while the agent changes

Changing an agent’s prompt, tools or harness creates a practical problem: how do you know whether the new version improved anything? Xander Song, an engineer at Arize AI working on Phoenix, introduces Harbor as a way to answer that question with realistic tasks and observable results. He approaches it as a Harbor power user rather than a maintainer, with a goal he calls shipping on “evidence rather than vibes.”

Harbor gives an agent natural-language instructions inside a fresh sandbox containing relevant data. After the agent finishes, task-specific scripts called verifiers inspect what it left behind. A verifier produces a reward—for example, 0 or 1 for failure or success. The test can therefore ask whether a database changed correctly or a file was repaired, rather than whether the agent’s final message sounds convincing.

The useful comparison holds the task, starting data and success criteria constant while changing the agent implementation. Run both versions through the same setup, and the rewards provide a signal of improvement or regression. Collecting many tasks extends that signal across the capabilities you care about. Its value still depends on two choices: whether the tasks resemble real work, and whether the verifiers actually describe success.

0:120:43
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:12 · section reference included

Tasks define the work; trials start fresh

A Harbor task packages three distinct inputs:

  • Instructions: A Markdown file contains the request the agent should receive.
  • Environment: Typically described by a Dockerfile, this supplies the software and starting data. The data might be a SQLite file or a Git repository checked out at a particular commit; Harbor does not prescribe one data type.
  • Verifiers: Scripts in a test directory inspect the environment after the agent runs and determine whether the intended effects occurred.

A collection of tasks is a dataset. For each task, Harbor can run a configurable number of trials, because an agent may behave differently on repeated attempts. Each trial gets a new container seeded with the task’s data, receives the instruction file and runs independently. The agent may edit source files, alter a database or make other changes. Once it stops, the verifier runs against that resulting environment and writes rewards to reward.txt.

Rewards tell you whether the run met the test. Optional ATIF trajectories help explain how it got there. ATIF stands for Agent Trajectory Interchange Format, Harbor’s JSON-based tracing specification for recording the agent’s internal steps. A job is the complete execution: tasks, their trials, verifier rewards and any collected trajectories.

A dataset supplies a task, a trial runs in an environment, and a verifier returns the reward.Open full source frame
Follow one task from the dataset through a fresh trial. The verifier scores the environment the agent leaves behind, while the trajectory records how it got there.

Before paying for agent runs, you can substitute an oracle: a script that solves the task. Running a known solution through the environment and verifiers helps expose bad data, configuration problems and checks that reject a valid result. This separates debugging the evaluation setup from debugging the agent. An oracle is especially useful when a failed run could otherwise mean either “the agent failed” or “the test never worked.”

3:353:58
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:35 · section reference included

A migration test grades the database it leaves behind

Consider Song’s hypothetical coding task: fix a failing migration. The instruction file asks for the repair in ordinary user language. The environment contains SQL, database definitions and source code for an up migration, with starting data arranged so that the migration fails. The agent under test could be a coding agent alone, one configured with skills, or one connected to a custom MCP server.

What would count as a repair? The verifier can run the up migration and require exit status 0, then inspect the resulting tables for the expected columns and indexes. These checks cover both execution and effect: a command that exits successfully is insufficient if it leaves the wrong schema. An optional oracle script supplies a literal repair, allowing the same checks to be exercised before the agent enters the picture.

Five cards show configuration, an instruction, a container image, an optional solution and a verifier.Open full source frame
The instruction, container, optional reference solution and verifier are separate parts of the migration task. The verifier can grade the resulting database even when two agents repair it differently.

The agent edits the broken migration file and stops. In the typical local Docker setup Song describes, the verifiers are mounted into the container only afterward; the agent cannot access the verifiers or reference solution during its attempt. That keeps the repair task separate from the machinery that grades it. The verifier then executes the migration and inspects the schema the agent’s changes actually produce.

This is the integration-testing analogy behind Harbor. The canonical verifier treats the agent as a black box: it cares about the final files, output and system state, not the precise sequence of tool calls. A different implementation can pass the same test. That makes it possible to “refactor under green”—replace the agent’s internals while retaining a meaningful check of its behavior.

The distinction came from maintenance pain on the Phoenix team. Tests coupled to particular tools broke when a tool changed even though the agent could still accomplish the task. Outcome-based tests support the team’s effort to consolidate reusable skills across CLI, MCP and server-side agent experiences. Harbor can inspect trajectories too, but making a particular execution path part of the test is a separate decision with a maintenance cost.

7:498:19
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

7:49 · section reference included

Independent trials can run in parallel

Fresh environments make the work horizontally scalable. One trial does not need another trial’s files or database changes, so a sandbox provider can run many at once. Song names Daytona, Modal and E2B as compatible examples; his team uses Daytona with about 200 concurrent workers. The point is to shorten the feedback time for a suite without making trials share state.

How should you read the task-by-agent-by-trial comparison? Song’s example has four tasks, two coding agents and three attempts per task-agent combination. Every attempt occupies a separate container with an identical starting state for that task. The agents can then take different paths. The comparison separates differences between agents from the variation across repeated attempts; it does not show one agent progressively repairing the same environment.

A color-coded matrix compares agents across repeated tasks, including passes, failures and timeouts.Open full source frame
Each row is one task. The six result cells show three independent attempts by each of two agents, so you can see both a successful attempt and how often that success repeats.

Those individual outcomes can be aggregated. In the example, pass@3 asks whether an agent solved a task in at least one of three independent attempts. Both agents pass the migration task under that criterion, while another task is passed by only one. Aggregating across the dataset gives a broader comparison, but the interpretation remains success within the allowed attempts—not a guarantee that the next single attempt will succeed.

14:1915:26
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

14:19 · section reference included

Phoenix adds experiment history and inspectable traces

Phoenix maps Harbor’s evaluation objects into its own observability and experimentation model. Datasets remain datasets, jobs become experiments, trials are recorded, and verifier rewards become evaluations. This gives the team a place to compare runs and track whether the metrics it cares about move in the intended direction.

The integration also converts ATIF trajectories into OpenInference, semantic conventions built on OpenTelemetry. Those conventions describe AI operations in forms Phoenix can interpret. An LLM call and a tool call can receive specialized displays instead of appearing only as undifferentiated log records. The migration example’s sequence of agent steps can therefore be inspected as a trace.

Dataset versioning preserves what a task looked like for a particular job, while experiment views show rewards and other metrics across changes. Phoenix also exposes this data through skills, CLIs and an MCP server so agents can help investigate runs. None of that makes Phoenix a prerequisite: Song notes that Harbor has its own local open-source UI and a hosted service. Phoenix is an additional way to organize and inspect the evidence.

17:2619:24
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

17:26 · section reference included

Put the application and its agent interfaces in the sandbox

Phoenix is both the tool used to inspect these experiments and the application being tested. In its ordinary architecture, an instrumented user application emits traces over OTLP, the OpenTelemetry Protocol. Phoenix ingests those traces into SQLite or Postgres and exposes services for traces, spans, sessions, datasets, experiments and evaluations through REST and GraphQL.

The team maintains three ways for agents to work with those services:

  • Coding agent with a CLI: Commands invoke the REST or GraphQL APIs.
  • Coding agent with MCP: An MCP server exposes access to those APIs.
  • Server-side agent: The team’s own harness uses the MCP server and also has a Bash emulator.

Which routes reach the same application services? The diagram keeps the three alternatives separate while showing their shared backend. They are supported experiences to test, not three agents that must cooperate on every task.

Phoenix has a convenient testing property: the server runs on a single node. The team can package the running application, APIs and a SQLite file containing realistic data into Docker, replacing the browser with a test client. A task can then ask an agent to find hallucinated LLM calls from the last week of traces and add them to a dataset. This exercises a product workflow rather than merely asking the agent to edit a repository.

After the attempt, a verifier can check the result through a REST endpoint or query SQLite directly—for example, to confirm that the expected dataset record was written. The same application state can support different agent interfaces while the success check remains about the intended database change.

Three agent experiences reach shared Phoenix services

Coding agent + CLIagent-experiences — Uses Phoenix commands to access application data.
Coding agent + MCPagent-experiences — Uses the configured Phoenix MCP server.
Server-side agentagent-experiences — Team-maintained harness with MCP access and a Bash emulator.
Phoenix CLIinterfaces — Invokes REST or GraphQL.
Phoenix MCP serverinterfaces — Exposes tools backed by REST and GraphQL.
REST / GraphQL → core servicesapplication — Traces, spans, sessions, datasets, experiments and evaluations over SQL-backed data.
Arrows show API access, not trial order. A trial tests one selected experience; the CLI and MCP routes lead to the same application services.
Read the diagram as text
  • Coding agent + CLI. agent-experiences — Uses Phoenix commands to access application data.
  • Coding agent + MCP. agent-experiences — Uses the configured Phoenix MCP server.
  • Server-side agent. agent-experiences — Team-maintained harness with MCP access and a Bash emulator.
  • Phoenix CLI. interfaces — Invokes REST or GraphQL.
  • Phoenix MCP server. interfaces — Exposes tools backed by REST and GraphQL.
  • REST / GraphQL → core services. application — Traces, spans, sessions, datasets, experiments and evaluations over SQL-backed data.
  • Coding agent + CLI → Phoenix CLI: Runs commands.
  • Coding agent + MCP → Phoenix MCP server: Calls exposed tools.
  • Server-side agent → Phoenix MCP server: Uses MCP access.
  • Phoenix CLI → REST / GraphQL → core services: Invokes REST or GraphQL.
  • Phoenix MCP server → REST / GraphQL → core services: Invokes REST or GraphQL.
20:3721:07
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

20:37 · section reference included

Harbor orchestrates from outside the environment

An audience question clarifies two important details. First, each trial tests only one selected agent configuration. Second, Harbor itself runs on a host machine, not inside the task’s Docker environment. That host might be a laptop or a GitHub Actions runner. The environment can run locally in Docker or remotely through a sandbox provider such as Daytona.

The host-side CLI coordinates execution through scripts and directory conventions. The setup supplies scripts for starting work and running verifiers. A task can also supply an oracle script. Harbor starts the appropriate stages, and the task environment supplies the application-specific behavior. This is why a sandbox can contain more than source files: Phoenix includes a running server, APIs, a CLI, data and skills available on the filesystem.

A distributed application makes the packaging problem harder. Song has not solved that case in his own work. He suggests exploring a stable interface around the agent’s data access or Harbor’s ability to start multiple Docker images. Neither is presented as a finished recipe. If the agent directly queries S3, replacing those calls with mocks can bring back the maintenance burden that realistic environments were meant to avoid.

Data volume is another choice the task must justify. The team generally uses cleaned subsets of existing data rather than its entire trace collection. Multiple tasks can reuse one SQLite file, but a task that needs only a subset need not carry everything. Conversely, testing behavior against a full database would require a large fixture. Song distinguishes that possible test from the smaller-data evaluations the team has actually run.

25:2125:51
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

25:21 · section reference included

A SQL tool still needs a tool-selection policy

The concrete optimization begins with aggregate questions. Users wanted to know their busiest day over several months, how many traces arrived that day, or which models accounted for the most spending. Phoenix’s available REST routes and GraphQL resolvers did not directly answer all of those aggregates. Agents fetched pages of records and calculated the answer themselves. More data meant more pagination and more tool calls.

The team added an MCP tool that lets an agent write and execute a read-only SQL query against the database. That provided a more direct way to answer aggregate questions, but simply adding it did not reliably change behavior. An agent now had REST, GraphQL and SQL available and still sometimes chose an inefficient route. The missing piece was guidance about when to select the new capability.

What did the prompt change tell the agent to do differently? The earlier prompt mentioned GraphQL; the revised guidance directed aggregate work, such as counts and grouping, toward the SQL tool before GraphQL. This was a small policy change on top of an existing tool. Harbor supplied a way to test whether that policy changed actual tool use rather than merely sounding sensible.

The evaluation started with a semi-realistic SQLite file collected during the team’s own product development. Song asked Claude to examine domains including traces, spans, datasets and experiments, then brainstorm both aggregate and non-aggregate user questions and write oracle solutions. Those solutions could be REST calls or database queries. He spot-checked the synthetic questions, and the verifiers included SQL checks, REST calls and LLM-as-a-judge checks. Running them against the oracles helped debug the setup before evaluating the agent.

The changed prompt led to more frequent SQL use and a lower average tool-call count. That matched the intended direction: Song preferred aggressive SQL use to agents paginating through records for an aggregate. The fixture was small enough that pagination was limited, so he suspected the modest observed reductions understated the problem at larger volumes. That larger-volume effect remained an expectation, not a measured result of this experiment.

33:2133:51
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

33:21 · section reference included

Keep correctness separate from the optimization target

The Phoenix demo opens the versioned dataset and a concrete aggregate task: identify the three most expensive models by total cost and report each cost. How is that question paired with an expected answer? Its reference answer comes from an oracle SQL query run against the fixture. This makes the result repeatable for that query and data, but its correctness still depends on the generated query expressing the intended question—a degree of trust Song explicitly places in Claude.

The experiment views then show jobs and metrics such as cost, run latency, reward, tool count and SQL use. Correctness rewards were already high before the prompt change: the system generally found the right answers, but used too many calls to get them. The experiment was an efficiency optimization, not a repair of an answer that had been consistently wrong.

A reward is not a universal score Harbor discovers for you. The task author writes the verifier, chooses what success means and gives the reward a name. In the binary examples, the script emits 0 or 1. The job view aggregates those task outcomes. Checking whether the expected database state exists and checking whether SQL was used answer different questions, even if both appear as evaluation metrics.

That prompts a useful objection from the audience: if verifiers are supposed to ignore the agent’s implementation, why inspect whether it used SQL? The distinction is the purpose of the test.

  • Correctness regression: Inspect the answer or side effects. Let any implementation pass if it accomplishes the task.
  • Targeted optimization: Inspect the trajectory when the behavior of interest is tool selection or tool-call count.

Trajectory-aware checks make the optimization visible, but they are less tolerant of future implementation changes. Song accepts that tradeoff for this particular investigation.

Phoenix also supports sharing and curating data, adding traces to datasets and loading externally stored data into its database. Those facilities can help a team maintain evaluations, but Song does not present a proven large-team operating model: this work is being done by him and a handful of colleagues.

38:3139:01
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

38:31 · section reference included

Compare jobs, then compare the same task

The next view puts two jobs side by side: before and after the single prompt change, using the same model configuration. At the aggregate level, GraphQL calls decrease, SQL calls increase, the average number of tool calls per trajectory falls and agent turns decrease. Rewards stay the same, with the displayed tasks still passing. The explanation is concrete: SQL can perform work in one query that previously required multiple retrieval and aggregation steps.

Do individual tasks follow the aggregate pattern? The task table places expected answers beside each job’s actual answer and metrics. For the three-most-expensive-models question, tool count and turn count both fall—but SQL-call count falls too. That is a useful surprise: more SQL across the dataset does not mean every improved task must make more SQL calls. The task-level comparison prevents an average trend from becoming an invented explanation of every run.

44:1544:45
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

44:15 · section reference included

Trace the change from GraphQL retrieval to SQL

Filtering helps narrow the comparison to consequential changes. Song tries a natural-language request for runs where the second experiment has a lower tool count than the first. A local model in the browser is meant to translate that request into a query DSL, but the attempt does not work as intended. He falls back to pasting a known-correct expression and filters the table, then examines an example that changed from no SQL use to SQL use. The fallback is part of the demonstration: the structured filter remains useful even when its language interface fails.

Where did the execution actually change? The corresponding traces reveal the distinction. Before the prompt change, the agent used Bash, whose emulator called GraphQL underneath. In the later job, the analogous operation issued a SQL query directly against the database. The ATIF-to-OpenInference conversion makes those steps inspectable. This is stronger explanatory evidence than a SQL-use counter alone: it identifies the old access path and the replacement in a particular task.

The team can also ask an agent to inspect a selected experiment and task. That fits its working practice: use the product, record the agent’s traces, collect useful data and enlist the agent to help investigate its own behavior. The traces provide concrete material for that investigation rather than requiring the agent to reconstruct what happened from memory.

45:4546:14
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

46:14 · section reference included

Test skill outcomes and skill triggering deliberately

The team’s regression suite is still early, focused on happy paths for its most important skills across supported models and providers. Skills are useful units because they describe workflows: when a condition occurs, take the appropriate action. Song compares these tests to Playwright-style end-to-end tests, except that an agent and LLM calls sit inside the execution path.

Skill selection introduces a different failure. A useful skill may stop triggering when another skill’s front matter overlaps with its description. There are two ways to test that risk:

  • Check the workflow outcome: Ignore which skill fired and verify that the task succeeded. This preserves flexibility when another route is equally valid.
  • Check the trigger itself: Inspect the trajectory to see whether a particular question invoked the intended skill. This directly tests routing, even if the rest of the workflow is not exercised.

A one-turn trigger test may need little realistic data: ask the question and inspect whether the intended skill fired. Its cost is coupling. Rename the skill and the assertion may break even though the underlying capability still works. Song’s position is practical rather than doctrinaire: such tests can be useful, provided the suite treats them as focused implementation-aware checks rather than confusing them with end-to-end proof.

48:2348:53
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

48:53 · section reference included

Evaluate the hybrid system—and invest in its starting data

The closing questions sharpen what the Phoenix evaluation includes. In production, an instrumented customer application sends traces through OTLP into the database. In these Harbor tests, that application and ingestion path are absent. The sandbox starts with an already-populated SQLite file. It assumes the upstream application has produced the data; it does not emulate that application or test the entire production ingestion path.

The unit under evaluation is a hybrid system. The Phoenix team does not modify Claude Code or Codex themselves. It modifies the MCP server, skills, CLI and APIs that help those agents interact with Phoenix. Packaging the supporting application and agent-facing infrastructure together allows the team to evaluate those combinations as a unit—the system its changes can actually improve or break.

Getting that setup working still requires care. Phoenix’s single-node architecture makes containerization comparatively easy; distributed systems face more work. Harbor’s reliance on scripts in particular locations also gives coding agents fewer automatic checks: Song notes that they cannot rely on a linter to establish that the setup follows all the execution conventions. Harbor’s own skills can help assemble the project correctly, but the script-driven setup remains something that must run successfully.

The data workflow is still fairly ad hoc. Song reuses an existing SQLite file when it fits the task; otherwise he asks Claude to remove unneeded material and upload the smaller file to cloud storage. His final emphasis is that obtaining realistic data is the hard part. The sandbox can only reveal problems represented by its starting conditions. Better fixtures make it more likely that an evaluation uncovers the failures users will actually encounter.

51:4352:13
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

51:43 · section reference included

Resources

Transcript download
Full transcript

Loading transcript…