What Does an AI Engineer Do Now? - Aparna Dhinakaran, Arize AI

Loading slides…

Read the talk

What Does an AI Engineer Do Now?

The AI engineering loop connects building, evaluation, shipping, monitoring and improvement.Open full source frame
The AI engineering loop connects building, evaluation, shipping, monitoring and improvement.

Aparna Dhinakaran explains the engineering behind Arize Signal: agents that investigate telemetry, preserve state across disposable runs, propose fixes and depend on evaluators calibrated against human judgment.

From a talk by Aparna Dhinakaran

At a glance

Ideas worth remembering

  • Signal investigates configurable batches of traces and remembers prior issues, allowing later evidence to change the significance of an earlier observation.

  • Disposable sandboxes and persistent external state serve different purposes: temporary access ends with a run, while findings and recovery state support future work.

  • Evaluators need their own validation: compare their decisions with human judgments on ground-truth datasets before relying on them to assess agent work.

  • Trace both production agents and the coding agents behind them. Online evaluations over those traces can supply the next investigation with behavioral evidence and judgments.

From writing features to building the system that improves them

Using an LLM occasionally and operating thousands of agents are very different engineering practices. Aparna Dhinakaran of Arize AI opens with that gap, drawing on a funnel she attributes to Andrej Karpathy. It places roughly six billion people in a group who have barely touched an LLM, one to two billion among casual free-tier users, about 20 million among math and coding professionals using agents to compress weeks of work, and roughly 5,000 among users with frontier access running thousands of agents. Dhinakaran presents these figures as estimates. The question is how the rest of the field moves toward operating those systems.

The progression from writing code through local and cloud agents leads toward teams of agents.Open full source frame
The progression from writing code through local and cloud agents leads toward teams of agents.

The progression raises the level at which engineers work: from writing code in higher-level languages, through copying generated code and using local agents, to delegating work to cloud agents and eventually teams or “factories” of agents. Dhinakaran presents this as both a roadmap and a direction Arize has already begun pursuing internally. As more execution moves into agents, the engineer has to design how that execution gets checked and improved.

That changes several responsibilities together:

  • Feature development: Build the loop that produces and improves a feature, extending the job beyond implementing its first version.
  • Review: Set up systems and agents to review work, with evaluations that can assess tasks whose outputs vary between runs.
  • Observability: Give agents access to telemetry so they can inspect behavior and find recurring patterns.

The familiar sequence—build, evaluate, ship, monitor, improve—remains. The new engineering work is making the system carry out those activities.

0:120:42
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:12 · section reference included

Signal reads traces and carries an investigation across runs

Arize’s concrete implementation is Signal, an always-on agent that reads traces, identifies patterns in telemetry and can propose a fix or open a pull request when it has repository access. Its starting task is deliberately small: read a configurable volume of traces collected over a configurable interval and look for problems. The trace data supplies the behavior to investigate; repository access gives the investigation a possible route to a code change.

What must persist when the next batch of traces arrives? Signal needs to remember previously identified issues, rather than treating every batch as unrelated evidence. Dhinakaran gives a useful progression: something initially looks suspicious without becoming a captured issue; later traces make the same concern look more critical. Keeping that history lets a later run reconsider an earlier observation as evidence accumulates. It also gives the agent context about problems already found.

Signal reads traces to find issues such as claiming actions without tool calls, missing balance checks and unchecked pre-approval.Open full source frame
Signal reads traces to find issues such as claiming actions without tool calls, missing balance checks and unchecked pre-approval.

This makes agents primary consumers of observability data. Telemetry becomes input to an ongoing investigation, with memory connecting one run to the next. Designing that consumption path is a different task from putting metrics on a dashboard for a person to inspect: the agent must receive enough recorded behavior and prior issue history to decide what deserves attention.

3:123:41
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:12 · section reference included

A trigger starts a fresh sandbox with the context to investigate

An always-on service still needs a rule for when work begins. Signal supports several alternatives:

  • Scheduled sweeps: Wake every three or six hours, for example, and pull the configured amount of recent trace data.
  • Events: Start an investigation from a Slack message or a user complaint.
  • Metrics: Start when a configured metric condition triggers the agent.

These are alternative ways to initiate a run. A scheduled sweep does not need to wait for a complaint, and an event need not wait for the next scheduled interval.

Once triggered, the run receives the material it needs to investigate: trace context, additional third-party telemetry such as APM logs and metrics, repository access, and any necessary keys. Signal starts a sandbox for that run and places the required context and access inside it. The sandbox is the working environment for the investigation.

How does information cross that environment’s boundary? In Dhinakaran’s architecture, a single proxy is the only route in or out of the sandbox. Everything happening inside is traced, and generated outputs—including identified issues—are saved to memory. This arrangement makes the investigator observable too: its actions produce records that can be inspected, while its findings survive beyond the process doing the work.

Admission and job composition launch an isolated sandbox, which produces retained results and traces before teardown.Open full source frame
Admission and job composition launch an isolated sandbox, which produces retained results and traces before teardown.
5:125:42
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:12 · section reference included

Keep the findings and recovery state; tear down the runtime

A sweep can finish with an issue or without one. At completion, Signal separates what future work needs from what belonged only to this execution. The useful distinction is between retained knowledge and temporary access:

  • Retained: Issue outputs and history, traces of the agent’s calls and PR activity, and any learned skills saved to the repository.
  • Removed: The sandbox, credentials and repository access associated with the finished run.

A later run can therefore begin with the investigation’s history while receiving a newly created working environment.

State saved outside the sandbox lets a replacement sandbox and harness resume from the interrupted step.Open full source frame
State saved outside the sandbox lets a replacement sandbox and harness resume from the interrupted step.

The same separation supports recovery during a run. If the sandbox or agent harness fails, Signal is designed to resume from the failed step rather than repeat the investigation from the beginning. Here, the harness is the software running the agent’s work. State lives outside the sandbox, so a replacement sandbox and harness can recover it. The disposable environment can fail without taking all progress with it.

The recovery diagram shows what happens when step 3 fails. Steps 1 and 2 have already been saved, along with the conversation, memory and which step comes next. A new sandbox loads that state and restarts at step 3. Issue history serves a different purpose in a later sweep, helping Signal recognize a problem it has seen before. The talk leaves unspecified what happens if the interrupted step already produced an external effect.

Persistent state supports two kinds of continuity

Current executiontemporary — Sandbox and harness investigate the telemetry.
Retained informationpersistent — Recovery state outside the sandbox and retained issue history, grouped by purpose without specifying storage layout.
Replacement executiontemporary — New sandbox and harness pick up the interrupted run.
Later investigationtemporary — Starts with the history of issues already found.
The failure arrow describes a control action: launch a replacement execution. Other arrows describe saved information. The shared information node groups recovery state and issue history conceptually; it does not specify a shared storage layout.
Read the diagram as text
  • Current execution. temporary — Sandbox and harness investigate the telemetry.
  • Retained information. persistent — Recovery state outside the sandbox and retained issue history, grouped by purpose without specifying storage layout.
  • Replacement execution. temporary — New sandbox and harness pick up the interrupted run.
  • Later investigation. temporary — Starts with the history of issues already found.
  • Current execution → Retained information: Saves progress and findings.
  • Current execution → Replacement execution: On sandbox or harness failure: relaunch.
  • Retained information → Replacement execution: Restores progress to resume.
  • Retained information → Later investigation: Supplies prior issue history.
7:127:42
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

7:12 · section reference included

Make model and harness choices testable

Signal’s architecture also aims to avoid dependence on one model or harness. Dhinakaran says it began with Claude Code and, at recording time, also supported Cursor and Codex. Supporting other major harnesses, including open-source ones, is a goal. She describes broader multiple-model support as forthcoming while also describing model selection within the Anthropic family when using Claude Code.

Knowledge, tools and controls remain fixed while harnesses and models are compared with the same evaluation set.Open full source frame
Knowledge, tools and controls remain fixed while harnesses and models are compared with the same evaluation set.

That flexibility creates an evaluation question with a practical cost attached: can a cheaper configuration preserve the quality of detected issues and proposed PRs? More powerful models may produce better work, but their value needs to be assessed through the agent’s results. Dhinakaran presents cost and quality comparison as a reason for the design; she does not give a measured configuration comparison here.

Producing a PR then creates another piece of work to evaluate. Signal runs evaluations over the agent’s proposed change. Finding a plausible problem and writing a patch do not by themselves establish that the change solves the problem or preserves existing behavior. The evaluation stage supplies the check between a proposed improvement and confidence in its outcome.

9:129:42
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:12 · section reference included

Define good behavior, then test whether the judge recognizes it

An evaluation can begin with a dataset, a prompt and outputs checked for regressions as part of development. The harder responsibility is deciding what those checks should reward or reject. Dhinakaran recommends spending time with traces and agents, identifying common errors and translating human judgment into evaluations. This is where the engineer defines what good behavior means for the particular product.

An illustrative judge-human comparison agrees on four of five cases; it is not a production accuracy estimate.Open full source frame
An illustrative judge-human comparison agrees on four of five cases; it is not a production accuracy estimate.

Skills can assist with writing those evaluations. Dhinakaran mentions Arize’s Evaluator skill as help for both creating evals and aligning them with human judgment. Assistance with authoring still leaves a separate question: does the resulting evaluator actually make the judgments the team intends? A judge that misses the errors people care about cannot reliably check an automated repair.

Arize addresses that question by evaluating its evaluators. The team builds ground-truth datasets, runs the candidate evaluations and compares their decisions with human outputs. It then works on alignment before releasing a new evaluator. The human judgments provide reference decisions; agreement with them is something to check, rather than assume because an evaluator returns a confident verdict. The talk gives the procedure without specifying an agreement threshold.

Once aligned, evaluations can serve two settings:

  • Offline testing: Check a proposed change during development, including whether it introduces regressions on the chosen examples.
  • Online evaluation: Judge recorded production behavior as the application runs.

The same effort to define useful judgments can therefore support both release testing and continued observation after release.

10:4211:12
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:42 · section reference included

Trace the coding agents as well as the product

After pre-release evaluations and deployment, the instruction is to “trace everything.” That includes the product’s agents and the coding agents that built them. Claude Code, Cursor and Codex runs are themselves observable work. Recording their behavior helps explain what happened inside the factory producing the application, extending investigation beyond the application’s final responses.

Online evaluations turn traced failures into patterns, test cases and proposed improvements.Open full source frame
Online evaluations turn traced failures into patterns, test cases and proposed improvements.

Production traces then receive online evaluations. Those results become additional data for agents investigating the system: the investigator reads telemetry together with evaluation results to find problems within traces. The causal connection matters. Instrumentation records execution; evaluations attach judgments to that behavior; agents consume both to locate gaps and propose improvements. An evaluation result becomes an input to subsequent engineering work.

This returns to the opening gap between frontier labs and ordinary businesses. Dhinakaran suggests that part of the difficulty is business data: it is messier than the constrained, testable work available in a lab. AI engineers have to make testing and improvement work with that operational reality. Her closing direction is concrete: trace the agents, learn to evaluate the products, and build systems that use collected telemetry to improve them. The engineering responsibility extends through the behavior of the system after it ships.

12:4213:11
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

12:42 · section reference included

Resources

Transcript download
Full transcript

Loading transcript…