We Kept Rebuilding Our AI Agent. Here's What Survived - Fuad Ali, Arize AI
Loading slides…
Read the talk
We Kept Rebuilding Our AI Agent. Here’s What Survived
Fuad Ali follows Arize’s Alyx from a late-2023 prototype to one agent with on-demand skills, explaining why planning, stopping, context management and regression testing became problems for the harness to solve.
From a talk by Fuad Ali
At a glance
Ideas worth remembering
Harness-owned plan state reduced redundant to-do bookkeeping. Alyx presents the plan on every model call and was testing removal of the to-do tool entirely.
Completion needs an explicit check: Alyx can reject a finish call while plan items remain, warn before the iteration limit, and force a bounded stop.
Oversized tool results, growing history and cross-session memory need different mechanisms: selective file queries, replacement checkpoints, and a per-user Markdown file capped at 8,000 characters.
Alyx replaced ten page-specific specialists with one agent that loads skills and tools on demand. Narrow features such as filter generation may need only one model call.
Production sessions supply regression fixtures because an okay status can hide repeated calls or excessive latency. Human labels calibrate the judges, and people still choose the speed–quality tradeoff.
From a useful-but-unreliable prototype to planned work
Arize’s first prototype of its in-product agent was, in one staff engineer’s recollection, about half hallucination and half something useful. Fuad Ali, a product manager at Arize, uses that late-2023 starting point to explain the rebuilds behind Alyx. The estimate is an informal description of the prototype, not a measured accuracy result.
The setting is appropriately recursive: Alyx helps debug Alyx, while Signal, a separate trace-reading agent, examines the traces Alyx produces. “Agents all the way down.” Keeping those roles separate matters: Alyx is the user-facing AI engineering agent; Signal helps find problems in its execution.
Inside Arize AX, Alyx can read execution traces, turn findings into evaluations and datasets, run experiments, generate synthetic data from traces, and carry out multistep tasks. That range makes it more than a chat window over documentation. It needs access to product data and tools, a way to keep track of unfinished work, and enough context to choose what to do next.
Prompt optimization supplies a concrete before-and-after example. The earlier product could retrieve prompts stored in Arize and accept a request to optimize them. In the later interface, Ali asks Alyx to optimize its own system prompt, and Alyx creates a plan that includes checking resources around the application. What changes between the two interfaces? The important distinction is the visible move from retrieving a prompt to organizing an investigation around it—not evidence that this particular request produced a better final prompt.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The rebuilds happened around the model
One early correction was to move math out of the model and into code. The late-2023 models were unreliable at computation, so the application’s harness—the code that runs the model, tools and surrounding state—took responsibility for it. That division survived the subsequent rebuilds: code computes; the model interprets and explains.
The first official product, then called Copilot, routed requests through a decision tree over a set of skills. An August 2025 rewrite introduced iterative tool calling and a to-do tool, allowing work to continue across multiple model calls. Alyx 2.0 relaunched in February 2026 with a new default model. In June 2026, the team merged specialist agents into one Alyx that loads skills on demand. Long-term user memory followed, written in the background.
These changes did not involve training the underlying models. Arize wrote its own agent loop, plan handling, stopping rule and memory management rather than using an agent framework. Better models opened up new options, but the team still had to decide how to use them. Owning the harness gave it control over those decisions—and responsibility for discovering where its assumptions broke.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Move plan bookkeeping out of the model’s way
Planning began with an apparently reasonable tool: give the model a to-do list and let it check items off. Ali compares the choice to assigning chores. You can hand someone the whole list and ask them to maintain it, or hold the list yourself and keep supplying the next task. Alyx initially used the first arrangement.
The model became too enthusiastic about maintaining the list. Calls went into checking off tasks rather than doing useful work, and after several tool calls it could lose track of its focus. More than 20 iterations on the to-do tools eventually pushed the design toward harness-owned state.
The correction happened in stages. First, an in progress status made the current task explicit. Then the harness began rendering the plan into a message on every model call, with the rules beside the state they governed. The next call no longer depended on the model reconstructing its position from a growing history of tool interactions.
Once the plan was kept in ranked order, the extra status became redundant: the next unfinished item already identified the next task. Maintaining in progress was creating two calls for bookkeeping that the ordering could express, so the team removed it. At the time of the talk, it was testing the further step of removing the to-do tool entirely, keeping plan state and completion checks in the harness and presenting them inline in messages.
The result is a different division of labor. The harness remembers what remains and checks completion; the model consumes that state and does the next piece of work. A plan can help an agent stay focused, but making the model continually administer that plan can consume the calls the plan was meant to organize.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A text response does not necessarily mean “done”
The original stopping rule treated a model response without tool calls as the end of the turn. That assumption failed when a model announced its next action in one response and made the tool call in the next. The harness could stop after the announcement, before the promised work happened.
This was more than a model-upgrade nuisance. Alyx lets customers choose their own provider, including customers committed exclusively to Anthropic or OpenAI. Arize benchmarks models and runs regressions, but cannot guarantee that every customer-selected model has been tested. A stopping rule based on one model’s conversational habits was too fragile for that flexibility.
Finishing therefore became an explicit call that the harness checks. Three controls address different failure modes:
- Premature completion: If plan items remain open, the harness can bounce a finish call back up to three times.
- Approaching the budget: A warning arrives one step before the iteration limit.
- Failure to stop: A forced finish provides a final bound on execution.
The completion check gives unfinished work another chance; the iteration limit prevents that chance from turning into an unbounded loop.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Treat a huge result like a file
Context management contains several different problems. Ali’s analogy is a desk covered in books: one book can be too long to read, and enough individually manageable books can still fill the desk. For Alyx, a single tool result can overwhelm the context window, while the accumulated history of its own work can exhaust the budget over time.
Trace analysis makes the first problem concrete. Alyx may be reading another agent’s trace, containing a large JSON result and a great deal of nested activity. Dumping that blob into context can leave little room for useful work. The replacement is the book-on-a-shelf arrangement: keep the large result available elsewhere and leave a compact indication of its contents and location in the working context.
For JSON, the harness truncates within the result and lets the agent use code, including jq, to retrieve what it needs from the full data. The model’s ability to write queries becomes a way to inspect a large result selectively.
The pointer preserves access; truncation reduces the immediate context load. This is the practical meaning of “treat a huge result like a file and just grep it”: make the data searchable rather than assuming it must all be present in the model’s current input.
Growing execution history needs a separate correction. When Alyx’s trajectory exceeds a window, the harness makes the model write a checkpoint, then drops the history that checkpoint replaces. A fixed token budget limits retained history. Selective file access handles oversized tool data; checkpointing handles accumulated work. Neither is the same as remembering a user across sessions.
A large trace has an immediate path and an on-demand path
Read the diagram as text
- Large trace result. A JSON blob can overwhelm the model’s context window.
- Full result kept available. on-demand — The book stays on the shelf rather than on the desk.
- Compact result in context. immediate — Truncation and a reference reduce what the model reads immediately.
- Alyx model call. Uses the compact result and requests more data when needed.
- Large trace result → Full result kept available: Retain for later access.
- Large trace result → Compact result in context: Truncate for context.
- Compact result in context → Alyx model call: Supply compact information.
- Alyx model call → Full result kept available: Query with code such as jq.
- Full result kept available → Alyx model call: Return requested data.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
One 8,000-character file per user
For cross-session memory, Arize tried vector retrieval, episodic and semantic stores, and temporal knowledge graphs. It ultimately chose one Markdown file per user, capped at 8,000 characters. Ali describes the small file as beating the more elaborate alternatives for Alyx; the comparison is a product-design judgment here, without a reported benchmark or numerical quality margin.
The read and write schedules are deliberately different. Every full request loads the file. After a session finishes, a background job edits it. Memory maintenance therefore does not require the interactive agent to run a retrieval-and-update process at every step, and the size cap bounds how much persistent memory each request introduces into context.
The small artifact also makes review practical. A human labeler can inspect a user’s file, decide whether the right information was retained, and adjust the instructions used by the background writer. Concentrating writes in one end-of-session job gives the team a focused place to enforce user separation, including checking that another user’s information does not enter the file.
The appeal is specific: a bounded amount of user history, loaded consistently, with an inspectable writer and an artifact people can audit. It also expresses a limit. Alyx’s persistent memory must fit into that small file; this design is not an attempt to preserve every detail of every session. For this product, the team preferred that constraint to the machinery of its earlier memory systems.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Ten specialists became one agent—and some tasks need no loop
The agent topology followed another expansion and rollback. Alyx began with one trace-analysis agent in 2025, grew to ten specialists, and assigned different specialists to different product pages. On the dataset page, for example, a specialist could inspect datasets and generate synthetic data.
The replacement was one agent that loads skill instructions and tools on demand, then decides which tools to use. The dataset capability remains available without requiring a separate dataset agent as the organizing unit. Ali credits both improving model capability and a desire to simplify; the rollback does not establish that specialists are always the wrong architecture.
Model selection follows the same task-specific reasoning. Arize generally defaults to “fast and good enough,” investing heavily in the harness rather than automatically switching to the newest model. Leading a general benchmark and being the best default for Alyx are different questions. The team does upgrade defaults, but treats that as a product decision to test.
At the time of the talk, the team was exploring routing models by task and training or post-training small models for narrow features. The “filter with AI” feature illustrates why: generating a filter can be simple enough for a very small model and a single call. Giving that feature the entire planning-and-tool loop may add machinery it does not need. Simplification can mean consolidating agents, but it can also mean declining to use an agent loop at all.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Production sessions become living regression fixtures
The debugging example returns to the recursive setup: Alyx’s own requests produce traces inside Arize, in a project still named Copilot Prod. Where does a user interaction become inspectable engineering evidence? In the live trace view, the in-product request and the record of the agent’s execution meet. The agent helping investigate a system is itself a system to investigate.
Those production records supply regression cases. Useful user queries and newly discovered failures enter datasets that grow as behavior changes. Session datasets can retain an entire session and let the team replay a particular turn through the agent. That provides a way to test a decision made partway through an interaction, rather than reducing every failure to a fresh opening prompt.
Arize Experiments then supports A/B comparisons of harness changes. The reason to retain old cases is blunt: “the fix can also be the next bug.” A change that repairs today’s failure may break behavior that previously worked, so the growing dataset becomes a record of obligations the next harness version must still satisfy.
A successful status alone is a weak definition of agent health. Two examples show what can hide behind it:
- Repeated work: A turn with 43 identical tool calls may report an okay status while plainly wasting execution.
- Excessive waiting: A response taking 227 seconds can be unacceptable even without an explicit error. Ali regards that wait as too long; the appropriate latency threshold still depends on the use case.
These are behavioral failures that require looking inside the trajectory, not merely checking whether a request returned.
Signal automates more of this trace review, reducing the time staff engineers spend reading executions by hand. Automation does not settle whether its judgments are right. Arize still aligns evaluators with human assessments, checking that judges reflect what its engineers consider good and bad behavior.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
People still define the baseline and choose the tradeoff
Human work remains concentrated in defining the target. Gold labels and golden datasets for judges come from people. An evaluator needs examples of the judgments it is supposed to reproduce; an automated verdict cannot supply its own independent standard of correctness.
Speed versus quality is another decision the metrics inform without resolving. A model change might increase latency by one amount and improve response quality by another. Measuring both makes the choice clearer, but does not determine how much waiting a particular workflow should tolerate or which quality losses are acceptable. Those remain human product judgments.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
What survived every rebuild
The ending distinguishes the machinery that changed from six commitments the team kept. Some have a detailed mechanism behind them; others remain concise design requirements:
- Trace everything into the team’s own product. Alyx’s execution stays available for investigation rather than disappearing behind its final response.
- Keep math and data work in code. Code computes aggregations, exposes them through tools, and lets the model read and explain the results.
- Keep golden datasets in CI. Preserve known expectations as part of checking subsequent changes.
- Return errors with an obvious fix. Make recoverable problems actionable.
- Keep streaming and persistence. These remain requirements across the rewrites.
- Keep skills. Names such as routers and tools changed, but the lasting purpose was to give the model a capability around an external system.
The surviving questions are concrete enough to use in a design review: Who holds the plan? What permits a turn to end? Where does an oversized result live? Who writes persistent memory? Does the task need several agents, one agent, or one model call? Which model is fast enough and good enough? And who checks the agent’s own fixes? Alyx’s answers changed as the team encountered failures; tracing, code-based computation, regression cases and human judgment made those changes inspectable.
“Every day we use Alyx to build Alyx” is the closing tagline. Its practical substance is that the product’s own sessions become material for debugging and testing the next version. The agent can help investigate its failures, while people still supply the standards by which an apparent improvement earns its place.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
Further reading
A practical starting point for using the in-product AI engineering agent discussed in the recording.
Related reading for the discussion of repeated tool calls and response latency.
A related repository for readers interested in tracing agent execution.
Related talks
- Building Self-Learning Loops for Your Agent — Fuad Ali, Arize AI
Ali’s talk on self-learning loops relates to his question about who checks an agent’s own fixes.
- Break It 'Til You Make It: Building the Self-Improving Stack for AI Agents
Related to the use of production failures to test the next version of an agent.
Full transcript
Loading transcript…