▶ Watch ↗AI Engineer Europe 202616:17
Conference affiliation: Arize
The organizer explicitly identifies DeLucia as a core contributor to Alyx, Arize’s AI engineering agent, and credits her with helping evolve the earlier Copilot through her product work at Arize. Her attributed contributions concern tools for investigating model behavior, planning work, managing context, and improving agents from evidence of failures. This is a contribution to an employer’s product; the evidence does not establish sole creation or personal ownership.
arize.com · Source ↗Authored by DeLucia in July 2024. Introduces Arize’s assistant for model-performance investigation, prompt optimization, custom evaluation creation, and natural-language data search and curation. The article establishes her authorship of the announcement and describes the team’s product capabilities.
arize.comAuthored by DeLucia in July 2025. Explains how Copilot’s isolated answers and generic next steps motivated its evolution into Alyx, with users supplying relevant traces, evaluation explanations, and datasets. Planning, retained state, and sustained progress toward outcomes are presented as development goals that the team had not yet fully achieved.
arize.comFebruary 2026 release announcement coauthored with Jack Zhou, Chris Cooning, and Aman Khan. Describes Alyx planning and executing work across Arize AX, including creating datasets, comparing prompt variants, building evaluations, and investigating traces. The announced workflows include asking for approval at relevant decision points.
arize.comMarch 2026 technical article coauthored with Chris Cooning, Priyan Jindal, and Jack Zhou. Documents Alyx stopping after the first part of a request and the team’s response: structured planning tools, four task statuses, a current plan placed after the system prompt on every iteration, and a completion gate. It also explains blocked tasks, selective use of planning, and user redirection.
arize.comMarch 2026 technical article coauthored with Chris Cooning, Priyan Jindal, and Jack Zhou. Describes retaining full trace and tool data outside the immediate context, exposing previews and stable identifiers, retrieving targeted portions, and isolating intensive searches in subagents. It reports failures from aggressive truncation and generated summaries and explains evaluating an eleventh conversation turn after preloading ten earlier turns.
arize.comMay 2026 article coauthored with Chris Cooning, Priyan Jindal, and Jack Zhou. Explains converting production failures into regression cases containing the original inputs, context, and expected behavior. Covers explanatory model judges, deterministic checks, experiments over time, automated checks for changes to prompts and tools, and testing model upgrades for behavioral regressions.
arize.comMay 2026 article coauthored with Chris Cooning, Priyan Jindal, and Jack Zhou. Describes using Alyx to search dense traces, aggregate metrics, categorize errors, create evaluation datasets, and investigate failures in Alyx itself. Colleagues report trace identifiers and expected behavior, allowing a smaller group to conduct investigations with the agent.
arize.comSallyAnn DeLucia is head of product at Arize AI and a core contributor to Alyx, the company’s agent for AI engineering. Her work focuses on the gap between seeing an AI system fail and knowing what to change.
DeLucia began her career in data science. She earned a master’s degree in applied data science. At Arize, she moved from machine-learning solutions engineering into product leadership.
In 2023, the Lost in the Middle paper challenged her assumption that giving a language model more context would improve its answers. She revisited applications she had built and emphasized inspecting retrieved information, trying fewer documents, and moving relevant passages toward the start of the input.
In July 2024, she introduced Arize Copilot, an assistant that helped users inspect model behavior, improve prompts, and build evaluations. A year later, she offered a candid assessment of its limits: a useful chart or explanation often led to generic advice, leaving users to finish the job. That gap shaped Copilot’s evolution into Alyx. Users could supply relevant traces—records of an application’s execution—along with evaluation results or datasets. The team’s ambition was to give those interactions continuity, with planning and retained state that would let the agent pursue an outcome across several steps.
Alyx 2.0, released in February 2026, could plan and execute work across Arize AX, the company’s platform for inspecting and evaluating AI applications. An engineer could ask it to compare two prompts, and Alyx would organize the experiment, request approval when needed, run it, and analyze the results. In technical accounts coauthored with Chris Cooning, Priyan Jindal, and Jack Zhou, DeLucia describes how the team addressed recurring failures:
In her prompt-learning work with Fuad Ali, DeLucia advocates revising agent instructions using failed examples, evaluator explanations, and feedback from domain experts. The evaluator needs its own improvement cycle: unreliable judgments can steer prompt revisions toward the wrong behavior. She also argues for distinguishing useful domain specialization from overfitting. She recommends testing revised instructions on examples kept separate from those used for optimization to check whether the learned rules generalize.
The team uses Alyx to debug Alyx. Colleagues who encounter a failure record its trace identifier and describe what they expected. A smaller group can then use the agent to group related problems and investigate causes, without requiring everyone who finds a bug to diagnose it.
In July 2026, DeLucia and her Arize colleagues introduced Signal, a managed agent built into Arize AX. Signal reviews production traces for recurring failures and groups them into issues with supporting evidence and proposed fixes. With a repository connected, Arize’s managed agents can carry an investigation into the code and open a pull request for human review.
DeLucia’s vision for AI operations teams gives agents the initial investigation and the selection of cases that need expert attention. People define quality, supply domain judgments, and decide what ships. In an education product, for example, an agent could select a small set of flagged responses for teachers to annotate, bringing their knowledge of what makes a good answer into the evaluation process.
▶ Watch ↗AI Engineer Europe 202616:17
Loading more talks and highlights…