← All speakers

Sally-Ann DeLucia

Conference affiliation: Arize

  • Alyx ↗

    The organizer explicitly identifies DeLucia as a core contributor to Alyx, Arize’s AI engineering agent, and credits her with helping evolve the earlier Copilot through her product work at Arize. Her attributed contributions concern tools for investigating model behavior, planning work, managing context, and improving agents from evidence of failures. This is a contribution to an employer’s product; the evidence does not establish sole creation or personal ownership.

    arize.com · Source ↗
  • Introducing Arize Copilot ↗

    Authored by DeLucia in July 2024. Introduces Arize’s assistant for model-performance investigation, prompt optimization, custom evaluation creation, and natural-language data search and curation. The article establishes her authorship of the announcement and describes the team’s product capabilities.

    arize.com
  • Meet Alyx: Arize’s Evolving AI Agent ↗

    Authored by DeLucia in July 2025. Explains how Copilot’s isolated answers and generic next steps motivated its evolution into Alyx, with users supplying relevant traces, evaluation explanations, and datasets. Planning, retained state, and sustained progress toward outcomes are presented as development goals that the team had not yet fully achieved.

    arize.com
  • Alyx 2.0: The AI Agent That Actually Plans ↗

    February 2026 release announcement coauthored with Jack Zhou, Chris Cooning, and Aman Khan. Describes Alyx planning and executing work across Arize AX, including creating datasets, comparing prompt variants, building evaluations, and investigating traces. The announced workflows include asking for approval at relevant decision points.

    arize.com
  • How to Build Planning Into Your Agent (The Architecture That Actually Works) ↗

    March 2026 technical article coauthored with Chris Cooning, Priyan Jindal, and Jack Zhou. Documents Alyx stopping after the first part of a request and the team’s response: structured planning tools, four task statuses, a current plan placed after the system prompt on every iteration, and a completion gate. It also explains blocked tasks, selective use of planning, and user redirection.

    arize.com
  • Managing Memory in AI Agents: Beyond the Context Window ↗

    March 2026 technical article coauthored with Chris Cooning, Priyan Jindal, and Jack Zhou. Describes retaining full trace and tool data outside the immediate context, exposing previews and stable identifiers, retrieving targeted portions, and isolating intensive searches in subagents. It reports failures from aggressive truncation and generated summaries and explains evaluating an eleventh conversation turn after preloading ten earlier turns.

    arize.com
  • How Arize tests AI agents with traces, evals, and CI/CD ↗

    May 2026 article coauthored with Chris Cooning, Priyan Jindal, and Jack Zhou. Explains converting production failures into regression cases containing the original inputs, context, and expected behavior. Covers explanatory model judges, deterministic checks, experiments over time, automated checks for changes to prompts and tools, and testing model upgrades for behavioral regressions.

    arize.com
  • How we use Alyx to build Alyx: How to build an AI agent feedback loop ↗

    May 2026 article coauthored with Chris Cooning, Priyan Jindal, and Jack Zhou. Describes using Alyx to search dense traces, aggregate metrics, categorize errors, create evaluation datasets, and investigate failures in Alyx itself. Colleagues report trace identifiers and expected behavior, allowing a smaller group to conduct investigations with the agent.

    arize.com

SallyAnn DeLucia is head of product at Arize AI and a core contributor to Alyx, the company’s agent for AI engineering. Her work focuses on the gap between seeing an AI system fail and knowing what to change.

On this page

From data science to product leadership

DeLucia began her career in data science. She earned a master’s degree in applied data science. At Arize, she moved from machine-learning solutions engineering into product leadership.

In 2023, the Lost in the Middle paper challenged her assumption that giving a language model more context would improve its answers. She revisited applications she had built and emphasized inspecting retrieved information, trying fewer documents, and moving relevant passages toward the start of the input.

In July 2024, she introduced Arize Copilot, an assistant that helped users inspect model behavior, improve prompts, and build evaluations. A year later, she offered a candid assessment of its limits: a useful chart or explanation often led to generic advice, leaving users to finish the job. That gap shaped Copilot’s evolution into Alyx. Users could supply relevant traces—records of an application’s execution—along with evaluation results or datasets. The team’s ambition was to give those interactions continuity, with planning and retained state that would let the agent pursue an outcome across several steps.

Making Alyx follow through

Alyx 2.0, released in February 2026, could plan and execute work across Arize AX, the company’s platform for inspecting and evaluating AI applications. An engineer could ask it to compare two prompts, and Alyx would organize the experiment, request approval when needed, run it, and analyze the results. In technical accounts coauthored with Chris Cooning, Priyan Jindal, and Jack Zhou, DeLucia describes how the team addressed recurring failures:

  • Plans the system can enforce. Early Alyx could sort model calls by latency and then stop before investigating the slowdown. The team’s planning architecture stores tasks and their statuses as structured data and places the current plan immediately after the system instructions on each iteration. A completion check prevents the agent from finishing while planned tasks remain pending or in progress. Work can be marked blocked when information or user input is missing, and users can redirect the plan as their needs change.
  • Memory the agent can retrieve. Aggressive truncation broke follow-up questions, while generated summaries dropped important details unpredictably. The team developed memory tools that preserve access to the full data: large trace records and tool outputs are stored outside the model’s immediate input, and Alyx receives previews and identifiers it can use to retrieve the portions it needs. Search subagents process data in separate contexts and return their findings to the main conversation. To test continuity, the team can preload ten conversation turns and evaluate the eleventh. DeLucia distinguishes this session history from memory across separate sessions, which poses a different challenge.
  • Regression tests from real failures. DeLucia and her colleagues turn production traces into test cases, pairing the original input and context with a plain-language description of the expected behavior. Model-based judges assess whether the agent met that expectation and explain failures; code-based checks handle requirements such as output formats or tool-call sequences. The suite runs when prompts, tools, or agent logic change, and the team also tests model upgrades before release. They learned that a more capable model could still disrupt established tool calls and response formats.

In her prompt-learning work with Fuad Ali, DeLucia advocates revising agent instructions using failed examples, evaluator explanations, and feedback from domain experts. The evaluator needs its own improvement cycle: unreliable judgments can steer prompt revisions toward the wrong behavior. She also argues for distinguishing useful domain specialization from overfitting. She recommends testing revised instructions on examples kept separate from those used for optimization to check whether the learned rules generalize.

The team uses Alyx to debug Alyx. Colleagues who encounter a failure record its trace identifier and describe what they expected. A smaller group can then use the agent to group related problems and investigate causes, without requiring everyone who finds a bug to diagnose it.

From debugging to ongoing AI operations

In July 2026, DeLucia and her Arize colleagues introduced Signal, a managed agent built into Arize AX. Signal reviews production traces for recurring failures and groups them into issues with supporting evidence and proposed fixes. With a repository connected, Arize’s managed agents can carry an investigation into the code and open a pull request for human review.

DeLucia’s vision for AI operations teams gives agents the initial investigation and the selection of cases that need expert attention. People define quality, supply domain judgments, and decide what ships. In an education product, for example, an agent could select a small set of flagged responses for teachers to annotate, bringing their knowledge of what makes a good answer into the evaluation process.

Read the topics behind these talks

3 conference talks

Loading more talks and highlights…

References