[
  {
    "startMs": 12600,
    "text": "Uh, what's up, guys? Thank you guys for coming to this talk. I know we're getting to the end of the day. You had lunch, you're getting tired. So really appreciate the patience. You've been hearing a lot from all of us at Arize. You just heard Sophia's talk, maybe if you were in the room earlier. So thank you again.  My name is Fuad. Uh, did a talk yesterday, if you weren't here. Uh, I work on product at Arize, so I'm what they call an AI PM, which I'll get into a little bit about, you know, kind of how that relates to shipping agents. Um, today I wanna talk about, uh, two years of shipping our"
  },
  {
    "startMs": 42680,
    "text": "in-product production agent, uh, and some of the lessons we've learned. Hopefully, there are good takeaways for you guys as you are kinda building your own agents in the wild. Cool. So before I start, we're gonna get a little bit meta here. Uh, and hopefully, you guys are all kinda following along. But, you know, this slide was made with an agent. Yes, it was. Um, we have coding agents, um, that kind of operate with Alyx, with Signal, with other parts of our product that are themselves agents. Uh, we"
  },
  {
    "startMs": 72400,
    "text": "have Signal, which is kind of the trace reading agent that reads our agent's traces, right? So we have an agent reading the agent traces. We have Alyx, which is the in-product agent. You can think of it like kinda like Cursor for AI engineering. That's the agent I'm talking about today. Uh, we actually use Alyx to debug Alyx, um, so it gets a little bit meta. Um, and then hopefully you get, you know, kind of an idea of how Alyx could help you evaluate your agent and some of the design patterns to be taken from Alyx and how that could be translated to your agent."
  },
  {
    "startMs": 102200,
    "text": "So anyways, turtles all the way down, agents all the way down. Uh, and if you are confused about which agent we're talking about, I will make sure to make that clear. So, um, what we're gonna cover is kind of like how Alyx got here. I've got some nice throwback screenshots from 2023, which feels like ages ago, about, uh, of how Alyx used to look, uh, where it used to be just a router over skills, uh, tool calling loop, um, and kind of how we've changed that. Uh, some of the hardest"
  },
  {
    "startMs": 131960,
    "text": "decisions we made along the way, so specifically who owns the plan, how to end a turn, uh, how we dealt with memory across the agent, uh, should we use one agent, should we use many sub-agents, uh, wha-what default model to choose, and sort of how we test it as well. Um, then, like I mentioned, it gets a bit meta, but we debug Alyx with Alyx. So, uh, and now we have Signal as well, so how Signal is finding Alyx's retry loops and some of the real issues we found with our agent using Signal. Uh, and"
  },
  {
    "startMs": 161880,
    "text": "then i-importantly, what, what still needs a person as well, and so where the human value kind of adds judgment and the AI PM world kinda comes in, uh, and some of the decisions we kept, um, that have lasted throughout the architecture of Alyx. Um, Alyx is the agent inside Arize AX. Alyx can read your traces, turn findings into tests, so building your evals for you, datasets, experiments. It can plan work. It can, uh, run multi-step, step tasks. It can create synthetic"
  },
  {
    "startMs": 191800,
    "text": "data for you based on your traces. Um, and it can do a lot of different things. I would say, yeah, this is a quote from one of our staff engineers. Uh, on the first prototype back in late 2023 with like 3.5-era models, it was about fifty percent hallucination, fifty percent something useful. Proud to say we're getting a bit better, um, as models have gotten better as well. Uh, this is just a little before and after of the, uh, visuals, just to give you an idea. So you can give it, you know, prompts that you've stored in Arize. You can ask it to"
  },
  {
    "startMs": 221600,
    "text": "optimize those prompts. Um, and obviously it had, you know, kind of tools to read prompts from our database, uh, and things like that. Now, uh, we have, you know, much more in-depth flow with planning, uh, with much more beautiful UI. Alyx kind of explains what it's doing. Uh, it'll reason through tasks, uh, much more strongly. So here I have like a request to optimize Alyx's own system prompt, and it actually makes a plan to check, you know, kind of resources around the application as well. Um, so yeah, this window shows up in Arize. You can"
  },
  {
    "startMs": 251640,
    "text": "use it to debug your traces. Um, how did we get here? So quick kinda like timeline. We started the prototype in late 2023. Um, this was like with 3.5-era models. Um, we then, um, you know, very quickly decided to move a lot of math out of the model and into the harness code because if you remember asking agents to do math back then, I mean, nowadays they're solving every Stokes level problems, but, uh, back then, they were not very good at math."
  },
  {
    "startMs": 281440,
    "text": "Um, then we kind of had the official launch. We called it the Copilot back then when things were still called Copilots. Uh, a router over a set of skills, a decision tree. We had some rails. Uh, we rewrote a lot of it in August 2025. Um, iterative tool calling and a to-do tool come in, um, so the agent can actually kinda plan what it's doing. Uh, Alyx 2.0 was our big relaunch in February 2026. Uh, Sonnet 4.5 became the default model, and we'll talk through some of how we"
  },
  {
    "startMs": 311420,
    "text": "test that model as well. Um, one Alyx was just released in June. So instead of specialist, uh, agents, we merged it into kind of like one agent that just loads skills on demand. Uh, and finally, memory. Um, we now have long-term memory, so as the user kind of interacts with Alyx throughout the course of their session, uh, they are kind of continuously, um, uh, the agent continuously knows, uh, sort of the history of that user, uh, and that's written in the background. Um, and"
  },
  {
    "startMs": 341560,
    "text": "obviously we didn't train these models. Everything we rebuilt was harness code, which I think, um, actually gives me a bit of faith in the world of engineering that there's still very interesting problems to solve here, and it's not all up to the model, and you can't just run a frontier model and expect it to do everything for you. Uh, models got better on their own schedule for sure, and we made some decisions around that, but Pretty much everything we rebuilt was that harness code. Uh, we don't use an agent framework. Um, the loop, the plan, the stopping rule, and the memory are all"
  },
  {
    "startMs": 371304,
    "text": "ours. Um, what that means is we have a lot of control over it, um, and, you know, we get to learn a lot along the way as well."
  },
  {
    "startMs": 379624,
    "text": "Uh, so first one, planning and to-do lists. Um, the analogy, I, I have a bunch of stuff on the slides, but the analogy I kind of think about around, you know, planning is like, uh, let's say you... I don't, I don't have kids. Maybe some of you guys have kids, and you can kind of relate to this analogy. But let's say you want, you know, your kid to do some chores. Um, you can give it a list of chores and ask it to, uh, or ask them to check off the lists of chores as they go. Um, you can check in with"
  },
  {
    "startMs": 409584,
    "text": "them continuously and hold the list yourself and not tell them about the list except for the next task, right? So that's one paradigm. Um, and there are trade-offs to kind of like each, uh, scenario. We initially just gave the model a to-do list, and what we saw was it spent most of its time checking off the to-do list. Uh, so a lot of LLM calls for, um, pretty much nothing, like no useful unit of work other than updating a to-do list. It als- we also realized"
  },
  {
    "startMs": 439204,
    "text": "it tended to lose focus after multiple tool calls. Um, and yeah, 20, 20-plus iterations on the to-do tools themselves. So we added this as an extra tool in its arsenal. And, uh, we eventually moved the plan into the harness one piece at a time. So what I mean by that is, first thing we did was we added an in progress so the, uh, the model wouldn't lose track of where it was. Um, then we decided to put the plan as a message, so the harness renders the plan to the model call every"
  },
  {
    "startMs": 469264,
    "text": "time, um, the model gets a new call and the rules are next to the data, so it's a little bit easier for the model to interpret where it's at. Uh, we actually ended up dropping the in progress identifier because we realized we're just making two calls for effectively, you know, anything that isn't finished is in, in a stack ranked order is the next thing to do, so we removed that. Um, and we're actually testing now without a to-do tool at all, uh, just keeping everything in line in the messages, uh, completely. So the harness is keeping the state and the completion check,"
  },
  {
    "startMs": 499564,
    "text": "and the model just consumes that and goes and does the next thing. Um, so maybe some lessons for you guys as you're building to-dos, uh, models can be very eager with, uh, lists. Um, then kind of like what ends the turn, uh, finishing became a gate the harness enforces. Um, so what we at first tried was, you know, kinda text without tool calls, uh, meant done. That broke on models that announce a step in one turn and"
  },
  {
    "startMs": 529504,
    "text": "make the call in the next, which, um, you know, as you change models, you'll, you'll quickly find some of these breaking things. And one of the things we do in Alyx is we actually allow users to choose their own provider, uh, so we can't necessarily guarantee that we've tested every single model someone will try with Alyx. So a lot of, um, you know, some of the decisions we made here are to guard against those. And yes, we run regression tests, and we check against a bunch of different models, and we benchmark different models on Alyx. We have an opinion on a default, but"
  },
  {
    "startMs": 559504,
    "text": "you know, obviously, some of our customers are exclusive Anthropic customers or exclusive OpenAI customers. So to actually empower them to use their own models, we kinda had to design against everything that was available, right? Um, so now we have an explicit finish call that bounces back, uh, when plan items are open up to three times, uh, a warning set one step before the iteration limit and then a force finish as well. Um, memory, this is a good one. Um, there are actually a number of problems here, and I'm-- I, I like analogies, so I'm gonna use"
  },
  {
    "startMs": 589424,
    "text": "another analogy here. It's like, you know, you're, you're working at your desk, um, and a big book lands on your desk, and maybe one big book is fine. You can open the book, kind of search through the book. Uh, if the book gets too long, then it's kinda useless now 'cause you're gonna spend your entire time reading that book. That's one issue. Uh, and that's sort of like, you know, one big tool result comes back, and it's too big for you to read, right? Um, but you can imagine there's actually multiple books on your desk, right? So eventually, not"
  },
  {
    "startMs": 619284,
    "text": "just are the books too long for you to read on your desk, but there's no space left on your desk, right? You just have too many books, right? So there's a, you know, you know, in this analogy, um, there's a lot of ways you can approach this, right? You can decide to store that book and just have a pointer to the book, right? So that's kind of what we did with the tool results where, okay, I now have a sticky note on my desk that says where the book is and what, you know, kind of high level the book contains. And so if I need to go"
  },
  {
    "startMs": 649164,
    "text": "read that book, I'll just reference my sticky note and then go find the book on the shelf. My desk clears up a little bit, right? So that's kind of what we do with tool calls, uh, and tool results. For example, like Alyx's, uh, you know, can help you with trace analysis. Some of these traces are other agents' traces, and they can have a shit ton of stuff in the trace, right? So, um, if it's a big JSON blob, for example, that just blows up our context window, and then you can pretty much do no useful work. So instead, what we do is we truncate within, uh, the JSON, and we just let the"
  },
  {
    "startMs": 679104,
    "text": "agent code to go find it if it needs the full context with jq. Um, one thing the agents are very, very good at is writing code and using code-like tools to, uh, kind of infer things. And so that's kind of what we try to leverage with that. Um, the agent's own trajectory also going past a, a certain, uh, window. Um, so what we now have is the harness makes the model write a checkpoint, uh, and then drops what it replaced. Uh, we have fixed token budget for"
  },
  {
    "startMs": 708644,
    "text": "history, um, and across sessions, so I kinda got to like the long-term memory piece. Uh, basically, we just ask them a background job to edit one small file per user. I'll dive a little bit more into like why we decided to do this with, with long-term memory, uh, later in the presentation, but essentially that's how it works. Um- Kind of talked through this. So treat a huge result like a file and just grep it. Uh, don't dump it all into context. Don't worry about doing anything fancy."
  },
  {
    "startMs": 739180,
    "text": "Uh, for long-term memory, a small file actually beat a knowledge graph, and we, we tried things like vector retrieval, episodic and semantic stores, uh, temporal knowledge graphs, et cetera, et cetera. What we eventually landed on was we just have one file per user. It's a markdown file. It's capped at eight thousand characters. Um, and, you know, I, I think there, there is a lesson here that, you know, as engineers, um, we like to build things, and it's really fun to build things, and sometimes we get in our own way, and we"
  },
  {
    "startMs": 768600,
    "text": "over-engineer. Um, and this file is just loading in on every full request. Uh, it's written into in a background job after the session finishes. It's small, it's exact, it's easy to load in, it doesn't blow up context, and it's very easy for a human to audit as well. Uh, so you can actually kind of have a human labeler go look at these memory files and, and determine whether the right things are being stored and kind of edit instructions for the background job, so. Um, and then you also only have one"
  },
  {
    "startMs": 798640,
    "text": "job at the very end, which, uh, you can make very safe as well. Uh, so you're, you're making sure you're not writing memory from other users into the file. Um, so yeah, very good lesson on not over-engineering, uh, your problems. Um, briefly on specialists, um, you know, we had one trace agent that initially just did trace analysis back in twenty twenty-five. Then we had ten specialists, so every page kinda had its own specialist. If you're on the dataset page, there's, you know, a special sub-agent that can"
  },
  {
    "startMs": 828360,
    "text": "generate synthetic data for you and look, look and introspect on all, all your datasets. Uh, we ended up kind of rolling that back as well and just using one agent where the agent can load skill instructions on demand and tools on demand, and we just let the agent decide what tools to do, right? So it is a bit of an evolution also in, in model capability, but I think simplification as well. Um, we-- O-on model capability, we also kinda stand by that the newest"
  },
  {
    "startMs": 858180,
    "text": "model isn't always the best by default. Um, there are times where we upgrade the model by default, right? And so we have a perspective on that. Um, I think we have, um, you know, uh, what is the best model is a very subjective question. Uh, what is the best model on a generalized frontier is very different from what may be best for you guys in what you're building or what we're building. Uh, we generally just default to fast and good enough and spend a lot of time on the harness instead. Um, we are now"
  },
  {
    "startMs": 888120,
    "text": "starting to explore auto-routing, um, so choosing models per task and training our own models or small post-training SLMs, things like that, on very narrow surfaces. Um, and some of these services actually don't even need the loop at all. You can just make a single model call. So an example of that is-- I wonder if I can pull it up. We have this, like, you know, filter with AI. Um, we've come a long way from SQL generation in the, in the past, and it's actually very, very easy to generate filters with very, very small models now. So,"
  },
  {
    "startMs": 918520,
    "text": "um, that's an example where maybe we don't need the entire loop and we're just over-engineering again. Um,"
  },
  {
    "startMs": 926600,
    "text": "cool. So that's a good takeaway. Um, production sessions becoming our test. So this is where I'll talk a little bit more about Arize as a platform. Uh, tracing everything is really important, right? So we trace-- Actually, if you look at this, these are the traces from this agent. Uh, so it's very meta here. Um, but every time you ask Alyx a quest-question here, the, the traces, these are the traces from Alyx. We, we still call it Copilot Prod. Um, so those are our live traces coming in from the tool. Um, this"
  },
  {
    "startMs": 956680,
    "text": "makes it very easy to add, you know, regression datasets and, and keep track of different user queries that you wanna go test against and iterate against. Um, and so those production sessions become our test cases and become our living fixtures over time. They grow, they change, um, as we see new error cases. Um, and we also now have the ability to do session datasets as well. So you can store an entire session as a dataset and kind of play a specific turn through the agent if you need. Um, finally, we use Arize Experiments"
  },
  {
    "startMs": 986620,
    "text": "throughout all of this as well to make sure that we're, you know, kinda A/B testing, uh, harness changes as we go through this. Uh, one takeaway I want you guys to, to kind of keep in mind also is that in agents, bugs rarely show up as explicit errors. Um, in the old world software, this was a lot. I, I don't know if it was easier, but it was different. Um, now it's more of like a qualitative, okay, the status says okay, but if there are forty-three identical tool calls in the same turn, that's"
  },
  {
    "startMs": 1016360,
    "text": "probably something we wanna fix. Um, softer metrics like latency, right? So I think we can all agree two hundred and twenty-seven seconds is way too long to wait for a response, but what actually is too r- too long, it really depends on your use case. Um, so we use Signal a lot now to kind of automate this review. We, we use less and less of the human reading the trace. Um, our staff engineers are very grateful for that. Uh, but we still do use kind of that human expertise, which I'll get to in a bit. Um,"
  },
  {
    "startMs": 1047120,
    "text": "the fix can also be the next bug, so these regression datasets are, are really important, and I would encourage you guys to keep them in mind. Judges need that human baseline, so that's where we kind of impart the human task. We align our evaluators, and we make sure that our judges actually accurately reflect what, you know, our engineers think as well. Uh, so that's very important. Um, I think we, we still keep a lot of things with the human. Those gold labels, those golden datasets, uh, for judges come from people. Um,"
  },
  {
    "startMs": 1077400,
    "text": "the speed versus quality is a human judgment call. I don't think there's any single answer across the stack on when you should be a little bit faster, but it's okay to degrade quality. It is still, you know, kind of like a human judgment call, even if you have the metrics around it, like, okay, ch-changing models will increase latency by X and decrease, uh, or increase quality response by Y. If we just use Astra everywhere, would that be the right decision? Still a human judgment call, I think. We have the metrics to support that, but, uh, we still have to make"
  },
  {
    "startMs": 1107400,
    "text": "those decisions. Um,"
  },
  {
    "startMs": 1111520,
    "text": "and these are kind of the, the six things that survived all these rebuilds. Uh, tracing into our own products, so we s- of course, still trace everything that Alyx does. Uh, math and data in code, so code computes the aggregations, the model reads them and explains. Uh, and then exposing those as tools to the, to the model. So we're not giving the model plain text math questions. We're actually asking it to use code to go do that. Uh, golden datasets in CI, errors with an obvious fix, streaming and persistent, and skills. Um,"
  },
  {
    "startMs": 1142080,
    "text": "we've changed what we call skills. I think, you know, routers, tools, et cetera, but really, it's give the model a capability around, um, an external system. Uh, so a lot of decisions you'll face, uh, and here's some of what we chose. So feel free to take a picture of the slide. Uh, who owns the plan? What ends a turn? Um, if your tool output is too big for your context, what do you do? Who should write the long-term memory? Do you-- Should you have one? Should you have multiple sub-agents? What model is good enough"
  },
  {
    "startMs": 1172140,
    "text": "and fast enough? Um, and who checks the agent's own fixes? Um, so yeah, that's me. Uh, on the bottom right of the screen there, my LinkedIn and X. Uh, the tagline is, \"Every day we use Alyx to build Alyx.\" We have a bunch of blog posts on sort of like technical deep dives into how we built this. Um, and yeah, encourage you guys to all reach, reach out, and a big shout out to the Alyx team and our engineers for building this. Thank"
  },
  {
    "startMs": 1198080,
    "text": "you."
  }
]
