[
  {
    "startMs": 12620,
    "text": "Hello, everyone. Thank you for joining me today. Uh, it's time to begin. And, uh, for years, uh, many of us, uh, monitored our applications, uh, with the golden signals, which is, was latency, error, traffic, and saturation, and shortly we called them LATS. And they were, and still, uh, they are and will be remain sufficient and critical for determine the health of our applications. However,"
  },
  {
    "startMs": 42420,
    "text": "today when we enter the new era, era of generative AI, it's becoming not enough. And today, we're gonna talk about what are the new metrics we need to look if we wanna build the healthy GenAI application. Very shortly, my name is Marina Petzel. I am GenAI advocate here in Datadog, and today I will walk you through the metrics that we need to look for. So, uh, when we talk about traditional applications, um, the"
  },
  {
    "startMs": 72460,
    "text": "main specific of them is that the same input always yield the same output, right? It's predictable. But with generative AI applications, we all know that the same prompt can generate, um, different responses every time we run it, and which means we cannot rely on the standard regression testing alone. And, uh, you must constantly evaluate quality in the live environment. The second difference between traditional applications and"
  },
  {
    "startMs": 102440,
    "text": "GenAI applications is variable cost structure. Traditional compute costs are predictable, right? Um, we pay for machine that is running our code, but for GenAI, this, this is dynamic variable here. They might, uh, they vary from the number of tokens, the model that we are using, or the size of your context window that you have. And it's become essential to do the real-time, uh, granular cores, uh,"
  },
  {
    "startMs": 132380,
    "text": "cost, um, to prevent the spending out of control. The third problem is that we face the new attack vectors. We used to worry about SQL injection or cross-site scripting before, but today we must defend against the prompt injection, jailbreaking, or PII, uh, outage in our output. And that's becoming a big problem. And these are security issue, not a simple 500 error code can--"
  },
  {
    "startMs": 162620,
    "text": "you'll never catch, and demand an entirely new forms of security monitoring. And finally, quality is very subjective for GenAI applications. The traditional systems, uh, either work or do not work, right? But generative AI output is judged on the spectrum. Is it relevant? Is it accurate? Is it complete or not? And the problem is that 200 OK response from your server doesn't necessarily mean"
  },
  {
    "startMs": 192520,
    "text": "that the answer was helpful or correct for your final user. And we must now integrate the continuous, uh, quality evaluation, uh, directly in your monitoring stack. And at the beginning, we all were thinking that these golden signals will be enough for our systems, but unfortunately, they are not. Traditional metrics will tell you if your system is up and running, but it won't tell you how well it's"
  },
  {
    "startMs": 222560,
    "text": "running. And established metrics LATS will be still essential, but we're gonna talk about what else on addition to that we need to implement into our systems. And first, I wanna start with the cost monitoring, how we can track the spending before it, uh, spirals out of control. And I think we all know the financial implications of GenAI, and a lot of companies are facing the"
  },
  {
    "startMs": 252840,
    "text": "huge outage of the, uh, cost for their monthly usage. And while the same is true as for traditional applications, there are more linear scale between the throughput and the, uh, uh, the cost. If the user loads, um, load goes up, our resources, uh, usage goes up, and sequentially the cost goes up. But the cost structure for LLM is highly dynamic and unpredictable, which means we can-- we don't"
  },
  {
    "startMs": 282880,
    "text": "have this visibility, and we need to really pay attention how we track that. And what are the problems you might encounter with the cost? And here we talk about the three major way for cost tracking, uh, and so you can track them, um, to prevent the cost outage. So the first is the token creep. Uh, this happens when engineers or product managers trying to improve the quality,"
  },
  {
    "startMs": 312760,
    "text": "quietly increasing the context window. And let's say the context window was four thousand tokens, and then it's, like, blow to thirty-two without any financial review. And what it can cost, like, the, the cost is-- depends very, uh, significantly to the context count, and that might result up to 8x cost increased. And in a month, it might cost you thousands of dollars. Second is the model drift. Uh, that is"
  },
  {
    "startMs": 342660,
    "text": "also the pretty typical problem when the team switches from cheaper and faster model, like let's say Haiku 4.5, to the more powerful and more expensive model like Opus 4.8, and which promise you obviously the better quality. But at the same time, Opus, for example, uh, can be higher, fifteen times higher, uh, per the cost request. And this change in model, even with the same usage volume,"
  },
  {
    "startMs": 373068,
    "text": "might blow your cost significantly within a month, too. And finally, it's uncached calls. In generative AI, the same query can hit the API repeatedly if, uh, you haven't implemented the effective caching layer. And what we've been seeing from, um, our, uh, own research that 70% of the spend sometimes are redundant because the system is paying for exact the same expensive model"
  },
  {
    "startMs": 403428,
    "text": "completion, uh, over and over again. And with this granular visibility into these three layers, you would be able to escalate your cost rapidly. And now let's talk about how you can prevent that and get more granular data. And when we, um, talk about that, we, uh, try to think about the implementation of cost attribution strategy, which is you must tag"
  },
  {
    "startMs": 433368,
    "text": "absolutely everything. For example, uh, tagging, uh, can help you to-- uh, why it's important, right? You receive a lot of, uh, interactions with your agents and-- but all of these, uh, interactions are undifferentiated. You don't know what's below your actual cost. That is why first layer what you can implement is the feature level, level. By tagging the intention of the feature, like it was a chat or summarization, you can"
  },
  {
    "startMs": 462788,
    "text": "attribute the cost for the specific product areas. And this data is crucial not only for engineering your cost management, but also for the product roadmap decisions, like knowing which features are actually the budget costs. The second level is the use-user level. Using the tags like the user ID or your org, um, name is essential for chargebacks, accurate billing, but also quickly identify the patterns of abusive"
  },
  {
    "startMs": 492628,
    "text": "usage before they drain your budget. Third is the model level. Uh, also tag your models like GPT-5.5 or the provider, OpenAI, Anthropic. They'll give you visibility into the higher level areas, uh, of cost optimization. And you can benchmark the true expenses, uh, of using one model versus another. And finally, uh, endpoint-level tagging."
  },
  {
    "startMs": 522908,
    "text": "Attributing cost to regions or to your production or, like, to your environment, like production or staging, helps your infrastructure planning. And you can see how regional distribution or staging versus production environment can impact your total spend. And by making this tagging mandatory practice, you gain an esse- an essential visibility to understand exactly where your money is going and how important,"
  },
  {
    "startMs": 553148,
    "text": "uh, to control it. Well, but now let's talk if, like, let's, let's imagine that we handle all the cost issues, right? We took that under control, and your finance tea-team is obviously get very happy. Uh, but the next one, uh, problem is the safety. And your AI can be cheap, but s-still leak some of the sensitive or personal information, which is not great. And even though, like, your"
  },
  {
    "startMs": 582988,
    "text": "lab scores can be perfect, uh, it cannot tell you if there are some, um, metrics went wrong. So what we can track here in the safety metrics to, um-- First is the prompt injection rate, which has become the very crucial problem. This metrics detects the manipulation attempts, uh, into your user prompts. There are if infrastructure-- if, uh, the infrastructure design doesn't have any guardrails. And"
  },
  {
    "startMs": 613128,
    "text": "this can be, um, prevent by using some advanced techniques like, uh, pattern matching or specialized classifier models. And for this, we usually need to target for very aggressive thresholds. The second is PII detection rate. Um, PII, uh, so leakage is a crucial and zero-tolerance issue here. We can scan all the model outputs, uh, for sensitive data"
  },
  {
    "startMs": 642968,
    "text": "like Social Security, like credit cards or personal information. And you can use some, um, um, methods like regex or, uh, named entity recognition here, and only acceptable target should be zero percent of, uh, your production outputs. The third is the content moderation score, how you can determine if your content is toxic, harmful, or biased, uh, towards"
  },
  {
    "startMs": 672988,
    "text": "some of the topics. And by running all outputs through toxicity classifier can gi-give you a scores, uh, how-- what is the risk of your output for being exposed to this, um, toxic, uh, um, output. And this allows you to set some clear thresholds, and we always target to, um, minimize that as much as we can. And finally, um, the jailbreak attempts, and there are"
  },
  {
    "startMs": 702528,
    "text": "persistent attempts by users to bypass the safety guardrails, and often, um, through the elaborate scenarios. And tracking p- uh, tracking patterns to indicate the system prompt overwrites, uh, you should design and become-- so it become the absolute blocker for you. So ideally, we should block all the hundred percent of the attempts of that."
  },
  {
    "startMs": 730332,
    "text": "Now let's imagine that cost is checked, security is checked, uh, but now you arrive to arguably the most crucial, um, dimension and the quality monitor that at least the most crucial for our final users. And even though your LATS metrics, um, showing that your service is up and around, uh, and running, your cost and safety tells you that your service is budget-fri-"
  },
  {
    "startMs": 759772,
    "text": "compliant and also secure, but ultimately, for any application, but more for generative application, it's whenever output is good. But we all know that 200 OK response from your server sometimes can be a lie, uh, if the answer is hallucination. And I'm sure many of you got some response from modern AI applications, right? But, like, we know that it can confidently by lie-- but lie to you."
  },
  {
    "startMs": 790792,
    "text": "And now let's understand what type of metrics you can imple- implement to your application right now to prevent it from being, uh, unsafe for your final user. We recommend to focus on these five metrics. Uh, one of them is hallucination rate. Um, this measures the percentage of response that contain claims unsupported by the grounded data. Um, we need to mix that with the manual review as well as with, uh,"
  },
  {
    "startMs": 820332,
    "text": "automated fact checks to achieve the aggressive targets here, too. The final-- uh, the next one is the relevance scores, which helps you to understand if, uh, the response actually address the user question. Um, te- techniques like y- that you can use here is similarity using BERT or embedding similarity that can score you, um, how relevant is the final answer was to the user."
  },
  {
    "startMs": 850392,
    "text": "The next one is user satisfaction, which directly pulse into your end user. Uh, you must implement-- You can implement some mechanism like thumbs up, thumbs down, or some of the stars ranking, and also NPS scores, um, to capture the subjective data, but also you should target that you got more than 80 or 85% of the positive feedback at least. Next one is the"
  },
  {
    "startMs": 880432,
    "text": "answer completeness, and this, uh, metric is often evaluated by LLM-as-a-judge, uh, answering whenever the response was fully or holistically addressed, um, the user intent. Um, and final one, the fifth one, um, if you are using RAG, retrieval-augmented, uh, generation system, you should always track the quality of your RAG. What-- If all the documents that were fetched were relevant to the user"
  },
  {
    "startMs": 910292,
    "text": "query, and here you have a lot of metrics and mechanism how you can calculate that. For example, top-K accuracy or normalized discounted gains. And that is, uh, essential to track because, uh, whenever, like, your context is exploded, it's getting harder and harder to pick the most relevant one. And by implementing these, uh, metrics, uh, you can gain some understanding of, like, more dimensional view of the quality that necessary you, you need"
  },
  {
    "startMs": 940332,
    "text": "to optimize to improve your GenAI application. Well, and now let's summarize, um, what we need, uh, to implement to make sure that our GenAI application is healthy, right? We are not saying that we need to get rid of traditional understanding. We always need to look into our metrics, LA- or LATS, latency, error, uh, traffic, and saturation. But also on top of that, we need to put another layer, layer of"
  },
  {
    "startMs": 970612,
    "text": "which is more relevant to generative applications, and in this case, we talked about the cost, safety, and also quality. And I know that for us as an engineer, it's becoming more and more critical right now to track it all, not be-- don't need just build application and make sure that it's running in production, but now we also need to care about additional layer of complexity. And centrally, Datadog has you back here since we are working"
  },
  {
    "startMs": 1000532,
    "text": "in observability and monitoring space for, for really long time. So we do have our Agent Observability, uh, product, which allows you to quickly and easily understand if your AI application is healthy, if it's running, or there, there is any problem that you need to fix. And we do have a lot of different way, um, uh, things that you can track inside of that. And, um, if you wanna hear more about that, please feel free"
  },
  {
    "startMs": 1030632,
    "text": "to see me afterwards. We also do have the booth here, uh, which you can check it out. We do have a really, uh, good, um, offers for you if you wanna just try it out and see how Agent Observability can help your application. We have some QR codes for the free trials. And, uh, with that, I wanna thank everyone. Again, my name is Marina Petzel. If you wanna contact me, please feel free to, um, use this QR code. And yeah, thank you for"
  },
  {
    "startMs": 1060672,
    "text": "being here today."
  }
]
