# Your LLM App Returned 200 OK. It Was Still Wrong. — Marina Petzel, Datadog

Marina Petzel, Datadog

[Talk page](https://ai.engineer/talks/rTojoVotlD8-your-llm-app-returned-200-ok-it) · [Video](https://www.youtube.com/watch?v=rTojoVotlD8)

## Your LLM App Returned 200 OK. It Was Still Wrong.

Marina Petzel explains why latency, errors, traffic and saturation need company: cost attribution, safety checks and continuous quality evaluation reveal failures that a successful HTTP response cannot.

![Generative AI adds nondeterministic behavior, variable costs, new attack vectors and subjective quality to application monitoring.](https://ai.engineer/api/talk-essay-media/69c95a550e2298a0b7538beea35e813d4e13478a26bd466b0da9ad770ff8f72d)

Generative AI adds nondeterministic behavior, variable costs, new attack vectors and subjective quality to application monitoring.

### A healthy server can deliver an unhealthy answer

[0:12](https://www.youtube.com/watch?v=rTojoVotlD8&t=12s)

A generative AI application can return `200 OK` while giving its user an irrelevant, incomplete or incorrect answer. The request succeeded at the server level; the response failed at the task level. Marina Petzel, introducing herself as a GenAI advocate at Datadog, starts with the familiar golden signals—latency, errors, traffic and saturation, or LATS—and asks what else production monitoring must measure to catch that difference.

The first change is variability. The same prompt can produce different responses across runs, so a successful regression test does not establish how every live response will behave. Regression testing still matters, but quality also needs continuous evaluation in the production environment. The system must be observed while it answers real questions, rather than treated as correct once a fixed set of examples passes.

Three other changes widen the monitoring problem:

- **Variable cost:** Token counts, model selection and context size affect spending. Request volume alone cannot explain the bill.
- **New attack vectors:** Prompt injection, jailbreaks and exposure of personally identifiable information can occur without producing a server error.
- **Graded quality:** Relevance, accuracy and completeness lie on a spectrum. A response can be technically delivered yet only partly useful.

These changes do not retire LATS. They add questions that LATS was never meant to answer: Is this response affordable? Is it safe? Does it help the user? Petzel develops those questions in that order, beginning with costs that can grow even when traffic looks ordinary.

### Three ways spending grows without more users

[4:12](https://www.youtube.com/watch?v=rTojoVotlD8&t=252s)

In the simplified traditional-compute picture, more traffic requires more resources, and spending rises with resource use. LLM spending has additional moving parts inside each request. A product change can increase the amount of context, select a more expensive model or repeat work that could have been reused. Monitoring the total bill reveals the consequence; monitoring these drivers helps explain it.

![Token creep, model changes and uncached calls can raise costs; the displayed multipliers are the speaker’s examples.](https://ai.engineer/api/talk-essay-media/29ef755d4a63b952db4136869cbf494de8c6811ca084a2d236d7324cc402aeed)

Token creep, model changes and uncached calls can raise costs; the displayed multipliers are the speaker’s examples.

**Token creep** begins with an apparently helpful quality change. Engineers or product managers expand the context supplied to the model without a financial review. Petzel’s example grows from 4,000 tokens to 32,000—eight times as much context—and warns of a potential eightfold cost increase, accumulating to thousands of dollars over a month. The useful comparison is between context actually sent and its cost: a larger permitted context window does not, by itself, establish that every request uses all of it.

**Model drift**, as used here, means a team switches from a cheaper, faster model to a more powerful, more expensive one. The motivation is better output quality, but the financial effect persists even at unchanged usage volume. Petzel illustrates this with a model that could cost fifteen times more per request. That multiplier is an example of the risk, not a verified price comparison: the decision to upgrade needs visibility into both the added expense and the quality benefit it is supposed to buy.

**Uncached calls** pay for the same expensive completion repeatedly. The same query reaches the API again because an effective caching layer is missing, so repeated demand becomes repeated model work. Petzel reports that, in Datadog’s own research, redundant work sometimes accounted for 70% of spending; the recording does not specify the workloads or measurement method, so that figure should not be read as an expected saving for every application. The mechanism to investigate is concrete: which repeated calls are generating a fresh bill instead of reusing an appropriate result?

### Tag requests so the bill can answer a question

[6:43](https://www.youtube.com/watch?v=rTojoVotlD8&t=403s)

A stream of undifferentiated agent interactions does not tell a team where its money went. Cost attribution adds identifying dimensions to those interactions so spending can be grouped by the product behavior, customer, model or deployment that produced it. Petzel’s deliberately broad instruction is to “tag absolutely everything”; the practical substance is four complementary views of the same spending.

![Attribution by feature, user, model and endpoint connects spending to product, billing and infrastructure decisions.](https://ai.engineer/api/talk-essay-media/4fb308508ddfb3774a966f36ffc49d2f2fd588f8e28b186c3ecbc548721a36c8)

Attribution by feature, user, model and endpoint connects spending to product, billing and infrastructure decisions.

- **Feature level:** Identify the intention of the interaction, such as chat or summarization. This assigns costs to product areas and gives roadmap discussions a financial basis: which features consume the budget?
- **User or organization level:** Attach a user ID or organization identifier. This supports chargebacks and accurate billing, while making unusually expensive or abusive usage easier to locate before it drains the budget.
- **Model and provider level:** Record which model and provider handled the work. This makes actual spending across alternatives comparable and helps locate expensive model choices.
- **Endpoint and deployment level:** Attribute costs to regions and environments such as production or staging. This reveals how deployment choices and nonproduction activity contribute to total spend.

Making tagging mandatory turns cost monitoring into a way to investigate decisions, rather than merely notice an overrun. Feature tags serve product planning; user tags serve billing and abuse investigation; model tags serve comparison; deployment tags serve infrastructure planning. But an inexpensive application can still expose sensitive information. Getting the finance team comfortable leaves the next problem untouched.

### Measure attacks and unsafe output separately

[9:13](https://www.youtube.com/watch?v=rTojoVotlD8&t=553s)

Safety monitoring has to inspect the content and behavior that ordinary status codes miss. Good laboratory scores do not tell a team whether a live prompt is trying to manipulate the application or whether a delivered answer contains personal information. Petzel separates four signals, with different detection methods and different desired outcomes.

![Safety monitoring tracks prompt injection, sensitive data, content moderation and jailbreaks; the displayed thresholds are proposed targets.](https://ai.engineer/api/talk-essay-media/a547296589e68cd8f653bd18816a9881262f76794878e9a5b10e3931890bdb4a)

Safety monitoring tracks prompt injection, sensitive data, content moderation and jailbreaks; the displayed thresholds are proposed targets.

- **Prompt injection rate:** Detect manipulation attempts in user prompts. Pattern matching and specialized classifier models are suggested ways to recognize suspicious inputs. The slide gives an example target of fewer than 0.5% of requests flagged for prompt injection. Guardrails determine what the application does with a flagged request.

- **PII detection rate:** Scan model outputs for sensitive information, including Social Security numbers, credit-card information and other personal data. Regular expressions can detect recognizable patterns, while named entity recognition can identify entities in text. Petzel sets a zero-tolerance objective: sensitive-data leakage should occur in 0% of production outputs.

- **Content moderation score:** Run outputs through a toxicity classifier to score the risk of toxic, harmful or biased content. A score allows the application to apply a threshold rather than rely on a binary server-success signal. The slide gives an example target of “< 0.3% flagged for review.” It leaves the denominator and the classifier score that triggers review unspecified.

- **Jailbreak attempts:** Track persistent efforts to bypass safety guardrails, including elaborate scenarios intended to overwrite the system prompt. The intended response is blocking, with an ideal target of stopping 100% of attempts.

The zero-leakage and complete-blocking figures are safety objectives, not demonstrated detector performance. That distinction matters operationally: recognizing a suspicious prompt, finding sensitive text in an output and preventing its delivery are different jobs. A detected problem needs an enforcement response; merely counting it does not make the interaction safe.

### Five questions a successful response still has to pass

[12:10](https://www.youtube.com/watch?v=rTojoVotlD8&t=730s)

Now suppose the service is running, spending is controlled and safety checks are in place. The user still needs a good answer. A confident hallucination can satisfy the server’s success conditions while failing the user completely. Quality monitoring therefore needs several measurements that distinguish factual support, relevance, satisfaction, completeness and the quality of retrieved context.

- **Hallucination rate: Are the claims supported?** Measure the percentage of responses containing claims unsupported by the grounding data. Petzel recommends combining manual review with automated fact checks. This evaluates the relationship between an answer and its evidence, rather than treating fluent or confident wording as a sign of correctness.

- **Relevance score: Does the answer address the question?** BERT-based similarity or embedding similarity can score the relationship between the user’s question and the response. This asks a different question from factual support: an answer may discuss the right subject yet contain unsupported claims, so a relevance score does not replace the hallucination check.

- **User satisfaction: Did the user find it useful?** Thumbs-up and thumbs-down controls, star ratings and NPS provide direct feedback on the experience. Petzel suggests aiming for more than 80–85% positive feedback. That is a recommended target for positive responses, not a measured result or an interchangeable interpretation of the NPS scale. User feedback supplies subjective information that content-based evaluation alone cannot capture.

- **Answer completeness: Did it cover the full intent?** An LLM-as-a-judge can assess whether the response fully addresses what the user wanted. Relevance checks whether the answer addresses the question; completeness asks whether it leaves important parts unanswered. Separating them makes a partially helpful answer easier to diagnose.

- **RAG retrieval quality: Did the system fetch useful evidence?** In retrieval-augmented generation, evaluation must also examine the documents fetched for the user’s query. Petzel names top-K accuracy and normalized discounted gain measures as options. These evaluate the retrieval side of the system, rather than only the final response. As the pool of context grows, selecting the most relevant material becomes harder; checking retrieved documents helps distinguish a context-selection problem from a problem in the answer produced from that context.

Together, these measurements make quality more diagnosable. Unsupported claims call for examining grounding; a poorly matched answer calls for examining relevance; missing coverage calls for examining completeness; unhelpful retrieved documents call for examining retrieval. User satisfaction adds the final user’s judgment. No one score answers all five questions.

### Keep LATS and add the missing dimensions

[15:40](https://www.youtube.com/watch?v=rTojoVotlD8&t=940s)

The closing recommendation is additive: retain latency, errors, traffic and saturation, then monitor cost, safety and quality alongside them. These dimensions are complementary, not successive gates that can be checked once and forgotten. A running service can become too expensive; an affordable service can leak information; a safe service can still answer badly.

![Traditional operational monitoring is combined with cost, safety and answer-quality signals.](https://ai.engineer/api/talk-essay-media/10dcb836b36ea7113c9c4cab46bdaa32435788a71ae7d78981156e34ad718973)

Traditional operational monitoring is combined with cost, safety and answer-quality signals.

Petzel ends by presenting [Datadog Agent Observability](https://www.datadoghq.com/products/ai/agent-observability/) as a product for investigating AI application health and identifying problems to fix, with an invitation to try it. The [LLM Observability documentation](https://docs.datadoghq.com/llm_observability/) offers a practical next step for instrumenting that approach. The engineering responsibility established by the talk remains the same whichever tool carries it: production monitoring must evaluate what the application delivered, not stop at whether it delivered something.

## Key takeaways

- LATS remains essential for operational health, but successful requests also need live evaluation for cost, safety and answer quality.
- Context growth, more expensive model choices and repeated uncached completions can raise spending without an increase in user traffic.
- Tag interactions by feature, user or organization, model and provider, and deployment location or environment so costs can be traced to useful decisions.
- Safety monitoring distinguishes manipulation attempts from unsafe outputs. Zero PII leakage and blocking every jailbreak are objectives, not evidence that detectors achieve perfect coverage.
- Evaluate factual support, relevance, user satisfaction, completeness and RAG retrieval quality separately; each identifies a different way an answer can fail.

## Resources

- [Datadog LLM Observability documentation](https://docs.datadoghq.com/llm_observability/) — Explains instrumentation, request traces and spans, operational monitoring, and evaluations for quality, privacy and safety—the implementation counterpart to the talk’s monitoring dimensions.
- [Datadog Agent Observability](https://www.datadoghq.com/products/ai/agent-observability/) — The product introduced at the end of the recording; provides an overview of tracing, evaluations and production monitoring, with options to explore or try it.

## Related talks

- [Build a Platform and Watch It Burn — Michael Forrester, Accenture & Whitney Lee, Datadog](https://ai.engineer/talks/1tpqdglI3T0-build-platform-watch-it-burn-michael-forrester) — Develops the safety discussion through concrete prompt-injection, input-blocking and output-redaction demonstrations, including failed protections and the difference between detection and enforcement.
- [Is a Faster Model Actually Faster? Test the Whole Job — Morgan Suo, MiniMax](https://ai.engineer/talks/YvQ4BVTWfHI-is-faster-model-actually-faster-test-whole) — Extends the cost–quality tradeoff into model-switch evaluation: measure the time and total expense of producing an acceptable result, including failed attempts and human repair.

## Slides

### Beyond Golden Signals

Latency, errors, traffic, and saturation remain critical, but no longer suffice for GenAI. Petzel introduces the need for additional metrics to assess application health.

[0:17](https://www.youtube.com/watch?v=rTojoVotlD8&t=17s) [Datadog title slide reading “Beyond Golden Signals” with the subtitle “Monitoring in the Age of GenAI” on a purple geometric background.](https://assets.ai.engineer/sites/aiecode2025/sha256/63268fabf4f4e4c27e7181b9df3cf573e8d61fd1927914678bde78685d7e7c3a.webp)

Beyond Golden Signals

Monitoring in the Age of GenAI

DATADOG

### Marina Petzel — Sr. GenAI Advocate

Marina Petzel, Sr. GenAI Advocate at Datadog, introduces her talk on the additional metrics needed to assess GenAI application health.

[0:56](https://www.youtube.com/watch?v=rTojoVotlD8&t=56s) [Speaker introduction slide with a circular portrait, the name Marina Petzel, the role Sr. GenAI Advocate, and Datadog branding.](https://assets.ai.engineer/sites/aiecode2025/sha256/d74067dacc6d4d119ae6af5c08589eb2b82d28bf9cb629847d4db5d767972bf2.webp)

Marina Petzel
Sr. GenAI Advocate

2
DATADOG

### Four GenAI Monitoring Challenges

A GenAI app can return 200 OK with a wrong or unhelpful answer. Monitoring must track variable outputs and costs, detect new attacks, and continuously evaluate subjective answer quality.

[2:47](https://www.youtube.com/watch?v=rTojoVotlD8&t=167s) [Four colored hexagonal icons in a two-by-two layout labeled Non-Deterministic Behavior, Variable Cost Structure, New Attack Vectors, and Quality is Subjective.](https://assets.ai.engineer/sites/aiecode2025/sha256/09a354617c7e5ffcc6e3c34b4abde8cd6a4c6918c1f355a67855124bc95c528c.webp)

Non-Deterministic Behavior
Variable Cost Structure
New Attack Vectors
Quality is Subjective

DATADOG
3

### Running Versus Running Well

Traditional metrics show whether a system is running. Petzel calls for continuous quality evaluation alongside those metrics to assess whether LLM answers are helpful and correct.

[3:33](https://www.youtube.com/watch?v=rTojoVotlD8&t=213s) [White text on a blue-purple background states that traditional metrics tell you whether a system is running, but not whether it is running well.](https://assets.ai.engineer/sites/aiecode2025/sha256/b6b1314408f1caa0d713bdd557d4733c803d9e77bd3ce5d7c50c8fa8ca37f3ae.webp)

“Traditional metrics tell you if the system is running, but not if it’s running well.”

DATADOG

### Cost Monitoring

Petzel introduces cost monitoring alongside established system metrics to track GenAI spending before it spirals out of control.

[3:53](https://www.youtube.com/watch?v=rTojoVotlD8&t=233s) [Cost Monitoring section slide with the subtitle: Track spending before it spirals out of control.](https://assets.ai.engineer/sites/aiecode2025/sha256/600456a44887c0c5ffa1f633318126dcc9c9d9674015980719c1c7353d6c414b.webp)

Cost Monitoring

Track spending before it spirals out of control

DATADOG 5

### Cloud Cost Spending Trends

Amortized daily usage spending rises from January into March. March 1–25 spending exceeds February’s entire monthly total; the dashboard flags March’s daily average as 203.7% higher than the previous two months.

[4:03](https://www.youtube.com/watch?v=rTojoVotlD8&t=243s) [Cloud cost dashboard with January, February and March spending totals and stacked daily spending bars. An annotation reports that March's daily average is higher than the previous two months.](https://assets.ai.engineer/sites/aiecode2025/sha256/c7fb0b5c3435c651f5d271d26d9762de4d8096141de760b97f331ec7d7553231.webp)

[illegible]DERSTAND SPEND TRENDS

March daily average is $149.85 higher (+203.7%) than previous 2 months

JANUARY TOTAL SPEND
$631.81

FEBRUARY TOTAL SPEND
$3,709.48

MARCH 1–25 TOTAL SPEND
$5,154.07

Amortized Daily Spend
Usage Charges Only

JANUARY
Daily Avg $20.38

FEBRUARY
Daily Avg $132.48

MARCH
Daily Avg $223.43

Y-axis ticks: $0, $100, $200, $300, $400, $500
Timeline: Jan 1, Jan 5, Jan 9, Jan 13, Jan 17, Jan 21, Jan 25, Jan 29, Feb 2, Feb 6, Feb 10, Feb 14, Feb 18, Feb 22, Feb 26, Mar 2, Mar 6, Mar 10, Mar 14, Mar 18, Mar 22, Mar 26, Mar 30
Legend: aws, azure, datadog

Interface text:
[illegible]oud Cost
Overview
Analyze
Recommendations
Plan
Settings
[illegible]mary
Containers
Networking
Storage
SaaS Costs
AI Costs
NEW
Map

Provider
All
Servicename
All
Sub Account Name
All
Region Name
All
Team
All
My Teams

Cloud Cost Checklist
Build your FinOps practice with Cloud Cost Management
27% completed
Check It Out

3 Months
6 Months
1 Year
Year to Date

### The Three Creeps

Token creep can raise costs 8x overnight; pricier models can cost up to 15x more per request at the same volume. Uncached calls pay for repeated completions—Petzel says Datadog sometimes found 70% of spend redundant.

[6:15](https://www.youtube.com/watch?v=rTojoVotlD8&t=375s) [Three rows list Token Creep, Model Drift and Uncached Calls. The slide's callouts state an eightfold overnight cost increase, up to fifteenfold higher per-request cost, and 70 percent redundant spending, respectively.](https://assets.ai.engineer/sites/aiecode2025/sha256/12107b7e79fa6ae5948bc7d3b84e11c4bd6fc8a6aa4af2032aa6f2751d593f36.webp)

The ‘Three Creeps’

Token Creep
8x cost increase overnight

Model Drift
Up to 15x higher per-request cost

Uncached Calls
70% of spend is redundant

DATADOG

### Cost Attribution Strategy

Feature tags tie spending to roadmap decisions; user tags support billing and abuse detection. Model tags enable cost comparisons; endpoint tags inform infrastructure planning across regions and environments.

[8:38](https://www.youtube.com/watch?v=rTojoVotlD8&t=518s) [Cost tagging at feature, user, model, and endpoint levels supports product roadmap decisions, billing and abuse detection, cost optimization, and infrastructure planning.](https://assets.ai.engineer/sites/aiecode2025/sha256/64cfe97185f0f3ec6df4630ecb1410011b68ce58ff01b380aa09e1e3557e6e03.webp)

Cost Attribution Strategy
Tag diligently to understand costs

Feature Level → Product roadmap decisions
User Level → Billing & abuse detection
Model Level → Cost optimization
Endpoint Level → Infrastructure planning

DATADOG

### Safety & Security

An inexpensive AI app can still leak sensitive or personal information. Petzel shifts from cost control to detecting threats before they reach users.

[9:13](https://www.youtube.com/watch?v=rTojoVotlD8&t=553s) [Safety and Security section divider with the subtitle: Detect threats before they reach your users.](https://assets.ai.engineer/sites/aiecode2025/sha256/fc0c28bad408236e7b1c8f75507e5fa19de8220f12f2a91220775dcd93ee5605.webp)

Safety & Security

Detect threats before they reach your users

DATADOG 9

### Safety Metrics to Monitor

Petzel’s safety targets pair low flag rates for prompt injection (<0.5% of requests) and moderation (<0.3% flagged for review) with 0% PII in production outputs and, ideally, blocking 100% of jailbreak attempts.

[10:28](https://www.youtube.com/watch?v=rTojoVotlD8&t=628s) [Safety metric targets: prompt injection flags below 0.5% of requests and zero PII in production outputs.](https://assets.ai.engineer/sites/aiecode2025/sha256/de4c1118cb57b67a46bf18055ab7d28dee999c953829f3322e4096b4e53fe09d.webp)

Safety Metrics to Monitor

Prompt Injection Rate
🎯 < 0.5% of requests flagged

PII Detection Rate
🎯 0% PII in production outputs

DATADOG

[11:34](https://www.youtube.com/watch?v=rTojoVotlD8&t=694s) [Four safety targets: prompt injection below 0.5% of requests flagged, zero PII in production outputs, content moderation below 0.3% flagged for review, and blocking 100% of jailbreak attempts.](https://assets.ai.engineer/sites/aiecode2025/sha256/5ffe50042600fd3aca0e11bcccd10a47dafd9e00b63e0e4f53d9e63d94f7409c.webp)

Safety Metrics to Monitor

Prompt Injection Rate
🎯 < 0.5% of requests flagged

PII Detection Rate
🎯 0% PII in production outputs

Content Moderation Score
🎯 < 0.3% flagged for review

Jailbreak Attempts
🎯 Block 100% of attempts

DATADOG

### Quality Monitoring

Quality monitoring focuses on output quality and user satisfaction: a 200 OK response can still contain a hallucinated answer.

[12:07](https://www.youtube.com/watch?v=rTojoVotlD8&t=727s) [Quality Monitoring section slide with the subtitle Output Quality and User Satisfaction on a blue and green geometric background.](https://assets.ai.engineer/sites/aiecode2025/sha256/4afdf2d450af5112ede0027180a725f15d789ee67efdbb59f1739695cf4dc981.webp)

Quality Monitoring

Output Quality and User Satisfaction

DATADOG
11

### Example Quality Metrics

Four cards pair answer quality with user satisfaction; completeness asks whether the response fully addresses the user’s intent. A fifth adds RAG retrieval quality: whether fetched documents are relevant to the query.

[14:37](https://www.youtube.com/watch?v=rTojoVotlD8&t=877s) [Example Quality Metrics slide with four cards: Hallucination Rate, Relevance Score, User Satisfaction, and Answer Completeness.](https://assets.ai.engineer/sites/aiecode2025/sha256/3b3e6b2d8c5b69c1b1b46e9f021ec88a1ccb207b4536de3488e74ceac00400cc.webp)

Example Quality Metrics

Hallucination Rate
Relevance Score
User Satisfaction
Answer Completeness

DATADOG

[14:53](https://www.youtube.com/watch?v=rTojoVotlD8&t=893s) [Five metric cards: Hallucination Rate, Relevance Score, User Satisfaction, Answer Completeness, and Retrieval Quality (RAG).](https://assets.ai.engineer/sites/aiecode2025/sha256/c1e4485b5559c0f3235737d66bf2366ae0de63a137ad8f9a0aec4f9a26d2fae3.webp)

Example Quality Metrics

Hallucination Rate
Relevance Score
User Satisfaction
Answer Completeness
Retrieval Quality (RAG)

DATADOG

### Traditional Understanding Plus Cost, Safety, and Quality

The plus sign adds cost, safety, and quality to traditional monitoring: assessing GenAI application health means tracking these alongside latency, errors, traffic, and saturation.

[15:41](https://www.youtube.com/watch?v=rTojoVotlD8&t=941s) [A Traditional Understanding block and plus sign face three stacked bars labeled Cost, Safety, and Quality.](https://assets.ai.engineer/sites/aiecode2025/sha256/efec834ce6228dc0b12b59389164d7cbf0ccfe505f40d4e328933a08c8cb67ab.webp)

Traditional
Understanding

+

Cost
Safety
Quality

DATADOG

### Agent Observability

Datadog’s Agent Observability pairs runtime metrics with quality, cost and security tracking. Its summary places alerts, error rate and p95 duration above trace volume to help engineers assess AI application health.

[16:39](https://www.youtube.com/watch?v=rTojoVotlD8&t=999s) [Agent Observability slide lists standard metrics, quality and cost, and safety and security beside a dashboard screenshot showing summary metrics and a traces chart.](https://assets.ai.engineer/sites/aiecode2025/sha256/a42b3b5be6c0ff6f0073cbdeb1c394a0ad7f85f1c3937f5202a8d9d749736704.webp)

Agent Observability

Standard Metrics
Latency, errors, tokens/s, API rate limits

Quality and Cost
Prompts, Tools, Evals, Patterns, Quality and Costs

Safety and Security
Security, PII detection, etc

Summary
5 ALERT
Error Rate 1.6%
Duration (p95) 2.92min

Total Traces
Y-axis ticks: 0, 50, 100, 150
Timeline: Tue 16; 12:00; Wed 17; 12:00; Th[illegible]
Legend: workflow

DATADOG
14

Interface text:
Agent Observability
cashri-v2
Env
All
Model Name
All
+ Filter

OVERVIEW
cashri-v2
Summary
Errors
Evaluations
Cost
Tools
Prompts
Latency
Security & Safety
Cost PREVIEW
Patterns PREVIEW

INVESTIGATE
Traces
Annotations

### Marina Petzel — Closing Contact Slide

Marina Petzel, Datadog’s Sr. GenAI Advocate, invites attendees to contact her using the large QR code on the right.

[17:30](https://www.youtube.com/watch?v=rTojoVotlD8&t=1050s) [Purple closing slide with a speaker portrait, the name Marina Petzel, the role Sr. GenAI Advocate, a large QR code, and Datadog branding.](https://assets.ai.engineer/sites/aiecode2025/sha256/153f9cadd06df602c62fc8df8322bc6f53ef32af86eaf2aabf5ea88629c810f9.webp)

Marina Petzel
Sr. GenAI Advocate

DATADOG

### AI Engineer World's Fair

[17:50](https://www.youtube.com/watch?v=rTojoVotlD8&t=1070s) [White AI Engineer World's Fair lettering inside a rectangular outline on black, with the event website below.](https://assets.ai.engineer/sites/aiecode2025/sha256/280c372716db1d895a99875af474cbafffbd7d2cdcf5013022331d9ee6fc522f.webp)

AI Engineer
World’s Fair

HTTPS://AI.ENGINEER

## Full transcript

[0:12](https://www.youtube.com/watch?v=rTojoVotlD8&t=12s) Hello, everyone. Thank you for joining me today. Uh, it's time to begin. And, uh, for years, uh, many of us, uh, monitored our applications, uh, with the golden signals, which is, was latency, error, traffic, and saturation, and shortly we called them LATS. And they were, and still, uh, they are and will be remain sufficient and critical for determine the health of our applications. However,

[0:42](https://www.youtube.com/watch?v=rTojoVotlD8&t=42s) today when we enter the new era, era of generative AI, it's becoming not enough. And today, we're gonna talk about what are the new metrics we need to look if we wanna build the healthy GenAI application. Very shortly, my name is Marina Petzel. I am GenAI advocate here in Datadog, and today I will walk you through the metrics that we need to look for. So, uh, when we talk about traditional applications, um, the

[1:12](https://www.youtube.com/watch?v=rTojoVotlD8&t=72s) main specific of them is that the same input always yield the same output, right? It's predictable. But with generative AI applications, we all know that the same prompt can generate, um, different responses every time we run it, and which means we cannot rely on the standard regression testing alone. And, uh, you must constantly evaluate quality in the live environment. The second difference between traditional applications and

[1:42](https://www.youtube.com/watch?v=rTojoVotlD8&t=102s) GenAI applications is variable cost structure. Traditional compute costs are predictable, right? Um, we pay for machine that is running our code, but for GenAI, this, this is dynamic variable here. They might, uh, they vary from the number of tokens, the model that we are using, or the size of your context window that you have. And it's become essential to do the real-time, uh, granular cores, uh,

[2:12](https://www.youtube.com/watch?v=rTojoVotlD8&t=132s) cost, um, to prevent the spending out of control. The third problem is that we face the new attack vectors. We used to worry about SQL injection or cross-site scripting before, but today we must defend against the prompt injection, jailbreaking, or PII, uh, outage in our output. And that's becoming a big problem. And these are security issue, not a simple 500 error code can--

[2:42](https://www.youtube.com/watch?v=rTojoVotlD8&t=162s) you'll never catch, and demand an entirely new forms of security monitoring. And finally, quality is very subjective for GenAI applications. The traditional systems, uh, either work or do not work, right? But generative AI output is judged on the spectrum. Is it relevant? Is it accurate? Is it complete or not? And the problem is that 200 OK response from your server doesn't necessarily mean

[3:12](https://www.youtube.com/watch?v=rTojoVotlD8&t=192s) that the answer was helpful or correct for your final user. And we must now integrate the continuous, uh, quality evaluation, uh, directly in your monitoring stack. And at the beginning, we all were thinking that these golden signals will be enough for our systems, but unfortunately, they are not. Traditional metrics will tell you if your system is up and running, but it won't tell you how well it's

[3:42](https://www.youtube.com/watch?v=rTojoVotlD8&t=222s) running. And established metrics LATS will be still essential, but we're gonna talk about what else on addition to that we need to implement into our systems. And first, I wanna start with the cost monitoring, how we can track the spending before it, uh, spirals out of control. And I think we all know the financial implications of GenAI, and a lot of companies are facing the

[4:12](https://www.youtube.com/watch?v=rTojoVotlD8&t=252s) huge outage of the, uh, cost for their monthly usage. And while the same is true as for traditional applications, there are more linear scale between the throughput and the, uh, uh, the cost. If the user loads, um, load goes up, our resources, uh, usage goes up, and sequentially the cost goes up. But the cost structure for LLM is highly dynamic and unpredictable, which means we can-- we don't

[4:42](https://www.youtube.com/watch?v=rTojoVotlD8&t=282s) have this visibility, and we need to really pay attention how we track that. And what are the problems you might encounter with the cost? And here we talk about the three major way for cost tracking, uh, and so you can track them, um, to prevent the cost outage. So the first is the token creep. Uh, this happens when engineers or product managers trying to improve the quality,

[5:12](https://www.youtube.com/watch?v=rTojoVotlD8&t=312s) quietly increasing the context window. And let's say the context window was four thousand tokens, and then it's, like, blow to thirty-two without any financial review. And what it can cost, like, the, the cost is-- depends very, uh, significantly to the context count, and that might result up to 8x cost increased. And in a month, it might cost you thousands of dollars. Second is the model drift. Uh, that is

[5:42](https://www.youtube.com/watch?v=rTojoVotlD8&t=342s) also the pretty typical problem when the team switches from cheaper and faster model, like let's say Haiku 4.5, to the more powerful and more expensive model like Opus 4.8, and which promise you obviously the better quality. But at the same time, Opus, for example, uh, can be higher, fifteen times higher, uh, per the cost request. And this change in model, even with the same usage volume,

[6:13](https://www.youtube.com/watch?v=rTojoVotlD8&t=373s) might blow your cost significantly within a month, too. And finally, it's uncached calls. In generative AI, the same query can hit the API repeatedly if, uh, you haven't implemented the effective caching layer. And what we've been seeing from, um, our, uh, own research that 70% of the spend sometimes are redundant because the system is paying for exact the same expensive model

[6:43](https://www.youtube.com/watch?v=rTojoVotlD8&t=403s) completion, uh, over and over again. And with this granular visibility into these three layers, you would be able to escalate your cost rapidly. And now let's talk about how you can prevent that and get more granular data. And when we, um, talk about that, we, uh, try to think about the implementation of cost attribution strategy, which is you must tag

[7:13](https://www.youtube.com/watch?v=rTojoVotlD8&t=433s) absolutely everything. For example, uh, tagging, uh, can help you to-- uh, why it's important, right? You receive a lot of, uh, interactions with your agents and-- but all of these, uh, interactions are undifferentiated. You don't know what's below your actual cost. That is why first layer what you can implement is the feature level, level. By tagging the intention of the feature, like it was a chat or summarization, you can

[7:42](https://www.youtube.com/watch?v=rTojoVotlD8&t=462s) attribute the cost for the specific product areas. And this data is crucial not only for engineering your cost management, but also for the product roadmap decisions, like knowing which features are actually the budget costs. The second level is the use-user level. Using the tags like the user ID or your org, um, name is essential for chargebacks, accurate billing, but also quickly identify the patterns of abusive

[8:12](https://www.youtube.com/watch?v=rTojoVotlD8&t=492s) usage before they drain your budget. Third is the model level. Uh, also tag your models like GPT-5.5 or the provider, OpenAI, Anthropic. They'll give you visibility into the higher level areas, uh, of cost optimization. And you can benchmark the true expenses, uh, of using one model versus another. And finally, uh, endpoint-level tagging.

[8:42](https://www.youtube.com/watch?v=rTojoVotlD8&t=522s) Attributing cost to regions or to your production or, like, to your environment, like production or staging, helps your infrastructure planning. And you can see how regional distribution or staging versus production environment can impact your total spend. And by making this tagging mandatory practice, you gain an esse- an essential visibility to understand exactly where your money is going and how important,

[9:13](https://www.youtube.com/watch?v=rTojoVotlD8&t=553s) uh, to control it. Well, but now let's talk if, like, let's, let's imagine that we handle all the cost issues, right? We took that under control, and your finance tea-team is obviously get very happy. Uh, but the next one, uh, problem is the safety. And your AI can be cheap, but s-still leak some of the sensitive or personal information, which is not great. And even though, like, your

[9:42](https://www.youtube.com/watch?v=rTojoVotlD8&t=582s) lab scores can be perfect, uh, it cannot tell you if there are some, um, metrics went wrong. So what we can track here in the safety metrics to, um-- First is the prompt injection rate, which has become the very crucial problem. This metrics detects the manipulation attempts, uh, into your user prompts. There are if infrastructure-- if, uh, the infrastructure design doesn't have any guardrails. And

[10:13](https://www.youtube.com/watch?v=rTojoVotlD8&t=613s) this can be, um, prevent by using some advanced techniques like, uh, pattern matching or specialized classifier models. And for this, we usually need to target for very aggressive thresholds. The second is PII detection rate. Um, PII, uh, so leakage is a crucial and zero-tolerance issue here. We can scan all the model outputs, uh, for sensitive data

[10:42](https://www.youtube.com/watch?v=rTojoVotlD8&t=642s) like Social Security, like credit cards or personal information. And you can use some, um, um, methods like regex or, uh, named entity recognition here, and only acceptable target should be zero percent of, uh, your production outputs. The third is the content moderation score, how you can determine if your content is toxic, harmful, or biased, uh, towards

[11:12](https://www.youtube.com/watch?v=rTojoVotlD8&t=672s) some of the topics. And by running all outputs through toxicity classifier can gi-give you a scores, uh, how-- what is the risk of your output for being exposed to this, um, toxic, uh, um, output. And this allows you to set some clear thresholds, and we always target to, um, minimize that as much as we can. And finally, um, the jailbreak attempts, and there are

[11:42](https://www.youtube.com/watch?v=rTojoVotlD8&t=702s) persistent attempts by users to bypass the safety guardrails, and often, um, through the elaborate scenarios. And tracking p- uh, tracking patterns to indicate the system prompt overwrites, uh, you should design and become-- so it become the absolute blocker for you. So ideally, we should block all the hundred percent of the attempts of that.

[12:10](https://www.youtube.com/watch?v=rTojoVotlD8&t=730s) Now let's imagine that cost is checked, security is checked, uh, but now you arrive to arguably the most crucial, um, dimension and the quality monitor that at least the most crucial for our final users. And even though your LATS metrics, um, showing that your service is up and around, uh, and running, your cost and safety tells you that your service is budget-fri-

[12:39](https://www.youtube.com/watch?v=rTojoVotlD8&t=759s) compliant and also secure, but ultimately, for any application, but more for generative application, it's whenever output is good. But we all know that 200 OK response from your server sometimes can be a lie, uh, if the answer is hallucination. And I'm sure many of you got some response from modern AI applications, right? But, like, we know that it can confidently by lie-- but lie to you.

[13:10](https://www.youtube.com/watch?v=rTojoVotlD8&t=790s) And now let's understand what type of metrics you can imple- implement to your application right now to prevent it from being, uh, unsafe for your final user. We recommend to focus on these five metrics. Uh, one of them is hallucination rate. Um, this measures the percentage of response that contain claims unsupported by the grounded data. Um, we need to mix that with the manual review as well as with, uh,

[13:40](https://www.youtube.com/watch?v=rTojoVotlD8&t=820s) automated fact checks to achieve the aggressive targets here, too. The final-- uh, the next one is the relevance scores, which helps you to understand if, uh, the response actually address the user question. Um, te- techniques like y- that you can use here is similarity using BERT or embedding similarity that can score you, um, how relevant is the final answer was to the user.

[14:10](https://www.youtube.com/watch?v=rTojoVotlD8&t=850s) The next one is user satisfaction, which directly pulse into your end user. Uh, you must implement-- You can implement some mechanism like thumbs up, thumbs down, or some of the stars ranking, and also NPS scores, um, to capture the subjective data, but also you should target that you got more than 80 or 85% of the positive feedback at least. Next one is the

[14:40](https://www.youtube.com/watch?v=rTojoVotlD8&t=880s) answer completeness, and this, uh, metric is often evaluated by LLM-as-a-judge, uh, answering whenever the response was fully or holistically addressed, um, the user intent. Um, and final one, the fifth one, um, if you are using RAG, retrieval-augmented, uh, generation system, you should always track the quality of your RAG. What-- If all the documents that were fetched were relevant to the user

[15:10](https://www.youtube.com/watch?v=rTojoVotlD8&t=910s) query, and here you have a lot of metrics and mechanism how you can calculate that. For example, top-K accuracy or normalized discounted gains. And that is, uh, essential to track because, uh, whenever, like, your context is exploded, it's getting harder and harder to pick the most relevant one. And by implementing these, uh, metrics, uh, you can gain some understanding of, like, more dimensional view of the quality that necessary you, you need

[15:40](https://www.youtube.com/watch?v=rTojoVotlD8&t=940s) to optimize to improve your GenAI application. Well, and now let's summarize, um, what we need, uh, to implement to make sure that our GenAI application is healthy, right? We are not saying that we need to get rid of traditional understanding. We always need to look into our metrics, LA- or LATS, latency, error, uh, traffic, and saturation. But also on top of that, we need to put another layer, layer of

[16:10](https://www.youtube.com/watch?v=rTojoVotlD8&t=970s) which is more relevant to generative applications, and in this case, we talked about the cost, safety, and also quality. And I know that for us as an engineer, it's becoming more and more critical right now to track it all, not be-- don't need just build application and make sure that it's running in production, but now we also need to care about additional layer of complexity. And centrally, Datadog has you back here since we are working

[16:40](https://www.youtube.com/watch?v=rTojoVotlD8&t=1000s) in observability and monitoring space for, for really long time. So we do have our Agent Observability, uh, product, which allows you to quickly and easily understand if your AI application is healthy, if it's running, or there, there is any problem that you need to fix. And we do have a lot of different way, um, uh, things that you can track inside of that. And, um, if you wanna hear more about that, please feel free

[17:10](https://www.youtube.com/watch?v=rTojoVotlD8&t=1030s) to see me afterwards. We also do have the booth here, uh, which you can check it out. We do have a really, uh, good, um, offers for you if you wanna just try it out and see how Agent Observability can help your application. We have some QR codes for the free trials. And, uh, with that, I wanna thank everyone. Again, my name is Marina Petzel. If you wanna contact me, please feel free to, um, use this QR code. And yeah, thank you for

[17:40](https://www.youtube.com/watch?v=rTojoVotlD8&t=1060s) being here today.
