Your LLM App Returned 200 OK. It Was Still Wrong. — Marina Petzel, Datadog
Loading slides…
Read the talk
Your LLM App Returned 200 OK. It Was Still Wrong.
Marina Petzel explains why latency, errors, traffic and saturation need company: cost attribution, safety checks and continuous quality evaluation reveal failures that a successful HTTP response cannot.
From a talk by Marina Petzel
At a glance
Ideas worth remembering
LATS remains essential for operational health, but successful requests also need live evaluation for cost, safety and answer quality.
Context growth, more expensive model choices and repeated uncached completions can raise spending without an increase in user traffic.
Tag interactions by feature, user or organization, model and provider, and deployment location or environment so costs can be traced to useful decisions.
Safety monitoring distinguishes manipulation attempts from unsafe outputs. Zero PII leakage and blocking every jailbreak are objectives, not evidence that detectors achieve perfect coverage.
Evaluate factual support, relevance, user satisfaction, completeness and RAG retrieval quality separately; each identifies a different way an answer can fail.
A healthy server can deliver an unhealthy answer
A generative AI application can return 200 OK while giving its user an irrelevant, incomplete or incorrect answer. The request succeeded at the server level; the response failed at the task level. Marina Petzel, introducing herself as a GenAI advocate at Datadog, starts with the familiar golden signals—latency, errors, traffic and saturation, or LATS—and asks what else production monitoring must measure to catch that difference.
The first change is variability. The same prompt can produce different responses across runs, so a successful regression test does not establish how every live response will behave. Regression testing still matters, but quality also needs continuous evaluation in the production environment. The system must be observed while it answers real questions, rather than treated as correct once a fixed set of examples passes.
Three other changes widen the monitoring problem:
- Variable cost: Token counts, model selection and context size affect spending. Request volume alone cannot explain the bill.
- New attack vectors: Prompt injection, jailbreaks and exposure of personally identifiable information can occur without producing a server error.
- Graded quality: Relevance, accuracy and completeness lie on a spectrum. A response can be technically delivered yet only partly useful.
These changes do not retire LATS. They add questions that LATS was never meant to answer: Is this response affordable? Is it safe? Does it help the user? Petzel develops those questions in that order, beginning with costs that can grow even when traffic looks ordinary.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Three ways spending grows without more users
In the simplified traditional-compute picture, more traffic requires more resources, and spending rises with resource use. LLM spending has additional moving parts inside each request. A product change can increase the amount of context, select a more expensive model or repeat work that could have been reused. Monitoring the total bill reveals the consequence; monitoring these drivers helps explain it.
Token creep begins with an apparently helpful quality change. Engineers or product managers expand the context supplied to the model without a financial review. Petzel’s example grows from 4,000 tokens to 32,000—eight times as much context—and warns of a potential eightfold cost increase, accumulating to thousands of dollars over a month. The useful comparison is between context actually sent and its cost: a larger permitted context window does not, by itself, establish that every request uses all of it.
Model drift, as used here, means a team switches from a cheaper, faster model to a more powerful, more expensive one. The motivation is better output quality, but the financial effect persists even at unchanged usage volume. Petzel illustrates this with a model that could cost fifteen times more per request. That multiplier is an example of the risk, not a verified price comparison: the decision to upgrade needs visibility into both the added expense and the quality benefit it is supposed to buy.
Uncached calls pay for the same expensive completion repeatedly. The same query reaches the API again because an effective caching layer is missing, so repeated demand becomes repeated model work. Petzel reports that, in Datadog’s own research, redundant work sometimes accounted for 70% of spending; the recording does not specify the workloads or measurement method, so that figure should not be read as an expected saving for every application. The mechanism to investigate is concrete: which repeated calls are generating a fresh bill instead of reusing an appropriate result?
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Tag requests so the bill can answer a question
A stream of undifferentiated agent interactions does not tell a team where its money went. Cost attribution adds identifying dimensions to those interactions so spending can be grouped by the product behavior, customer, model or deployment that produced it. Petzel’s deliberately broad instruction is to “tag absolutely everything”; the practical substance is four complementary views of the same spending.
- Feature level: Identify the intention of the interaction, such as chat or summarization. This assigns costs to product areas and gives roadmap discussions a financial basis: which features consume the budget?
- User or organization level: Attach a user ID or organization identifier. This supports chargebacks and accurate billing, while making unusually expensive or abusive usage easier to locate before it drains the budget.
- Model and provider level: Record which model and provider handled the work. This makes actual spending across alternatives comparable and helps locate expensive model choices.
- Endpoint and deployment level: Attribute costs to regions and environments such as production or staging. This reveals how deployment choices and nonproduction activity contribute to total spend.
Making tagging mandatory turns cost monitoring into a way to investigate decisions, rather than merely notice an overrun. Feature tags serve product planning; user tags serve billing and abuse investigation; model tags serve comparison; deployment tags serve infrastructure planning. But an inexpensive application can still expose sensitive information. Getting the finance team comfortable leaves the next problem untouched.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Measure attacks and unsafe output separately
Safety monitoring has to inspect the content and behavior that ordinary status codes miss. Good laboratory scores do not tell a team whether a live prompt is trying to manipulate the application or whether a delivered answer contains personal information. Petzel separates four signals, with different detection methods and different desired outcomes.
- Prompt injection rate: Detect manipulation attempts in user prompts. Pattern matching and specialized classifier models are suggested ways to recognize suspicious inputs. The slide gives an example target of fewer than 0.5% of requests flagged for prompt injection. Guardrails determine what the application does with a flagged request.
- PII detection rate: Scan model outputs for sensitive information, including Social Security numbers, credit-card information and other personal data. Regular expressions can detect recognizable patterns, while named entity recognition can identify entities in text. Petzel sets a zero-tolerance objective: sensitive-data leakage should occur in 0% of production outputs.
- Content moderation score: Run outputs through a toxicity classifier to score the risk of toxic, harmful or biased content. A score allows the application to apply a threshold rather than rely on a binary server-success signal. The slide gives an example target of “< 0.3% flagged for review.” It leaves the denominator and the classifier score that triggers review unspecified.
- Jailbreak attempts: Track persistent efforts to bypass safety guardrails, including elaborate scenarios intended to overwrite the system prompt. The intended response is blocking, with an ideal target of stopping 100% of attempts.
The zero-leakage and complete-blocking figures are safety objectives, not demonstrated detector performance. That distinction matters operationally: recognizing a suspicious prompt, finding sensitive text in an output and preventing its delivery are different jobs. A detected problem needs an enforcement response; merely counting it does not make the interaction safe.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Five questions a successful response still has to pass
Now suppose the service is running, spending is controlled and safety checks are in place. The user still needs a good answer. A confident hallucination can satisfy the server’s success conditions while failing the user completely. Quality monitoring therefore needs several measurements that distinguish factual support, relevance, satisfaction, completeness and the quality of retrieved context.
- Hallucination rate: Are the claims supported? Measure the percentage of responses containing claims unsupported by the grounding data. Petzel recommends combining manual review with automated fact checks. This evaluates the relationship between an answer and its evidence, rather than treating fluent or confident wording as a sign of correctness.
- Relevance score: Does the answer address the question? BERT-based similarity or embedding similarity can score the relationship between the user’s question and the response. This asks a different question from factual support: an answer may discuss the right subject yet contain unsupported claims, so a relevance score does not replace the hallucination check.
- User satisfaction: Did the user find it useful? Thumbs-up and thumbs-down controls, star ratings and NPS provide direct feedback on the experience. Petzel suggests aiming for more than 80–85% positive feedback. That is a recommended target for positive responses, not a measured result or an interchangeable interpretation of the NPS scale. User feedback supplies subjective information that content-based evaluation alone cannot capture.
- Answer completeness: Did it cover the full intent? An LLM-as-a-judge can assess whether the response fully addresses what the user wanted. Relevance checks whether the answer addresses the question; completeness asks whether it leaves important parts unanswered. Separating them makes a partially helpful answer easier to diagnose.
- RAG retrieval quality: Did the system fetch useful evidence? In retrieval-augmented generation, evaluation must also examine the documents fetched for the user’s query. Petzel names top-K accuracy and normalized discounted gain measures as options. These evaluate the retrieval side of the system, rather than only the final response. As the pool of context grows, selecting the most relevant material becomes harder; checking retrieved documents helps distinguish a context-selection problem from a problem in the answer produced from that context.
Together, these measurements make quality more diagnosable. Unsupported claims call for examining grounding; a poorly matched answer calls for examining relevance; missing coverage calls for examining completeness; unhelpful retrieved documents call for examining retrieval. User satisfaction adds the final user’s judgment. No one score answers all five questions.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keep LATS and add the missing dimensions
The closing recommendation is additive: retain latency, errors, traffic and saturation, then monitor cost, safety and quality alongside them. These dimensions are complementary, not successive gates that can be checked once and forgotten. A running service can become too expensive; an affordable service can leak information; a safe service can still answer badly.
Petzel ends by presenting Datadog Agent Observability as a product for investigating AI application health and identifying problems to fix, with an invitation to try it. The LLM Observability documentation offers a practical next step for instrumenting that approach. The engineering responsibility established by the talk remains the same whichever tool carries it: production monitoring must evaluate what the application delivered, not stop at whether it delivered something.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Explains instrumentation, request traces and spans, operational monitoring, and evaluations for quality, privacy and safety—the implementation counterpart to the talk’s monitoring dimensions.
The product introduced at the end of the recording; provides an overview of tracing, evaluations and production monitoring, with options to explore or try it.
Related talks
- Build a Platform and Watch It Burn — Michael Forrester, Accenture & Whitney Lee, Datadog
Develops the safety discussion through concrete prompt-injection, input-blocking and output-redaction demonstrations, including failed protections and the difference between detection and enforcement.
- Is a Faster Model Actually Faster? Test the Whole Job — Morgan Suo, MiniMax
Extends the cost–quality tradeoff into model-switch evaluation: measure the time and total expense of producing an acceptable result, including failed attempts and human repair.
Full transcript
Loading transcript…