# 1,368 AI Failures and What They Teach About Evals — Vishal Srivastava, New York Life & Tanmay Sah

Vishal Srivastava, New York Life · Tanmay Sah, New York Life

[Talk page](https://ai.engineer/talks/mMw3arwQsL0-1-368-ai-failures-what-they-teach) · [Video](https://www.youtube.com/watch?v=mMw3arwQsL0)

## 1,368 AI Failures and What They Teach About Evals

Vishal Srivastava and Tanmay Sah connect technical failures, deployment controls and business consequences through a retrospective incident study. Their proposed evaluations challenge system claims, test whether errors reach real decisions, and use an airline chatbot’s incorrect refund advice to make the approach concrete.

![Recording frame at 1:45: 1,368 AI Failures and What They Teach About Evals](https://ai.engineer/api/talk-essay-media/e115e30ccafe6e4e7b29a3cdff908fac6e028af9c4966bd2ff102900d5710dc4)

1,368 AI Failures and What They Teach About Evals — recording at 1:45

### Follow an error beyond the model

[0:12](https://www.youtube.com/watch?v=mMw3arwQsL0&t=12s)

An enterprise can accumulate more AI use cases than its reviewers can assess one by one. Vishal Srivastava and [Tanmay Sah](https://tradertanmay.github.io/) approach that capacity problem through a study of 1,368 incidents from the public AI Incident Database. Their starting observation is a sharp rise in recorded failures, particularly over the preceding four years. Incident counts alone do not establish a failure probability per deployment, but they motivate a practical question: how should limited evaluation effort be directed? [0:35](https://www.youtube.com/watch?v=mMw3arwQsL0&t=35s)

![Recording frame at 1:23: Follow an error beyond the model](https://ai.engineer/api/talk-essay-media/fd5739c72a17677a4e756a62f7255a2cb3bee7c71b55409a12fd1a0e7c1ef313)

Follow an error beyond the model — recording at 1:23

The framework begins with development. Problems in data quality, tool architecture or system design can create a technical fault. That fault can propagate into production, where deployment controls determine whether it is contained. A model can have design weaknesses while the surrounding system still prevents a particular error from reaching the people who would act on it.

The third layer is business context: what follows when an error escapes those controls. Reputational damage, legal consequences and financial loss depend on how the system is being used. Evaluating technical strength therefore leaves an essential question open: what happens to the model’s mistakes after they leave the model?

### Thirteen dimensions make the investigation concrete

[3:23](https://www.youtube.com/watch?v=mMw3arwQsL0&t=203s)

The three layers become 13 dimensions that an evaluator can investigate. Six concern **technical strength**:

- **Data quality:** What weaknesses in the data used to build the system could affect its behavior?
- **System design:** Does the design fit the task the system is supposed to accomplish?
- **Predictive validity and outcome analysis:** Does the system produce the intended outcomes?
- **Fairness:** Are fairness-related checks part of the assessment?
- **Failure modes:** Where, when, how and for whom can the system break?
- **Explainability:** Can its decisions be explained to regulators or other bodies that need to understand them?

![Recording frame at 6:38: Thirteen dimensions make the investigation concrete](https://ai.engineer/api/talk-essay-media/8b7283210d8dc74a39704ebc66bd6443858103f36218e61a18dee4749cd91e79)

Thirteen dimensions make the investigation concrete — recording at 6:38

Four concern **deployment controls**, where the questions shift from model behavior to how that behavior is used:

- **Decision authority:** Is the AI supplying information, or can it act on someone’s behalf?
- **Human mediation:** How effective is the human review process? Having a person in the loop is useful only to the extent that the process catches or contains mistakes.
- **Exposure control:** Who encounters the output? Internal employees may detect and contain an error; direct exposure to external customers can let it propagate quickly.
- **Monitoring and harm detection:** Can production monitoring detect an issue in time for the organization to contain it?

Three describe **business impact**:

- **Financial impact:** What financial consequences could an error create?
- **Legal consequences:** What legal consequences could follow from the system’s behavior?
- **Scale:** How many users could be affected?

Applied retrospectively, these 13 dimensions cover almost 98% of the incidents in the study. An incident has an average of six dimensions present. This is coverage of a classification: several categories can describe the same event. The overlap is the useful finding. An investigation needs to account for combinations of weaknesses across the system, including the fault itself and the conditions that let it become consequential.

### The most frequent dimensions are not always the most harmful

[7:35](https://www.youtube.com/watch?v=mMw3arwQsL0&t=455s)

The plot compares 13 dimensions across 1,368 documented incidents, including 206 with material harm. Its horizontal axis shows, on a log scale, the percentage of all incidents in which a dimension received a severe rating of 3. Its vertical axis shows how much more common that rating was in harm cases, expressed as a multiple. Human mediation sits near the upper left at 0.95%. The slide labels its harm multiple 4.8× but does not identify the comparison group. Harm detection also appears toward the upper left. Scale and system design received severe ratings more often, but their association with harm was weaker. [7:35](https://www.youtube.com/watch?v=mMw3arwQsL0&t=455s)

![Recording frame at 8:23: The most frequent dimensions are not always the most harmful](https://ai.engineer/api/talk-essay-media/123cee45bf274218dc4adfe37b5522079fd27b9311215036fab33f8b0e104dc4)

The most frequent dimensions are not always the most harmful — recording at 8:23

That separation changes how evaluation time should be spent. A review process organized entirely around common failure categories can underweight control failures that are disproportionately common in material-harm cases. Human review needs to be tested for whether it catches mistakes, and monitoring for whether it detects trouble early enough to contain it. Their presence on an architecture diagram would not answer those questions.

Grouping the dimensions into the three larger layers puts deployment-control failures first in association with harm, followed by business impact and technical failure. These are retrospective associations; they do not establish how much harm a particular intervention would prevent. They do make the deployment configuration a central object of evaluation, especially for systems whose mistakes could be costly.

### Materiality directs attention to the uses with more at stake

[9:43](https://www.youtube.com/watch?v=mMw3arwQsL0&t=583s)

Materiality turns the findings into a way to prioritize use cases. The proposed scale ranks AI applications by how much is at stake, with greater materiality reported to align with greater associated harm. Its purpose is to allocate review effort: a more consequential application should receive more investigation and resources than one whose errors have limited effects.

![Recording frame at 11:01: Materiality directs attention to the uses with more at stake](https://ai.engineer/api/talk-essay-media/1de53ce188839a0a926ad58f3ae60b813f366c6117c05819269a5681e8351f97)

Materiality directs attention to the uses with more at stake — recording at 11:01

Sah then separates risk into four questions:

- **Model failure:** How often does the model fail?
- **Reaching a decision:** Does that failure influence a real decision?
- **Escaping detection:** Does the failure go uncaught?
- **Consequences:** How severe is the resulting harm?

This decomposition explains why the same model can create different risks in different deployments. Giving it more decision authority changes the route from an error to action. Human review and monitoring can interrupt that route if they work in time. Evaluating the complete setup therefore means testing the model together with the permissions, review processes and monitoring under which it will actually operate.

### Test the agent’s environment—and its evaluator

[11:38](https://www.youtube.com/watch?v=mMw3arwQsL0&t=698s)

Traditional machine learning already has a discipline of challenging assumptions, input data and behavior under stress. Extending it to an agent makes more of the deployed system part of the test:

- **Retrieval:** Does it bring back the right information?
- **Tools:** What happens when a tool fails?
- **Human review:** Does the reviewer catch a mistake?
- **Permissions:** Does the agent stay within the actions it is allowed to take?
- **Recovery:** Can the system recover when something goes wrong?

The comparison between traditional models and agents asks what evidence a deployment decision rests on. A traditional model can be evaluated on held-out data, using metrics such as accuracy and ROC AUC. Those results still depend on the data and assumptions behind the test; the metric does not remove the need to examine them.

An agent introduces many possible inputs, tool responses and action sequences. Evaluation samples selected scenarios from those possibilities. Another AI may generate the tests, judge the results, or do both. That creates two distinct ways to get misleading evidence: the selected scenarios can miss an important failure, and the judge can misclassify a result that was tested.

![Recording frame at 13:00: Test the agent’s environment—and its evaluator](https://ai.engineer/api/talk-essay-media/8d42b488f2351f55114406137d280adc30d3cf0430f234cd6bd1ec6f279ae5c1)

Test the agent’s environment—and its evaluator — recording at 13:00

A passing set of scenarios therefore needs three further checks:

- **Evaluator reliability:** Can the evaluator make the judgments being relied on?
- **Coverage:** Which behaviors and failures does the evidence actually cover?
- **Deployment match:** Does the evidence apply to the system being deployed, with its particular setup?

The evaluator becomes part of the system that must be governed. An AI judge’s verdict cannot, on its own, establish the reliability of the process that produced that verdict. [13:08](https://www.youtube.com/watch?v=mMw3arwQsL0&t=788s)

### Choose tests that can challenge the expected rule

[13:48](https://www.youtube.com/watch?v=mMw3arwQsL0&t=828s)

The confirmation-bias example begins with `2, 4, 6`. It is easy to infer a rule of even numbers increasing by two and continue with `8, 10`. Those continuations also fit the broader rule given in the example: any increasing numbers. Their compatibility therefore cannot distinguish the narrow rule from the broad one. The mistaken confidence lies in treating an example that fits as if it had established the rule.

![Recording frame at 14:35: Choose tests that can challenge the expected rule](https://ai.engineer/api/talk-essay-media/da6fc9fdfa206c22e504363615b50cddff9484a1601ae0066d3fcf81d4165485)

Choose tests that can challenge the expected rule — recording at 14:35

That is the trap for an evaluation designed around the behavior its builders already expect. More confirming examples can leave the important uncertainty untouched. Failure-driven evaluation deliberately looks for evidence that would falsify the system’s claim. The test has to create an opportunity for the claim to fail, so the result can change what the team believes the system is capable of doing.

### Build, break, fix turns failed tests into deployment limits

[15:19](https://www.youtube.com/watch?v=mMw3arwQsL0&t=919s)

Build, break, fix begins with a system claim, then deliberately tests where it breaks. A failed test identifies a condition under which the claimed outcome cannot yet be trusted. The next decision concerns what the system should be allowed to do in that condition.

![Recording frame at 16:34: Build, break, fix turns failed tests into deployment limits](https://ai.engineer/api/talk-essay-media/3bc0677be59f4a5dd5a98dafed38944e8ee81d78fa4225a0ed14c8dcbea7653c)

Build, break, fix turns failed tests into deployment limits — recording at 16:34

The framework offers two responses:

- **Restrict the use:** Prevent the AI from operating in the region where failures occur. Reducing the allowed scope can be a valid fix without changing the underlying model.
- **Add supporting controls:** Introduce a supporting system that the team has reason to trust, then test the combined behavior again.

The iteration continues until the evidence supports sufficient confidence in the intended outcomes.

The result should be a map of where outcomes are trusted, where uncertainty remains, and where failures carry high consequences. Reliability and materiality describe different properties: confidence in performance and the stakes of getting it wrong. Routing needs to consider both. For a high-risk use case, the proposed rule is to keep AI operation within the region where the team has adequate confidence, while further tests and controls can change that scope.

### The airline chatbot’s 90-day promise

[17:03](https://www.youtube.com/watch?v=mMw3arwQsL0&t=1023s)

A customer needed to travel after a death in the family and asked an airline’s chatbot about a discounted fare. The chatbot said he could buy a regular ticket and claim the discount afterward, within 90 days. He relied on that advice. The airline later refused the request because its policy did not permit it, and a tribunal held the airline responsible. [17:03](https://www.youtube.com/watch?v=mMw3arwQsL0&t=1023s)

The failure develops through observable changes: an incorrect policy statement reaches a customer, the customer acts on it, and the organization faces the consequences. The consequential action here is the customer’s reliance on the advice. An evaluation of answer quality needs to account for the delivery and decision process that allows a wrong answer to become a real commitment.

The proposed evaluation starts with a requirement: **advice must match the approved policy**. Questions about refunds after travel directly challenge that requirement. A response that permits a later discount when the approved policy does not allow it exposes a specific failure. The test now identifies the discrepancy the system must prevent, rather than merely collecting another plausible answer.

Sah proposes adding a policy check or human escalation, then repeating the challenge to establish whether the control catches the error before it reaches the customer. This is a proposed prevention test whose effectiveness would have to be demonstrated.

Where could the incorrect advice be stopped?
Both paths begin with the same 90-day promise. The observed path reaches the customer. The proposed path adds a check or escalation before delivery; its conditional branches distinguish catching the error from allowing it through.
- Incorrect policy advice — Buy a regular ticket and claim the discount afterward, within 90 days.
- Customer acts on the advice — The customer relies on the promised later discount.
- Airline held responsible — The airline refuses the request under its actual policy; the tribunal holds it responsible.
- Policy check or human escalation — Proposed review of the advice before it reaches the customer.
- Error caught before delivery — The intended outcome to verify by testing the policy questions again.
- Incorrect policy advice → Customer acts on the advice (Observed: advice reaches the customer)
- Customer acts on the advice → Airline held responsible (Customer acts; the request is later refused)
- Incorrect policy advice → Policy check or human escalation (Proposed: review before delivery)
- Policy check or human escalation → Error caught before delivery (If the control catches the policy mismatch)
- Policy check or human escalation → Customer acts on the advice (If incorrect advice still reaches the customer)

### Make deployment depend on the supported uses

[18:49](https://www.youtube.com/watch?v=mMw3arwQsL0&t=1129s)

The closing deployment decision combines the materiality of a use case with evidence from stress testing the full system. An organization can use its own materiality scale, and the risk assessment needs to accommodate changes in the use case. Technical behavior, deployment controls and business context all remain part of that assessment. Deployment should proceed within the region where the tests establish sufficient confidence. When confidence is missing in a high-stakes region, restrict the scope; discovering the gap through a customer may be too expensive. [18:49](https://www.youtube.com/watch?v=mMw3arwQsL0&t=1129s)

## Key takeaways

- The study’s 13 dimensions cover almost 98% of the incidents retrospectively, with an average of six dimensions present per incident. That overlap calls for investigating combinations of technical weaknesses, deployment controls and business consequences.
- Deployment-control failures show the strongest association with harm in the study. Human mediation and harm detection deserve attention even when other dimensions occur more frequently; the associations guide priorities without establishing the effect of a particular intervention.
- Risk depends on how often a model fails, whether the failure reaches a decision, whether it goes uncaught, and how severe the consequences are. Materiality helps direct scarce evaluation resources toward the uses with more at stake.
- Agent evaluations need to test retrieval, tools, human review, permissions and recovery. When AI generates tests or judges results, evaluator reliability, scenario coverage and relevance to the deployed setup also need examination.
- Failure-driven evaluation starts with a claim and tries to break it. Failed tests can justify restricting use or adding controls, followed by another test of whether errors are caught before they affect customers. The airline example makes that final interception test concrete.

## Resources

- [The Verifier Tax: Horizon Dependent Safety Success Tradeoffs in Tool Using LLM Agents](https://arxiv.org/abs/2603.19328) — 

## Related talks

- [Why 80% Reliability Isn't Good Enough — Felipe Blanes, Amazon AGI Lab](https://ai.engineer/talks/Emo5FGGY-wM-why-80-reliability-isnt-good-enough-felipe) — 
- [What Technical Due Diligence Finds When the Deck Says “70% Automated” — Soukaïna Abelhad](https://ai.engineer/talks/gDRCpTMCqEU-what-technical-due-diligence-finds-when-deck) — 

## Slides

### 1368 Failures: What They Teach Us About System Evals

Vishal Srivastava and Tanmay Sah examine 1,368 AI failures from the AI Incident Database (2010–2025) to ask how lessons from incidents can guide system-level evaluation.

[0:16](https://www.youtube.com/watch?v=mMw3arwQsL0&t=16s) [Title slide for 1368 Failures: What They Teach Us About System Evals, crediting Vishal Srivastava and Tanmay Sah and citing AIID 2010–2025. A disclaimer says the views and research are the presenters' own. Conference branding and a presenter inset surround the slide.](https://assets.ai.engineer/sites/aiecode2025/sha256/e02f95369a3e07095f713fb08f7f5f32ef07d3ee4b51342ea98fa371fd4413fc.webp)

1368 Failures
What They Teach Us About System Evals

AI ENGINEER NYC

Vishal Srivastava, PhD • Tanmay Sah, PhD
Vishal Srivastava: AI Governance at New York Life Insurance
Views and research presented are our own and do not represent our employers.

Source AIID 2010–2025

Interface text:
AI Engineer
NEW YORK
PRESENTED BY
arize
OCTOBER 13, 2026
BREAKOUT SESSION
Turing Room
Speaker

### AI Failures Are Increasing Exponentially

Annual incidents in the AI Incident Database rise from 1 in 2010 to 399 in 2025, with a sharp climb in the foundation-model era. The speakers argue that this growth calls for structured investigation.

[1:00](https://www.youtube.com/watch?v=mMw3arwQsL0&t=60s) [Slide states 399× growth in annual incidents and shows a bar chart from 2010 through 2025. Labeled counts include 1 in 2010, 100 in 2022, 169 in 2023, 292 in 2024, and 399 in 2025. A bottom banner calls for a structured way to investigate AI failures.](https://assets.ai.engineer/sites/aiecode2025/sha256/211d519e724b6359a0908f3809e67471cf661378fd6a6ed23361d05b35648a1d.webp)

AI Failures Are Increasing Exponentially

399× growth in annual incidents

FOUNDATION-MODEL ERA →

Timeline: '10, '11, '12, '13, '14, '15, '16, '17, '18, '19, '20, '21, '22, '23, '24, '25
Visible bar labels: '10: 1; '22: 100; '23: 169; '24: 292; '25: 399

We need a structured way to investigate AI failures

Interface text:
AI Engineer
NEW YORK
PRESENTED BY
arize
OCTOBER 13, 2026
BREAKOUT SESSION
Turing Room

### What Can Lead to Material AI Failures?

Deployment controls can contain development flaws before they reach business use. Assessing material impact also requires the business context: model quality is only one part of the system.

[1:41](https://www.youtube.com/watch?v=mMw3arwQsL0&t=101s) [Three connected stages read Develop, Deploy, and Use. Their boxes pair Technical Failure with where the system can break, Deployment Controls with whether failure can reach the real world, and Business Impact with what becomes material. The bottom banner states that the model is only one part of the system.](https://assets.ai.engineer/sites/aiecode2025/sha256/7161213ba41833bece6b7322ca57e50d45c0271038f7486259f3b49f65c40625.webp)

What can lead to material AI failures?

DEVELOP
Technical Failure
Where can the system break?

DEPLOY
Deployment Controls
Can failure reach the real world?

USE
Business Impact
What becomes material?

The model is only one part of the system.

Interface text:
AI Engineer
NEW YORK
PRESENTED BY
arize
Speaker
OCTOBER 13, 2026
BREAKOUT SESSION
Turing Room

### AI Failures Map to the Same 13 Dimensions

In a retrospective review, 97.7% of 1,368 AI incidents map to at least one of 13 dimensions. The categories extend from data and system design to human mediation, finance and law.

[6:34](https://www.youtube.com/watch?v=mMw3arwQsL0&t=394s) [Donut chart reports that 97.7% of 1,368 incidents map to at least one of 13 dimensions: Data Foundations, System Design, Predictive Validity, Fairness, Failure Modes, Explainability, AI Authority, Human Mediation, Exposure Control, Harm Detection, Financial, Legal and Scale. The takeaway reads, “AI failures have structure. We can map it.”](https://assets.ai.engineer/sites/aiecode2025/sha256/cb67cf72e39682b417d91a5ea9e7c52e6d9e650e1ee94847441f450ba0208281.webp)

AI Failures Map to the Same 13 Dimensions

97.7%
of 1,368 incidents
map to at least one of the 13 dimensions

1 Data Foundations
2 System Design
3 Predictive Validity
4 Fairness
5 Failure Modes
6 Explainability
7 AI Authority
8 Human Mediation
9 Exposure Control
10 Harm Detection
11 Financial
12 Legal
13 Scale

AI failures have structure. We can map it.

Interface text:
AI Engineer
NEW YORK
PRESENTED BY
arize
OCTOBER 13, 2026
BREAKOUT SESSION
Turing Room
Speaker
[illegible]

### AI Failures Occur Across Multiple Dimensions Simultaneously

AI failures in the study often span several dimensions at once. The histogram peaks at seven dimensions per incident, and the dashed line marks a median of six.

[7:02](https://www.youtube.com/watch?v=mMw3arwQsL0&t=422s) [Histogram shows incident counts by the number of failure dimensions spanned, from 0 through 13. A dashed vertical line marks median 6, and the tallest bar is at 7 dimensions. The takeaway reads, “The typical AI failure spans nearly 6 dimensions.”](https://assets.ai.engineer/sites/aiecode2025/sha256/e54fb39492a85c9ed70465698aad2803bc1de2f9129d875cf08e90947aca2872.webp)

AI Failures Occur Across Multiple Dimensions Simultaneously

median 6
Y-axis ticks: 0, 50, 100, 150
X-axis: Failure dimensions spanned per incident
X-axis ticks: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13

The typical AI failure spans nearly 6 dimensions.

Interface text:
AI Engineer
NEW YORK
PRESENTED BY
arize
OCTOBER 13, 2026
BREAKOUT SESSION
Turing Room
Speaker

### What Fails Most Often Isn’t Necessarily What Matters Most

Across 1,368 incidents (206 with material harm), severe human mediation failures are rare (0.95%) yet 4.8× more common in harm cases. Severe system design failures are far more frequent (38%) but have a weaker link to harm (≈1×).

[7:34](https://www.youtube.com/watch?v=mMw3arwQsL0&t=454s) [Scatterplot of 13 failure dimensions compares frequency of severe ratings on a logarithmic horizontal axis with relative prevalence in harm cases vertically. Colors distinguish Technical Failure, Deployment Controls and Business Impact. Human Mediation is annotated 0.95% and 4.8×; System Design is annotated 38% and approximately 1×. The takeaway recommends prioritizing potential harm over frequency.](https://assets.ai.engineer/sites/aiecode2025/sha256/f643d69bef717605268f2d3871882374fe573ab03911dcf9f1e7b6a3a0967d50.webp)

What Fails Most Often Isn’t Necessarily What Matters Most

Each dot is one of 13 failure dimensions, measured across 1,368 documented incidents (206 with material harm).

Legend:
Technical Failure
Deployment Controls
Business Impact

Y-axis: TIED TO HARM → × more common in harm cases
Y-axis ticks: 0×, 1×, 2×, 3×, 4×, 5×
X-axis: HOW COMMON → % of all incidents with a severe (3) rating · log scale

Chart labels:
Human Mediation: rare (0.95%)
but 4.8×
Harm Detection
Exposure Control
Fairness
Predictive Validity
Legal
Explainability
AI Authority
Scale
System Design: most common
(38%) but ≈1×
Financial
Data Foundations
Failure Analysis

Prioritize by potential harm, not frequency.

Interface text:
AI Engineer
NEW YORK
PRESENTED BY
arize
Speaker
OCTOBER 13, 2026
BREAKOUT SESSION
Turing Room

### Deployment Controls Shape Whether Failure Reaches Harm

In 1,368 documented incidents (206 with material harm), deployment controls show the strongest association with harm: 2.43× as common in harmful versus other cases, compared with 1.46× for business impact and 1.10× for technical failure.

[8:57](https://www.youtube.com/watch?v=mMw3arwQsL0&t=537s) [Bar chart compares how often each pillar appears in harmful versus other incidents, with 1× indicating equal prevalence. Technical Failure is 1.10×, Deployment Controls 2.43× and Business Impact 1.46×. The slide defines material harm as fatality, legal action or discrimination and reports 206 such cases among 1,368 documented incidents. The takeaway says deployment controls shape whether an error becomes harm.](https://assets.ai.engineer/sites/aiecode2025/sha256/547da303e49777da3c693464ca318a67c9e36d717a63c8f814c56c1efb136d65.webp)

Deployment Controls Shape Whether Failure Reaches Harm

Overall material harm = fatality, legal action or discrimination · 206 of 1,368 documented incidents

How much more often each pillar appears when harm occurs
Harmful vs. other incidents · 1× = equally common in both

Y-axis ticks: 0×, 1×, 2×, 3×, 4×
Technical Failure: 1.10×
Deployment Controls: 2.43×
Business Impact: 1.46×

Deployment controls shape whether an error becomes harm.

Interface text:
AI Engineer
NEW YORK
PRESENTED BY
arize
Speaker
[illegible]
OCTOBER 13, 2026
BREAKOUT SESSION
Turing Room

### Materiality Determines Evaluation Depth

Expert-designed materiality tiers were not fitted to harm data. Across 1,368 retrospectively assessed incidents, material-harm rates rise from 9.1% (36/396 low) to 33.3% (17/51 critical), supporting deeper evaluation for higher tiers.

[9:43](https://www.youtube.com/watch?v=mMw3arwQsL0&t=583s) [Slide pairs materiality scoring based on technical failure, deployment controls and business impact with a retrospective test on 1,368 incidents. The chart reports material-harm rates of 9.1% for low materiality (36 of 396 incidents), 15.9% for medium (87 of 548), 17.7% for high (66 of 373) and 33.3% for critical (17 of 51). A banner states that materiality sets how deeply to evaluate.](https://assets.ai.engineer/sites/aiecode2025/sha256/3514c850cd5ac8baee8b6cf9413b3fa80b89e5d2957f968af2e186d16c0be76d.webp)

Materiality Determines Evaluation Depth

1 DESIGN
Expert heuristics, not fitted to harm data

Technical Failure — 6 inputs
×
Deployment Controls — 4 inputs
×
Business Impact — 3 inputs

Each input scored 1–3 against written anchors; when unsure, rate higher.

Low ≤4
Medium ≤8
High ≤15
Critical >15

2 TEST
Applied retrospectively to 1,368 incidents

Y-axis ticks: 0%, 10%, 20%, 30%, 40%
LOW: 9.1%; 36 of 396 incidents
MEDIUM: 15.9%; 87 of 548 incidents
HIGH: 17.7%; 66 of 373 incidents
CRITICAL: 33.3%; 17 of 51 incidents

Material-harm rate by materiality tier. Harm rises with tier; Critical includes 51 incidents.

Materiality sets how deep we evaluate.

Interface text:
AI Engineer
NEW YORK
PRESENTED BY
arize
OCTOBER 13, 2026
BREAKOUT SESSION
Turing Room
Speaker

### Design System-Level Evals

The talk shifts from learning from failures to designing evaluations of the deployed system. The roadmap highlights this middle stage, with case applications still ahead.

[10:29](https://www.youtube.com/watch?v=mMw3arwQsL0&t=629s) [Section slide titled Design System-Level Evals, with a subtitle connecting lessons from failures to evaluation of the deployed system. Three navigation boxes read Learn from Failure, Design System Evals and Apply Through Cases; the middle box is highlighted with a purple fill and yellow border.](https://assets.ai.engineer/sites/aiecode2025/sha256/f63b8920bfb67f63d0fc7ad6d712a98620cd58dc39cd5b046828188c0989813b.webp)

Design System-Level Evals

From what failures taught us to how we evaluate the deployed system.

Learn from Failure
Design System Evals
Apply Through Cases

Interface text:
AI Engineer
NEW YORK
PRESENTED BY
arize
OCTOBER 13, 2026
BREAKOUT SESSION
Turing Room

### Decomposing AI System Level Risk

Expected harm multiplies model failure probability, the chances an error reaches a decision and goes uncaught, and severity. Authority, exposure, review and monitoring can change risk even with the same model.

[10:48](https://www.youtube.com/watch?v=mMw3arwQsL0&t=648s) [Diagram shows P(fail) multiplied by P(reach given fail), P(miss given reach) and severity, equaling expected harm Eloss given A. The factors describe model error, an error reaching a real decision, failure to catch it in time and the severity of consequences. A denotes deployment authority, exposure, human review and monitoring. The takeaway states that the same model can have different risk under different deployment setups.](https://assets.ai.engineer/sites/aiecode2025/sha256/b1317964e603d26ea39d2357492cb39a2ee92016443b81bae4e55e141ecd0261.webp)

Decomposing AI System Level Risk

P(fail)
The model gets it wrong
Set by: model quality, inputs

×

P(reach | fail)
The error reaches a real decision
Set by: authority, exposure

×

P(miss | reach)
Nobody catches it in time
Set by: human review, monitoring

×

severity
How bad it is when it lands
Set by: financial, legal, scale

=

E[loss | A]
expected harm

A = the deployment setup around the model: authority · exposure · human review · monitoring

Same model, different A, different risk.

Source: Srivastava & Sah (2026), arXiv:2602.18986 (preprint), adapted.

Interface text:
AI Engineer
NEW YORK
PRESENTED BY
arize
OCTOBER 13, 2026
BREAKOUT SESSION
Turing Room

### We already challenge models. Extend the boundary.

Validation expands from the model to the deployed system, including retrieval and tools. Green boxes mark human workflow, authority, exposure and monitoring—the setup that shapes whether errors cause harm.

[11:45](https://www.youtube.com/watch?v=mMw3arwQsL0&t=705s) [Two panels compare traditional model validation with deployed AI system evaluation. Model validation covers assumptions, input data, methodology, sensitivity, stress conditions, model limitations, performance monitoring and outcomes analysis. An arrow extends the evaluation scope to model, data and retrieval, rules, tools, controls, recovery, human workflow, authority, exposure and monitoring. Green highlights mark the deployment setup. The bottom banner reads: Same logic of challenge. Bigger boundary.](https://assets.ai.engineer/sites/aiecode2025/sha256/e96757fa012d9651b5a1f84a342b10ba3f68a3dd7f0246219460b1bc0d60752a.webp)

We already challenge models. Extend the boundary.

TRADITIONAL MODEL VALIDATION
Where can the model fail?
Assumptions
Input data
Methodology
Sensitivity
Stress conditions
Model limitations
Performance monitoring
Outcomes analysis

EXTEND THE
UNIT

DEPLOYED AI SYSTEM
green = A, the deployment setup
Where can the system fail?
Model
Data / retrieval
Rules
Tools
Human workflow
Authority
Controls
Exposure
Monitoring
Recovery

Same logic of challenge. Bigger boundary.

Interface text:
AI Engineer
NEW YORK
PRESENTED BY
arize

### We already challenge models. Extend the boundary.

The arrow extends validation from the model to the deployed system: can human review catch mistakes, can the agent stay within its permissions, and can the system recover? Green marks the deployment setup that shapes risk.

[11:52](https://www.youtube.com/watch?v=mMw3arwQsL0&t=712s) [Slide contrasts traditional model validation with evaluation of a deployed AI system. An arrow labeled 'Extend the unit' connects the panels. The system panel lists model, data/retrieval, rules, tools, controls, and recovery, with human workflow, authority, exposure, and monitoring highlighted in green. The takeaway reads, 'Same logic of challenge. Bigger boundary.'](https://assets.ai.engineer/sites/aiecode2025/sha256/14c46187b79bc6d5751a9d54497a7314dd58a2d1244cc1aaebb0ff6ab1af36e3.webp)

We already challenge models. Extend the boundary.

TRADITIONAL MODEL VALIDATION
Where can the model fail?
Assumptions
Input data
Methodology
Sensitivity
Stress conditions
Model limitations
Performance monitoring
Outcomes analysis

EXTEND THE
UNIT

DEPLOYED AI SYSTEM
green = A, the deployment setup
Where can the system fail?
Model
Data / retrieval
Rules
Tools
Human workflow
Authority
Controls
Exposure
Monitoring
Recovery

Same logic of challenge. Bigger boundary.

Interface text:
AI Engineer
NEW YORK
PRESENTED BY
arize
Engineering the future of AI

## Full transcript

[0:12](https://www.youtube.com/watch?v=mMw3arwQsL0&t=12s) A big part of my research is looking into AI failures, investigating it, and then learning from it. And joining me is Tanmay Sah.

[0:21](https://www.youtube.com/watch?v=mMw3arwQsL0&t=21s) Yeah. Hi, everyone. I'm Tanmay Sah. I am a senior quantitative modeler and an AI researcher. This is my second time here, so it's great to be back. So I will hand it over to Vishal to get us started.

[0:35](https://www.youtube.com/watch?v=mMw3arwQsL0&t=35s) So we looked at AI Incident Database. It's a publicly available AI incident database. And, uh, we looked at, um, the AI failures and asked one simple question, what these, uh, failures, um, uh, teach us about how the sys... how the AI system should be evaluated, and if we can, um, apply those learnings to system-level evaluation.

[1:02](https://www.youtube.com/watch?v=mMw3arwQsL0&t=62s) So the first thing that we see is that AI failures, um, have been increasing exponentially. And if you look at, um, the last four, um, years of, um, AI failures, particularly in the foundational model era, you see that the jump is almost, uh, vertical. Um, and many of us feel that, um, the number of AI use cases that we have in enterprises, um, they, um, they are much larger than our

[1:32](https://www.youtube.com/watch?v=mMw3arwQsL0&t=92s) capacity to review them each one by one. So what we argue is that we need a structured way, um, to investigate these AI failures.

[1:43](https://www.youtube.com/watch?v=mMw3arwQsL0&t=103s) And to do that, um, we look at the AI failures essentially into three dimensions. The first dimension is the development stage when, um, the model and or the AI system is developed. So it could be issues with respect to, uh, data quality or tool architecture or system design. Any of these things during the development phase can, um, can break, and once it happens,

[2:13](https://www.youtube.com/watch?v=mMw3arwQsL0&t=133s) then those errors can be propagated to the next dimension, which is the deployment phase. Now, even if the model or the system, um, has some faulty design issues, but if you have sufficient controls in deployment, those errors can be contained. And that is why it is very important to kind of understand what kind of controls do we have in deployment and whether, uh, we can contain those

[2:43](https://www.youtube.com/watch?v=mMw3arwQsL0&t=163s) failures, uh, from reaching from deployment, uh, production platform to the final stage, which is the context in which the AI system is being used from a business standpoint. It could lead to, uh, reputational risk, it could lead to legal or financial risk, and all those business context really matters, um, for the AI system in which it is being used. And what we say-- what we propose is that the model alone or the

[3:13](https://www.youtube.com/watch?v=mMw3arwQsL0&t=193s) technical strength of the model is just one part of the system. We have to look at the system holistically.

[3:23](https://www.youtube.com/watch?v=mMw3arwQsL0&t=203s) So after defining those three core dimensions, the technical strength where the system, um, should be strong or in opposite way where the system can break. The second dimension is how do, uh, we ensure that the deployment controls are in place. And third, the business context, uh, the AI system is being used. We divide each of these core dimensions into several sub-dimensions.

[3:52](https://www.youtube.com/watch?v=mMw3arwQsL0&t=232s) So in the... from the technical strength point, right, there could be six major sub-dimensions which could be affecting the technical strength of the AI system. It could be, uh, first related with the data quality that we use for to build the AI system, how the system has been de-designed for the appropriate task that it is supposed to accomplish. How is the predictive validity or outcome analysis of the out-- of, of the AI system? Does that system perform as we

[4:22](https://www.youtube.com/watch?v=mMw3arwQsL0&t=262s) intend? Um, there could be fairness-related, um, checks. Um, failure modes essentially looks at where, when, how, and for whom the, uh, system can break. Uh, so basically mapping out all the failure modes of the system. And finally, can we explain for, uh, explain the decisions for the regulators or other bodies, uh, so that we know how the system is arriving at its decision? So these six sub-dimensions

[4:52](https://www.youtube.com/watch?v=mMw3arwQsL0&t=292s) are essentially part of the technical strength of the AI system.

[5:00](https://www.youtube.com/watch?v=mMw3arwQsL0&t=300s) The next four are essentially looking at how we are using the model. Um, what kind of authorities we have given to the AI, right? Is the AI for informing us or it, it is out there to act on our behalf? What kind of decision authority are we associating with respect to AI? Uh, second one, human mediation. Um, how effective is human-in-the-loop process is, right? That's really important. The third one, exposure control. Who's actually using the, uh,

[5:29](https://www.youtube.com/watch?v=mMw3arwQsL0&t=329s) system? Is it internal employees so that they see the error and they can contain it, or it is exposed to the out there in the wild external customers, so that any small issue can really propagate fast? And finally, what kind of monitoring, uh, controls that we have so that we see some, uh, issues in production and we detect it and we can contain it.

[5:54](https://www.youtube.com/watch?v=mMw3arwQsL0&t=354s) Finally, in the financial, um, or the business impact phase, we see these three core sub-dimensions. What kind of financial impact it would make if there is an error, there could be legal consequences, and how many users would get affected? That's the, the last one, the scale sub-dimension.

[6:15](https://www.youtube.com/watch?v=mMw3arwQsL0&t=375s) So we, we, we look at this, uh, this as one framework which has thirteen, uh, dimensions and three core questions as we explained. Once we define these thirteen sub-dimensions and look back at AI incident retrospectively, we find that using these thirteen sub-dimensions, almost ninety-eight percent of AI failures can be covered. Meaning that AI failures are ha-- are associated with some of these

[6:45](https://www.youtube.com/watch?v=mMw3arwQsL0&t=405s) dimensions. In some cases, mul-multiple of these dimensions, and in some cases, few of these dimensions. But these dimensions are involved. So failure, um, have... AI failures have structures. We need to map it, and we need to act accordingly, um, a-according to these sub-dimensions. Now, as I mentioned that some of these, um, failure sub-dimensions can be present in, uh, these failures simultaneously, meaning that when something goes wrong with

[7:15](https://www.youtube.com/watch?v=mMw3arwQsL0&t=435s) respect to a particular failure, there may be multiple things going, uh, wrong. What we find is that on average, there are six, uh, failure sub-dimensions that can be present, um, in each of these AI incident failures that, um, we, we, we have investigated.

[7:35](https://www.youtube.com/watch?v=mMw3arwQsL0&t=455s) Now, the question comes to our mind is that, uh, are all of these sub-dimensions are equal, um, or some are more important than others? And the answer is that, um, some of the sub-dimensions can be, uh, more critical than others. And what I'm showing you here in the, in the plot on the y-axis is the association to harm when these sub-dimensions are present. And on x-axis, we have the frequency of these sub-dimensions being present. What we see on the left

[8:05](https://www.youtube.com/watch?v=mMw3arwQsL0&t=485s) upper side of the plot, uh, sub-dimensions like human mediations or effectiveness of human-in-the-loop, harm detection, meaning that whether we have monitoring control in place. On, on x-axis, we see things like scale and system design, although they are present in many cases, but they are not associated so much, uh, with harm as some of the, some of, some of the earlier sub-dimensions that I have mentioned. So when we are really looking at the

[8:35](https://www.youtube.com/watch?v=mMw3arwQsL0&t=515s) AI systems, some of these sub-dimensions can be really critical. So we need to prioritize these, uh, sub-dimensions by the potential of making harm and then allocate appropriate resources to the, to the investigation of these sub-dimensions rather than looking at all the failure sub-dimensions equally.

[8:59](https://www.youtube.com/watch?v=mMw3arwQsL0&t=539s) And when we club the, uh, when we group these sub-dimensions together, we see that controls that we have in deployment are more critically associated with harm, followed by the business impact, and followed by the technical failure. So this plot is essentially telling us that once, um, once we have deployment controls taken care of, um, for the high-risk, uh, system, um, that gives us some level of, uh, assurance that, uh, we can take care

[9:29](https://www.youtube.com/watch?v=mMw3arwQsL0&t=569s) of, uh, the failures in ad-- in advance, in contrast to just looking at the technical failure alone. So all these sub-dimensions are important, but deployment controls are far more important.

[9:43](https://www.youtube.com/watch?v=mMw3arwQsL0&t=583s) And, uh, what-- In the next phase, what we did was we, um, we designed a materiality, uh, scale, meaning that if we can understand which AI use cases are more risky compared to some others, we can, uh, we can prioritize, uh, these AI use cases based on materiality, because materiality directly rank orders with respect to the harm associated with these AI use cases. Meaning that,

[10:13](https://www.youtube.com/watch?v=mMw3arwQsL0&t=613s) uh, we, uh, can spend... Suppose we know which AI cases are mo-more material, then we can allocate more resources to much higher material AI use cases than compared to some other AI use cases which may not be as material. With that, I would hand over to Tanmay, who would explain how we can use those learnings to build system-level evals.

[10:37](https://www.youtube.com/watch?v=mMw3arwQsL0&t=637s) Thanks, Vishal. Also, so far we have seen what this failure teaches. So now let us use those lessons, uh, to build system-level evaluations. So first, first of all, what we did is that we have decomposed risk into four components, which are how often the model fails, whether that failure reaches a real decision, whether it cause-- whether it goes uncaught, and how severe the

[11:08](https://www.youtube.com/watch?v=mMw3arwQsL0&t=668s) consequences are. So in this scenario, the deployment setup matters. How much authority does the system have? Does someone review its output? Can monitoring catch error before it causes harm? So the same model can create different risk depending on how we deploy it. So what we are saying is that if risk depends on the whole deployment system, our evaluation

[11:38](https://www.youtube.com/watch?v=mMw3arwQsL0&t=698s) needs to cover that setup too. So that brings us to the next question: What should we test in an agentic system? So see, in traditional machine learning, we already know how to challenge models. We test their assumptions, input data, and behavior under stress. Now, when we extend that discipline to the whole deployment system, then, like, does retrieval, uh, bring back the right in-information? Or what happens when a tool

[12:08](https://www.youtube.com/watch?v=mMw3arwQsL0&t=728s) fails? Does human review catch mistake? Or does the agent stay within its permission or boundary? And can the system finally recover? So here the logic stays the same. The boundary of what we test gets bigger. So Now the question is: How should we test an agentic system? So on the right, we have the familiar traditional ML. We test a model on held-out data, calculate metrics like accuracy and AUC,

[12:38](https://www.youtube.com/watch?v=mMw3arwQsL0&t=758s) ROC, use those evidence to inform deployment. But we still need to check the data and assumptions behind those metrics. But on the left-hand side, we have an-- this agentic system, which adds new complexity. So the deployment system, like, it has many possible inputs, tool responses, and action sequences or trajectory or rollout, whatever you desc-- say. So we test selected scenarios. Sometimes another

[13:08](https://www.youtube.com/watch?v=mMw3arwQsL0&t=788s) AI, uh, agent or AI judge generates those tests or judges the result. So, but that judge can make mistakes, and the test can miss important failures. So this is where we need to check whether the evaluator is reliable, what the evidence covers, and whether it applies to the system we are actually deploying it. That's why the evaluator is the part of the system we govern. So

[13:37](https://www.youtube.com/watch?v=mMw3arwQsL0&t=817s) now I will hand it over to Vishal to explain how we can check whether an evaluation is reliable and what to do when it is not.

[13:48](https://www.youtube.com/watch?v=mMw3arwQsL0&t=828s) Thank you, Tanmay. So, uh, we have seen the system-level evaluation for evaluator as well as, uh, you know, the whole AI system. The question is that how should we really evaluate? Um, and, uh, when I was looking at, uh, some of the previous research, particularly in the area of cognitive neuroscience, what I found was the, um, this j-- um, this scientist Pa-- uh, Peter Wason, he's the one who, uh, coined this term, uh, uh,

[14:18](https://www.youtube.com/watch?v=mMw3arwQsL0&t=858s) confirmation bias. And essentially what he did was he, uh, did a bunch of experiments. But essentially, uh, in this case here, uh, the series that you see two, four, six, right? He was asking people to predict the next number, and more often than not, they were predicting eight, ten, like, even numbers. But, um, th-those answer was-- those answers were incorrect because essentially the correct number was any increasing numbers. Essentially proving that all humans, like most humans, um, in this case it, uh, seventy-nine percent of the people

[14:48](https://www.youtube.com/watch?v=mMw3arwQsL0&t=888s) tested w-wrong, uh, may have some sort of confirmation bias. And AI systems, as we know, uh, are trained on human, uh, generated data. So, uh, what, what he proposed was that any evaluation or any criti-- any evaluation that needs to be done has to falsify, not to confirm, and that, that's the way to go about it. And we take that, uh, learning from his research and propose that eval system needs to be failure-driven.

[15:19](https://www.youtube.com/watch?v=mMw3arwQsL0&t=919s) So if the eval systems are failure-driven, what would that mean? That would mean that you would start right from looking at your system claim and, uh, testing it and trying to break it, and then i-identify where the system claims are breaking, right? And then what you do for fix, either you, uh, don't let the AI system run in, you know, in the, in the, in the territory where, um, there are failures, or you come up with, uh, supporting

[15:50](https://www.youtube.com/watch?v=mMw3arwQsL0&t=950s) systems, right? Which you are confident of. And you keep doing it iteratively, uh, until you get, um, you to the, to the level of confidence, right? This is essentially the first principles ba-based approach that we propose for any evaluation system. So essentially, you ta-take your AI systems, perform this build, break, fix loop, understand where the AI systems could be trusted in terms of its, uh, uh, final outcomes, and also t--

[16:20](https://www.youtube.com/watch?v=mMw3arwQsL0&t=980s) map out, uh, the uncertain region and high-risk region where you can not be very sure of the system's performance. And you map out the entire failure region map and route it based on the reliability and materiality. Meaning that, particularly if you are using AI for, uh, any high-risk related system, make sure that your AI is only being used for, um, the, the, the region that you are confident

[16:50](https://www.youtube.com/watch?v=mMw3arwQsL0&t=1010s) of.

[16:53](https://www.youtube.com/watch?v=mMw3arwQsL0&t=1013s) And with this, I will hand over to Tanmay, who w- who, who, who will apply this approach to a real, uh, AI failure case.

[17:03](https://www.youtube.com/watch?v=mMw3arwQsL0&t=1023s) Okay, yeah, sure. So, uh, so, uh, let's make this framework concrete and see how a narrow evaluation can miss failures, uh, that matters in the real world. So I will quickly go through the story behind it. So a customer needed to travel after a death in the family. He asked an airlines chatbot about a discounted fare. The chatbot said he could buy a regular ticket and claim the discount afterwards within ninety days. So he relied on that advice, but the airline l-later refused because its

[17:33](https://www.youtube.com/watch?v=mMw3arwQsL0&t=1053s) policy, uh, did not allow that request. Then finally, the tribunal held the airline responsible. So through this example, we, we are seeing the failure chain, so it's crystal clear. So incorrect advice reaches the customer, the customer acts on it, and the organization faces the consequences. So if we apply our build, break, fix, and break again approach, uh, that could have exposed the failure way before the deployment.

[18:03](https://www.youtube.com/watch?v=mMw3arwQsL0&t=1083s) So we've-- ideally, we would start with a clear requirement like advice must match the approved policy. Then we would challenge the chatbot with questions about refunds after travel. If those tests exposed incorrect advice, then we could add a policy check or human escalation. Then we would test again to see whether those controls caught the errors before it reached the customer. So that's our framework in practice. We have to define the claim,

[18:34](https://www.youtube.com/watch?v=mMw3arwQsL0&t=1114s) actively challenge it, and use the evidence to guide changes and make a deployment decision. So finally, I will hand it over to Vishal to bring the pieces together and conclude.

[18:49](https://www.youtube.com/watch?v=mMw3arwQsL0&t=1129s) And this is our final slide. So essentially, what we are saying is that your evals should be risk-based, um, rather than frequency-based. You should look at, uh, your, uh, entire system and then break it into the core components like technical, uh, failure, deployment controls, and business context. And then assess your AI use cases materiality with some, uh, scale, uh, and that could be your company's specific scale, and then that, that, that risk should--

[19:19](https://www.youtube.com/watch?v=mMw3arwQsL0&t=1159s) that risk could change. And then once you have that, uh, apply the system-level stress testing on the AI use case. And then once you are confident about the region, um, then only make the deployment decision. Otherwise, it, it, it, it may be too expensive. Our final takeaway message is evaluate the full AI system based on risk, not just w-one component. Um, thank you so much for your time and attention. Uh, you can feel free to

[19:49](https://www.youtube.com/watch?v=mMw3arwQsL0&t=1189s) reach out to us outside. Um, we'll be happy to take any questions later.
