# Guardrails First: Engineering Member-Facing Health AI

Rashi Agrawal

AI Engineer World's Fair 2026

[Talk page](https://ai.engineer/talks/YXEqC05WEI0-guardrails-first-engineering-member-facing-health) · [Video](https://www.youtube.com/watch?v=YXEqC05WEI0)

## Guardrails First: Engineering Member-Facing Health AI

Healthcare AI needs privacy boundaries before storage, deterministic controls before generation, and continuous evaluation backed by people who can decide when to hold a launch.

Familiarity with LLM prompts, application routing, and basic production monitoring will help; no clinical background is required.

### When dietary advice becomes an emergency

[0:18](https://www.youtube.com/watch?v=YXEqC05WEI0&t=18s)

How do you ship AI to someone who already trusts you with their health? Rashi Agrawal, introducing herself as the leader of AI and ML at Hinge Health, starts with the scale of that responsibility. She cites 40 million people using frontier models for healthcare triage; the supporting [ECRI account](https://home.ecri.org/blogs/ecri-news/misuse-of-ai-chatbots-tops-annual-list-of-health-technology-hazards) describes a broader activity: more than 40 million daily users of **ChatGPT** seeking health information.

The danger becomes concrete in her account of a healthy person who asked an AI assistant how to reduce dietary salt. The assistant reportedly suggested sodium bromide as a substitute. Agrawal recounts that after three months the person developed paranoia and hallucinations, had bromide levels reported as 200 times the safe limit, and spent three weeks in hospital. An ordinary dietary question had become an emergency.

Agrawal next cites roughly 50% under-triage of life-threatening emergencies in a Mount Sinai evaluation of **ChatGPT Health**. This was a structured scenario evaluation at one point in time, not a measured failure rate across all patient encounters. Her examples include diabetic ketoacidosis and respiratory failure: the system recommended seeing a doctor in a day or two when the appropriate response was immediate emergency care. ECRI ranked AI chatbot misuse first among its health technology hazards for 2026, announcing the list in January. These failures establish the production problem: a useful conversational answer can still direct someone toward the wrong action.

### Make privacy a property of the architecture

[2:39](https://www.youtube.com/watch?v=YXEqC05WEI0&t=159s)

Agrawal organizes the response around three foundations. **Constraints shape the architecture** before a model generates a token. Deterministic rules govern behavior that cannot be left to probability. Safety evaluation continues after launch, when real production risk begins. Those foundations also establish the limits of automation: when the architecture cannot settle whether something should ship, a human must make the decision.

Start with **protected health information**, or PHI. A reactive design lets sensitive data enter storage and depends on redaction when a log appears in a dashboard. The architecture Agrawal describes removes PHI at the ingestion boundary, before it reaches the data lake. Developers then open dashboards whose underlying stored data never contained that PHI. Moving the control upstream removes the need for every downstream viewer to get redaction right.

That boundary extends to environments and access. Production and non-production remain separate, without connecting pipelines that can carry member data into development. Raw-PHI access depends on both an authorized role and an approved geographic region; in the policy Agrawal describes, an engineer outside that region cannot reach the data. These geographic restrictions describe her system's controls, not a universal HIPAA prohibition on overseas processing.

HIPAA, the FDA's [Good Machine Learning Practice principles](https://www.fda.gov/medical-devices/software-medical-device-samd/good-machine-learning-practice-medical-device-development-guiding-principles), and Texas's [Responsible Artificial Intelligence Governance Act](https://capitol.texas.gov/tlodocs/89R/billtext/html/HB00149F.HTM) enter the design before implementation. Their legal scope differs: GMLP supplies guiding principles for medical-device development, while statutes impose applicable legal obligations. The engineering consequence is shared: identify the relevant constraints first, then build the data flows and access paths around them. A privacy policy becomes stronger when the system removes the path by which a prohibited disclosure could occur.

### Run code before the model on every turn

[5:56](https://www.youtube.com/watch?v=YXEqC05WEI0&t=356s)

Probabilistic models are useful for generation, but a high-stakes decision needs an execution boundary outside generation. Agrawal's stack puts **code above the model**: every conversation turn passes through the code layer first. That layer decides whether to escalate an emergency or bring in a clinician. Most turns continue to the model, which handles the long tail of member conversations; it does not get to overrule an escalation.

![Slide showing a code layer stacked above a model layer, with three panels describing generation, irreversible decisions, and long-tail model behavior.](https://ai.engineer/api/talk-essay-media/3b10facc87cb302c1341cdfddc6ffe9fd34a20b1d99f37ae0da4b5860e6b2035)

Must-not-fail decisions belong in the code layer above the model.

A model alone is not a guardrail, and adding a system prompt does not make it a security boundary. The [OpenAI Model Spec](https://model-spec.openai.com/2025-12-18.html) defines the authority order `root → system → developer → user → guideline`. That hierarchy specifies intended instruction priority. Agrawal warns that prompt injection can undermine prompt-based controls, using the stronger formulation that every layer above the user is one injection away from override. The specification does not establish that categorical claim; the actionable architectural point is that instruction priority cannot replace an independently enforced security check.

Three controls make the distinction concrete:

- **Emergency escalation:** self-harm, suicidal ideation, or an acute medical emergency triggers an appropriate route to 911 or 988. In Agrawal's design, the conversational model never sees that turn. The talk specifies the routing requirement but does not detail how emergencies are reliably detected.
- **Capability routing:** a multi-agent system may contain clinical, technical-support, educational, and exercise-recommendation capabilities. A model can assist with classification, but high-stakes paths need enforced routing. A clinical question must not silently fall through to generic technical support.
- **Identity verification:** an operation touching member data must verify that the person at the other end is the correct member. Authentication supplies the security boundary; a prompt asking the model to respect identity does not.

A TypeScript dispatch boundary can express that separation. Here, `decision` is the result of upstream controls, including authentication where member data is involved. Only an explicitly permitted model route can reach generation:

```typescript
type Decision =
  | { kind: "emergency"; destination: "911" | "988" }
  | { kind: "clinician" }
  | { kind: "authentication-required" }
  | {
      kind: "model";
      capability: "clinical" | "tech-support" | "education" | "exercise";
    };

type Action =
  | Exclude<Decision, { kind: "model" }>
  | { kind: "generate"; capability: string; text: string };

function dispatch(text: string, decision: Decision): Action {
  switch (decision.kind) {
    case "emergency":
    case "clinician":
    case "authentication-required":
      return decision;
    case "model":
      return {
        kind: "generate",
        capability: decision.capability,
        text,
      };
  }
}
```

The function returns a proposed action; it does not contact emergency services or execute a model call. Its useful property is the control flow: once an upstream control selects escalation or authentication, generation is no longer an available branch.

### Evaluate live conversations—and staff the response

[9:35](https://www.youtube.com/watch?v=YXEqC05WEI0&t=575s)

Pre-launch tests remain necessary, but a saved golden dataset cannot reveal every failure that emerges in live conversations. The third layer continuously scores production traffic across multiple dimensions. Agrawal combines three signal sources because each exposes a different part of the system's behavior.

- **Automated judges:** Agrawal suggests 30–40 judges as an illustrative scale, covering clinical accuracy, safety, escalation, relevance, drift, refusal, and other dimensions. The judge set keeps evolving as the team learns what to detect.
- **Member feedback:** per-message thumbs up and thumbs down provide a direct signal from the person receiving the answer. That signal can reveal tone problems and failures that automated judges miss.
- **Trace review:** random samples span capabilities, while Agrawal calls for review of 100% of high-stakes cases. People still need to read the underlying conversations and interpret patterns that no single score captures.

**The bottleneck is the capacity to interpret signals and act on them.** More compute or more judges does not supply the people needed to investigate a concerning trace, recognize a recurring pattern, and decide what to change. Review capacity therefore belongs in the operating design, alongside inference and evaluation capacity.

Some failures return after a prompt fix because the surrounding conditions change: a new prompt, a new tool, or a different model behavior exposes another version of the problem. Successive fixes may yield diminishing returns without eliminating the failure. Agrawal's response is to turn a newly observed production failure into a new judge, and to make the monitoring architecture capable of accommodating that growing set. Monitoring is how the team checks whether its architectural protections continue to hold as usage expands.

### One unresolved issue, five different risks

[12:39](https://www.youtube.com/watch?v=YXEqC05WEI0&t=759s)

Monitoring can also reveal a problem that the architecture does not resolve. Agrawal introduces a launch scenario: one issue remains five days before a new healthcare AI capability is due to ship. Five stakeholders examine the same issue and disagree because they are evaluating different consequences.

| Stakeholder | Risk they see | Launch concern |
| --- | --- | --- |
| Clinical | Member safety | Hold the launch to prevent harm |
| Legal | Regulatory exposure | Consequences of noncompliance |
| Compliance | Audit risk | Ability to satisfy scrutiny |
| Product | Adoption risk | A broken feature may not land |
| Engineering | Velocity risk | A fix means slipping the date |

Clinical wants to hold; engineering wants to ship. The disagreement does not require anyone to be irrational. Each function sees a real risk, but identifying those risks still leaves the release decision unresolved.

![Five panels pair Clinical with member safety risk, Legal with regulatory exposure, Compliance with audit risk, Product with adoption risk, and Engineering with velocity risk.](https://ai.engineer/api/talk-essay-media/c4d7537a0f359f4a08b72c51870c3da08a0fa0a1479016cb17b46397cbbbca73)

Five stakeholders see five different launch risks.

### Set severity by harm, then choose the safer mistake

[14:06](https://www.youtube.com/watch?v=YXEqC05WEI0&t=846s)

The first of Agrawal's five decision rules sets severity by the **worst plausible outcome**, rather than the average experience. Her comparison is deliberately asymmetric: minor annoyance affecting 100% of users is less severe than potential serious harm in 0.1% of cases. The triage question starts with the worst credible version of the failure; frequency alone cannot set its severity.

The second rule separates **severity from capacity**. Ownership, staffing, and the difficulty of a fix do not change the harm a bug can cause. Once severity is established, the available responses are to fix it, delay launch, or accept the risk with explicit sign-off. Quietly downgrading an issue because the team cannot reach it substitutes a scheduling constraint for a safety assessment.

The third rule gives uncertain decisions an asymmetric default:

| Issue type | More costly mistake | Default under uncertainty |
| --- | --- | --- |
| Safety | Shipping a real harmful defect | Hold and fix |
| Polish | Delaying over a small flaw | Ship |

For safety, an unnecessary delay is preferable to exposing members to a real hazard. For polish, the cost of delaying can exceed the cost of the imperfection. This framework supplies a direction for judgment, not an automatic release verdict.

### Use the organization's actual bar—and fund the follow-through

[16:28](https://www.youtube.com/watch?v=YXEqC05WEI0&t=988s)

The fourth rule distinguishes **revealed risk tolerance** from stated risk tolerance. An organization may say it accepts no production bugs while continuing to ship behavior that contradicts that standard. Agrawal argues that if a behavior has been live for weeks or months without escalation, member complaints, or leadership concern, it should not become a blocker solely because it appears in a new feature. Existing production behavior supplies the calibration floor for a consistent launch bar.

The fifth rule returns to the human constraint. Judges can score traces automatically and dashboards can refresh every few hours, but people must interpret patterns and act on them. A launch plan that includes monitoring without allocating review capacity leaves that responsibility unfulfilled. And work deferred to a fast follow is already committed debt: it does not become an optional backlog item once the launch succeeds.

### Verify the scorer before changing the agent

[17:56](https://www.youtube.com/watch?v=YXEqC05WEI0&t=1076s)

All five rules depend on trustworthy evidence. Yet an LLM judge is itself nondeterministic. In Agrawal's hypothetical example, a clinical-accuracy score stays at 4.9 for weeks, then drops to 4.5 and remains there the next day. Immediately changing the agent prompt could introduce a regression while leaving the actual source of the score change untouched. The first investigation is whether the judge is right.

Her caffeine example separates two superficially similar alerts. In scenario A, a member asks about caffeine and receives general guidance mentioning 400 mg per day for most adults, with qualifications concerning pregnancy and medications. The [FDA guidance](https://www.fda.gov/consumers/consumer-updates/spilling-beans-how-much-caffeine-too-much) describes that amount as not generally associated with negative effects for most adults; it advises consulting a healthcare provider about restrictions in circumstances such as pregnancy or medication use. It is not a personalized safe dose. The judge nevertheless labels the response a hallucination because the agent mentioned pregnancy and medications without first checking whether those circumstances apply. In this example, those qualifications supply clinical context rather than assert facts about the member. The judge is over-calling.

| Scenario | Agent response | Judge behavior | Component to fix |
| --- | --- | --- | --- |
| A | General caffeine guidance with conditional qualifications | Incorrectly flags the qualifications | Judge |
| B | Says 1,000 mg of caffeine daily is fine | Correctly flags the unsafe endorsement | Agent |

The second scenario changes the answer, not the member's question. Here the judge correctly identifies a problem in the agent's guidance. The same alert category can therefore require changes to different components; a lower score alone does not tell the team which component is wrong.

The resulting investigation follows a short sequence:

1. Inspect the response and the judge's reason for flagging it.
2. Establish whether the judgment is correct before changing the agent.
3. Correct the judge when it over-calls valid context; correct the agent when its answer is wrong.

A judge prompt is software that needs maintenance. Correcting an erroneous evaluator preserves the meaning of its score; it is not cheating the evaluation.

### Build a system worthy of member trust

[20:18](https://www.youtube.com/watch?v=YXEqC05WEI0&t=1218s)

Agrawal closes the architectural argument with three compact formulations: “Don't policy what you can architect.” “Don't prompt what you can code.” “Don't gate what you can monitor.” Each moves a responsibility into a mechanism that can enforce or continually check it. Human release decisions then address the residual risk, using worst-case severity, safer defaults, an explicit organizational bar, and committed follow-through.

Building these guardrails first can slow development. For member-facing healthcare AI, that cost is intentional: the product must be worthy of a person's trust in matters affecting their health. Architecture determines how the system can operate; accountable decisions determine when it should ship. As Agrawal puts it, “The architecture is how, the decisioning is when, and member trust is why.”

![Closing slide pairs a sunlit wall photograph with three statements: “Architecture is how,” “Decisioning is when,” and “Member trust is why.”](https://ai.engineer/api/talk-essay-media/9e0bdab9ce6ac5006d90db4eb6be69b5b088bef2808b21e0ac0d7695cca925b4)

Architecture is how. Decisioning is when. Member trust is why.

## Resources

- [ECRI’s 2026 health technology hazards](https://home.ecri.org/blogs/ecri-news/misuse-of-ai-chatbots-tops-annual-list-of-health-technology-hazards) — ECRI’s ranked hazards and recommendations for responsible use of healthcare chatbots.
- [FDA guidance on caffeine consumption](https://www.fda.gov/consumers/consumer-updates/spilling-beans-how-much-caffeine-too-much) — Explains the 400 mg daily figure for most adults, individual sensitivity and circumstances requiring medical advice.
- [Good Machine Learning Practice principles](https://www.fda.gov/medical-devices/software-medical-device-samd/good-machine-learning-practice-medical-device-development-guiding-principles) — FDA’s overview of international guiding principles for developing AI and machine learning medical devices.
- [Texas Responsible Artificial Intelligence Governance Act](https://capitol.texas.gov/tlodocs/89R/billtext/html/HB00149F.HTM) — Enrolled legislation covering AI governance, prohibited uses and healthcare disclosure requirements.
- [OpenAI’s instruction hierarchy](https://model-spec.openai.com/2025-12-18.html) — The December 2025 Model Spec defines instruction authority from root rules through guidelines.

## Slides

### State of Healthcare AI

A poisoning case, a test in which a health AI under-triaged 51.6% of life-threatening emergencies, and ECRI’s 2026 hazard ranking frame the challenge of delivering AI to people who trust providers with their health.

[0:52](https://www.youtube.com/watch?v=YXEqC05WEI0&t=52s) [Healthcare AI safety headlines and a question about delivering AI to people who trust providers with their health.](https://assets.ai.engineer/sites/aiecode2025/sha256/fed9dafb79b9bdbe06c2d03159d7b503e58d2061a5fa823cd0e729eb9d1fb132.webp)

State of healthcare AI.

Poisoned by a chatbot — 2025
A 60-yr old man landed in ER because a popular AI chatbot told it to replace salt with sodium bromide.

Half of Emergencies Missed — 2026
The first independent safety test, out of Mount Sinai, found it under-triaged 51.6% of life-threatening emergencies.

#1 Health Hazard — 2026
ECRI, the patient-safety group hospitals trust to rank their top risks, named AI chatbot misuse the number one health technology hazard of 2026

"So how do you ship AI to someone who has already trusted you with their health?"

Sources: Live Science (Bromism syndrome), Nature Medicine (Ichan School of Medicine), ECRI (Becker's Hospital Review)

### Three Non-Negotiable Foundations

Healthcare AI safety starts with architectural constraints and deterministic rules that run before the model. Evaluation continues after launch, when operational risk persists.

[2:36](https://www.youtube.com/watch?v=YXEqC05WEI0&t=156s) [Three foundations: architectural constraints, deterministic rules above the model, and continuous safety evaluation.](https://assets.ai.engineer/sites/aiecode2025/sha256/0ab1daef471495c9c19bcc15eb39942fc8e96523987eb1d9fec82aafb6bb8f93.webp)

3 Non-Negotiable Foundations

These are not aspirations or guidelines. They are the structural commitments that determine whether a healthcare AI system is safe to operate.

1
The constraint is the architecture.
Most AI safety failures aren't model failures. They're architectural decisions made before a single token was generated.

2
Deterministic rules belong above the model, not inside it.
What can never be wrong must never be left to probability. Code runs before the model on every turn.

3
Safety is a continuous evaluation layer, not a one-time gate.
Launch is the beginning of sustained operational risk, not the finish line.

Views expressed are my own and do not represent Hinge Health

### Layer 1: Policy and Architecture

A one-way gate sanitizes PHI before observability data crosses the trust boundary. Production and non-production are isolated, while access depends on both role and geography.

[3:33](https://www.youtube.com/watch?v=YXEqC05WEI0&t=213s) [A trust boundary diagram shows sanitization between production data and non-production observability, with isolation and access controls.](https://assets.ai.engineer/sites/aiecode2025/sha256/8450697f5b8568cfa032b00753e1fe42a3414b315e70ae250e4637032b1025e3.webp)

Layer 1 — Policy sets the rule. Architecture enforces it

TRUST ZONE BOUNDARY

PRODUCTION ZONE
RAW PHI DATA
SANITIZATION

ONE-WAY SANITIZATION GATE
NOTHING CROSSES THE BOUNDARY WITHOUT SANITIZATION

NON-PRODUCTION ZONE
SANITIZED DASHBOARDS
LOGS
TRACES

Sanitize at the pipeline boundary
PHI is sanitized before it reaches any dashboard, log, or telemetry stream — not after.

Hard isolation between environments
Production and non-production zones have no bridges. Isolation is enforced by architecture, not policy.

Telemetry sanitized before it leaves the trust zone
No observability data exits the production boundary in a form that contains identifiable information.

Access restricted by role AND geography
Authorization is a compound check — who you are and where you are both determine what you can see.

Regulatory inputs: HIPAA · FDA GMLP · Texas TRAIGA · EU AI Act

Views expressed are my own and do not represent Hinge Health

### Layer 2: Must-Not-Fail Decisions

A deterministic code layer runs first on every turn and makes irreversible decisions, such as crisis routing. The model handles long-tail generation within the boundaries that code sets.

[5:54](https://www.youtube.com/watch?v=YXEqC05WEI0&t=354s) [A code layer sits above a model layer, with text assigning high-stakes decisions to code and generation to the model.](https://assets.ai.engineer/sites/aiecode2025/sha256/a60b397dd43b35961d76c333dcff9dc7607cabb7ac55fa52825bdab00f6e9584.webp)

Layer 2 — Must-not-fail belongs above the prompt.

CODE LAYER
(deterministic):
runs first, cannot
be overridden

flow

MODEL LAYER
(probabilistic):
handles long-tail generation,
operates within boundaries
set by the code layer

"A model is not a guardrail. A model with a system prompt is not a guardrail. Code that runs above the model is closer."

Probabilistic systems excel at generation
They are unreliable for must-not-fail decisions, where any error rate is unacceptable.

Code makes the irreversible decisions
Routing to a crisis line, blocking a session, or rejecting a turn cannot be left to inference.

The model handles the long tail
Within the boundaries the code layer sets, the model operates freely — that's the correct division of labor.

Views expressed are my own and do not represent Hinge Health

### What Lives in the Code Layer

Code handles three decisions before the model: escalating emergencies, routing requests to the right capability, and verifying identity before member data is accessed.

[7:53](https://www.youtube.com/watch?v=YXEqC05WEI0&t=473s) [Three columns describe emergency escalation, intent routing, and identity verification as code-layer capabilities.](https://assets.ai.engineer/sites/aiecode2025/sha256/65db4ce47fd7c1e995f77152abb94ff1ab4f4a0c2ac2913724b3d9c11ceb93a8.webp)

What Lives in the Code Layer

Three concrete examples of capabilities that cannot be prompt instructions — and the architectural reason each one lives in deterministic code.

Emergency Escalation
Trigger: Member mentions self-harm, suicidal ideation, or acute medical emergency.
Action: Route to 911 or 988. The model never sees the turn.
Why code, not prompt: Models can ignore "please remember to escalate." Code can't.

Intent Routing
Trigger: A turn that could be clinical, technical, or general in nature.
Action: A deterministic classifier decides which capability handles the request.
Why code, not prompt: A silent misroute of a clinical question to a generic chatbot is unrecoverable.

Identity Verification
Trigger: Any turn that touches member-specific data.
Action: An authentication check confirms the correct member before any data is accessed.
Why code, not prompt: Authentication is a security boundary. Prompts aren't.

Views expressed are my own and do not represent Hinge Health

### Layer 3: Continuous Safety Evaluation

Automated judges score live traffic alongside member feedback and human-reviewed traces. High-stakes scenarios receive 100% sampling; the bottleneck is having enough people to read the signals.

[9:32](https://www.youtube.com/watch?v=YXEqC05WEI0&t=572s) [A gear diagram connects production traffic, automated judges, a scoring dashboard, and human triage; panels describe judges, member feedback, and sampled traces.](https://assets.ai.engineer/sites/aiecode2025/sha256/d5a240192a72d213eb6a6d420d84389e67c101f810a6a49c3c2157d7862aefad.webp)

Layer 3 — Safety is a continuous evaluation layer.

Production Traffic
Automated Judges
Scoring Dashboard
Human Triage Queue

Automated Judges
30+ judges covering clinical accuracy, safety, escalation correctness, relevance, refusal calibration, and off-topic drift. Runs continuously, refreshed every few hours. Catches drift, regressions, and capability-specific quality drops.

Member Feedback
Thumbs up/down at the message level. The only signal that comes directly from the person being served. Catches tone failures, dissatisfaction, and failure modes that automated judges miss.

Sampled Traces
Random sampling plus stratified sampling — 100% coverage on high-stakes scenarios. Reviewed by humans. Catches the patterns that no single metric surfaces on its own.

"The bottleneck isn't compute. It's having enough people to read the signal."

"Some failure modes can't be prompt-hardened away."

Views expressed are my own and do not represent Hinge Health

### When the architecture isn't enough

Monitoring can reveal when architecture is not enough. The second part turns to the human decisions that follow.

[12:37](https://www.youtube.com/watch?v=YXEqC05WEI0&t=757s) [Section title introducing the decisioning framework beside a photograph of a corridor.](https://assets.ai.engineer/sites/aiecode2025/sha256/a381e122bd1d4505a2499a8ae82b4d0e66fd0dc2406cd893708d9ce573caa156.webp)

When the architecture isn't enough.

Part II — The Decisioning Framework

### The Decisioning Framework

Set severity by the worst plausible harm, then consider capacity separately. When unsure, hold safety risks and ship polish flaws; calibrate to production tolerance and leave humans room to interpret judge signals.

[14:03](https://www.youtube.com/watch?v=YXEqC05WEI0&t=843s) [Five decision rules cover worst-case severity, capacity, safer defaults, revealed tolerance and human interpretation.](https://assets.ai.engineer/sites/aiecode2025/sha256/90349a2e8ee1a89db8425fbe473c4c27d0c652d782dfd417cbbff158c79a95db.webp)

The Decisioning Framework

1 WORST CASE WINS
Severity is set by the worst plausible outcome, not the average. A bug that harms one in a thousand is a severity-one bug.

2 SEVERITY ≠ CAPACITY
The bug's harm sets the severity. Your team's bandwidth is a different conversation — one that happens after the severity call, not before.

3 ASYMMETRIC DEFAULT
Default to the safer mistake. Safety risks delay the launch. Polish risks ship. These are not equivalent errors.

4 REVEALED > STATED TOLERANCE
Your launch bar is what your organization already accepts in production, not what it says it will accept. Calibrate to actual behavior.

5 HUMANS ARE THE CONSTRAINT
Judges scale. Pattern interpretation doesn't. Design for the human in the loop before you design anything else in the eval stack.

Views expressed are my own and do not represent Hinge Health

### Judge Wrong vs. Agent Wrong

Two caffeine examples show why a falling judge score needs inspection: one flag incorrectly rejects general guidance, while the other catches an unsafe agent answer. Fix whichever system is wrong.

[17:54](https://www.youtube.com/watch?v=YXEqC05WEI0&t=1074s) [Side-by-side caffeine examples contrast a judge's false positive with a correct flag of an agent's unsafe answer.](https://assets.ai.engineer/sites/aiecode2025/sha256/5464b9859dd0e6c9c49ecaf23af2a337f252ff4a3f9a58379a5b3d2ac2bec8aa.webp)

Judge Wrong vs. Agent Wrong

Your clinical-accuracy judge drops from 4.9 to 4.5 for two days. Your team's instinct: start changing prompts. But in a non-deterministic system, the judge is also non-deterministic. Before you trust the score, verify the scorer.

PANEL A — JUDGE WRONG (FALSE POSITIVE)

Member asks: "How much caffeine can I have per day?"
Agent replies: "Up to about 400mg for most healthy adults. Less if you're pregnant or on certain medications."
Judge flags: "Hallucination — agent gave specific medical guidance without verifying member's conditions."
Reality: 400mg is the FDA's published general guidance. Standard context. The agent is correct. The judge is overcalling.
→ Fix the judge. Its prompt is too sensitive.

PANEL B — AGENT WRONG (TRUE POSITIVE)

Member asks: "How much caffeine can I have per day?"
Agent replies: "Up to 1000mg per day is generally safe."
Judge flags: "Hallucination — recommended limit is incorrect."
Reality: FDA general guidance is ~400mg per day. 1000mg is well above safe and could cause direct harm. The agent is wrong. The judge caught it.
→ Fix the agent through the eval system, not reactive prompt patching.

Decision rule: Always ask "is the judge correct?" before changing the agent.

Views expressed are my own and do not represent Hinge Health

### 6 Takeaways

Architecture sanitizes PHI, runs deterministic safeguards first, and monitors after launch. Decisioning scores worst cases, calibrates to what the organization ships, and keeps humans in the loop.

[20:16](https://www.youtube.com/watch?v=YXEqC05WEI0&t=1216s) [Two columns summarize three architecture principles and three decisioning principles.](https://assets.ai.engineer/sites/aiecode2025/sha256/9286fcc05a040568e3cebcb1c2a6c184b53849b61f77d566c14abe48feb110bf.webp)

6 Takeaways

ARCHITECTURE
How the system holds the line

01
Don't policy what you can architect.
Sanitize PHI at the boundary. The guarantee lives in the design, not the documentation.

02
Don't prompt what you can code.
Must-not-fail lives above the model. Deterministic code runs first, every time, on every turn.

03
Don't gate what you can monitor.
Safety is a continuous layer. Launch begins the risk period — it doesn't end it.

DECISIONING
How humans decide when systems can't

01
Score by worst case.
Default to the safer mistake. Severity is set by the worst plausible outcome, not the median.

02
Calibrate to what your org actually ships.
Revealed tolerance is what matters. Stated tolerance is not a launch bar.

03
Design for the human in the loop.
Fast follows are debt, not backlog. Judges scale. Pattern interpretation doesn't.

Views expressed are my own and do not represent Hinge Health

### Architecture, Decisioning, and Member Trust

Building guardrails first is slower than bolting them on later, the speaker says. Member trust is the reason to build a system worthy of someone’s health.

[21:01](https://www.youtube.com/watch?v=YXEqC05WEI0&t=1261s) [Closing slide states that architecture is how, decisioning is when, and member trust is why.](https://assets.ai.engineer/sites/aiecode2025/sha256/22b7ed00984fd177fbbd534eedd6f533e3863df63440a587339992b224629726.webp)

Architecture is how.
Decisioning is when.
Member trust is why.

Views expressed are my own and do not represent Hinge Health

## Full transcript

[0:00](https://www.youtube.com/watch?v=YXEqC05WEI0&t=0s) [on-hold music] Hello and good morning. Uh, Chetana gave us a great overview of, you know, what Abridge does.

[0:18](https://www.youtube.com/watch?v=YXEqC05WEI0&t=18s) Uh, today I'm here to talk more from a practitioner's view of, you know, how we are building healthcare AI within Hinge Health. So hi, I'm Rashi Agrawal. I lead AI and ML at Hinge Health, and today I will be talking about guardrails that are needed to build member-facing healthcare AI.

[0:39](https://www.youtube.com/watch?v=YXEqC05WEI0&t=39s) I want to talk a little bit about, uh, the state of healthcare AI right now. We do have a lot of frontier models which are running, and believe it or not, forty million people actually use these models for triaging their healthcare issues.

[0:53](https://www.youtube.com/watch?v=YXEqC05WEI0&t=53s) But there is a caveat, and these are some of the headlines that have been happening in the past few, uh, in the past one year or so. Poisoned by a chatbot.

[1:02](https://www.youtube.com/watch?v=YXEqC05WEI0&t=62s) Um, let's start with this one person. A [REDACTED:age] healthy [REDACTED:gender] asked a popular AI assistant how to cut salt from his diet. The LLM told him to swap it with bromium, uh, sodium bromide.

[1:15](https://www.youtube.com/watch?v=YXEqC05WEI0&t=75s) He did it for three months. He landed in the ER with paranoia and hallucinations. Bromide levels two hundred times the safe limit. Three weeks in the hospital. For what?

[1:27](https://www.youtube.com/watch?v=YXEqC05WEI0&t=87s) For following diet advice? Let's look at another pattern. The first independent safety test of a consumer health AI out of Mount Sinai found that this health AI is under-triaging life-threatening emergency fifty percent of the times, diabetic ketoacidosis, respiratory failure, and it told, uh, the people to go see a doctor in

[1:52](https://www.youtube.com/watch?v=YXEqC05WEI0&t=112s) a day or two. The right answer was ER right now, and this isn't fringe. In February, ECRI, the patient safety group that hospitals trust to ra-rank their top risks, named AI chatbot misuse as the number one health technology hazard of 2026.

[2:16](https://www.youtube.com/watch?v=YXEqC05WEI0&t=136s) Number one on the list that they publish every year. So this is not neces-- This is not really a frontier problem.

[2:23](https://www.youtube.com/watch?v=YXEqC05WEI0&t=143s) This is the production baseline that we are working with right now. So the question comes: How do you ship AI to somebody who's already trusted you with their health?

[2:34](https://www.youtube.com/watch?v=YXEqC05WEI0&t=154s) The next twenty minutes are all about that.

[2:39](https://www.youtube.com/watch?v=YXEqC05WEI0&t=159s) It starts with three non-negotiable foundations. One, the constraint is the architecture. Most AI safety failures in healthcare are not model failures. They are architectural decisions that were made before even a single token was generated.

[2:57](https://www.youtube.com/watch?v=YXEqC05WEI0&t=177s) Two, deterministic rules belong above the model, not inside it. What can never be wrong cannot be left to probability. And three, safety is a continuous evaluation layer, not a one-time gate.

[3:15](https://www.youtube.com/watch?v=YXEqC05WEI0&t=195s) Launch of your product is where the real risk starts, not where it ends. That's the first part of what I want to talk about today. The second half is what happens when the architecture is not enough, and a human has to make a decision of what ships versus what holds.

[3:35](https://www.youtube.com/watch?v=YXEqC05WEI0&t=215s) Let's start with layer one. Protecting PHI takes both policy and architecture. Policy tells you what to protect, and architecture makes sure that it actually happens. The first thing that sh-changes when you start shipping member-facing health AI is where PHI lives.

[3:57](https://www.youtube.com/watch?v=YXEqC05WEI0&t=237s) Most teams treat PHI as a runtime problem, something to redact when a log gets written to a dashboard. That's the reactive version.

[4:08](https://www.youtube.com/watch?v=YXEqC05WEI0&t=248s) The architecture version strips PHI at the pipeline boundary, at ingestion, before it ever reaches the data lake. By the time the data is stored, the PHI is gone. So a developer opens a dashboard, there's nothing to redact.

[4:26](https://www.youtube.com/watch?v=YXEqC05WEI0&t=266s) The PHI was never there. The rest of the architecture works in a similar way.

[4:33](https://www.youtube.com/watch?v=YXEqC05WEI0&t=273s) Production and non-production stay completely separate. No pipes in between, because even a single pipe is all that it takes for member data to leak into a dev environment. And HIPAA laws are very stringent, especially in healthcare.

[4:48](https://www.youtube.com/watch?v=YXEqC05WEI0&t=288s) You know, the regulatory bar is much, much higher. So you have to be very careful about the architecture that you're designing. And a big-- another big thing, access depends on two things: your role and your geographic region.

[5:01](https://www.youtube.com/watch?v=YXEqC05WEI0&t=301s) We all work with, uh, teams which are geographically distributed, but not everybody has access to PHI. That is a certification, a policy that is applied to specific regions only.

[5:12](https://www.youtube.com/watch?v=YXEqC05WEI0&t=312s) An engineer outside the regulated region cannot reach raw PHI at all. And the compliance rules, HIPAA, FDA's Good Machine Learning Practice, state laws like Texas, TRAIGA, they are not afterthoughts.

[5:27](https://www.youtube.com/watch?v=YXEqC05WEI0&t=327s) They are the grounding input in how you actually design your systems.

[5:32](https://www.youtube.com/watch?v=YXEqC05WEI0&t=332s) You cannot slap on HIPAA on top of, you know, an underlying system or an architecture. You start with it and let the architecture grow around it.

[5:42](https://www.youtube.com/watch?v=YXEqC05WEI0&t=342s) When PHI is protected at the architecture level, you're not just trusting that the policies will get followed. You're actually relying on a system that's incapable of certain failures.

[5:56](https://www.youtube.com/watch?v=YXEqC05WEI0&t=356s) Let's move to the layer two. Probabilistic systems are great at generation. We all know that.

[6:03](https://www.youtube.com/watch?v=YXEqC05WEI0&t=363s) However, they are unreliable for things that cannot, can never be wrong. So the rule is very simple: must not fail behavior belongs above your prompt, above the model. And what does above the prompt actually mean?

[6:19](https://www.youtube.com/watch?v=YXEqC05WEI0&t=379s) It means that there is a code layer that runs first on every turn before the model even runs. The code layer is what makes your irreversible decisions. The decision of, you know, whether this is an emergency escalation, should they be routed to [REDACTED:phone_number]?

[6:36](https://www.youtube.com/watch?v=YXEqC05WEI0&t=396s) Should a clinician step into the loop? All of those are irreversible decisions which need to lie at a deterministic code level layer. The model handles the long tail of your conversations and interactions with your members.

[6:51](https://www.youtube.com/watch?v=YXEqC05WEI0&t=411s) The picture to hold in your head is a stack. Code on top, model below. Every turn goes through the code layer first.

[7:03](https://www.youtube.com/watch?v=YXEqC05WEI0&t=423s) Most turns do reach the model, but the model never gets a vote on high-stake calls.

[7:13](https://www.youtube.com/watch?v=YXEqC05WEI0&t=433s) Here's how you can think about it in a different way. A model is not a guardrail. A model with a system prompt is also not a guardrail. Code that runs above the model is closer.

[7:27](https://www.youtube.com/watch?v=YXEqC05WEI0&t=447s) Even the labs that bu-build these frontier models publish the authority hierarchy: root, system, developer, user, guideline.

[7:38](https://www.youtube.com/watch?v=YXEqC05WEI0&t=458s) Every layer above user is one prompt injection away from being overridden.

[7:45](https://www.youtube.com/watch?v=YXEqC05WEI0&t=465s) If the labs themselves don't trust the prompt as a security boundary, neither should you.

[7:52](https://www.youtube.com/watch?v=YXEqC05WEI0&t=472s) So what does live in this code layer? Let's examine it a little bit. Let's take three examples. First, very, very relevant to healthcare, which is emergency escalation. If a member mentions self-harm, suicidal ideation, or an acute medical emergency, the system must route to [REDACTED:phone_number] or [REDACTED:phone_number].

[8:14](https://www.youtube.com/watch?v=YXEqC05WEI0&t=494s) The model should not even see this turn. Code runs first, decides and routes, and makes a decision right away.

[8:23](https://www.youtube.com/watch?v=YXEqC05WEI0&t=503s) Another example, intent routing. Which capability in your underlying multi-agentic system, multi-agentic architecture handles a conversation turn? Is it clinical? Is it tech support? Is it education from the millions of, you know, credited articles?

[8:40](https://www.youtube.com/watch?v=YXEqC05WEI0&t=520s) Is it exercise recommendation? The model can help to classify, but high-stakes path mu-must again take a deterministic route at the top itself. You, you don't want, like, a clinical question quietly being routed to your generic tech support agent.

[9:00](https://www.youtube.com/watch?v=YXEqC05WEI0&t=540s) That's unrecoverable. Third, identity verification. Anything that touches member data has to check that the right member is at the other end.

[9:15](https://www.youtube.com/watch?v=YXEqC05WEI0&t=555s) That's an authentication check, and authentication is a security bround-- boundary. Prompts are not.

[9:22](https://www.youtube.com/watch?v=YXEqC05WEI0&t=562s) The underlying pattern across all three, code runs first. Code makes the irreversible decisions. The model handles what's left.

[9:35](https://www.youtube.com/watch?v=YXEqC05WEI0&t=575s) Last but not least, layer three. As we all know, safety is not a gate you pass once. It is a continuous layer that runs the whole time. Most teams treat evals as a pre-launch checklist.

[9:51](https://www.youtube.com/watch?v=YXEqC05WEI0&t=591s) You run your tests, you ship, you move on. That's necessary, of course, but that's hardly enough.

[9:58](https://www.youtube.com/watch?v=YXEqC05WEI0&t=598s) What actually holds up in production is judges that continuously keep scoring real conversations as they happen, not a saved golden dataset. Live traffic

[10:11](https://www.youtube.com/watch?v=YXEqC05WEI0&t=611s) scored on a lot of dimensions all the time.

[10:16](https://www.youtube.com/watch?v=YXEqC05WEI0&t=616s) These signals come from three sources, and each one catches something different.

[10:21](https://www.youtube.com/watch?v=YXEqC05WEI0&t=621s) First, automated judges, thirty, forty, name it, you know, as, as much as you can scale. Automated judges with multiple dimensions, always refreshing.

[10:32](https://www.youtube.com/watch?v=YXEqC05WEI0&t=632s) Clinical accuracy, safety, escalation, relevance, drift, refusal, et cetera, et cetera. I can keep going on, but you, you get the point. These are the automated judges that are always going to catch regressions and any even sensitive drops in quality.

[10:50](https://www.youtube.com/watch?v=YXEqC05WEI0&t=650s) Second, your goldmine of information. That's going to be member feedback. Thumbs up, thumbs down on each and every single message. That's the truth signal. That's your member communicating with you, and it's the only one that comes straight from the person that you're serving it to.

[11:07](https://www.youtube.com/watch?v=YXEqC05WEI0&t=667s) It catches tone problems and things that judges miss.

[11:13](https://www.youtube.com/watch?v=YXEqC05WEI0&t=673s) Third, sample traces. Random samples spread across capabilities with high-stake cases checked every single time. Hundred percent sampling on those.

[11:25](https://www.youtube.com/watch?v=YXEqC05WEI0&t=685s) Ultimately, people need to read these signals. People are going to catch what no single metric is going to catch. And here's the part that nobody really warns you about.

[11:37](https://www.youtube.com/watch?v=YXEqC05WEI0&t=697s) The bottleneck is not the compute, the models, the capability. It's actually having enough people to read the signal and act on it.

[11:49](https://www.youtube.com/watch?v=YXEqC05WEI0&t=709s) One more thing about layer three. Some failures, you can't just prompt away. You ship the fix, it comes back under new conditions. New prompts, new tools, the model shifts.

[12:01](https://www.youtube.com/watch?v=YXEqC05WEI0&t=721s) You ship the fix again. Each round buys you less and less. The rate never hits zero.

[12:09](https://www.youtube.com/watch?v=YXEqC05WEI0&t=729s) At this point, monitoring is not a last resort. It is the first resort which is always on. A new failure that you see in production simply means you now have a new judge.

[12:21](https://www.youtube.com/watch?v=YXEqC05WEI0&t=741s) Your underlying architecture and your system needs to be able to keep scaling with new judges, new monitoring, as you keep scaling your, you know, consumers, and that's the point.

[12:32](https://www.youtube.com/watch?v=YXEqC05WEI0&t=752s) Monitoring is how you know that the architecture is still holding.

[12:39](https://www.youtube.com/watch?v=YXEqC05WEI0&t=759s) But monitoring also tells you when the architecture is not enough. And when the architecture is not enough, a human has to decide. And this is the second part of my talk, where I want to focus on the decisioning frameworks.

[12:56](https://www.youtube.com/watch?v=YXEqC05WEI0&t=776s) Let's take an example. You're about to ship, you know, um, consumer AI, again, in the healthcare space, and you have a feature, a specific capability that you're about to launch.

[13:05](https://www.youtube.com/watch?v=YXEqC05WEI0&t=785s) And there is one issue left on the board five days before your launch, and you have multiple different stakeholders. Five stakeholders look at the same issue. Each one sees a different risk,

[13:18](https://www.youtube.com/watch?v=YXEqC05WEI0&t=798s) and they don't agree what to do about it.

[13:22](https://www.youtube.com/watch?v=YXEqC05WEI0&t=802s) Clinical sees member safety risk. They want to hold the launch. Legal sees regulatory exposure. Compliance sees audit risk. Product sees adoption risk.

[13:36](https://www.youtube.com/watch?v=YXEqC05WEI0&t=816s) The fe-- if the sh-- if it ships broken, the feature won't land. And engineering sees velocity risk. They can't fix it without slipping the date. They want to ship.

[13:47](https://www.youtube.com/watch?v=YXEqC05WEI0&t=827s) Five rational people, five different risks, and five very different fixes.

[13:54](https://www.youtube.com/watch?v=YXEqC05WEI0&t=834s) So what do you do? Do you hold the launch and fix or do you actually ship?

[14:01](https://www.youtube.com/watch?v=YXEqC05WEI0&t=841s) The next slide is the framework I actually use for making these decisions.

[14:06](https://www.youtube.com/watch?v=YXEqC05WEI0&t=846s) Five rules. This is how I think about decisions when stakeholders disagree.

[14:12](https://www.youtube.com/watch?v=YXEqC05WEI0&t=852s) Rule one: worst case always wins. Severity is set by the worst pos- plausible outcome, not the average, and this is extremely relevant in healthcare. A bug that lightly annoys one hundred percent of users is way less severe than one that could cause serious harm in zero point one percent of cases.

[14:36](https://www.youtube.com/watch?v=YXEqC05WEI0&t=876s) This is non-negotiable. The worst case matters more than the average case, always. So when you're triaging, don't ask, "How often does this happen?" Ask, "What's the worst version of this?"

[14:52](https://www.youtube.com/watch?v=YXEqC05WEI0&t=892s) That sets the severity. Rule two: severity is not capacity. This one keeps politics out of it. As we all know, as we ship features, there's always a little bit of contention between timelines, features, deliverables.

[15:07](https://www.youtube.com/watch?v=YXEqC05WEI0&t=907s) But a bug's severity comes from the harm that it causes, not who owns it, not whether your team has the capacity to fix it, not how hard the fix is.

[15:20](https://www.youtube.com/watch?v=YXEqC05WEI0&t=920s) You have three options in front of you at this point: fix, delay the launch, or accept the risk with explicit sign-off. Those are the three.

[15:31](https://www.youtube.com/watch?v=YXEqC05WEI0&t=931s) You never quietly downgrade a bug just because you can't get to it.

[15:38](https://www.youtube.com/watch?v=YXEqC05WEI0&t=938s) Rule three: asymmetric default. When you don't know what to do, always pick the safer mistake.

[15:47](https://www.youtube.com/watch?v=YXEqC05WEI0&t=947s) And there are two spectrums to it. One is safety bugs and polish... The other side is polish bugs. For safety bugs, the math is one-sided. Shipping a real safety bug is much worse than delaying for a false alarm.

[16:02](https://www.youtube.com/watch?v=YXEqC05WEI0&t=962s) So for safety bugs, when you're not sure, always hold and fix. On the other side, for polish bugs, the math runs the other way. Delaying a launch costs more than shipping a small flaw.

[16:18](https://www.youtube.com/watch?v=YXEqC05WEI0&t=978s) So when you're not sure, ship in case of polish bugs.

[16:23](https://www.youtube.com/watch?v=YXEqC05WEI0&t=983s) Ultimately, the framework doesn't decide for you. It just tells you which way to lean.

[16:28](https://www.youtube.com/watch?v=YXEqC05WEI0&t=988s) Rule four: revealed risk tolerance, not stated risk tolerance.

[16:35](https://www.youtube.com/watch?v=YXEqC05WEI0&t=995s) Your launch bar is what your org already accepts in production, not what it says it will accept.

[16:43](https://www.youtube.com/watch?v=YXEqC05WEI0&t=1003s) If a behavior has been live in your existing product for weeks, months, without escalation, without member complaints, without leadership concern, you cannot, you cannot call it a launch blocker just for a new thing.

[16:57](https://www.youtube.com/watch?v=YXEqC05WEI0&t=1017s) Your stated risk tolerance might be no bugs in production, but your revealed risk tolerance is what's actually shipping today. Calibrate to the revealed one. That's the floor.

[17:14](https://www.youtube.com/watch?v=YXEqC05WEI0&t=1034s) Rule five: humans are the constraint. Judges scale, pattern interpretation doesn't. Always, always design for human in the loop.

[17:27](https://www.youtube.com/watch?v=YXEqC05WEI0&t=1047s) Judges score traces automatically. Dashboards refresh every few hours. None of that is hard anymore. But what's hard is having enough people to read the signal and act on it.

[17:41](https://www.youtube.com/watch?v=YXEqC05WEI0&t=1061s) One more piece around this. Fast follows are committed debt, not an optional backlog. If you didn't ship it at launch, it's not a wish list item. It's already committed.

[17:56](https://www.youtube.com/watch?v=YXEqC05WEI0&t=1076s) The five rules tell you how to decide, but they all assume one thing, that your underlying signal is true. So here's the discipline that needs to come first. In a non-deterministic system, the judge is also non-deterministic.

[18:13](https://www.youtube.com/watch?v=YXEqC05WEI0&t=1093s) Before you trust the score, verify the scorer. And here's what it looks like in practice.

[18:21](https://www.youtube.com/watch?v=YXEqC05WEI0&t=1101s) Say you're watching a clinical accuracy judge in production. The score has been steady for-- at four point nine for weeks. Today, it drops to four point five, and tomorrow it stays at four point five.

[18:35](https://www.youtube.com/watch?v=YXEqC05WEI0&t=1115s) The immediate instinct is, "Let's start changing the prompts. The agent is broken. Let's fix the agent." That's reactive, and it's risky.

[18:45](https://www.youtube.com/watch?v=YXEqC05WEI0&t=1125s) You fix one thing, and you break another. Worse, you're changing the agent based on a signal that might not be true, and the discipline needs to be different. First, ask whether the judge is right.

[19:03](https://www.youtube.com/watch?v=YXEqC05WEI0&t=1143s) We can solidify that with an, with a concrete example. Sp-- Let's take it side by side. In scenario A, same question, member asks about caffeine. The agent gives FDA standard guidance, four hundred milligrams for most adults, less if pregnant or on certain medications.

[19:22](https://www.youtube.com/watch?v=YXEqC05WEI0&t=1162s) The judge flags it as a hallucination because the agent mentioned pregnancy and medications without checking.

[19:32](https://www.youtube.com/watch?v=YXEqC05WEI0&t=1172s) But that's just clinical context. The judge is over-calling in this case.

[19:38](https://www.youtube.com/watch?v=YXEqC05WEI0&t=1178s) Fix the judge in this scenario. For the same question, scenario B, the agent says thousand milligrams a day is fine. That's well above the safety limits. The judge correctly flags it, and the agent is wrong.

[19:54](https://www.youtube.com/watch?v=YXEqC05WEI0&t=1194s) In this case, fix the agent. The rule is always ask, is the judge right, before changing the agent's response.

[20:03](https://www.youtube.com/watch?v=YXEqC05WEI0&t=1203s) Fixing a judge prompt is not cheating. Judges are software too, and they need to continuously evolve. This is what production discipline looks like when the system is not deterministic.

[20:18](https://www.youtube.com/watch?v=YXEqC05WEI0&t=1218s) Here's the whole talk in one slide. If you screenshot one thing, this would be it. Six takeaways, three from architecture, three from decisioning. On the architecture side, the pattern is very s-simple.

[20:31](https://www.youtube.com/watch?v=YXEqC05WEI0&t=1231s) Don't X what you can Y. Don't policy what you can architect. Don't prompt what you can code. Don't gate what you can monitor.

[20:42](https://www.youtube.com/watch?v=YXEqC05WEI0&t=1242s) On the decisioning side, the pattern is how humans decide when the system cannot. Score by the worst case and default to the safer mistake. Calibrate to your org and always design for the human in the loop.

[20:58](https://www.youtube.com/watch?v=YXEqC05WEI0&t=1258s) Fast follows are debt, not backlog. Yes, building guardrails first is slower than bolting them on later, but that's a design, not limitation. We are not building a generic low-stakes chatbot.

[21:15](https://www.youtube.com/watch?v=YXEqC05WEI0&t=1275s) We are building a system that has to be worthy of someone's health. The architecture is how, the decisioning is when, and member trust is why. Thank you. Let's continue the conversation on LinkedIn.

[21:30](https://www.youtube.com/watch?v=YXEqC05WEI0&t=1290s) Thank you. [audience applauding] [outro jingle]
