AI Engineer World's Fair 2026

Guardrails First: Engineering Member-Facing Health AI

Loading slides…

Read the talk

Guardrails First: Engineering Member-Facing Health AI

Healthcare AI needs privacy boundaries before storage, deterministic controls before generation, and continuous evaluation backed by people who can decide when to hold a launch.

From a talk by Rashi Agrawal

Before you start: Familiarity with LLM prompts, application routing, and basic production monitoring will help; no clinical background is required.

When dietary advice becomes an emergency

How do you ship AI to someone who already trusts you with their health? Rashi Agrawal, introducing herself as the leader of AI and ML at Hinge Health, starts with the scale of that responsibility. She cites 40 million people using frontier models for healthcare triage; the supporting ECRI account describes a broader activity: more than 40 million daily users of ChatGPT seeking health information.

The danger becomes concrete in her account of a healthy person who asked an AI assistant how to reduce dietary salt. The assistant reportedly suggested sodium bromide as a substitute. Agrawal recounts that after three months the person developed paranoia and hallucinations, had bromide levels reported as 200 times the safe limit, and spent three weeks in hospital. An ordinary dietary question had become an emergency.

Agrawal next cites roughly 50% under-triage of life-threatening emergencies in a Mount Sinai evaluation of ChatGPT Health. This was a structured scenario evaluation at one point in time, not a measured failure rate across all patient encounters. Her examples include diabetic ketoacidosis and respiratory failure: the system recommended seeing a doctor in a day or two when the appropriate response was immediate emergency care. ECRI ranked AI chatbot misuse first among its health technology hazards for 2026, announcing the list in January. These failures establish the production problem: a useful conversational answer can still direct someone toward the wrong action.

0:180:39
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:18 · section reference included

Make privacy a property of the architecture

Agrawal organizes the response around three foundations. Constraints shape the architecture before a model generates a token. Deterministic rules govern behavior that cannot be left to probability. Safety evaluation continues after launch, when real production risk begins. Those foundations also establish the limits of automation: when the architecture cannot settle whether something should ship, a human must make the decision.

Start with protected health information, or PHI. A reactive design lets sensitive data enter storage and depends on redaction when a log appears in a dashboard. The architecture Agrawal describes removes PHI at the ingestion boundary, before it reaches the data lake. Developers then open dashboards whose underlying stored data never contained that PHI. Moving the control upstream removes the need for every downstream viewer to get redaction right.

That boundary extends to environments and access. Production and non-production remain separate, without connecting pipelines that can carry member data into development. Raw-PHI access depends on both an authorized role and an approved geographic region; in the policy Agrawal describes, an engineer outside that region cannot reach the data. These geographic restrictions describe her system's controls, not a universal HIPAA prohibition on overseas processing.

HIPAA, the FDA's Good Machine Learning Practice principles, and Texas's Responsible Artificial Intelligence Governance Act enter the design before implementation. Their legal scope differs: GMLP supplies guiding principles for medical-device development, while statutes impose applicable legal obligations. The engineering consequence is shared: identify the relevant constraints first, then build the data flows and access paths around them. A privacy policy becomes stronger when the system removes the path by which a prohibited disclosure could occur.

2:392:57
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:39 · section reference included

Run code before the model on every turn

Probabilistic models are useful for generation, but a high-stakes decision needs an execution boundary outside generation. Agrawal's stack puts code above the model: every conversation turn passes through the code layer first. That layer decides whether to escalate an emergency or bring in a clinician. Most turns continue to the model, which handles the long tail of member conversations; it does not get to overrule an escalation.

Slide showing a code layer stacked above a model layer, with three panels describing generation, irreversible decisions, and long-tail model behavior.Open full source frame
Must-not-fail decisions belong in the code layer above the model.

A model alone is not a guardrail, and adding a system prompt does not make it a security boundary. The OpenAI Model Spec defines the authority order root → system → developer → user → guideline. That hierarchy specifies intended instruction priority. Agrawal warns that prompt injection can undermine prompt-based controls, using the stronger formulation that every layer above the user is one injection away from override. The specification does not establish that categorical claim; the actionable architectural point is that instruction priority cannot replace an independently enforced security check.

Three controls make the distinction concrete:

  • Emergency escalation: self-harm, suicidal ideation, or an acute medical emergency triggers an appropriate route to 911 or 988. In Agrawal's design, the conversational model never sees that turn. The talk specifies the routing requirement but does not detail how emergencies are reliably detected.
  • Capability routing: a multi-agent system may contain clinical, technical-support, educational, and exercise-recommendation capabilities. A model can assist with classification, but high-stakes paths need enforced routing. A clinical question must not silently fall through to generic technical support.
  • Identity verification: an operation touching member data must verify that the person at the other end is the correct member. Authentication supplies the security boundary; a prompt asking the model to respect identity does not.

A TypeScript dispatch boundary can express that separation. Here, decision is the result of upstream controls, including authentication where member data is involved. Only an explicitly permitted model route can reach generation:

typescript

type Decision =
  | { kind: "emergency"; destination: "911" | "988" }
  | { kind: "clinician" }
  | { kind: "authentication-required" }
  | {
      kind: "model";
      capability: "clinical" | "tech-support" | "education" | "exercise";
    };

type Action =
  | Exclude<Decision, { kind: "model" }>
  | { kind: "generate"; capability: string; text: string };

function dispatch(text: string, decision: Decision): Action {
  switch (decision.kind) {
    case "emergency":
    case "clinician":
    case "authentication-required":
      return decision;
    case "model":
      return {
        kind: "generate",
        capability: decision.capability,
        text,
      };
  }
}

The function returns a proposed action; it does not contact emergency services or execute a model call. Its useful property is the control flow: once an upstream control selects escalation or authentication, generation is no longer an available branch.

5:566:03
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:56 · section reference included

Evaluate live conversations—and staff the response

Pre-launch tests remain necessary, but a saved golden dataset cannot reveal every failure that emerges in live conversations. The third layer continuously scores production traffic across multiple dimensions. Agrawal combines three signal sources because each exposes a different part of the system's behavior.

  • Automated judges: Agrawal suggests 30–40 judges as an illustrative scale, covering clinical accuracy, safety, escalation, relevance, drift, refusal, and other dimensions. The judge set keeps evolving as the team learns what to detect.
  • Member feedback: per-message thumbs up and thumbs down provide a direct signal from the person receiving the answer. That signal can reveal tone problems and failures that automated judges miss.
  • Trace review: random samples span capabilities, while Agrawal calls for review of 100% of high-stakes cases. People still need to read the underlying conversations and interpret patterns that no single score captures.

The bottleneck is the capacity to interpret signals and act on them. More compute or more judges does not supply the people needed to investigate a concerning trace, recognize a recurring pattern, and decide what to change. Review capacity therefore belongs in the operating design, alongside inference and evaluation capacity.

Some failures return after a prompt fix because the surrounding conditions change: a new prompt, a new tool, or a different model behavior exposes another version of the problem. Successive fixes may yield diminishing returns without eliminating the failure. Agrawal's response is to turn a newly observed production failure into a new judge, and to make the monitoring architecture capable of accommodating that growing set. Monitoring is how the team checks whether its architectural protections continue to hold as usage expands.

9:359:51
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:35 · section reference included

One unresolved issue, five different risks

Monitoring can also reveal a problem that the architecture does not resolve. Agrawal introduces a launch scenario: one issue remains five days before a new healthcare AI capability is due to ship. Five stakeholders examine the same issue and disagree because they are evaluating different consequences.

StakeholderRisk they seeLaunch concern
ClinicalMember safetyHold the launch to prevent harm
LegalRegulatory exposureConsequences of noncompliance
ComplianceAudit riskAbility to satisfy scrutiny
ProductAdoption riskA broken feature may not land
EngineeringVelocity riskA fix means slipping the date

Clinical wants to hold; engineering wants to ship. The disagreement does not require anyone to be irrational. Each function sees a real risk, but identifying those risks still leaves the release decision unresolved.

Five panels pair Clinical with member safety risk, Legal with regulatory exposure, Compliance with audit risk, Product with adoption risk, and Engineering with velocity risk.Open full source frame
Five stakeholders see five different launch risks.
12:3912:56
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

12:39 · section reference included

Set severity by harm, then choose the safer mistake

The first of Agrawal's five decision rules sets severity by the worst plausible outcome, rather than the average experience. Her comparison is deliberately asymmetric: minor annoyance affecting 100% of users is less severe than potential serious harm in 0.1% of cases. The triage question starts with the worst credible version of the failure; frequency alone cannot set its severity.

The second rule separates severity from capacity. Ownership, staffing, and the difficulty of a fix do not change the harm a bug can cause. Once severity is established, the available responses are to fix it, delay launch, or accept the risk with explicit sign-off. Quietly downgrading an issue because the team cannot reach it substitutes a scheduling constraint for a safety assessment.

The third rule gives uncertain decisions an asymmetric default:

Issue typeMore costly mistakeDefault under uncertainty
SafetyShipping a real harmful defectHold and fix
PolishDelaying over a small flawShip

For safety, an unnecessary delay is preferable to exposing members to a real hazard. For polish, the cost of delaying can exceed the cost of the imperfection. This framework supplies a direction for judgment, not an automatic release verdict.

14:0614:12
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

14:06 · section reference included

Use the organization's actual bar—and fund the follow-through

The fourth rule distinguishes revealed risk tolerance from stated risk tolerance. An organization may say it accepts no production bugs while continuing to ship behavior that contradicts that standard. Agrawal argues that if a behavior has been live for weeks or months without escalation, member complaints, or leadership concern, it should not become a blocker solely because it appears in a new feature. Existing production behavior supplies the calibration floor for a consistent launch bar.

The fifth rule returns to the human constraint. Judges can score traces automatically and dashboards can refresh every few hours, but people must interpret patterns and act on them. A launch plan that includes monitoring without allocating review capacity leaves that responsibility unfulfilled. And work deferred to a fast follow is already committed debt: it does not become an optional backlog item once the launch succeeds.

16:2816:35
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

16:28 · section reference included

Verify the scorer before changing the agent

All five rules depend on trustworthy evidence. Yet an LLM judge is itself nondeterministic. In Agrawal's hypothetical example, a clinical-accuracy score stays at 4.9 for weeks, then drops to 4.5 and remains there the next day. Immediately changing the agent prompt could introduce a regression while leaving the actual source of the score change untouched. The first investigation is whether the judge is right.

Her caffeine example separates two superficially similar alerts. In scenario A, a member asks about caffeine and receives general guidance mentioning 400 mg per day for most adults, with qualifications concerning pregnancy and medications. The FDA guidance describes that amount as not generally associated with negative effects for most adults; it advises consulting a healthcare provider about restrictions in circumstances such as pregnancy or medication use. It is not a personalized safe dose. The judge nevertheless labels the response a hallucination because the agent mentioned pregnancy and medications without first checking whether those circumstances apply. In this example, those qualifications supply clinical context rather than assert facts about the member. The judge is over-calling.

ScenarioAgent responseJudge behaviorComponent to fix
AGeneral caffeine guidance with conditional qualificationsIncorrectly flags the qualificationsJudge
BSays 1,000 mg of caffeine daily is fineCorrectly flags the unsafe endorsementAgent

The second scenario changes the answer, not the member's question. Here the judge correctly identifies a problem in the agent's guidance. The same alert category can therefore require changes to different components; a lower score alone does not tell the team which component is wrong.

The resulting investigation follows a short sequence:

  1. Inspect the response and the judge's reason for flagging it.
  2. Establish whether the judgment is correct before changing the agent.
  3. Correct the judge when it over-calls valid context; correct the agent when its answer is wrong.

A judge prompt is software that needs maintenance. Correcting an erroneous evaluator preserves the meaning of its score; it is not cheating the evaluation.

17:5618:13
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

17:56 · section reference included

Build a system worthy of member trust

Agrawal closes the architectural argument with three compact formulations: “Don't policy what you can architect.” “Don't prompt what you can code.” “Don't gate what you can monitor.” Each moves a responsibility into a mechanism that can enforce or continually check it. Human release decisions then address the residual risk, using worst-case severity, safer defaults, an explicit organizational bar, and committed follow-through.

Building these guardrails first can slow development. For member-facing healthcare AI, that cost is intentional: the product must be worthy of a person's trust in matters affecting their health. Architecture determines how the system can operate; accountable decisions determine when it should ship. As Agrawal puts it, “The architecture is how, the decisioning is when, and member trust is why.”

Closing slide pairs a sunlit wall photograph with three statements: “Architecture is how,” “Decisioning is when,” and “Member trust is why.”Open full source frame
Architecture is how. Decisioning is when. Member trust is why.
20:1820:31
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

20:18 · section reference included

Resources

From the talk

Transcript download
Full transcript

Loading transcript…