AI Engineer World's Fair 2026
Guardrails First: Engineering Member-Facing Health AI
Loading slides…
Read the talk
Guardrails First: Engineering Member-Facing Health AI
Healthcare AI needs privacy boundaries before storage, deterministic controls before generation, and continuous evaluation backed by people who can decide when to hold a launch.
From a talk by Rashi Agrawal
Before you start: Familiarity with LLM prompts, application routing, and basic production monitoring will help; no clinical background is required.
When dietary advice becomes an emergency
How do you ship AI to someone who already trusts you with their health? Rashi Agrawal, introducing herself as the leader of AI and ML at Hinge Health, starts with the scale of that responsibility. She cites 40 million people using frontier models for healthcare triage; the supporting ECRI account describes a broader activity: more than 40 million daily users of ChatGPT seeking health information.
The danger becomes concrete in her account of a healthy person who asked an AI assistant how to reduce dietary salt. The assistant reportedly suggested sodium bromide as a substitute. Agrawal recounts that after three months the person developed paranoia and hallucinations, had bromide levels reported as 200 times the safe limit, and spent three weeks in hospital. An ordinary dietary question had become an emergency.
Agrawal next cites roughly 50% under-triage of life-threatening emergencies in a Mount Sinai evaluation of ChatGPT Health. This was a structured scenario evaluation at one point in time, not a measured failure rate across all patient encounters. Her examples include diabetic ketoacidosis and respiratory failure: the system recommended seeing a doctor in a day or two when the appropriate response was immediate emergency care. ECRI ranked AI chatbot misuse first among its health technology hazards for 2026, announcing the list in January. These failures establish the production problem: a useful conversational answer can still direct someone toward the wrong action.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make privacy a property of the architecture
Agrawal organizes the response around three foundations. Constraints shape the architecture before a model generates a token. Deterministic rules govern behavior that cannot be left to probability. Safety evaluation continues after launch, when real production risk begins. Those foundations also establish the limits of automation: when the architecture cannot settle whether something should ship, a human must make the decision.
Start with protected health information, or PHI. A reactive design lets sensitive data enter storage and depends on redaction when a log appears in a dashboard. The architecture Agrawal describes removes PHI at the ingestion boundary, before it reaches the data lake. Developers then open dashboards whose underlying stored data never contained that PHI. Moving the control upstream removes the need for every downstream viewer to get redaction right.
That boundary extends to environments and access. Production and non-production remain separate, without connecting pipelines that can carry member data into development. Raw-PHI access depends on both an authorized role and an approved geographic region; in the policy Agrawal describes, an engineer outside that region cannot reach the data. These geographic restrictions describe her system's controls, not a universal HIPAA prohibition on overseas processing.
HIPAA, the FDA's Good Machine Learning Practice principles, and Texas's Responsible Artificial Intelligence Governance Act enter the design before implementation. Their legal scope differs: GMLP supplies guiding principles for medical-device development, while statutes impose applicable legal obligations. The engineering consequence is shared: identify the relevant constraints first, then build the data flows and access paths around them. A privacy policy becomes stronger when the system removes the path by which a prohibited disclosure could occur.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Run code before the model on every turn
Probabilistic models are useful for generation, but a high-stakes decision needs an execution boundary outside generation. Agrawal's stack puts code above the model: every conversation turn passes through the code layer first. That layer decides whether to escalate an emergency or bring in a clinician. Most turns continue to the model, which handles the long tail of member conversations; it does not get to overrule an escalation.
A model alone is not a guardrail, and adding a system prompt does not make it a security boundary. The OpenAI Model Spec defines the authority order root → system → developer → user → guideline. That hierarchy specifies intended instruction priority. Agrawal warns that prompt injection can undermine prompt-based controls, using the stronger formulation that every layer above the user is one injection away from override. The specification does not establish that categorical claim; the actionable architectural point is that instruction priority cannot replace an independently enforced security check.
Three controls make the distinction concrete:
- Emergency escalation: self-harm, suicidal ideation, or an acute medical emergency triggers an appropriate route to 911 or 988. In Agrawal's design, the conversational model never sees that turn. The talk specifies the routing requirement but does not detail how emergencies are reliably detected.
- Capability routing: a multi-agent system may contain clinical, technical-support, educational, and exercise-recommendation capabilities. A model can assist with classification, but high-stakes paths need enforced routing. A clinical question must not silently fall through to generic technical support.
- Identity verification: an operation touching member data must verify that the person at the other end is the correct member. Authentication supplies the security boundary; a prompt asking the model to respect identity does not.
A TypeScript dispatch boundary can express that separation. Here, decision is the result of upstream controls, including authentication where member data is involved. Only an explicitly permitted model route can reach generation:
typescript
type Decision =
| { kind: "emergency"; destination: "911" | "988" }
| { kind: "clinician" }
| { kind: "authentication-required" }
| {
kind: "model";
capability: "clinical" | "tech-support" | "education" | "exercise";
};
type Action =
| Exclude<Decision, { kind: "model" }>
| { kind: "generate"; capability: string; text: string };
function dispatch(text: string, decision: Decision): Action {
switch (decision.kind) {
case "emergency":
case "clinician":
case "authentication-required":
return decision;
case "model":
return {
kind: "generate",
capability: decision.capability,
text,
};
}
}
The function returns a proposed action; it does not contact emergency services or execute a model call. Its useful property is the control flow: once an upstream control selects escalation or authentication, generation is no longer an available branch.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Evaluate live conversations—and staff the response
Pre-launch tests remain necessary, but a saved golden dataset cannot reveal every failure that emerges in live conversations. The third layer continuously scores production traffic across multiple dimensions. Agrawal combines three signal sources because each exposes a different part of the system's behavior.
- Automated judges: Agrawal suggests 30–40 judges as an illustrative scale, covering clinical accuracy, safety, escalation, relevance, drift, refusal, and other dimensions. The judge set keeps evolving as the team learns what to detect.
- Member feedback: per-message thumbs up and thumbs down provide a direct signal from the person receiving the answer. That signal can reveal tone problems and failures that automated judges miss.
- Trace review: random samples span capabilities, while Agrawal calls for review of 100% of high-stakes cases. People still need to read the underlying conversations and interpret patterns that no single score captures.
The bottleneck is the capacity to interpret signals and act on them. More compute or more judges does not supply the people needed to investigate a concerning trace, recognize a recurring pattern, and decide what to change. Review capacity therefore belongs in the operating design, alongside inference and evaluation capacity.
Some failures return after a prompt fix because the surrounding conditions change: a new prompt, a new tool, or a different model behavior exposes another version of the problem. Successive fixes may yield diminishing returns without eliminating the failure. Agrawal's response is to turn a newly observed production failure into a new judge, and to make the monitoring architecture capable of accommodating that growing set. Monitoring is how the team checks whether its architectural protections continue to hold as usage expands.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
One unresolved issue, five different risks
Monitoring can also reveal a problem that the architecture does not resolve. Agrawal introduces a launch scenario: one issue remains five days before a new healthcare AI capability is due to ship. Five stakeholders examine the same issue and disagree because they are evaluating different consequences.
| Stakeholder | Risk they see | Launch concern |
|---|---|---|
| Clinical | Member safety | Hold the launch to prevent harm |
| Legal | Regulatory exposure | Consequences of noncompliance |
| Compliance | Audit risk | Ability to satisfy scrutiny |
| Product | Adoption risk | A broken feature may not land |
| Engineering | Velocity risk | A fix means slipping the date |
Clinical wants to hold; engineering wants to ship. The disagreement does not require anyone to be irrational. Each function sees a real risk, but identifying those risks still leaves the release decision unresolved.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Set severity by harm, then choose the safer mistake
The first of Agrawal's five decision rules sets severity by the worst plausible outcome, rather than the average experience. Her comparison is deliberately asymmetric: minor annoyance affecting 100% of users is less severe than potential serious harm in 0.1% of cases. The triage question starts with the worst credible version of the failure; frequency alone cannot set its severity.
The second rule separates severity from capacity. Ownership, staffing, and the difficulty of a fix do not change the harm a bug can cause. Once severity is established, the available responses are to fix it, delay launch, or accept the risk with explicit sign-off. Quietly downgrading an issue because the team cannot reach it substitutes a scheduling constraint for a safety assessment.
The third rule gives uncertain decisions an asymmetric default:
| Issue type | More costly mistake | Default under uncertainty |
|---|---|---|
| Safety | Shipping a real harmful defect | Hold and fix |
| Polish | Delaying over a small flaw | Ship |
For safety, an unnecessary delay is preferable to exposing members to a real hazard. For polish, the cost of delaying can exceed the cost of the imperfection. This framework supplies a direction for judgment, not an automatic release verdict.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Use the organization's actual bar—and fund the follow-through
The fourth rule distinguishes revealed risk tolerance from stated risk tolerance. An organization may say it accepts no production bugs while continuing to ship behavior that contradicts that standard. Agrawal argues that if a behavior has been live for weeks or months without escalation, member complaints, or leadership concern, it should not become a blocker solely because it appears in a new feature. Existing production behavior supplies the calibration floor for a consistent launch bar.
The fifth rule returns to the human constraint. Judges can score traces automatically and dashboards can refresh every few hours, but people must interpret patterns and act on them. A launch plan that includes monitoring without allocating review capacity leaves that responsibility unfulfilled. And work deferred to a fast follow is already committed debt: it does not become an optional backlog item once the launch succeeds.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Verify the scorer before changing the agent
All five rules depend on trustworthy evidence. Yet an LLM judge is itself nondeterministic. In Agrawal's hypothetical example, a clinical-accuracy score stays at 4.9 for weeks, then drops to 4.5 and remains there the next day. Immediately changing the agent prompt could introduce a regression while leaving the actual source of the score change untouched. The first investigation is whether the judge is right.
Her caffeine example separates two superficially similar alerts. In scenario A, a member asks about caffeine and receives general guidance mentioning 400 mg per day for most adults, with qualifications concerning pregnancy and medications. The FDA guidance describes that amount as not generally associated with negative effects for most adults; it advises consulting a healthcare provider about restrictions in circumstances such as pregnancy or medication use. It is not a personalized safe dose. The judge nevertheless labels the response a hallucination because the agent mentioned pregnancy and medications without first checking whether those circumstances apply. In this example, those qualifications supply clinical context rather than assert facts about the member. The judge is over-calling.
| Scenario | Agent response | Judge behavior | Component to fix |
|---|---|---|---|
| A | General caffeine guidance with conditional qualifications | Incorrectly flags the qualifications | Judge |
| B | Says 1,000 mg of caffeine daily is fine | Correctly flags the unsafe endorsement | Agent |
The second scenario changes the answer, not the member's question. Here the judge correctly identifies a problem in the agent's guidance. The same alert category can therefore require changes to different components; a lower score alone does not tell the team which component is wrong.
The resulting investigation follows a short sequence:
- Inspect the response and the judge's reason for flagging it.
- Establish whether the judgment is correct before changing the agent.
- Correct the judge when it over-calls valid context; correct the agent when its answer is wrong.
A judge prompt is software that needs maintenance. Correcting an erroneous evaluator preserves the meaning of its score; it is not cheating the evaluation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Build a system worthy of member trust
Agrawal closes the architectural argument with three compact formulations: “Don't policy what you can architect.” “Don't prompt what you can code.” “Don't gate what you can monitor.” Each moves a responsibility into a mechanism that can enforce or continually check it. Human release decisions then address the residual risk, using worst-case severity, safer defaults, an explicit organizational bar, and committed follow-through.
Building these guardrails first can slow development. For member-facing healthcare AI, that cost is intentional: the product must be worthy of a person's trust in matters affecting their health. Architecture determines how the system can operate; accountable decisions determine when it should ship. As Agrawal puts it, “The architecture is how, the decisioning is when, and member trust is why.”
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
ECRI’s ranked hazards and recommendations for responsible use of healthcare chatbots.
Explains the 400 mg daily figure for most adults, individual sensitivity and circumstances requiring medical advice.
FDA’s overview of international guiding principles for developing AI and machine learning medical devices.
Enrolled legislation covering AI governance, prohibited uses and healthcare disclosure requirements.
The December 2025 Model Spec defines instruction authority from root rules through guidelines.
Full transcript
Loading transcript…