AI Engineer World's Fair 2026

200 Million Patient Interactions Later: What the Generic Voice Stack Misses

Vivek Muppalla· VP of Engineering, Hippocratic AI20:40

Loading slides…

Read the talk

200 Million Patient Interactions Later: What the Generic Voice Stack Misses

Hippocratic AI’s clinical voice system combines contextual speech recognition, parallel specialists, and layered verification to fit safer patient conversations into a telephone call’s latency budget.

From a talk by Vivek Muppalla

Before you start: Basic familiarity with language models and speech-to-text systems will help; the article explains the clinical workflow and latency mechanisms.

Who gets a proactive healthcare call?

Have you ever received a proactive call from your healthcare provider? When Vivek Muppalla asks the audience, only a few hands go up. Muppalla, who leads AI engineering at Hippocratic AI, is building a system that can make those calls and conduct clinical conversations. The problem begins with scarcity: too few clinicians, too little time, and too little money force healthcare systems to triage attention toward the sickest patients.

Falling conversation costs change the possible scope of outreach. Instead of reserving calls for the sickest five percent, the ambition is to call everyone who needs support. The goal is to extend clinicians’ reach, including patients they previously lacked the capacity to contact. Muppalla describes this as clinically safe abundance, built around three commitments: do no harm, put patients first, and provide access for all. Every employee takes that oath, including engineers.

Muppalla reports 200 million clinical interactions, zero significant safety incidents, deployment in more than 60 health systems, and patient satisfaction of 8.5/10. These are company-reported operating figures; the talk does not define the incident surveillance protocol. The accompanying slide displays 8.95/10 rather than the spoken 8.5/10, and labels its deployment figure as enterprise partners. Interactions should not be read as a count of completed calls.

Hippocratic AI slide with “Do No Harm,” a clinician image, and three boxes showing 200M+ clinical interactions without safety issues, 60+ enterprise partners, and an 8.95/10 average patient rating.Open full source frame
Hippocratic AI’s introductory slide presents clinical interaction, enterprise partner, and patient rating figures.
0:170:24
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:17 · section reference included

A conversation built from clarification

The demonstration starts with a health-monitoring-device check-in. The agent introduces itself as an AI care manager and asks whether the patient has time to talk. The patient wants to know whether the caller is human or a computer; the agent explicitly explains that it is artificial intelligence. The patient then describes feeling tired earlier, somewhat revived now, and amused by the encounter. Disclosure is part of the conversation, not an assumption left for the patient to resolve.

Medication recognition requires more than accepting the first transcript. Asked which medication they take, a patient names Prazosin but cannot remember another drug. The agent requests a spelling. The patient offers the uncertain P-R-O-Z-I-N; the agent proposes Prazosin and obtains confirmation. The useful operation is a sequence: hear a candidate, request additional evidence, propose an interpretation, and confirm it with the patient.

A vital-sign exchange has a different ambiguity. The patient gives a reading as 121 over 45, 91. The agent separates that utterance into blood pressure 121/45 and heart rate 91, then asks whether that interpretation is right. The patient confirms. Recognizing the numbers is only the first step; assigning them to the correct clinical fields is what makes the information usable.

Next, the agent asks whether the patient stopped taking Nifedipine as instructed. The answer reveals that the patient continued taking it because the replacement prescription was unavailable and they were concerned about blood-pressure control. The agent acknowledges that concern and directs the patient to their primary care doctor for medication guidance. It detects the discrepancy without improvising a replacement treatment plan.

The final exchange moves from general tiredness after hospitalization to weak legs while walking. The agent asks about chest discomfort and difficulty breathing; the patient reports having experienced shortness of breath. The agent then recommends speaking with a nurse immediately. The excerpt shows recognition of an escalation condition and the recommendation, but not a completed nurse transfer.

2:372:49
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:37 · section reference included

Buying time for more intelligence

Strip away the audio interfaces and the engineering problem becomes a joint constraint on intelligence and latency. Muppalla describes clinically capable models that take tens of seconds, sometimes more than a minute, to respond. That is unusable in a two-way telephone conversation. Fast models, meanwhile, did not provide the clinical accuracy the team needed. Hippocratic’s own models initially took several seconds; the team pursued a vertically integrated stack and hundreds of optimizations to improve responsiveness without sacrificing capability.

Muppalla groups voice architectures into ensembles, cascaded models, and real-time speech-to-speech models. Each can work well for different tasks, but a generic benchmark does not settle whether a system can handle a clinical workflow. Lab-results checking and interactive voice response, or IVR, navigation are two of his examples. The team required more than 99% accuracy on its lab-results-checking and IVR-navigation benchmarks. He points to the company’s published benchmarks; the current page is a changing evaluation resource, not a preserved copy of the presentation’s results.

Latency savings become an intelligence budget. An optimization first creates room within the conversational response window. The team spends that room on additional reasoning or safety checks, then optimizes again. This turns the apparent tug-of-war between speed and intelligence into a repeated engineering cycle: reduce delay, add capability, and recover the time that capability consumes.

Circular Hippocratic AI diagram with five segments labeled Latency Surplus, Smarter Models, Safety, Latency Increases, and Latency Innovations.Open full source frame
The Intelligence–Latency Flywheel connects latency savings, smarter models, and safety.
5:105:27
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:10 · section reference included

One voice, a constellation behind it

The Polaris constellation architecture divides the round trip into listening, reasoning, and responding. The listening side handles speech detection, bilingual switching, background noise, and conversational context. In the architecture described here, each conversation runs 31 models: one central conversation model and 30 specialists. Specialists cover domains such as laboratory results, medications, and scheduling, supplying information to the central model.

LayerResponsibilities
ListeningDetect speech, handle noise and language switching, incorporate context
ReasoningCoordinate the conversation with specialist input
RespondingProduce the voice and personality; document the conversation

The output side includes a custom personality, an HD-quality voice, and a clinical documentation engine. Information must reach both the patient and the health system. The motivation for multiple models is redundancy: Muppalla considers a single model an unacceptable single point of failure for a patient conversation. The patient should still experience one coherent voice.

7:558:07
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

7:55 · section reference included

Reasoning cannot repair everything the system mishears

Real calls contain noise that quiet-room speech benchmarks miss. Muppalla’s team found that apparent reasoning failures often began as recognition failures: the downstream model was reasoning over the wrong words. Spanish sí could become the letter C. He also cites an Arabic drug-name example with word error rate above 30%, illustrating how domain and language interact at the input boundary.

Instead of mapping audio directly to text in isolation, the system supplies audio, conversation history, and domain knowledge to an audio language model. Muppalla describes a decoder-only audio LLM supported by an encoder and projector:

  1. Encode the audio. The encoder is based on Whisper large-v3-turbo, which the team fine-tuned on millions of clinical conversations.
  2. Project it into language-model inputs. A conformer projector compacts the audio into tokens while retaining prosody, including pauses and stresses.
  3. Provide task context. Medication lists, the task being performed, required form fields, and policy give the model a more constrained interpretation space.

For a medication name, a patient-specific list provides relevant candidates instead of leaving the recognizer to choose from an effectively unbounded vocabulary. The system is trying to preserve both what was said and how it was said.

Contextual biasing also enters training. Synthetic patient records supply addresses, phone numbers, and other details so that the model learns to use contextual information during recognition. Muppalla describes an address utterance vulnerable to phonetic substitutions when heard without context; the address itself is redacted in the corrected public recording. Supplying the address as context helps recover the intended transcription before it reaches the reasoning system.

Short replies receive an additional check. A single word such as now can become no, or five can become fine; there may be little acoustic material to disambiguate the answer. When the patient gives a one-word response, a secondary scoring pass uses the whole conversation. Muppalla reports that the combined ASR techniques reduce medical word error rate by more than 50% relative to standard off-the-shelf models.

Muppalla also reports 3× faster P99 ASR latency, describing the comparison broadly as against other systems. The companion technical report supports a narrower comparison against the enterprise ASR previously used by Hippocratic. The useful performance target is tail latency: even an occasional slow recognition step can leave a patient waiting through an awkward conversational gap.

9:279:31
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:27 · section reference included

Parallel specialists, then verification

Running many models need not mean waiting for them sequentially. Polaris runs the models in parallel, and each specialist first performs a quick relevance check. If it has nothing to contribute, it short-circuits. This is the synchronous path: specialist participation is conditional, even though the checks run concurrently. Separate asynchronous models perform verification in the background.

An ibuprofen mention illustrates the division of labor. The medication specialist recognizes that the conversation concerns its domain and supplies context to the central model. The central model combines that input with the conversation’s history and goal, then steers the response. The specialist does not become a second independent voice speaking to the patient.

Tool use adds another failure surface. Background verifiers inspect tool-call parameters and returned responses, rather than assuming that a plausible conversation implies a correct external action. Muppalla attributes scheduling accuracy of 99.9%–99.99% to this verification approach. Where the workflow permits a later check, offline verification can identify an inaccurate appointment and support a correction in the back end or a callback and apology. The layers operate on different deadlines: some input must arrive before the next response, while some errors can still be repaired after the call.

Slide titled “Constellation of Models: orchestrator, supervisors, verifiers,” with orchestration and governance bullets plus medication supervisor and scheduling verifier examples.Open full source frame
Specialist supervision and verification support the main conversation model.
13:0413:16
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:04 · section reference included

Reducing inference cost without spending quality

The orchestration design depends on an inference stack fast enough to support it. Muppalla treats output quality as the constraint and speed as the optimization target. He calls the required optimizations “lossless,” including the move from 16-bit to 4-bit computation. Here, that expresses the team’s quality requirement; reduced numerical precision should not be interpreted as a guarantee of numerically identical computation.

Three techniques address different sources of inference work:

  • Four-bit quantization: reduces the precision used for model computation, helping reduce latency under the team’s quality constraint.
  • Speculative decoding: a smaller model drafts tokens ahead of time, and the main model checks them together.
  • KV-cache compression: retains more of a long conversation’s context in cache so that prior computation can be reused.

For the caching system, Muppalla reports a hit rate above 96% and an 18× improvement in prefill speed. The companion report describes steady-state prefix reuse and an estimated prefill improvement; this is not an 18× speedup of the entire call.

15:0115:12
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

15:01 · section reference included

A small error rate becomes a daily workload

At 10,000 scheduling calls per day, a 1% failure rate produces an expected 100 incorrect appointments per day. Muppalla uses this rounded calculation while discussing volumes above 10,000 calls daily. The failures can mean an incorrect appointment type, date, or time—and potentially a missed critical appointment. 99% accuracy can still create an unacceptable operational burden when the remaining errors reach real patients every day.

Rare failures also require substantial test sets. Muppalla estimates about 450 tests for a 99% chance of detecting a 1% error rate, and about 1,900 tests to see the failure ten times. Under independent trials with a fixed 1% failure probability, the binomial model makes the first estimate precise:

P(at least one failure)=1−(1−p)np=0.01nmin⁡=⌈log⁡(1−0.99)log⁡(1−0.01)⌉=459\begin{aligned} P(\text{at least one failure}) &= 1-(1-p)^n \\ p &= 0.01 \\ n_{\min} &= \left\lceil\frac{\log(1-0.99)}{\log(1-0.01)}\right\rceil = 459 \end{aligned}

At 450 independent tests, the detection probability is approximately 98.91%; 459 tests reaches 99%. Observing ten failures is likewise a probability, not a guarantee. At 1,900 tests, the expected count is 19.

The calculation can be made explicit in Python, including the probability of observing at least ten failures:

python

from math import ceil, comb, log1p

failure_rate = 0.01
confidence = 0.99
minimum_tests = ceil(log1p(-confidence) / log1p(-failure_rate))


def probability_at_least(n: int, k: int, p: float) -> float:
    below_k = sum(
        comb(n, failures) * p**failures * (1 - p)**(n - failures)
        for failures in range(k)
    )
    return 1 - below_k


print(f"Tests for 99% detection: {minimum_tests}")
print(f"Detection with 450 tests: {probability_at_least(450, 1, failure_rate):.2%}")
print(f"Ten or more in 1,900: {probability_at_least(1900, 10, failure_rate):.2%}")
print(f"Expected failures in 1,900: {1900 * failure_rate:.0f}")

These assumptions matter: repeating similar scenarios is not necessarily equivalent to drawing independent tests from a fixed failure distribution. The calculation explains the scale required to observe rare errors, not whether a particular clinical test set is representative.

Hippocratic combines synthetic evaluation with continuous clinician testing because the team found synthetic data alone insufficient. Muppalla reports more than 7,000 trained clinicians and over 700,000, approaching 800,000, evaluation conversations. The accompanying slide gives more specific figures of 7,750+ trained clinical testers and 775,000+ test calls. Human evaluation supplies the domain judgments that a large synthetic test set cannot automatically provide.

Slide titled “99% is not good enough for us,” showing 100 per day and approximately 36,000 per year, testing requirements, and a banner citing 7,750+ trained clinical testers and 775,000+ test calls.Open full source frame
The slide connects a 1% scheduling error rate with consequential failures and extensive human testing.
16:1516:29
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

16:15 · section reference included

Correctness, harm, and willingness to talk

Across five shipped product generations, Muppalla describes improving performance and a grading approach that extends beyond whether an answer is valid. The rubric distinguishes correctness and potential consequences: no harm, minor harm, severe harm, and death. A wrong answer is not merely a failed test case; its severity depends on what could happen if a patient acts on it.

Muppalla reports 99.89% for Polaris versus approximately 81% for humans, describing the comparison in terms of no harm. The published materials do not support treating those numbers as a clean no-harm comparison: the technical report places the human figure under Correct Advice and separates No Harm, while the company companion article calls 99.89% clinical accuracy. These are reported evaluation results, not demonstrated differences in patient outcomes. Muppalla attributes the system’s advantage to its lack of fatigue and the availability of many specialist supervisors.

Safety alone does not make a patient willing to open up. Voice and conversational behavior must also convey empathy so that patients want to engage. Finding existing benchmarks inadequate for this requirement, the team developed HEART and published the paper. Its role here is to make emotional-support quality an explicit evaluation target alongside clinical safety; an empathy benchmark does not itself establish adherence or better clinical outcomes.

17:5318:09
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

17:53 · section reference included

Building the team that sustains the stack

The closing investment is organizational. Muppalla describes changes to hiring and everyday work over the preceding years, including unlimited tokens for employees. Two training programs address different starting points:

  • Agent deployment residency: early-career engineers learn to build and deploy safer healthcare agents.
  • AI residency: experienced software engineers learn model training and contribute to the underlying systems.

These programs connect deployment experience with the engineering needed to improve the stack.

The final invitation is to build systems that pursue quality, speed, and safety together, rather than accepting a requirement to choose only two. In this architecture, that ambition depends on repeated work across recognition, specialist reasoning, verification, inference, and evaluation—and on people trained to keep improving those parts as one patient-facing product.

19:2719:43
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

19:27 · section reference included

Resources

From the talk

Transcript download
Full transcript

Loading transcript…