AI Engineer World's Fair 2024

Building an AI assistant that makes phone calls

Loading slides…

Read the talk

Building Floyd, an AI assistant that makes phone calls

A voice request becomes a phone conversation through reactive database queries, personal context, and a streaming speech pipeline—with live calls exposing the limits.

From a talk by Tom Redman

Before you start: Familiarity with React hooks, TypeScript, and HTTP or WebSocket connections will help with the code walkthrough.

What would a personal assistant actually do?

Calendar events, reminders, and timers cover only a small part of a personal assistant’s work. Beyond physical errands, much of the job happens on the phone and over email. Could an AI assistant know enough about its user to handle those conversations with another person, without making the interaction uncomfortable? Tom Redman, head of developer experience at Convex, starts with that problem: combine real-time speech recognition, personal context, and nearly real-time speech synthesis into an assistant that can act by phone.

Slide titled “Building a better AI assistant,” with a subtitle about making calls, a reaction image, and a speaker inset.Open full source frame
Building a better AI assistant: “This one can make calls.”

The prototype is called Floyd because Redman misremembered the name of Lloyd, the personal assistant in Entourage. Running it requires Google Cloud with speech-to-text enabled, a Convex account, an OpenAI key, and a Twilio account, with configuration for each in the environment. The Floyd workshop repository accompanies the project; during the workshop, Redman clarifies that it is not yet on GitHub and promises to share it afterward through Slack.

0:300:48
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:30 · section reference included

The database is the handoff

Start with a request such as booking a dentist appointment. The phone or web client streams the user’s voice to Google Cloud and receives a transcription. It then saves the request text and the requesting user in Convex. That write is the client’s handoff to the backend.

The server subscribes to a reactive query for new requests. It does not poll, and the client does not separately notify it that work has arrived. When the query result changes, the server picks up the request and writes a progress status. The client has its own subscription, so that status appears in the interface through the same database mechanism. Request dispatch and progress reporting share a durable record.

4:034:19
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:03 · section reference included

Prepare the context, then start the conversation

A short request depends on information the user has not repeated. Calling a child’s school requires knowing which school; booking car service may require the car model and a preferred mechanic. Floyd’s intended memory grows from requests, information the user supplies, and possible connections such as email. Users can provide as much or as little as they want, with the assistant asking for missing details. For each task, the server saves a moment-in-time context snapshot on the request, alongside the requester and the request text.

GPT-4o receives the request and its context before the call begins. The server prepares the model for a conversation in which the next participant will be the person answering the phone. Ordinarily, the destination number comes from existing user context. Looking up a vendor based on the user’s location is a proposed extension, not implemented here. If that search failed, Floyd could ask the user by text whether they have a preferred vendor.

Once connected, the conversation repeats a pipeline:

  1. The model generates a conversational response.
  2. OpenAI text-to-speech turns that response into audio.
  3. Twilio carries the audio to the person on the phone.
  4. The person’s reply streams to Google Cloud for transcription.
  5. The transcription returns to the model for the next turn.

The loop continues until the conversation finishes. An audience member suggests using multimodal Gemini to accept audio directly and remove the separate transcription step. Redman has not tried it; replaceable service boundaries are intended to make that kind of experiment possible.

During the call, the server appends transcript entries to the request in Convex. A React client using useQuery receives the changed result automatically. Where a local useState value would need explicit updates, the subscribed query lets database writes drive the displayed transcript. The user can watch the conversation develop without participating in the media connection.

5:576:14
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:57 · section reference included

Calling the school

For the live demonstration, every destination number is overridden to call Redman’s own phone. He plays the business or school receiving the call; the prototype is not contacting real vendors. He asks Floyd to call the school and explain that Mara is sick and staying home. When he stops speaking, the client submits the request automatically and displays its ID. His phone rings.

Redman answers as Mara’s school. Floyd introduces itself as calling on behalf of Tom Redman and reports that Mara Redman is staying home sick. The conversation continues through follow-up questions: is she going to be okay, and when will she return? Floyd says she hopes to be back by the end of the week. The spoken request supplied no return date; the demonstration does not establish where Floyd obtained that answer.

The exchange reaches a polite ending, but the pauses between turns are conspicuous. Redman identifies latency as crucial and proposes production configurations and alternative models as further experiments. He also describes using Opus for efficient, phone-quality streaming; the later media walkthrough distinguishes that codec discussion from the format sent to Twilio.

Behind the call is an action plan generated before any vendor-facing speech. OpenAI is prompted as a teammate: here is the request, here is what is known about the user, and here is the task to accomplish. For the school request, the plan extracts Mara’s full name, the school, the absence reason, the current date, and the steps needed to fulfill the request. The server feeds the plan back as additional conversation context; it is not read aloud.

Navigating an automated call tree remains separate work. Redman has neither tried it nor added specific support, although he considers it a solvable problem. His first experiment targets the less certain part: sustaining a conversation with a human. The request detail view also exposes the transcript as it arrives.

12:0512:15
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

12:05 · section reference included

Ordering flowers—and knowing when to stop

The next request is to order flowers for Krista. Redman answers as Sweet Violet’s Flowers while the action plan appears in the interface. Floyd asks for delivery that day. When asked which flowers, it says Krista likes the shop’s usual offerings and asks for a popular recommendation.

The shop offers peonies and roses for $560. Floyd initially responds, “That sounds good,” and asks whether they can deliver to an address in Guelph, ON that day. When the shop asks whether Tom’s information is already on file, Floyd asks whether the details are needed again, then abruptly says it will have to call back. No completed purchase is demonstrated.

That exit is deliberate: the assistant has a bailout instruction for situations where it lacks information the other person needs. Redman describes the next step as a text to the user explaining that a budget is missing and the flowers are expensive. The text is a prospective follow-up, not something shown arriving. The demonstrated behavior is narrower: Floyd can stop the call, but it has already verbally accepted the quoted offer before doing so.

Redman then opens the stored flower request in the Convex dashboard, adds that there is no budget, and saves it. The client immediately reflects the edited request. This is the same mechanism that makes appended transcript entries appear: a database mutation changes a subscribed query result. The dashboard also contains user context. Redman explicitly calls the implementation a proof of concept, far from production best practice; the edit demonstrates synchronization, not a resumed or completed order.

17:1117:26
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

17:11 · section reference included

Separate the phone connection from the interface

The original goal included streaming the call transcript back to the user and eventually letting the user listen to the call live. Redman first tried to keep the implementation together in a Next.js application hosted on Vercel. Socket.IO helped with parts of the initial transcription, but handling Twilio’s stateful WebSocket audio connection proved difficult in that serverless hosting setup. This describes the 2024 implementation constraint; Vercel later announced WebSocket support in Functions on Fluid compute in June 2026, subject to function limits.

The revised design assigns persistent media handling to an Express server. That server does the phone-call work and writes updates to Convex; the frontend subscribes only to the data it needs. It does not need to own the call’s socket or poll for progress. The side-by-side interface and database view show the resulting separation: a call transcript in Floyd, personal context in a Convex sessionUsers record.

Side-by-side browser windows show completed Floyd requests and a call transcript on the left, with an expanded sessionUsers context field in Convex on the right.Open full source frame
Floyd’s call transcript beside a Convex sessionUsers record.

Sharing a codebase is still possible. Convex schemas, backend functions, and APIs can live alongside frontend code while being deployed to Convex servers, rather than shipped to the browser. A host that supports the required persistent WebSockets could also change how the media server is packaged. Code location and execution location are separate decisions.

20:5221:01
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

20:52 · section reference included

A query connects the React dashboard to server state

In the frontend walkthrough, a single subscribed query produces the request array. The dashboard renders that array as a list and uses the selected request’s data for the detail pane. When a matching request changes, the query result updates and React renders the new state.

There is an important difference between the earlier architecture sketch and the actual query: the prototype’s get function is not user-specific, and the application has no authentication. The intended per-user subscription is therefore not an access-control boundary in this implementation.

The generated identifier api.requests.get connects the React subscription to the exported Convex query. This minimal component illustrates the data flow without adding another copy of the query result in local state:

tsx

import { useQuery } from "convex/react";
import { api } from "../convex/_generated/api";

export function RequestInspector() {
  const requests = useQuery(api.requests.get);

  if (requests === undefined) {
    return <p>Loading requests…</p>;
  }

  return <pre>{JSON.stringify(requests, null, 2)}</pre>;
}

Convex generates the typed API from the backend functions and updates it as those files change. The query implementation stays server-side; importing its API reference does not put the database function into the browser bundle.

Redman describes a V8 execution environment close to the database and emphasizes speed and relief from application-managed caching. More precisely, TypeScript is compiled into JavaScript for execution in V8; this does not require assuming every function and database operation occupies one physical machine. The architectural benefit for Floyd is that the client consumes a subscribed result rather than building its own invalidation and refresh system.

23:5224:02
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

23:52 · section reference included

Backend functions, HTTP actions, and debugging

Why use Convex instead of MongoDB, DynamoDB, or Postgres? Redman’s answer centers on developer ergonomics: schema-driven type safety through to the frontend, editor completions, and less infrastructure code to maintain. His broader claim is that this can reduce the need for dedicated backend or infrastructure staffing. For Floyd, the request backend consists of a small set of operations: retrieve requests, retrieve one by ID, retrieve pending requests, and post a request.

HTTP actions provide another entry point. They can define GET or POST endpoints with arbitrary responses, whether or not the endpoint accesses the database. In the walkthrough, a POST route runs an action that updates stored request data. The separate Express process remains because it handles the Twilio stream.

The debugging tools discussed have different roles:

ToolWhat Redman uses it for
Dashboard logsInspect requests and runtime output
Dashboard function runnerInvoke functions and inspect responses
Local compilation outputCatch errors while developing
Server-side breakpointsSupport left unanswered in the workshop

The dashboard exposes function definitions as well as logs. When asked specifically about breakpoints in Convex server code, Redman says he does not know and offers to follow up; his own workflow relies on logs and local compilation errors.

Terminal split into panes displaying transcription messages, Convex functions-ready messages, an ngrok status view, and phone-call server output.Open full source frame
Terminal panes show transcription output, Convex function readiness, and server logs.
27:2127:35
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

27:21 · section reference included

From pending request to connected call

The Express server contains its own Convex client. Its subscription watches pending requests, the default status assigned when the client creates one. The worker then follows this sequence:

  1. Check that the request exists.
  2. Change its status from pending to in-flight.
  3. Retrieve the full request and session name.
  4. Gather the user’s stored context.
  5. Filter that context to the information relevant to this task and save it on the request.
  6. Initiate the call.

This is the concrete implementation behind the earlier reactive handoff. Redman notes that the design still needs work; the status transition is the demonstrated workflow, not a complete account of production job coordination.

Calling Twilio starts an asynchronous interaction. The call configuration tells Twilio where to send status callbacks and streaming data. During development, ngrok exposes the Express server so Twilio can reach it. The server handles call lifecycle notifications and media arriving over its WebSocket connection.

The key to reconnecting those events with the user’s task is the request ID. Redman passes it through the call configuration into the callback location. When Twilio connects back, the media stream handler extracts that ID and looks up the request. The call can then recover its instructions and context without relying on the original function invocation still being active.

32:0632:25
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

32:06 · section reference included

Audio chunks become conversational turns

A repeatedly invoked message handler processes incoming media. Redman describes the application as receiving audio after utterances and discusses fixed chunks, pauses, and punctuation as possible boundaries. That description should not be confused with the transport contract: Twilio Media Streams carries raw audio chunks with sequence information and timestamps, not punctuation-delimited speech. The application’s turn handling sits above those media messages.

Incoming audio is converted into a format suitable for Google Cloud recognition. The returned transcription feeds the conversation, and the generated reply goes through the speech function. Outgoing audio is converted to μ-law and Base64-encoded for Twilio. The documented Twilio wire format is mono 8 kHz μ-law; Base64 is the payload encoding, not another audio codec. This is distinct from the earlier discussion of Opus as a streaming optimization.

The final serialization step can be expressed directly in TypeScript:

typescript

import { Buffer } from "node:buffer";

export function encodeTwilioMedia(
  streamSid: string,
  mulawAudio: Uint8Array,
): string {
  return JSON.stringify({
    event: "media",
    streamSid,
    media: {
      payload: Buffer.from(mulawAudio).toString("base64"),
    },
  });
}

Here, mulawAudio must already contain the correctly formatted audio bytes. Encoding them as Base64 does not resample or transcode them.

Alongside media processing, a client mutation appends each new Google Cloud transcript entry to Convex. The frontend’s query subscription delivers those changes to the transcript view. Audio transport and transcript presentation therefore share a request identity but do not require the same connection.

36:3536:47
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

36:35 · section reference included

Buy time at pickup, then establish trust

The first turn has a special latency problem. The person answers the phone, then the recognition, conversation, and synthesis pipeline starts working. Redman adds a pregenerated introduction so Floyd can begin speaking immediately rather than waiting for that entire chain. He suspects this introduction was not working during the live demonstrations.

While the recorded introduction plays, the first conversation loop runs concurrently. The introduction identifies Floyd as calling on behalf of a client, buying time for a task-specific continuation such as booking a car for an oil change at Brock Road Garage. This overlaps useful speech with startup work; it does not make the underlying services intrinsically faster. Redman describes it as one of several small changes that can improve the experience for the person answering.

The intended introduction also discloses that the caller is AI. Redman does not want Floyd to misrepresent itself. Its intended role is to buy services or make ordinary changes such as canceling appointments, never to sell. His hypothesis is that businesses will accept legitimate purchasing calls and may learn to interact efficiently with an assistant.

An audience suggestion that businesses could have their own agents exposes another boundary: the prototype has no guard against a person on the call telling it to forget its instructions. Redman explicitly identifies that prompt-injection risk as a reason not to deploy it. If both sides eventually use agents to negotiate an appointment, he observes, the exchange starts to resemble APIs talking to each other.

38:4038:55
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

38:40 · section reference included

Measure the stages that make people wait

An audience member asks whether serverless warm-up explains the initial delay. Redman attributes no warm-up delay to Convex, describing an always-running deployment built around WebSockets. He also makes a broad scalability claim, but the latency diagnosis he offers comes from instrumentation around the speech pipeline: time to transcribe, time to obtain a conversational response, and time to synthesize that response. No stage-by-stage benchmark table is presented.

One observed source of growing delay is conversation history. Each model request includes the preceding conversation, so the prompt grows as the call continues. Redman reports that delays tend to worsen with longer conversations. He proposes reducing prompt size and trying the OpenAI Threads API, which he had not used when building this functionality. Those are proposed experiments; the workshop does not demonstrate a latency benefit from Threads.

Redman reports that synthesizing three sentences with OpenAI text-to-speech could take two to five seconds in this prototype. The observation has no specified model, sample count, or timing boundary, so it is useful as a description of the demonstrated system’s delay rather than a service benchmark. With limited tuning available in his implementation, he would consider paying for lower latency or switching synthesis providers.

The optimization opportunities are distributed across the chain:

  • Conversation generation: reduce the amount of history sent with each turn.
  • Speech synthesis: investigate a lower-latency service or configuration.
  • End-to-end orchestration: reduce individual service delays, because their waits accumulate into the pause the caller hears.

Redman says several material optimizations remain untried. His expectation that provider progress could make the system twice as fast in six months without application changes is explicitly a bet, not a measured outcome. The modular architecture leaves room to test that expectation as services improve.

42:1442:26
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

42:14 · section reference included

The infrastructure bet behind the prototype

The final technical question concerns Convex’s internals: how queries are distributed and what database infrastructure runs underneath. Redman describes a database built from scratch in Rust and points to its open-source implementation. He credits the leadership team, Jamie and James, as a reason he joined. James Cowling’s MIT doctoral work concerns large-scale distributed transaction processing and consensus protocols—the database expertise Redman invokes in explaining the system’s origins.

In Redman’s account, an AWS cluster runs the Convex database and application, including the WebSocket connections and subscriptions. He does not attempt a detailed answer about physical distribution, which is outside his expertise, and offers to follow up. He reports large customers and no fundamental scaling problems so far, without quantitative results. For the implementation details, he points readers to the Convex backend repository and chief scientist Sujay Jayakar’s How Convex Works.

Choosing a custom database instead of a familiar option such as Postgres is a bet. Redman justifies it through developer ergonomics and speed of product development: let infrastructure engineers spend their time building what customers need instead of maintaining database backups. Floyd makes that trade-off tangible. Once request state and live updates have a shared home, the remaining engineering effort can focus on what makes a phone assistant useful—carrying out the task, handling missing information, and making the conversation work for the person answering.

47:0747:22
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

47:07 · section reference included

Resources

From the talk

Updates since the talk

Transcript download
Full transcript

Loading transcript…