AI Engineer Summit 2023

Building Reactive AI Apps

Matt Welsh· Co-Founder, Fixie.ai17:02

Loading slides…

Read the talk

Building Reactive AI Apps with AI.JSX

Compose prompts, tools, retrieval, and streaming output as a TypeScript component tree, then follow that application model into an embedded chat interface and a voice ordering demo.

From a talk by Matt Welsh

Before you start: Familiarity with TypeScript, JSX, and basic LLM prompts will help you follow the component examples.

The application is more than an LLM call

How do you turn a language model into a high-quality application without making every developer assemble the surrounding infrastructure? A useful product needs more than a prompt: vector databases, context-window management, retrieval-augmented generation, and tools all become part of the job. Matt Welsh introduces Fixie’s open-source AI.JSX framework to make those pieces easier to compose.

Slide stating that building and deploying high-quality LLM-powered experiences is still super hard, above two rows of challenge labels.Open full source frame
Building high-quality LLM applications involves grounding, context limits, tools, RAG, and observability.

The deliberately theatrical reveal is angle brackets: JSX, used to express AI applications in TypeScript. The intended audience is frontend and full-stack developers—the people building the interfaces through which users experience AI. Welsh argues that this community lacks good tools and, by his account, contains more JavaScript developers than the Python community. The goal is to bring AI application development into their existing programming environment.

0:360:51
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:36 · section reference included

A completion expressed as components

The first application asks for a Shakespearean sonnet about large language models. ChatCompletion represents the model call; its UserMessage child supplies the prompt. The essential JSX expression is small:

tsx

const sonnet = (
  <ChatCompletion>
    <UserMessage>
      Write a Shakespearean sonnet about large language models.
    </UserMessage>
  </ChatCompletion>
);

Rendering this component tree produces the model’s response. The sonnet is the Hello World example: a single prompt inside a single completion.

React provides a useful analogy for this programming model, but AI.JSX is a separate framework with React integration. The component names and behavior here follow the presentation’s API rather than a claim about current package compatibility. Welsh’s tour expands from the completion into several capabilities:

  • Model providers: support for Anthropic and OpenAI.
  • React integration: AI functionality that can be incorporated into a full-stack application.
  • Retrieval: built-in RAG support, multiple vector databases, and the option to connect your own database.
  • Tools: calls to custom functions and external APIs.
  • Generated interfaces: UI components produced through the application’s component representation.
  • Extensibility: custom behavior written in TypeScript.

The UI capability follows from working with a structured component representation: an application can produce interface components as well as text.

3:473:52
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:47 · section reference included

From a prompt to an asynchronous tree

A reusable MakeSetting component asks the model to write a one-paragraph description of a story setting. It accepts children and inserts them into that request. Those children can be a literal string describing the desired setting, or another JSX subtree whose rendered result becomes part of the prompt. Calling MakeSetting therefore means instantiating the component with the setting you want as its child. The component owns the instruction; its caller supplies the material.

That alone could look like ordinary function composition with extra punctuation. The substantive difference is the rendering tree: AI.JSX renders nodes asynchronously, in parallel, and as streams. Where React renders an interface toward the browser DOM, this tree assembles inputs and produces LLM output. Nesting expresses dependencies between pieces of generation, rather than merely formatting a prompt string.

The story example makes the execution model concrete. MakeStory contains separate children for the character, setting, and plot. In the story example, three child LLM calls run and stream concurrently. Their tokens flow back into the parent’s rendering process, and MakeStory streams the resulting story outward. This is the concurrency described for the component tree; it does not specify a provider-level guarantee about when a parent model request can begin consuming an unfinished prompt. The useful abstraction is that independent branches can generate their material concurrently while the application presents streaming output.

5:556:05
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:55 · section reference included

Use a wrapper to transform generated output

The next composition puts one generation around another. KidSafe supplies a system message asking the model to rewrite text so it is safe for children. Its children’s rendered content becomes the user message for that rewrite. Wrapping an existing component in KidSafe therefore applies the transformation without changing the wrapped component’s own prompt or implementation.

The separation is useful: the inner component generates content, and the outer component requests a transformation of that content. Here, “kid-safe” describes the requested rewrite, not a verified safety guarantee. The mechanism is another prompted model call over child output.

8:308:43
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:30 · section reference included

Make an external API available during rendering

The GitHub example extends the tree beyond text generation. A record defines the tools available to the LLM, with one entry for calling GitHub’s GraphQL API. That tool pairs an English description with a JavaScript function. The description explains what the tool does; the function performs the request using fetch. Welsh omits the function body from the presentation, so the example establishes the tool boundary rather than showing the complete HTTP implementation.

The application then instantiates UseTools and passes it the tool set. Components that need those tools during rendering can invoke them. Defining the operation and making it available to the application are separate steps: the record describes the callable capability, while UseTools supplies access within the rendering process.

Code slide titled Call out to APIs and tools, with the tool definition dimmed and the UseTools application snippet highlighted.Open full source frame
The example makes tools available through UseTools.
9:129:25
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:12 · section reference included

Put retrieved context inside the prompt

Retrieval-augmented generation uses the same composition mechanism. A ChatCompletion contains a SystemMessage and a UserMessage. The system message tells the model to use supplied information when answering the user, and a nested DocsQA component supplies that information. Welsh describes the displayed RAG composition as roughly ten lines of code. That compactness applies to the composition, not to the work of preparing the corpus.

The data path is explicit:

  1. Prepare the corpus. Documents have already been crawled, indexed, and placed in a vector database.
  2. Retrieve for the query. DocsQA receives the corpus configuration and the user’s query, then returns relevant chunks.
  3. Assemble the completion. Those chunks appear inside the SystemMessage, beside the instruction to use them.
  4. Ask the question. The UserMessage contains the query again, and the enclosing completion generates the answer.

The query serves two purposes: selecting context and asking the final question. Putting both uses in the component tree makes their relationship visible. Welsh argues that this representation is easier to understand, modify, and integrate than coordinating many separate library calls.

10:0910:31
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:09 · section reference included

Embed the interface and manage the backend

Once the AI application is built, FloatingFixieEmbed provides the transition into a React interface. Instantiating it supplies a chat window with session management, Markdown rendering, and custom UI support. Welsh’s one-line embedding description refers to instantiating this supplied interface, not implementing the application behind it.

The framework and the hosted service have different responsibilities. AI.JSX expresses the application; the Fixie platform presented here hosts, runs, deploys, and manages it. Its managed RAG pipeline covers document ingestion, chunking, embedding, the vector database, and document storage. It also manages conversational state so later user-agent exchanges retain context. Welsh describes the platform’s LLM interfaces as tuned for performance and low latency, without supplying benchmark results.

The platform offered two authoring paths: build through its no-code web interface, or write a custom AI.JSX application and embed it in a site. At presentation time, Welsh invited developers to sign up with a free starting option. Using the hosted service was optional: the open-source framework could also be used independently of Fixie.

11:1311:26
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:13 · section reference included

Extend the application to spoken interaction

The final capability is bidirectional voice: a user speaks to the agent, and the agent speaks back. Responsiveness becomes especially visible in this interface because a pause after every turn interrupts the conversation. Welsh estimates a four-to-five-second response gap in the recent ChatGPT voice demos he had seen. That is his estimate of particular demos, not a measured comparison with Fixie or a claim about present-day ChatGPT.

The voice system is presented as grounded in the application’s documents and data. Welsh calls its performance state of the art, but demonstrates it through a recording rather than a latency benchmark. To avoid venue Wi-Fi problems, CTO Justin recorded the interaction the previous day; Welsh says it was unstaged, untrimmed, and not sped up.

Slide titled AI.JSX supports Real-Time Voice, listing state-of-the-art performance, real-time bidirectional conversations, and grounding on documents and data.Open full source frame
AI.JSX presents real-time voice conversations grounded in documents and data.

Welsh points the audience to a demo address for trying it themselves, then sets up the scenario: a fictional restaurant called Dr. Donut. The customer is at the drive-through, and the AI plays the employee taking the order. This turns the voice interface into a specific application task with menu questions, quantities, and an eventual order recap.

13:3213:40
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:32 · section reference included

Take the order, answer a question, and retain context

The agent opens with a restaurant greeting. The customer asks for two pumpkin spice iced donuts, and the agent confirms adding them to the order. Next, the customer asks for the best coffee. Instead of treating that as another order item, the agent recommends seasonal pumpkin spice coffee and the year-round caramel macchiato. The customer chooses two macchiatos, and the agent confirms adding two caramel macchiatos. The exchange moves from ordering to recommendation and back to ordering without requiring the customer to repeat the donuts.

The customer then asks what the agent thinks about large language models. Its response preserves the restaurant role: “Sir, this is a Dr. Donut.” The customer returns to the order, and the agent recaps two pumpkin spice iced donuts and two caramel macchiatos, states a total of $9.56, and directs the customer to the next window. These are spoken confirmations within the demonstration; no underlying payment or order transaction implementation is shown.

The demonstration brings the earlier pieces into a user-facing interaction: conversational context survives a recommendation and an off-topic question, and the agent produces a coherent final recap. Welsh closes by explaining that the full Fixie platform supplies the backend and that developers can build the same kind of application with Fixie and AI.JSX. The latency and quality work serves that practical outcome: an application that can carry a task through successive spoken turns.

15:1715:22
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

15:17 · section reference included

Resources

From the talk

Updates since the talk

Transcript download
Full transcript

Loading transcript…