AI Engineer Summit 2023

Storyteller: Building Multi-modal Apps with TS & ModelFusion - Lars Grammel, PhD

Loading slides…

Read the talk

Storyteller: Building Multi-modal Apps with TS & ModelFusion

A spoken topic becomes an illustrated, multivoice children’s story by combining parallel generation, partial structure parsing, and playback before the full story is ready.

From a talk by Lars Grammel

Before you start: Familiarity with TypeScript, React state, and asynchronous iteration will help with the implementation details.

A forest in trouble, told in several voices

How do you turn a child’s spoken topic into an audio story with a narrator and distinct character voices? Storyteller, Lars Grammel’s TypeScript application, uses ModelFusion, the AI orchestration library he developed, to generate preschool stories about two minutes long from voice input. Its opening sample follows Benny into a troubled forest: the narrator describes brown leaves and unusually quiet animals, Benny asks what is wrong, and another character explains that the trees are dying. The voices make the dialogue audible as a conversation rather than a single narrator reading every line.

A story card titled “Benny and the Magical Forest Rescue” shows a bear in a boat and audio controls. OpenAI, LMNT, ElevenLabs, and stability.ai appear beside their respective capabilities.Open full source frame
Storyteller’s illustrated audio story alongside its model providers.
0:000:20
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:00 · section reference included

Make the first result visible

Storyteller has a React client and a custom Fastify server. Its design has three challenges: responsiveness, quality, and consistency. The interaction begins with a small screen containing a Record Topic button. Holding the button records audio; releasing it sends an audio buffer to the server for transcription with OpenAI Whisper. Grammel reports about 1.5 seconds to transcribe a short topic in this application.

As soon as the transcription becomes available, the server sends it back through server-sent events. The client updates React state and renders the result, giving the user visible evidence that the request is progressing. Meanwhile, the server starts generating the story outline. Client feedback does not have to wait for the next generation stage to finish.

1:001:13
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:00 · section reference included

Use the outline to start parallel work

The outline is generated with gpt-3.5-turbo-instruct, chosen for its speed. Grammel reports about four seconds to generate a story outline. Once that shared story foundation exists, three branches can proceed in parallel: title generation, illustration generation, and generation and narration of the audio story. These branches share an input, but one branch’s completion need not hold up the others.

A client-server diagram shows audio recording flowing into topic transcription and story-outline generation, then branching into three generation tasks.Open full source frame
The story outline branches into title, image-prompt, and audio-story generation.

The title branch uses gpt-3.5-turbo-instruct again. When the title is ready, the server sends an event and the client renders it. This adds another concrete result to the screen while the more expensive branches continue.

2:142:28
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:14 · section reference included

Derive the illustration from the story

The illustration branch first needs an image prompt. To keep the picture consistent with the story, GPT-4 receives the whole story and extracts representative keywords. Those keywords become the prompt for Stability AI’s Stable Diffusion XL. The text model therefore bridges the narrative and the image model: the image is grounded in the story’s content rather than generated independently from the initial spoken topic.

The server stores the generated image as a virtual file and sends its path to the client in an event. The browser then retrieves the image through a regular URL request from its image element. Events carry the notification and location; ordinary media requests carry the generated asset.

2:573:17
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:57 · section reference included

Stream completed passages, not just tokens

With the illustration appearing in the UI, the largest remaining task is the full audio story. A more complex prompt extends the story into a structure containing dialogue and speakers. GPT-4 runs at a low temperature to preserve the original story. Grammel reports about one and a half minutes for full structured-story generation with low-temperature GPT-4. Waiting for that entire result before beginning narration would make the interactive client feel stalled. These timings describe his application in the recording; input lengths, model snapshots, network conditions, and trial counts are not specified.

The useful streaming unit is a finished story passage. A stream of characters alone cannot tell the speech pipeline whether a passage has its complete text and speaker information. The application needs partial parsing of the generated structure, followed by a decision about which parts are ready to narrate. ModelFusion supplies an iterable of partially parsed result fragments; the application retains responsibility for identifying finished passages. A partially parsed object is not automatically a completed, validated passage.

A TypeScript consumer can separate these responsibilities explicitly. Here, completedPrefix represents the application’s completion check: it returns only the stable, finished prefix of the story. submit hands each newly finished passage to the narration pipeline without resubmitting earlier passages.

typescript

type Passage = {
  speaker: string;
  text: string;
};

async function consumeStory(
  fragments: AsyncIterable<unknown>,
  completedPrefix: (fragment: unknown) => readonly Passage[],
  submit: (passage: Passage) => void,
): Promise<void> {
  let submittedCount = 0;

  for await (const fragment of fragments) {
    const finished = completedPrefix(fragment);

    while (submittedCount < finished.length) {
      submit(finished[submittedCount]!);
      submittedCount += 1;
    }
  }
}

The completion check is the critical boundary: it must establish that a passage will no longer change, not merely that its current fields can be read. Each finished part can then enter narration while later parts are still being generated.

3:454:05
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:45 · section reference included

Give each speaker a consistent voice

Before synthesizing a passage, Storyteller resolves its speaker to a voice. The existing assignment determines how much work is needed:

Speaker stateNext action
NarratorUse the predefined narrator voice
Previously encountered characterReuse the assigned voice
New characterFind and assign a voice

Reusing assignments lets recurring characters proceed immediately and keeps their voices consistent across passages. A newly encountered speaker requires an additional selection step.

For a new character, the selection procedure is:

  1. Generate a voice description with GPT-3.5 Turbo, requesting structured gender and voice-description fields.
  2. Retrieve similar voices from a collection whose voice descriptions were embedded beforehand, filtering candidates by gender.
  3. Select a voice while ensuring that no two speakers receive the same voice.

The description provides a semantic query into the voice collection; the gender filter narrows the candidates; the uniqueness constraint preserves audible differences between characters. Once selection is complete, the passage can proceed to audio generation.

A flowchart routes a speaker with a voice ID directly to audio generation. The other path generates a voice description, retrieves similar voice IDs, and selects a voice ID before joining audio generation.Open full source frame
Voice selection feeds audio generation for each story part.
5:035:16
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:03 · section reference included

Start listening while generation continues

Storyteller supports LMNT and ElevenLabs for speech synthesis. The selected voice determines which provider synthesizes a passage. As with images, the server stores the resulting audio as a virtual file and sends its path to the client. The client reconstructs the URL and retrieves the file through a media element.

Playback begins when the first audio part is ready. The listener can hear that part while the server continues generating and narrating subsequent passages. Streaming thus changes the dependency that matters to the user: listening requires the first playable passage, rather than completion of the entire structured story and all of its audio.

5:546:07
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:54 · section reference included

Responsiveness across the whole pipeline

Storyteller’s loading state has multiple parts that update as results arrive. Backend streaming and parallel processing make those results available sooner, and passage-level audio lets listening overlap with the remaining work. Model choice also contributes: fast generation is a selection criterion for stages such as the outline. The resulting experience depends on the whole pipeline—visible progress, concurrent branches, completed-passage detection, and early playback—rather than the speed of a single model call. Grammel closes by pointing readers to the Storyteller and ModelFusion repositories for the application and its orchestration library.

6:366:48
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:36 · section reference included

Resources

From the talk

Updates since the talk

  • May 2024 announcement explaining ModelFusion's contribution to AI SDK Core and its text, structured-object and tool-call primitives.

Transcript download
Full transcript

Loading transcript…