AI Engineer World's Fair 2024
From Text to Vision to Voice: Exploring Multimodality with OpenAI
Loading slides…
Read the talk
From Text to Vision to Voice: Exploring Multimodality with OpenAI
Romain Huet traces the move from text generation to interruptible voice, shared visual context, and a documentary pipeline that combines video, vision, and a custom voice.
From a talk by Romain Huet
Before you start: Basic familiarity with model APIs, React components, and CSS breakpoints will help with the implementation example.
A world generated one text response at a time
What can you build when a model’s interface is text in, text out? For the early OpenAI developer platform, that question led from coding assistance and translation to entire fictional worlds. Romain Huet, introducing himself as OpenAI’s developer experience lead and a former founder, connects that experimentation to iterative deployment: putting technology into contact with real use early and often, with developers helping discover what it can become. Huet reports three million developers building on the platform at the time of the talk.
The developer API launched in 2020, before ChatGPT, with GPT-3. Early applications included basic coding assistance, copy editing, and simple translation. AI Dungeon made the open-ended nature of generation especially tangible: a player explored a role-playing world, looked around, and received newly generated descriptions rather than choosing only among prewritten scenes. Huet recalls it as one of the platform’s most popular early uses.
In 2023, GPT-4 expanded what applications could ask of a model: more complex reasoning, more specific and creative responses, stronger coding, tool use, and data interpretation. The application moved beyond generating the next passage of text toward reasoning over a user’s situation. Huet’s consumer example is Spotify, which he describes as using the models to generate playlists from music taste and listening history.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Removing the handoffs between modalities
GPT-4 also began the platform’s move into vision: applications could ask about images and photographs, not just text. Huet describes GPT-4 Turbo as bringing text and vision together in the same model, followed by GPT-4o, whose “o” stands for omni. Its intended interaction spans text, images, audio, and video in real time. The desktop demonstrations that follow show a preview of those capabilities; the May 2024 API launch exposed text and vision, with audio and video access planned initially for trusted partners.
The architectural distinction is clearest in voice. Previously, a developer assembled Whisper for transcription, GPT-4 for reasoning, and a text-to-speech model for the reply. Each handoff added latency, and converting speech into text lost some of the original context. A single model handling audio directly reduces the need for those intermediate translations.
| Voice architecture | Processing path | Interaction consequence |
|---|---|---|
| Separate models | Speech → transcription → reasoning → speech | Added handoffs and loss of audio context |
| Unified model | Audio input → multimodal model → audio output | Direct handling of spoken input and delivery |
The change concerns the conversation itself, not merely the number of services in a diagram. Tone, timing, and spoken content can participate in the same interaction.
Huet presents GPT-4o as retaining GPT-4-level reasoning while improving efficiency. OpenAI’s launch claims put GPT-4o at twice the speed, half the price, and five times the rate limits of GPT-4 Turbo. Those are launch comparisons, without a reproducible speed workload specified, rather than measurements from the stage demonstration. Huet adds that rate limits are continuing to rise; eliminating them altogether is an aspiration.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A conversation you can interrupt
On the Mac, Huet opens ChatGPT with Option-Space and activates voice mode. He introduces the event and asks the assistant to greet the audience. Before the greeting finishes, he interrupts: the response needs more energy, and it should include people watching the livestream. The assistant then delivers a more animated welcome addressing both groups. The correction happens inside the spoken exchange, without waiting for a completed answer and starting over.
Next, he asks it to whisper a valuable secret for people building AI-native products. Its advice concerns responsible and ethical AI. He then requests another secret, quieter and much slower; this time the advice concerns keeping products adaptable and scalable. The useful demonstration is the control over vocal delivery: the request changes how the answer is spoken as well as what it says.
Huet describes the latency as conversational and occasionally almost too fast. That is his qualitative impression, not a timed result. Alongside expressive speech, the key interaction property is interruption: he can resume speaking before the model’s audio finishes. A voice interface that supports this lets the user redirect an answer at the moment it becomes unhelpful.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From a handwritten greeting to a page in a book
Huet enables the camera in the existing conversation. The whispering style persists, so he explicitly tells the assistant to speak normally again. After a brief exchange about how he looks onstage, he draws something and holds it up. Asked to infer his location and translate the writing, the assistant identifies the Golden Gate Bridge, concludes San Francisco, and translates “Bonjour, développeurs” as “Hello, developers.” The same exchange now joins spoken instructions, a rough drawing, and handwritten text.
The next object is a physical book. The assistant identifies Poor Charlie’s Almanack by Charlie Munger and describes its collection of speeches, philosophy, and ideas about investing and decision-making. Huet asks for a random number between 1 and 400; the assistant chooses 126, and he opens that page.
Asked for an overview, the assistant summarizes the page as Coco Chanel’s rule for success: identify the qualities customers want and supply them. This is the assistant’s onstage reading of the page. Huet remarks that it responded faster than he could read a line. The progression matters: recognizing a cover establishes the object, while interpreting an open page brings the conversation down to the particular content the user is holding.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Diagnosing a layout through screen sharing
Huet replaces the camera feed with screen sharing while continuing to speak. He shows a travel application and switches to a mobile view whose responsiveness looks wrong. The assistant notices a cramped layout and squished images, then suggests media queries. Visual context establishes the symptom before either participant has identified the code responsible for it.
He first copies over code and requests a one-line explanation. The assistant identifies useAssistant, a React hook handling messages, file uploads, and streamed server responses. That description helps eliminate it as the likely layout target. Huet moves to Discover, and the assistant agrees that this component appears to set the layout for the items and descriptions. The diagnosis moves from rendered output to code responsibility rather than treating every nearby component as a candidate for editing.
Once Huet identifies Tailwind CSS, the assistant suggests adjusting the grid columns. Asked for the prefix for mobile screens, it answers sm:. Here the distinction in Tailwind CSS v3’s mobile-first rules matters: unprefixed utilities apply to the smallest screens, while sm: starts at 640px by default and continues upward. Huet proposes two columns on medium screens and retaining three on large screens. A minimal React implementation of that layout intent is:
jsx
export function Discover({ destinations }) {
return (
<div className="grid grid-cols-1 gap-4 md:grid-cols-2 lg:grid-cols-3">
{destinations.map((destination) => (
<article key={destination.id}>
<img
className="h-auto w-full"
src={destination.imageUrl}
alt={destination.imageAlt}
/>
<h2>{destination.name}</h2>
<p>{destination.description}</p>
</article>
))}
</div>
);
}
Here, grid-cols-1 supplies the base layout, and the larger breakpoints override the column count. This expresses the proposed behavior; the recording does not establish the complete final class list.
After the edit, Huet returns to the rendered application, and the assistant reports that the images and layout look better. The loop is now complete: observe the symptom, locate the relevant component, make the edit, and inspect the result. Huet calls this a trivial coding example, then points to a broader workflow he uses: reason aloud with ChatGPT, and ask it how to prompt Cursor to carry out the implementation. In that arrangement, conversation helps formulate the task before a second assistant performs the coding work.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Reasoning, affordability, customization, and agents
The roadmap begins with textual intelligence. Multimodal interfaces do not remove the need for better reasoning underneath them. Huet compares current models to first graders who can help but still make mistakes, then imagines future models reaching the level of master’s students across disciplines such as medical research and scientific reasoning. His suggestion that models could become unrecognizable within a year is a forecast of further progress, not a capability demonstrated here.
The second priority is affordability, with two different ways to reduce the cost of useful work:
- Choose an appropriate model. Not every task needs the highest available intelligence. Huet reports that GPT-4 pricing fell 80% over a year. He does not specify the baseline model or input/output mix for that comparison. He also describes plans for models of different sizes, without announcing a release timeline.
- Move eligible work out of the interactive path. Huet says the Batch API, launched a couple of months before the talk, supports asynchronous workloads such as analyzing documents, photos, and images. Huet describes a further 50% pricing discount for asynchronous batch processing. The practical tradeoff is that these jobs do not need an immediate conversational response.
Third comes model customization. Huet predicts that organizations will want models adapted to their own work, with options ranging from the publicly available fine-tuning API to assisted model development. Harvey, which builds software for law firms, is his example of the latter: it worked with OpenAI to customize GPT-4 around US case law. He praises the results without supplying a quantitative measure.
Fourth, the platform should enable agents that perceive and act. Building on the vision discussed at the previous November’s DevDay, Huet describes agents coordinating multiple AI systems, securely accessing a user’s data, and managing a calendar. Multimodality supplies ways to perceive the world in which those actions occur. Devin, from Cognition Labs, is the software-development example: breaking down complex tasks, browsing documentation, and submitting pull requests. Huet then attributes to Paul Graham the observation that younger programmers can match or exceed older programmers, connecting that possibility to access to AI tools. It is an attributed observation about changing developer capabilities, not a measured age comparison presented in the talk.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Six video frames become a documentary script
The final demonstration moves from using ChatGPT to composing a multimodal application. Huet introduces Sora as a preview diffusion model that generates video from text. Before the live session, he has submitted a detailed prompt for a tree-frog documentary. He plays the resulting clip, then turns to the work needed to give it narration.
The video-to-script operation has three concrete steps:
- Extract six frames from the generated video.
- Send those images to GPT-4o with vision, together with a prompt asking it to narrate what it sees.
- Generate the script by selecting
Analyze and Narrate.
This demonstration analyzes sampled images rather than making a direct video-input call. The frames provide visual evidence, while the prompt sets the desired narrative role. Huet generates the script live and says each run yields a new story, which he discovers along with the audience. The Sora clip is already prepared; the narration text is produced during the demonstration.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Giving the documentary a voice
The remaining ingredient is speech. Voice Engine creates a custom voice from a short recording. At the time of the talk, it remains a restricted preview because of safety considerations. Huet says OpenAI also uses it internally with actors to create voices heard in the API and ChatGPT.
Huet records a brief sample of himself speaking about being onstage at the event, then plays it back. He sends that recording together with the newly generated documentary script to Voice Engine. The recording supplies the voice reference; the script supplies what the synthesized voice should say. The demonstration does not state an exact required duration for his sample.
The resulting English narration describes a vibrant green frog moving along a moss-covered branch, with black and yellow markings against the surrounding foliage. Huet then demonstrates a French version and says it sounds like him speaking French, followed by a Japanese version. The pipeline has carried visual content into a script and then into multilingual speech using his voice reference.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Build more with the combined capabilities
The documentary joins Sora, GPT-4o vision, and the still-unreleased Voice Engine in one application. Huet returns to the four platform priorities behind that example: stronger textual intelligence, faster and more affordable models, customization, and multimodal agents. Their shared purpose is to make more applications feasible. As he puts it, the goal is for developers to “build more with OpenAI,” rather than simply spend more.
He closes by treating this as an early stage of a fundamental change in how software is built. Supporting developers and startups includes hearing what the platform still needs to do better, and he invites that feedback after the talk. The invitation to reinvent software 2.0 follows directly from the demonstrations: a conversation can be interrupted, an assistant can share the user’s visual context, and an application can pass useful content between modalities to produce something none of those steps delivers alone.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
The original announcement explains GPT-4o's unified modalities, launch capabilities, and API efficiency claims.
The 2024 preview describes custom voices from short recordings, multilingual applications, and consent safeguards.
A case study of Harvey and OpenAI collaborating on domain-specific model training for legal work.
Cognition's initial presentation of an agent that plans software tasks and uses developer tools.
The original announcement of OpenAI's general-purpose text API and request-access rollout.
Further reading
Examples explain mobile-first utilities, breakpoint prefixes, and responsive layout changes.
Explains Sora's diffusion-transformer architecture and representation of video as spacetime patches.
Updates since the talk
Current instructions for submitting asynchronous batches, retrieving results, and handling expiration.
Full transcript
Loading transcript…