AI Engineer Summit 2023

See, Hear, Speak, Draw

Logan Kilpatrick· Developer Relations, OpenAISimón Fishman· Applied AI Engineer, OpenAI18:43

Loading slides…

Read the talk

See, Hear, Speak, Draw: Connecting Multimodal Models Through Text

An image reconstruction loop and a video-to-article pipeline show how text can connect visual understanding, speech transcription, and image generation.

From a talk by Logan Kilpatrick and Simón Fishman

Before you start: Basic familiarity with language-model prompts and passing one model’s output into another is helpful.

What can you build beyond a chatbot?

How do you turn a research model into an application that solves a real problem? At OpenAI, Simón Fishman and Logan Kilpatrick work at that boundary: Fishman as an Applied engineer working with model users and unsolved problems, Kilpatrick in developer relations helping developers build products with the API. Their starting point is a company that both develops models and finds ways to put them to work.

Chatbots have already shown how much value a simple interface can deliver. Speaking in 2023, Kilpatrick calls it the year of chatbots and predicts that 2024 will be the year of multimodal models. The immediate examples are concrete: users are beginning to try image understanding in ChatGPT’s image-capability rollout, and image generation is advancing from DALL·E 2 to DALL·E 3. The question becomes what developers can build when their applications can work with more than typed conversation.

Black slide with a globe surrounded by books, cameras and screens, labeled “2024” and “MULTI-MODAL.”Open full source frame
“Where are we going?” introduces 2024 as the year of multimodal.
0:000:17
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:00 · section reference included

Text connects the model islands

The architecture starts with separate models, each handling a different conversion.

ModelInputOutput used by the application
DALL·ETextGenerated image
WhisperAudioTranscript
GPT-4 with Vision, also called GPT-VImages and textText describing or reasoning about the images

Text is the shared interface. A transcript can become material for another model to reason over; an image description can become a generation prompt. The models need not share a native representation for the application to connect them.

Fishman distinguishes this composition from the future he wants: a model that can consume and produce more modalities while reasoning across them together. In the architecture demonstrated here, developers still do the wiring. Kilpatrick says that connecting modalities accounted for much of the work of building the demos; a unified model accepting text, video, or speech could remove some of that integration work. The two examples that follow explore what developers can assemble with separate models while thinking ahead to products they might build over the next six months or year. That horizon is an invitation to plan, not a release commitment.

2:292:43
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:29 · section reference included

Describe, generate, compare, revise

The first demonstration is a DALL·E Vision loop. Start with a real photograph, ask GPT-4 with Vision for a readable description, then send that description to DALL·E 3 to generate a synthetic version. The pipeline takes enough time that the speakers bring prepared examples and also attempt a live run.

Fishman’s first photograph comes from the hotel lobby: a display of Halloween-themed painted ladies. Vision describes the scene and produces a prompt for DALL·E, picking up details including RIP on a tombstone and a sign welcoming all dogs. The generated image captures parts of the scene but introduces conspicuous differences: black marble and white spiders.

The application then closes the loop:

  1. Give GPT-4 with Vision both the original photograph and the generated image.
  2. Ask it to identify differences between them.
  3. Use those differences to produce revised generation instructions.
  4. Generate another image with DALL·E.

The next image removes the black marble and makes the spider black, although it also makes the spider larger. The result is still imperfect, but it demonstrates an operation beyond describing an isolated picture: compare a current result with a target, then turn the discrepancy into instructions.

5:115:22
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:11 · section reference included

Make visual evaluation part of the application

This loop resembles work a human would otherwise perform: inspect a visual output, compare it with a reference, revise the instructions, and run the model again. Fishman acknowledges that there is a long way to go, but the demonstration moves some of that evaluation into the model pipeline. Two operations matter:

  • Description: Convert an image into text that subsequent reasoning steps can consume.
  • Comparison: Examine the difference between a current state and a desired state, then use that difference to guide the next action.

The second operation gives the application a target to work toward rather than merely another caption to produce.

An imagined interior-design application extends the comparison pattern. Someone moving into a new room finds an Instagram image with the atmosphere they want. They could provide that reference and ask an application to search Amazon for lamps that match it. Here the desired result is not an exact reconstruction: it is a set of objects compatible with the reference’s visual style. Fishman and Kilpatrick discuss this as a product idea, not as a shopping crawler they have built.

The roughness of the lobby result also reflects a deliberately minimal implementation. Kilpatrick describes a one-hour demo with no prompt engineering; Fishman estimates roughly 50 lines of code for the image loop. Almost all completion output goes straight into the next model. They expect additional techniques to improve the results, but do not measure that improvement here.

7:387:47
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

7:38 · section reference included

Remove invented lights and restore a curtain

A second prepared example uses a photograph Fishman took backstage immediately before the talk. Vision describes monitors, boxes, and cables; DALL·E’s reconstruction includes blue carpet, cables, and boxes. But the generated room also contains vertical lights that were not in the photograph.

The comparison identifies those extra lights on the walls and ceiling. After the prompt is rewritten, the next generation removes them and adds the black curtain visible at the rear of the original scene. These are specific, observable corrections, even though the whole room is not faithfully reconstructed. The useful capability is that a visual discrepancy can become an instruction without a person first spelling out what is wrong.

Browser demo displaying a generated equipment room with cases on patterned carpet, cables, overhead lights and a dark curtain at the back.Open full source frame
The revised backstage image shows equipment cases, coiled cables and a dark rear curtain.
10:0610:17
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:06 · section reference included

A live woodland run exposes the limits

For the live attempt, Fishman selects a trail-running photograph of a woodland setting. Vision describes a wooden boardwalk or footbridge winding through dense forest, with light filtering through the trees. He passes that description directly to DALL·E.

While it runs, Kilpatrick explains another useful role for language models: writing image prompts. ChatGPT’s DALL·E mode does some prompt engineering for the user, although he does not know its exact prompt. In the earlier Labs workflow, users had to do that work themselves. He points to Midjourney and other image models as further examples of systems for which writing effective prompts can be difficult. A language model can mediate between what a person wants and the instructions an image generator needs.

The woodland comparison describes the generated image as more beautiful and detailed. That is a different judgment from whether it matches the photograph: an attractive reconstruction can still be an inaccurate one. The live attempt then runs into trouble, and the speakers return to the slides while leaving it running. Kilpatrick reports three successful runs before the onstage attempt, but the demonstration has not yet reached a completed result.

11:2211:34
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:22 · section reference included

A video contains more than its transcript

The second demonstration addresses a limitation of transcript-based video summaries. Sending only the spoken words to GPT-4 leaves out information conveyed visually. Instead, the application extracts frames and asks GPT-4 with Vision to describe them, while Whisper transcribes the audio. The two streams become a long textual representation of what the video says and shows.

The worked input is OpenAI’s GPT-4 introduction video from YouTube. Its relatively simple frames are easy to describe in text. Those descriptions join the transcript as source material for an article. Kilpatrick informally calls the visual material “image embeddings,” but the operation he describes is textual frame description, not an explicit vector-embedding step.

The assembly step can remain simple. In Python, keep the transcript and frame descriptions explicitly separated so the downstream writing prompt can refer to both:

python

def article_prompt(transcript: str, frame_descriptions: list[str]) -> str:
    frames = "\n\n".join(
        f"Frame {index}:\n{description}"
        for index, description in enumerate(frame_descriptions, start=1)
    )
    return (
        "Write an article using both the spoken content and visual details.\n"
        "Indicate which supplied frames belong beside relevant passages.\n\n"
        f"TRANSCRIPT\n{transcript}\n\n"
        f"FRAME DESCRIPTIONS\n{frames}"
    )

This is the text handoff: descriptions let the writing model use visual information, while references to the supplied frames let the application retain the actual pictures.

The resulting article, Don’t forget about GPT-4, begins with a DALL·E image and includes actual video frames matched to the surrounding discussion. The output therefore offers two kinds of access to the source: prose that incorporates visual details, and images that preserve the source’s appearance.

Browser displaying “Don’t forget about GPT-4” above an illustration of colorful balloons and a glowing circular doorway, with article text below.Open full source frame
The resulting article, “Don’t forget about GPT-4,” shown with an illustrated opening image.

Kilpatrick says the article was generated with a raw prompt, without hidden manual work, and that he could not open-source the code because it used unreleased APIs. This is a 2023 demonstration, not a current API recipe: ChatGPT’s image rollout was already underway, while public GPT-4 vision and DALL·E 3 API access followed in the November 2023 DevDay announcement. The product goal is straightforward—make video content accessible in text without throwing away everything the viewer could see.

13:4614:06
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:46 · section reference included

Products that can work with what they see

Before closing, Fishman checks the woodland run again. It still has not finished. The prepared image examples and the illustrated article remain the completed demonstrations; the live run does not acquire a successful ending.

Fishman’s closing advice is to start thinking about products that incorporate multiple modalities, including ideas that still feel out of reach. His six-month-to-one-year outlook is optimistic, while his immediate architectural guidance remains practical: text will continue to connect models, and image-based interaction patterns offer substantial room for exploration.

Kilpatrick ends with image-input agents. Much of the internet requires understanding what is visible, so an agent that can interpret images can engage with information unavailable through text alone. The demos provide an early vocabulary for that work: describe what is present, compare it with what is wanted, and use the resulting text to decide what comes next.

16:3917:11
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

16:39 · section reference included

Resources

From the talk

  • Logan Kilpatrick’s generated Summit demo article, combining a DALL·E 3 cover with frames from its source video.

  • OpenAI’s speech-recognition code, with installation and transcription examples.

  • An overview of DALL·E 3 image generation and ChatGPT-assisted prompt refinement.

  • OpenAI’s introduction to its evaluation and mitigation work for GPT-4 image inputs, with a link to the system card.

  • The September 2023 announcement of ChatGPT image and voice capabilities and their staged rollout.

Updates since the talk

Transcript download
Full transcript

Loading transcript…