AI Engineer Summit 2023
The Intelligent Interface
Loading slides…
Read the talk
The Intelligent Interface
A camera-aware laptop, an expressive headset, and an imagined image canvas explore how computers can choose their responses from the context around a request.
From a talk by Samantha Whitmore and Jason Yuan
Sensing, thinking, and reacting
What should our relationship with a computer look like when its interface can interpret context? For Samantha Whitmore and Jason Yuan, co-founders of New Computer, that question starts with reconsidering the metaphors and abstractions that computing has long treated as fixed. Before choosing a better interface, they ask what intelligence does: take information from different sources, make sense of it, and choose a response.
Yuan describes this as sense → think → react. People perceive their surroundings, process what they perceive, and decide how to respond. In his framing, processing multiple sensory inputs together supports short-term memory and a theory of mind: an interpretation of what is happening, how others relate to us, and what we should do next.
Language is one way to communicate and think, but it is only part of this loop. Onstage, Yuan makes eye contact, gestures with his gloves, looks toward Whitmore, and points at objects. Audience laughter provides another signal. These channels operate together; the words alone do not describe the whole interaction. A computer that responds only to explicit language has access to a narrower picture.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Language opens the loop
ChatGPT made an approximation of this loop tangible: language enters, some reasoning process intervenes, and language returns. Whitmore leaves the internal nature of that reasoning unspecified. The interesting design opportunity is what comes next—how interaction could feel more natural when the computer can work with more than the words a person deliberately supplies.
The demonstrations move from a prototype running on an ordinary computer to imagined interactions with future hardware. Their range includes gaze, implicit and explicit gestures, sound, expression, language, touch, attention, and drawing. These are ingredients for interaction, not a requirement to use every channel at once. The demonstrations are exploratory stories, and the founders explicitly distinguish them from the product New Computer was building.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
When should a laptop listen?
The laptop prototype changes the question from whether text or audio is the right interface to when each is appropriate. Its camera supplies real-time pose measurements describing Whitmore’s position relative to the screen. An LLM uses those measurements to select how the interaction should proceed.
| Context detected | Modality selected |
|---|---|
| User is close to the screen | Keyboard input |
| User is facing the screen | Text output |
Proximity and orientation answer different questions. Being near the keyboard suggests how the user can provide input; looking at the screen suggests where a response can be presented. The prototype exposes these decisions in a side panel.
Whitmore then walks away. Once the camera no longer detects her, the intended behavior is to start listening so she can converse while moving around the room. She calls out, but the system does not answer promptly. This is the counterpart to its intended behavior at the keyboard: when she is typing nearby, it should avoid attending to her voice or ambient sounds.
The system eventually speaks, after an awkward pause and some laughter. The demonstration therefore shows a delayed transition into spoken interaction, not a seamless handoff. No measured latency or established cause accompanies the delay. Behind that imperfect execution is a concrete design: use proximity and screen orientation to decide whether to listen, speak, or display text.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Context becomes world state
The goal is for the computer to adapt to a person’s circumstances instead of requiring a fixed posture and interaction method. The prototype separates sensing from response generation through world state, an interpretation of the user’s current situation. Its architecture proceeds in three stages:
- Process the image stream with pose detection and the audio stream with voice activity detection.
- Use an LLM to update world state from those signals, including where the user is and whether they are speaking.
- Generate a response using that state, selecting both what to say and how to present it.
Whitmore describes this as a simple, rough prototype that could be built with the technology available at the time. The architectural point is that context can change even when the user is not explicitly addressing the assistant.
That gives an interface two complementary sources of information:
- Explicit inputs: what the user types or says.
- Implicit inputs: where the user is, with possible extensions such as vocal tone or detected emotion.
These signals can affect more than the output medium. Yuan offers a hypothetical social constraint: if the assistant remembers Whitmore complaining about him, and he is now visible beside her, it should wait until he leaves before mentioning it. Context would govern whether a response is appropriate at all, not merely whether it should be spoken.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Blending images as a physical action
The next imagined interface is a Midjourney canvas for iPad. It begins with a generated image of Naruto dressed in Balenciaga. This is a proposed canvas experience; the linked blend documentation does not establish the touch and shaking interactions shown here as a shipped interface. To edit the image, Yuan first zooms out, creating room around it using an already familiar gesture.
He then imagines his cat in the outfit. An assistant with personal context retrieves pictures of his cat, making those images available alongside the generated character. He picks up a cat photo and brings it together with the other image to express blending. The selected photos glow: light becomes the visual metaphor for combining sources, drawing on the familiar idea of colors mixing and concentrating into a new form.
The resulting human–cat hybrid is not what he wants. Rather than switching to a separate editing vocabulary, he asks what physical action already means remixing. A blender bottle provides the answer: shake it. In the concept, shaking remixes the image, carrying the material metaphor through both the initial combination and the next attempt.
Generative intelligence is a probabilistic material. Yuan compares designing with it to designing a phone with an understanding of aluminum or titanium. If generated results are fluid and variable, light, fog, or mercury may offer more suitable metaphors than rigid wood, paper, or metal. The proposal is to make the material’s behavior legible through familiar actions and feedback, rather than merely decorate a conventional interface.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Selecting, correcting, and remembering at the desk
The final scenario moves the interface onto a physical desk. An imagined wearable has a camera, microphone, and the ability to project feedback. Yuan browses a Beyoncé tour book and points to an inspiring image while verbally expressing interest. The gesture identifies the object; speech supplies the intention. Projected feedback acknowledges that the device has understood and will act. The shown sequence lacks sound, but the proposed interaction combines voice and gesture simultaneously.
That interpretation can be wrong. After accidentally selecting a photo, Yuan flicks it away like dust. The correction uses the same physical vocabulary as the selection instead of requiring a separate undo control. Probabilistic interpretation needs a graceful way out: the interface must make a mistaken inference easy to reject.
With two pieces selected, he wants to share them with two friends but cannot remember which friend he had agreed to spend Halloween with. He asks the assistant to recall that person. The imagined response appears as interface cards beside the book, keeping the selected images and the contextual question in the same workspace. The scenario illustrates recall and recipient discovery; it does not establish that a message has been sent.
The input and output channels do not have to match. Voice input need not produce speech; gesture input need not produce a gesture-based response; visual input need not return another visual interface. Multiple simultaneous inputs can establish intent, while the computer chooses whichever output suits the current setting. Here, a spoken question can lead to projected information beside the physical objects already receiving the user’s attention.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Choosing what an interface should do
This design space does not depend entirely on future hardware. Existing laptops already offer possible contextual signals, including typing speed and vocal tone. Those signals can be processed before being passed to an LLM reasoning layer, which decides both the wording of a response and its presentation. The interface decision becomes part of responding, rather than a fixed wrapper around generated text.
More possible outputs do not make every output useful. Yuan challenges the idea of generating a photo on an Apple Watch when a phone would be more suitable. Intentional device and modality choices still matter. Familiar metaphors—from nature and physics to tools, materials, and ordinary buttons—can give users stable expectations even when the system produces variable results.
Social expectations provide another boundary. Yuan’s example is a banking assistant that should not drift into a deep philosophical conversation simply because its model can sustain one. An agent’s role helps determine which interactions make sense, just as the user’s surroundings help determine the right output channel.
All these interfaces borrow from what people already perceive and do: looking toward someone, dismissing a call, mixing materials, brushing away dust. That makes them understandable, but it also leaves an open question. If these familiar behaviors are the skeuomorphism of intelligent interfaces, what abstraction comes above them? The next interface language remains to be designed.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
The original ChatGPT announcement explains its conversational format, training approach, and limitations.
Further reading
Apple’s 2023 introduction describes a spatial interface controlled through eyes, hands, and voice.
A practical guide to detecting body landmarks in browser images and video with MediaPipe.
Updates since the talk
Instructions for combining images with Midjourney’s Discord blend command and web image prompts.
The founders explain their decision to wind down New Computer and Dot.
Full transcript
Loading transcript…