AI Engineer World's Fair 2024
The era of unbounded products: Designing for Multimodal I/O
Loading slides…
Read the talk
The era of unbounded products: Designing for Multimodal I/O
As AI expands what users can ask, show, and change, interface structure becomes essential: it reveals capabilities, keeps work stable, and makes personalization understandable.
From a talk by Ben Hylak
How do users know what a product can do?
How does someone discover the capabilities of a product that extends beyond a mouse and monitor? Ben Hylak calls these unbounded products. His experience with them began with robotics around eighth grade, continued with rockets at SpaceX, and included four years on the Apple Vision Pro design team developing the first visionOS. He introduces Dawn as a company helping teams including GitHub and Can of Soup build more predictable AI products. The recurring design problem is making a large space of possibilities understandable.
AI expands that space further. Users can type, talk, supply images, or show video. They can also plead, bargain, and confide—social behaviors that carry expectations beyond the format of the input. When the interface does not explain its boundaries, users assume it can do things it cannot. A failed attempt can then convince them that it cannot even do the things it supports. An unclear capability boundary can hide real capability.
Consider how people learn to use ChatGPT. Someone hears that a friend used it for travel planning and tries the same task. Technical users often keep experimenting after a failure because they already believe the underlying models are capable. Other users try once, find that it does not work for them, and never return. A product cannot depend on every user supplying that initial confidence. Understanding how to earn it requires looking first at earlier interfaces, then at contemporary AI products, and finally at more personalized forms of interaction.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From visible buttons to open-ended situations
Traditional screen software makes a relatively clear agreement with its users. They swipe, click, or tap; the developer specifies what happens. Visible buttons expose available actions. Those same actions are straightforward to instrument: attach an Amplitude or Mixpanel event to a button press, and the product team can see what someone did. The interface supplies both a map of capabilities and a vocabulary for observing usage.
Even a small expansion in input can change the design space dramatically. Multi-touch adds a second pointer, but that also introduces relative distance and rotation. Hylak credits those extra dimensions with helping make small smartphone screens practical. The expansion becomes more consequential when software operates in the physical world: his example of software roaming San Francisco streets is accompanied by a photograph of a burning car. The environment can now create situations far outside a developer's intended interaction sequence.
For Vision Pro, the questions begin with ordinary movement. Someone opens apps in the living room, walks into the bedroom, and lies down. What should happen to the apps? A laptop operating system can largely leave that question to the position of the laptop; a spatial interface must decide how software relates to the user and the room. Plane travel, a nearby friend, an inability to move one's neck, or being confined to bed introduce further requirements. These are the spatial equivalent of unexpected AI inputs and the evaluation cases they generate.
A blank slate does not resolve those possibilities. It leaves users and designers facing them all at once. Structure creates clarity by giving an otherwise open-ended product recognizable places, actions, and expectations.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Give the unfamiliar a recognizable shape
The first structural decision in visionOS is what to show immediately. Its home screen foregrounds apps, people, and environments: the categories the team wants users to recognize as central. This is familiar from the iPhone, but Hylak contrasts it with earlier VR menus whose contents made it harder to answer a basic question: what is this device good for? The first view should make the product's priorities legible quickly.
The next decision is hierarchy. In the visionOS hierarchy Hylak describes, the home menu is the point of departure and return. Apps occupy bounded windows that users can move and resize. An individual window can then go full screen. Each level gives the user a different scope of interaction without requiring them to invent their own organization of an unlimited space.
Familiarity makes that hierarchy easier to learn. Dawn's early prototype used an exploratory star cluster: enjoyable to navigate, but eventually replaced with tables, graphs, and examples. The same reasoning explains why the visionOS TV app resembles the tvOS TV app. Reusing a recognizable design gives people an orientation aid when the surrounding experience is unfamiliar; it is not evidence of laziness. Control Center serves a similar purpose. The three decisions reinforce one another: foreground what matters, establish a hierarchy, and use familiar forms to make that hierarchy understandable.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A journal and a search engine should not feel the same
These principles do not imply that every AI product needs the same layout. The appropriate structure depends on what the application is for. In the interfaces Hylak shows, Dot is a companion that he also uses as a journal. Pinching out separates the conversation into days. Tapping a person—or, in his example, his two co-founders—reveals structured information and a timeline of mentions. The conversation remains the source material, but days and people become ways to navigate it.
Perplexity uses a different arrangement to communicate a different purpose. It promotes the query to a page title, highlights the sources, and places the answer underneath. The answer occupies a full page, making the interaction feel more like a search result than one turn in an ongoing conversation. In the displayed example, the query concerns coffee shops to work from in Paris; the final slide outlines the query, source cards, and answer as distinct regions. Structure tells the user how to read the result before they read its contents.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keep the work from drifting away in chat
A control can be useful in isolation and still be confusing in its surrounding interface. Hylak illustrates this with a Vercel chatbot demo, while emphasizing his admiration for Vercel's design work more broadly. Asking to buy Doge produces a purchase interface with a quantity slider. That is a sensible improvement over repeatedly typing quantities into chat.
The difficulty appears as the conversation continues. Follow-up messages push the control away, and eventually there are two purchase interfaces in the transcript. Returning to the first one and pressing purchase means interacting with a different conversational state. Hylak compares the drifting interface to the floating house in Up: the user has a useful object, but its location and relationship to the current task are no longer stable.
Separate the work object from the conversation that changes it. A stable workspace can update as messages arrive without displacing the user's point of interaction. Claude Artifacts applies this pattern to an app being developed: the app occupies a separate area alongside chat, and the user can switch between versions without scrolling through messages. The conversation records the process; the artifact provides a persistent place to inspect the result.
A small TypeScript state model illustrates that separation. The artifact keeps its identity while a revision is appended and selected; viewing an older revision does not require modifying or searching the conversation.
typescript
type Revision = {
id: string;
source: string;
};
type Artifact = {
id: string;
revisions: Revision[];
selectedRevisionId: string;
};
function addRevision(
artifact: Artifact,
revision: Revision
): Artifact {
if (artifact.revisions.some(item => item.id === revision.id)) {
throw new Error("Revision already exists");
}
return {
...artifact,
revisions: [...artifact.revisions, revision],
selectedRevisionId: revision.id,
};
}
function selectRevision(
artifact: Artifact,
revisionId: string
): Artifact {
if (!artifact.revisions.some(item => item.id === revisionId)) {
throw new Error("Unknown revision");
}
return { ...artifact, selectedRevisionId: revisionId };
}
const first: Artifact = {
id: "greeting-app",
revisions: [{ id: "v1", source: "<h1>Hello</h1>" }],
selectedRevisionId: "v1",
};
const updated = addRevision(first, {
id: "v2",
source: "<h1>Hello, friend</h1>",
});
const viewingOriginal = selectRevision(updated, "v1");
Here, viewingOriginal still contains both revisions. The selected view changes, but the work object and its accumulated history remain available.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make changes recoverable and context understandable
Versioning addresses a related obstacle: the fear that another attempt will destroy something useful. Hylak points to message editing and version switching as features present in the original ChatGPT experience. Underneath that interaction is a branching conversation tree. Editing an earlier message creates an alternative path rather than simply replacing everything that followed. In v0, he sees the same benefit expressed through a familiar interface that resembles working in Google Slides: users can keep iterating on a UI while retaining access to earlier versions.
Familiar boundaries matter for memory as well. Hylak finds it uncomfortable that information about a medical problem could remain available while he works on JavaScript in a different chat. A project supplies a more understandable context boundary: material relevant to one body of work belongs together. Claude Projects makes that boundary concrete through supplied project knowledge and instructions; it should not be understood as automatic recall of every conversation within a project. The design advantage is that users can reason about why a piece of context is present.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make agent steps visible as columns
Agents introduce unfamiliar ideas: multiple tasks, intermediate results, and data passed from one step to another. A spreadsheet turns those ideas into a recognizable arrangement. Hylak says that the real-world agent uses he has personally encountered have been in spreadsheets, and uses Clay to explain why the format works. Each column represents a user-defined step. Moving across a row builds up the context required by later steps.
| Spreadsheet element | Workflow meaning |
|---|---|
| Column | A user-defined agent step |
| Row | One instance of the workflow |
| Successive columns | Accumulated context and intermediate results |
| Final column in the example | A personalized email |
This layout also gives users a natural way to expand execution gradually:
- Run a small initial set of rows and inspect the results.
- Adjust the steps until the workflow behaves as intended.
- Apply the workflow to a much larger set.
Hylak illustrates the progression with ten initial rows followed by fifty thousand or a hundred thousand. Those are example batch sizes for the procedure, not measured throughput. The useful mechanism is the ability to inspect intermediate work before applying it broadly.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Offer a useful starting point
A blank prompt field asks users to discover both what the product can do and how to ask for it. Examples and presets reduce that burden in several distinct ways:
- Suggested tasks: ChatGPT offers starting points such as comforting a friend or planning a relaxing day. These make possible uses visible.
- Working examples: v0 adds an Explore page to its suggestions, letting users see what other people are making successfully.
- Named transformations: Notion offers a tone menu instead of requiring users to compose an instruction about being a concise GPT. The product team can validate these predefined operations.
Each pattern moves some of the work of discovering a useful prompt from the user into the product.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Direct controls need more than broad labels
The next step is to make generation feel less like indirect manipulation through carefully chosen words. Hylak recalls Linus's earlier metaphor of a llama trying to drive a car from the back seat with a pool noodle. It captures the distance between an intended change and the prompt used to induce it. Hylak expects less prompt engineering as interfaces expose more direct controls. Apple's image-generation interface, which combines concepts, and demos that manipulate emotional expressions suggest that direction.
The Figma Slides launch provides another example. Hylak describes text controls spanning professional, casual, expanded, and concise. His reference to their release the previous day belongs to the June 2024 launch context. The appeal is immediate: users adjust the property they care about instead of writing instructions for how to change it.
But a label such as casual conceals important distinctions. A Fortune 500 company and a direct-to-consumer cosmetics brand advertising on TikTok may both want casual copy, yet expect very different language. The same person writes differently to a best friend and a coworker. A small menu can simplify interaction by collapsing distinctions that actually matter. The design challenge is to make control easier without making its vocabulary reductive.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From coarse presets to feature steering
Hylak's proposed response is to increase the number of available presets enormously. His millionfold or billionfold language expresses the ambition: enough specificity to cover distinctions that a few generic labels cannot. He sees sparse autoencoders as a promising route toward identifying more granular features that an interface could expose.
Golden Gate Claude demonstrates one form of feature steering. Anthropic increased an internal Golden Gate Bridge feature in Claude 3 Sonnet, causing the model to bring the bridge into its responses persistently. The intervention changed internal activations rather than supplying a prompt or performing traditional fine-tuning. It showed that an identified feature could influence behavior, but Anthropic also warned of unexpected effects. The temporary public experiment is no longer available; it does not establish reliable control over arbitrary user preferences.
The following image demonstration makes the interface possibility more tangible. Hylak shows controls for the play of light and shadows, serene forest streams, and Venetian canals, describing the changes as controllable and predictable. The selected frame, labeled @gytdau, shows a canal scene beside reference thumbnails and the feature controls. Here the user-facing vocabulary concerns properties of the image, rather than an elaborate instruction for generating it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Retrieve the right controls, then personalize them
A much larger preset space creates another discovery problem: how does anyone find the relevant options? Hylak proposes ranked presets that are personalized, searchable, and invocable through natural language. They need not all appear as visible controls. A request such as more friendly could retrieve dimensions involving kindness, interpersonal closeness, and confrontation. The user might adjust those dimensions directly, or the application might use them behind the interface. Natural language becomes a way to locate structured controls rather than the only mechanism for influencing output.
Once developers can define those features, Hylak proposes tuning them separately for each user. This is developer-defined personalization: the product supplies the dimensions, while their settings vary with the person. He contrasts that possibility with the fragility of text prompts, where removing a word can change the output. The proposed advantage is a more explicit set of personalization controls; the talk presents this as a future direction, not a demonstrated personalization architecture.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Measure whether the result fits the user
As an application behaves differently for different people, its measurement problem changes too. Hylak describes a shift from evals toward analytics. Some questions still have objectively correct answers: identifying the first president is his example. But the right tone for a particular user's summary is not determined in the same way. In these domains, a shared answer key cannot fully describe success.
The measurement target becomes whether the product understands what users ask for and meets their needs. That preserves a place for factual correctness while recognizing that personalized output also has to fit an individual. An increasingly unbounded product therefore needs more than the ability to generate many possible results: it needs a way to understand which results are useful to the person using it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Anthropic explains its experiment amplifying a Golden Gate Bridge feature in Claude 3 Sonnet. The temporary public demo is no longer available.
The original announcement explains project knowledge, custom instructions, and a separate Artifacts workspace alongside chat.
Figma’s launch article introduces Slides and its AI tools for changing text length, tone, and language.
Full transcript
Loading transcript…