AI Engineer Summit 2023
Climbing the Ladder of Abstraction
Loading slides…
Read the talk
Climbing the Ladder of Abstraction
From spreadsheet formulas to books and lodging searches, AI can help us move between useful views of the same information—and keep acting at every level.
From a talk by Amelia Wattenberger
What changed when accounting became a spreadsheet?
What did accounting look like in the early 1900s? Handwritten letters and numbers, annotations in margins, calculations performed by hand. Crossed-out entries and ink blots make the effort visible: maintaining the document was itself a substantial part of the work.
VisiCalc brought the spreadsheet to personal computers in 1979; Lotus 1-2-3 followed in 1983. The breakthrough was not automatic arithmetic. Calculators and computers already supplied that. It was a structured interface that connected calculations through formulas. Change a cell or add a row, and the dependent results update live.
That structure changed where an accountant could spend attention. Instead of repeatedly calculating values and maintaining rows and columns, they could concentrate on what the numbers meant. The interface made many small automations work together as a tool for a larger task.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Augmentation is built from smaller automations
The corresponding question for AI is how to combine new capabilities with interfaces people use today—and interfaces they might want tomorrow. Two goals often frame that discussion:
- Automation: Do a tedious task for the user, such as copying data into a table or calculating values.
- Augmentation: Give the user a new ability, or improve an existing one, especially in creative or nuanced work such as analyzing data.
Concerns about jobs being automated are valid, Amelia Wattenberger acknowledges. But treating automation and augmentation as opposing goals misses their relationship: augmentation can be composed of smaller automations. Aggregating data into a table and generating visualizations can both serve a larger activity that remains directed by a person: answering the question that motivated collecting the data.
The spreadsheet already works this way. Its cell calculations are automated, while Excel remains a tool people use to understand financial data. Automatically assembling the table or writing its formulas would extend that pattern, removing more of the preparation involved in analysis.
This is the reason to look beyond chatbots as the default AI interface. General tools such as calculators and chatbots are useful, but surrounding their capabilities with structure can make them much more powerful for particular tasks. The desired arrangement leaves the user directing the work while the model handles frustrating subtasks inside the interface.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Zoom changes which information matters
The ladder of abstraction describes how the same object can be represented at different levels of detail. Digital maps make this familiar. At a close view of the Monterey Bay Aquarium, buildings, names, icons, and routes help someone navigate within the aquarium. That representation is less useful for getting to the aquarium in the first place.
Zooming out does more than make buildings smaller. The map hides some information and foregrounds other information as the task changes.
| View | Information emphasized | Supported task |
|---|---|---|
| Neighborhood | City streets and restaurants | Find and reach a local destination |
| Region | Highways and terrain | Travel to and from Monterey |
| Broad geography | State and country shapes | Understand the wider geography |
Keeping every building label and street visible at every scale would make the broad views incomprehensible. Screen space and attention are limited, and most fine detail is irrelevant to the task at that level. Could AI bring this behavior to other kinds of information?
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Zooming out on Peter Pan
What would it mean to zoom out on a book? Reading usually exposes every word, but remembering a book or explaining it to a friend involves topics and plots rather than exact wording. Language models can summarize and transform text, making these different representations available inside a reading interface.
Wattenberger’s demo begins with the first five chapters of Peter Pan. Initially, she scrolls through the original first chapter. Zooming out replaces each paragraph with a one-sentence summary. A minimap on the right makes the reduction in text volume visible. At the next level, a summary stands in for a group of roughly ten paragraphs, reducing the text further.
At the highest level, each chapter becomes one sentence, so all five chapters fit on a single page. The unit of attention has moved from words, to paragraphs, to groups of paragraphs, to chapters.
That reading demonstration suggests a writing workflow: inspect the chapter-level representation, adjust pacing or plot structure there, and then zoom back in to examine how the full text changes. This editing behavior is a proposal, rather than a demonstrated round trip. Its attraction is concrete: the writer could work on the structure without holding every part of the book in memory while editing word by word.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From a text summary to a story arc
A book can also be represented by its emotional trajectory. A story arc maps mood across the narrative rather than summarizing its wording. Kurt Vonnegut’s familiar Man in a Hole shape follows a character into trouble, out again, and into better circumstances. Wattenberger offers The Hobbit, The Wizard of Oz, and Alice in Wonderland as examples of that pattern.
The proposed interface would assign sentiment values to sections of a book and plot them on a graph. A writer could then adjust part of that curve and inspect the resulting changes to the prose. Here the higher-level representation is no longer shorter text: it is a different editing surface for a property of the story. This is a direction for future writing tools, not a capability established by the reading demo.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Get information, reason about it, act
Combining smaller automations with movement between abstraction levels suggests a broader product direction. At the time of the talk, Wattenberger was on Adept’s design team, working in a company focused on training AI to read screens and use software through actions people perform. The goal was to make knowledge work—work done on a computer—easier.
Interviews about people’s daily work revealed a recurring sequence:
- Obtain information.
- Transform it or reason about it.
- Act on it.
Adept’s design exploration asks what it would mean to zoom out on information throughout that sequence. The lodging workflow that follows is a hypothetical interface sketch, including its booking and messaging actions.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A lodging page organized around your decision
Suppose you are attending a conference in San Francisco. You find Airbnb listings near the venue and open one. Its detail page contains information intended for everyone, but your decision depends on particular questions: How close is it to the venue? Is there a coffee maker? Is the Wi-Fi good? Answering them requires digging through the page.
Zooming out slightly would remove branding and generic detail that do not help with this decision. The new view would retain the listing name, rating, a short summary, and total price. Then it would add information specific to the trip: walking minutes to Hotel Nikko, where the conference is being held. Instead of repeatedly consulting a map, you could compare that value directly. If the walk is too long, the view could also identify the nearest BART station and the walk to it as an alternative.
Wi-Fi matters because the night before the conference may still involve working on the talk. AI could extract relevant reviews and summarize whether their evidence is positive or negative. That gives the traveler a quick judgment from reported experiences, rather than a measured connection speed. The interface could also select a useful bedroom, living-room, or kitchen photo from the larger gallery, reducing another search through presentation material.
The abstraction must preserve the ability to act. From this personalized view, the traveler would still be able to reserve the listing or message its host without returning to Airbnb. Removing detail should simplify the decision while leaving the person in control of what happens next.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From individual listings to a comparison table
The next question is whether a hotel would be a better deal. A shared representation could present hotel and Airbnb information in the same form, regardless of the source website. Putting those views side by side would make their relevant differences easier to compare.
But fifty individual views would still be cumbersome. Zooming out again would turn the hypothetical set of fifty listings into a spreadsheet, with the deciding factors laid out as columns. The traveler could scan the distribution of total prices, compare walking times, and inspect counts of positive Wi-Fi reviews. Booking would remain available at this level: a clear winner could be reserved directly from the table without returning to Airbnb or Hotels.com.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Select a cluster, then ask a new question
Sometimes neither the cheapest nor the closest listing is an obvious winner. A further abstraction would represent each listing as a circle on a scatterplot, making several criteria visible together.
| Encoding | Meaning |
|---|---|
| Horizontal position | Price increases to the right |
| Vertical position | Listings closest to the hotel sit near the bottom |
| Color | Wi-Fi review evidence |
This view would reveal a cluster of inexpensive, nearby listings with favorable Wi-Fi reviews. The traveler could identify that promising group without comparing every table cell individually.
Then another constraint appears: the flight arrives at 9 AM. The traveler would circle the promising cluster and message every selected property about early check-in. The proposed decision rule is to book the first property that replies that an 11 AM check-in is possible. Selection on the plot therefore becomes an input to an action across several listings; the visualization is more than a passive summary.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make it easy to move between levels
Each task benefits from a different representation. Today, much of that transformation happens in the user’s head: inspect one listing, remember its relevant details, inspect another, and try to maintain a comparison across fifty sources. That manual abstraction consumes memory and mental energy.
Wattenberger credits Bret Victor’s Up and Down the Ladder of Abstraction, whose verified publication is an interactive essay. The direction of travel matters: higher abstraction is not inherently better. The opportunity is to use AI to generate different levels, connect them, and make movement between them easy. A detailed listing, a table, and a scatterplot can remain useful parts of the same workflow.
Adept’s work here was an ongoing exploration. Its design ambition returns to the spreadsheet: automate the smaller, tedious parts so a person can do the larger task more effectively. For AI interfaces, that includes generating useful representations of information and preserving the ability to act through them. The gain comes from making the right level available for the task at hand.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Bret Victor's interactive essay explores how linked representations reveal patterns and let readers return to concrete examples.
Further reading
- VisiCalc — The Early HistoryArticle
A firsthand account of VisiCalc's development, publication, and early use.
Wattenberger examines the limitations of chat interfaces and the value of more structured tools.
Adept's 2022 introduction to a model trained to operate digital tools, with browser and spreadsheet examples.
- Shapes of StoriesArticle
A visual interpretation of Vonnegut's story shapes, including the Man in Hole arc.
Full transcript
Loading transcript…