AI Engineer Summit 2023

Climbing the Ladder of Abstraction

Loading slides…

Read the talk

Climbing the Ladder of Abstraction

From spreadsheet formulas to books and lodging searches, AI can help us move between useful views of the same information—and keep acting at every level.

From a talk by Amelia Wattenberger

What changed when accounting became a spreadsheet?

What did accounting look like in the early 1900s? Handwritten letters and numbers, annotations in margins, calculations performed by hand. Crossed-out entries and ink blots make the effort visible: maintaining the document was itself a substantial part of the work.

VisiCalc brought the spreadsheet to personal computers in 1979; Lotus 1-2-3 followed in 1983. The breakthrough was not automatic arithmetic. Calculators and computers already supplied that. It was a structured interface that connected calculations through formulas. Change a cell or add a row, and the dependent results update live.

Green-on-black VisiCalc payroll table with employee rows, tax columns and a populated totals row, shown in a YouTube player.Open full source frame
VisiCalc’s payroll spreadsheet with calculated totals.

That structure changed where an accountant could spend attention. Instead of repeatedly calculating values and maintaining rows and columns, they could concentrate on what the numbers meant. The interface made many small automations work together as a tool for a larger task.

0:160:22
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:16 · section reference included

Augmentation is built from smaller automations

The corresponding question for AI is how to combine new capabilities with interfaces people use today—and interfaces they might want tomorrow. Two goals often frame that discussion:

  • Automation: Do a tedious task for the user, such as copying data into a table or calculating values.
  • Augmentation: Give the user a new ability, or improve an existing one, especially in creative or nuanced work such as analyzing data.

Concerns about jobs being automated are valid, Amelia Wattenberger acknowledges. But treating automation and augmentation as opposing goals misses their relationship: augmentation can be composed of smaller automations. Aggregating data into a table and generating visualizations can both serve a larger activity that remains directed by a person: answering the question that motivated collecting the data.

The spreadsheet already works this way. Its cell calculations are automated, while Excel remains a tool people use to understand financial data. Automatically assembling the table or writing its formulas would extend that pattern, removing more of the preparation involved in analysis.

Slide with the statement “Each cell is automated, the overall task is augmented” above a multicolored spreadsheet.Open full source frame
Each cell is automated; the overall task is augmented.

This is the reason to look beyond chatbots as the default AI interface. General tools such as calculators and chatbots are useful, but surrounding their capabilities with structure can make them much more powerful for particular tasks. The desired arrangement leaves the user directing the work while the model handles frustrating subtasks inside the interface.

1:381:46
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:38 · section reference included

Zoom changes which information matters

The ladder of abstraction describes how the same object can be represented at different levels of detail. Digital maps make this familiar. At a close view of the Monterey Bay Aquarium, buildings, names, icons, and routes help someone navigate within the aquarium. That representation is less useful for getting to the aquarium in the first place.

Zooming out does more than make buildings smaller. The map hides some information and foregrounds other information as the task changes.

ViewInformation emphasizedSupported task
NeighborhoodCity streets and restaurantsFind and reach a local destination
RegionHighways and terrainTravel to and from Monterey
Broad geographyState and country shapesUnderstand the wider geography

Keeping every building label and street visible at every scale would make the broad views incomprehensible. Screen space and attention are limited, and most fine detail is irrelevant to the task at that level. Could AI bring this behavior to other kinds of information?

4:455:00
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:45 · section reference included

Zooming out on Peter Pan

What would it mean to zoom out on a book? Reading usually exposes every word, but remembering a book or explaining it to a friend involves topics and plots rather than exact wording. Language models can summarize and transform text, making these different representations available inside a reading interface.

Wattenberger’s demo begins with the first five chapters of Peter Pan. Initially, she scrolls through the original first chapter. Zooming out replaces each paragraph with a one-sentence summary. A minimap on the right makes the reduction in text volume visible. At the next level, a summary stands in for a group of roughly ten paragraphs, reducing the text further.

PenPal interface showing Peter Pan, short text blocks under Chapter 1 and Chapter 2 headings, and a narrow minimap on the right.Open full source frame
Peter Pan condensed into summaries under chapter headings.

At the highest level, each chapter becomes one sentence, so all five chapters fit on a single page. The unit of attention has moved from words, to paragraphs, to groups of paragraphs, to chapters.

That reading demonstration suggests a writing workflow: inspect the chapter-level representation, adjust pacing or plot structure there, and then zoom back in to examine how the full text changes. This editing behavior is a proposal, rather than a demonstrated round trip. Its attraction is concrete: the writer could work on the structure without holding every part of the book in memory while editing word by word.

6:537:00
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:53 · section reference included

From a text summary to a story arc

A book can also be represented by its emotional trajectory. A story arc maps mood across the narrative rather than summarizing its wording. Kurt Vonnegut’s familiar Man in a Hole shape follows a character into trouble, out again, and into better circumstances. Wattenberger offers The Hobbit, The Wizard of Oz, and Alice in Wonderland as examples of that pattern.

The proposed interface would assign sentiment values to sections of a book and plot them on a graph. A writer could then adjust part of that curve and inspect the resulting changes to the prose. Here the higher-level representation is no longer shorter text: it is a different editing surface for a property of the story. This is a direction for future writing tools, not a capability established by the reading demo.

8:318:44
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:31 · section reference included

Get information, reason about it, act

Combining smaller automations with movement between abstraction levels suggests a broader product direction. At the time of the talk, Wattenberger was on Adept’s design team, working in a company focused on training AI to read screens and use software through actions people perform. The goal was to make knowledge work—work done on a computer—easier.

Interviews about people’s daily work revealed a recurring sequence:

  1. Obtain information.
  2. Transform it or reason about it.
  3. Act on it.

Adept’s design exploration asks what it would mean to zoom out on information throughout that sequence. The lodging workflow that follows is a hypothetical interface sketch, including its booking and messaging actions.

9:319:43
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:31 · section reference included

A lodging page organized around your decision

Suppose you are attending a conference in San Francisco. You find Airbnb listings near the venue and open one. Its detail page contains information intended for everyone, but your decision depends on particular questions: How close is it to the venue? Is there a coffee maker? Is the Wi-Fi good? Answering them requires digging through the page.

Zooming out slightly would remove branding and generic detail that do not help with this decision. The new view would retain the listing name, rating, a short summary, and total price. Then it would add information specific to the trip: walking minutes to Hotel Nikko, where the conference is being held. Instead of repeatedly consulting a map, you could compare that value directly. If the walk is too long, the view could also identify the nearest BART station and the walk to it as an alternative.

Wi-Fi matters because the night before the conference may still involve working on the talk. AI could extract relevant reviews and summarize whether their evidence is positive or negative. That gives the traveler a quick judgment from reported experiences, rather than a measured connection speed. The interface could also select a useful bedroom, living-room, or kitchen photo from the larger gallery, reducing another search through presentation material.

The abstraction must preserve the ability to act. From this personalized view, the traveler would still be able to reserve the listing or message its host without returning to Airbnb. Removing detail should simplify the decision while leaving the person in control of what happens next.

10:3310:44
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:33 · section reference included

From individual listings to a comparison table

The next question is whether a hotel would be a better deal. A shared representation could present hotel and Airbnb information in the same form, regardless of the source website. Putting those views side by side would make their relevant differences easier to compare.

But fifty individual views would still be cumbersome. Zooming out again would turn the hypothetical set of fifty listings into a spreadsheet, with the deciding factors laid out as columns. The traveler could scan the distribution of total prices, compare walking times, and inspect counts of positive Wi-Fi reviews. Booking would remain available at this level: a clear winner could be reserved directly from the table without returning to Airbnb or Hotels.com.

12:5013:05
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

12:50 · section reference included

Select a cluster, then ask a new question

Sometimes neither the cheapest nor the closest listing is an obvious winner. A further abstraction would represent each listing as a circle on a scatterplot, making several criteria visible together.

EncodingMeaning
Horizontal positionPrice increases to the right
Vertical positionListings closest to the hotel sit near the bottom
ColorWi-Fi review evidence

This view would reveal a cluster of inexpensive, nearby listings with favorable Wi-Fi reviews. The traveler could identify that promising group without comparing every table cell individually.

Then another constraint appears: the flight arrives at 9 AM. The traveler would circle the promising cluster and message every selected property about early check-in. The proposed decision rule is to book the first property that replies that an 11 AM check-in is possible. Selection on the plot therefore becomes an input to an action across several listings; the visualization is more than a passive summary.

13:5514:07
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:55 · section reference included

Make it easy to move between levels

Each task benefits from a different representation. Today, much of that transformation happens in the user’s head: inspect one listing, remember its relevant details, inspect another, and try to maintain a comparison across fifty sources. That manual abstraction consumes memory and mental energy.

Wattenberger credits Bret Victor’s Up and Down the Ladder of Abstraction, whose verified publication is an interactive essay. The direction of travel matters: higher abstraction is not inherently better. The opportunity is to use AI to generate different levels, connect them, and make movement between them easy. A detailed listing, a table, and a scatterplot can remain useful parts of the same workflow.

Three diagonally arranged views show a hotel listing, a hotel comparison table and a colored scatterplot, each beside a vertical plus-and-minus control; a Send message popup appears over the plot.Open full source frame
Hotel information presented as a listing, comparison table and scatterplot.

Adept’s work here was an ongoing exploration. Its design ambition returns to the spreadsheet: automate the smaller, tedious parts so a person can do the larger task more effectively. For AI interfaces, that includes generating useful representations of information and preserving the ability to act through them. The gain comes from making the right level available for the task at hand.

14:5615:01
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

14:56 · section reference included

Resources

From the talk

Transcript download
Full transcript

Loading transcript…