AI Engineer World's Fair 2024

Second Order Effects

Cheng Lou· Member of Technical Staff, Midjourney21:46

Loading slides…

Read the talk

Second Order Effects: From GPU Art to Interfaces That Choose Their Shape

Cheng Lou explores how AI could change learning, communication, and interface design by widening context and making room for many more possibilities.

From a talk by Cheng Lou

Before you start: Basic familiarity with user interfaces and machine learning is helpful; the shader example assumes a little programming experience.

A GPU drawing without a scene

The opening animation visualizes matrix multiplication, but its construction is the more revealing part. Cheng Lou describes a single fragment shader running entirely on the GPU: a few hundred lines of GLSL, no imported assets, and no triangle meshes defining the pictured scene. The image comes from computation rather than a collection of modeled objects.

A small GLSL example makes that distinction tangible. This fragment shader rotates pixel coordinates with a matrix, then colors a rectangular region defined by a distance calculation. The visible shape needs no imported artwork:

glsl

#version 300 es
precision highp float;

uniform vec2 uResolution;
uniform float uTime;
out vec4 fragColor;

void main() {
    vec2 p = (2.0 * gl_FragCoord.xy - uResolution)
           / uResolution.y;
    float c = cos(uTime);
    float s = sin(uTime);
    p = mat2(c, s, -s, c) * p;

    vec2 q = abs(p) - vec2(0.45, 0.2);
    float distanceToBox = length(max(q, 0.0))
                        + min(max(q.x, q.y), 0.0);
    float coverage = 1.0 - smoothstep(-0.01, 0.01, distanceToBox);
    vec3 color = mix(vec3(0.03), vec3(0.2, 0.8, 1.0), coverage);
    fragColor = vec4(color, 1.0);
}

Shader art is a niche use of graphics hardware. Machine learning is a much larger departure from the GPU’s original purpose. Both suggest the opening question: what becomes possible after a technology escapes the use case it was built for?

0:000:18
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:00 · section reference included

From individual rules to collective behavior

When Deep Blue beat Garry Kasparov and AlphaGo beat Lee Sedol, a plausible pessimistic response was that chess and Go had lost their point. If humans could no longer be the best players, why keep playing? The historical milestones need a small distinction: Deep Blue won a game in 1996 and the match in 1997; AlphaGo’s victory over Lee Sedol came in March 2016. Lou then shows a graph of professional Go decision quality whose final years rise sharply. Machine superiority need not end human improvement.

Chart spanning 1950 to October 2021, with green points, vertical bars, a dashed horizontal baseline, and a tan highlighted region at the right.Open full source frame
Green points and vertical bars rise sharply in the highlighted final years.

Conway’s Game of Life makes the same forecasting difficulty visible in another form. Black-and-white cells follow a few simple local rules. Larger arrangements produce moving patterns and, eventually, an implementation of Game of Life inside Game of Life. The system is Turing complete: a small rule set can support general computation when organized at sufficient scale.

An order-of-magnitude increase in quantity, quality, or performance can change which level of description is useful. Knowing the cell rules does not make the behavior of a large construction immediately obvious. Lou describes emergent behavior broadly as “mostly NP-complete”; that is his characterization, not a complexity classification established for every system here. His practical point is that prediction often requires moving up to macro patterns, heuristics, and trial and error. The work starts to resemble biology more than deduction from a compact physical law.

Second-order effects are the ripples produced by a technology’s direct consequences. Lou introduces his forecasting approach with an unattributed science-fiction aphorism: “Good sci-fi predicts cars. Great sci-fi predicts traffic.” Predicting a better machine is only the beginning; the harder question concerns the behavior that gathers around it.

0:591:15
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:59 · section reference included

Who wants to do the learning?

The first question is whether the person wants an outcome or wants to acquire the ability to produce it. For chess, Lou proposes a reinforcing loop: free, always-available teachers remove dependence on a scarce local expert; more people learn the game; a more knowledgeable audience can sustain professional play. The machine’s skill helps distribute human skill rather than making it irrelevant.

The distinction is the same as going to a mental gym. A machine can practice indefinitely without giving you the benefit of practice. If your goal is your own learning, the fact that the machine performs better does not remove your reason to participate.

Drawing brings this distinction into interface design. A blank canvas is daunting for a novice. Lou imagines stroke autocompletion controlled by an assistance slider: at one end, the system does nothing; at the other, it produces the entire drawing. The interesting territory lies between those endpoints. A curved stroke on a chair could prompt the system to infer a Victorian style. A blue reflection on a face could invite a suggestion to shift toward green because the scene also contains sand.

These interventions shorten the feedback loop involved in learning color and form. Assistance can increase when the learner needs guidance and decrease as the lesson becomes internalized. In Lou’s imagined progression, successful use eventually moves the slider all the way back toward no assistance. A learning tool can succeed by making itself less necessary.

2:563:05
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:56 · section reference included

Instruments worth learning

Music offers another path beyond generating a finished artifact. One-shot song generation can satisfy a utilitarian need, but playing an instrument is a different activity. A piano has a particularly direct relationship between action and result: press a key and hear the expected sound. A theremin introduces more indirection, translating movement through space into sound. That extra distance between gesture and output can also create expressive possibilities.

Lou imagines manipulating a musical spectrogram with your fingers. Fingers cannot precisely specify all the fine structure of a sound, but an AI system with sufficient knowledge could fill in that detail—something like supersampling a coarse gesture. The performer would act more like a puppeteer, indirectly shaping an output that ordinary analog manipulation might not make available. This is a proposed instrument, not just a faster way to request a song.

That possibility leads back to conventional interfaces. Tapping controls and mastering a tool may be valuable because the person wants to learn, much as someone wants to exercise or draw. Once automation satisfies more utilitarian needs, Lou expects room for artisanal, quirky, niche interfaces used for self-fulfillment, lifestyle, or luxury. Their appeal would come from the experience of using them, rather than from beating the machine on efficiency.

4:475:03
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:47 · section reference included

A language for each recipient

The next forecasting move is to widen information bandwidth. A cluster visualization in Anthropic’s sparse-autoencoder research reminds Lou of Denis Villeneuve’s film Arrival, in which an alien language changes what humans can perceive. He pushes the association further: why stop at a shared new language? Why not a language for each person?

Standardized language compresses a large, fuzzy internal experience into a narrow communication channel. Lou takes AI having “basically solved translation” as the premise for a thought experiment, then extends translation from one national language to another person’s interpretation of the same language. If he says “I feel blue,” the intended feeling may not arrive intact. For a particular listener, “I’m feeling purple” might communicate it more faithfully. The target is preserved intent, not preserved wording.

The message need not remain text. A speech bubble could change according to the recipient’s aesthetic preferences. With sufficiently fast generation, each sentence could become a dynamic, four-dimensional artwork handed to someone in augmented reality. Lou imagines a surrounding visual cloud translated just in time for its observer, carrying more nuance than static emojis or a few Bézier curves.

At a speculative fifty-year horizon, Lou imagines verbal conflicts resolving in seconds rather than minutes or hours. The proposed mechanism is a richer, recipient-specific channel that loses less intent in transit. It depends on understanding the receiver, not merely producing more elaborate imagery.

6:306:43
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:30 · section reference included

A pencil stroke has more context than its path

After briefly pointing to the iPhone’s hardware interface, Lou turns to the iPad. In a canvas application, pressing a pencil against the screen usually draws a line. The same contact is also overloaded to select, move, or resize. Yet the user supplies more information than the application ordinarily considers. Several taps might follow a spoken question about why an area is red. A stroke might follow the observation that an object belongs somewhere else.

Without that context, moving an object becomes a sequence of interface acrobatics: find the lasso tool, select it, return to the canvas, circle the object, long-press to move it, then double-tap the pencil to return to the pen. Trying to eliminate these steps with more gesture rules can make the system harder to understand, because the rules become implicit.

Traditional gesture interpretation already considers more than geometry. It may depend on the selected tool, the object under the pencil, and a recent action that causes a tentative stroke to be reinterpreted as a tap. Lou proposes expanding that conditioning much further:

InterpretationAvailable contextDesign consequence
Tool-driven gesture rulesTool, target object, recent inputExplicit modes and carefully tuned exceptions
Proposed learned interpretationGesture plus speech, surrounding activity, and user behaviorInfer intent from a broader situation

The aim is to condition a stroke on the world around it, rather than only the beginning of its Bézier path.

A slightly overlong short press illustrates the difference. A rigid threshold may trigger the wrong gesture, while a human observer would recognize the intended action almost immediately. Lou asks why a learned system could not make that correction too, including through local inference. The opportunity is to use context the person already provides.

8:409:00
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:40 · section reference included

The cost of making every agent smart

A third forecasting technique is to push a quantity or quality to an extreme, then reason about what changes. Lou begins with Smalltalk, the programming environment developed in the 1970s that helped establish object-oriented programming and influenced Objective-C. Its message-passing model invites a view of objects as active participants, potentially communicating with remote objects across a LAN or the internet.

Lou recounts Alan Kay’s biological inspiration: objects resemble cells, and each object can be understood as a full computer that can be inspected and manipulated. One physical computer might host many objects, or an object might correspond to a computer. The difficult part is not merely sending a command; it is finding the receiver that should handle it.

Slide titled “Raise Order of Magnitude” with three statements about cells, each object being a full computer, and finding the receiver, beside a black-and-white photograph of a man at a computer and keyboard.Open full source frame
Objects as full computers: biological inspiration and the difficulty of finding the receiver.

Such an object could, in principle, go onto the internet, perform work, and return an answer. It starts to resemble a self-directed intelligent agent. But capability has a resource cost. In Lou’s resource-budget thought experiment, sufficiently expensive agents leave room for only single- or double-digit populations. Optimizing each member’s intelligence can prevent the system from reaching the scale where quantity and collaboration produce a different kind of behavior.

10:3510:52
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:35 · section reference included

Simple components can change the unit of intelligence

A multilayer perceptron offers a contrasting organization. Its graph specifies possible receivers through connectivity, and learned weights determine which connections matter most. The individual nodes are simple enough that a system can contain millions of them. The interesting capability then belongs to the aggregate.

OrganizationIndividual componentCoordination emphasis
Capable agent swarmA relatively expensive, self-directed processFinding work and suitable receivers
Multilayer perceptronA simple computational nodeGraph connectivity and learned weights

Lou uses this contrast to redirect attention from how impressive each agent is to what a population can do. As the population grows, the useful perspective moves outward, much as understanding civilization requires more than studying one person.

He closes the comparison with an alternate history: a few agents search Wikipedia and return snippets. In his analogy for the path actually taken, billions of simpler nodes absorb information into an aggregate that can sit in a phone, coordinated through a smart top-level process. The nodes are not literally independent web crawlers; the comparison shifts the design question from individual autonomy to collective capability.

12:5013:02
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

12:50 · section reference included

What if the problem is too little UI?

The final example starts with a small surprise in the Apple TV app on macOS. A description has a more button. Where should the expanded text appear? In the version Lou demonstrates, clicking it opens a large selectable text view that he identifies as a UITextView, resembling an unfinished notepad more than the surrounding interface.

Lou identifies the app as a Catalyst port of its iOS counterpart. On iOS, opening the description in another view feels relatively idiomatic; carried onto the Mac, the same interaction feels out of place. One explanation is insufficient design care. Lou offers another perspective: perhaps there are not enough suitable interfaces available.

What would it mean to increase the quantity of UI by two orders of magnitude? Instead of expecting one design to cover many different situations, a product could have a much larger collection of appropriate designs. The next problem becomes how to produce that collection and choose from it.

14:2214:36
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

14:22 · section reference included

Generate widely, curate offline

Lou’s proposed workflow begins with a twelve-column grid and an inventory of the discrete information that might appear across a view—or the entire application. AI generates thousands of possible layouts at design time. These are candidates for review, not interfaces immediately shipped to users.

The crucial contribution is semantic placement. Traditional generative algorithms can arrange boxes, but assigning a show’s title to an appropriately sized box in the right relationship to other content requires understanding what the boxes mean. The proposed pipeline combines that semantic generation with filtering:

  1. Generate candidate layouts from the content inventory and grid.
  2. Score candidates using traditional heuristics, AI-based aesthetic judgments, or both.
  3. Remove undesirable candidates during data generation.
  4. Have a designer curate the remaining collection offline.

Because this work is outside the interactive path, the algorithms can be slow and the designer can take time.

The scale matters. Lou’s target is roughly a thousand layouts rather than ten or a handful manually rearranged in Figma. The designer still participates, but the work moves beyond repeatedly shifting individual boxes in response to one more requested variation.

Rough drafts could also use diffusion, perhaps with ControlNet, to turn box arrangements into more tangible website concepts. That helps stakeholders judge whether a layout could work as an actual interface. This remains an optional design-time step in Lou’s proposal; runtime does not need to reproduce the image-generation process.

15:4716:00
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

15:47 · section reference included

At runtime, select rather than generate

Generating two or three designs, choosing one, and shipping it still produces a conventional application. Lou wants AI to change runtime behavior too. He frames the limitation provocatively: even AGI cannot make JavaScript faster than C++ if the web platform itself is the bottleneck. That categorical comparison is his assertion, rather than a demonstrated language benchmark. His architectural proposal is to replace some existing decision logic with a neural network where that changes what the application can do.

With a curated layout collection, runtime can use a quick decision tree or learned classifier to select an appropriate interface. Lou compares this with width-driven responsive design. Media queries already support more than width or visibility—including orientation, resolution, pointer, and hover conditions—but his proposed extension concerns richer user and task context.

Four rectangular wireframes in a two-by-two arrangement, each showing different outlined content boxes over pink vertical columns.Open full source frame
Four interface layouts arranged on shared column grids.

The selection inputs could include several parallel kinds of information:

  • Experience: Onboarding users may need a different arrangement from superusers, with layouts changing as expertise develops.
  • User context: Country or age could affect which information is relevant.
  • Current intent: A search query or chatbot exchange could determine which relationships deserve space.

Onboarding need not be a completely separate concept from choosing which boxes to show. It can be another situation handled by the layout selector.

Lou points to Uber and Facebook as companies already maintaining extensive contextual variations. He describes regional Uber interfaces as substantially different and attributes such adaptation to the effort of thousands of engineers, while acknowledging that business logic contributes to the competitive advantage too. His concern is that extensive bespoke logic becomes brittle and hard to inspect. A classifier over a known collection could provide a more controllable selection mechanism.

The movie-search example makes the runtime distinction concrete:

User requestProposed curated layout
What movies did Denis Villeneuve make?Emphasize the movies
What movies did Denis Villeneuve make, and with whom?Emphasize movies and collaborators

The second request changes which relationships the interface should expose. In a chatbot, the surrounding conversation could supply still more selection context. The application would choose among reviewed layouts rather than invent a new screen for each request.

Lou’s napkin estimate puts the needed collection in the thousands of layouts rather than tens. AI-assisted curation is what makes that scale seem feasible to him. Once the candidates are discrete, runtime layout choice is a classification problem. Autoregressive generation and diffusion may help create the collection, but they do not have to sit in the runtime selection path.

18:1118:23
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

18:11 · section reference included

Forecasting through invention

Second-order effects remain difficult to predict. These ways of thinking help make the uncertainty more approachable, but Lou closes by pairing them with historical study and practical work: read about what happened before, and build things that can reveal what happens next. His final line turns prediction into participation: “the best way to predict the future is to invent it.”

21:0021:12
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

21:00 · section reference included

Resources

From the talk

Transcript download
Full transcript

Loading transcript…