AI Engineer World's Fair 2024
Second Order Effects
Loading slides…
Read the talk
Second Order Effects: From GPU Art to Interfaces That Choose Their Shape
Cheng Lou explores how AI could change learning, communication, and interface design by widening context and making room for many more possibilities.
From a talk by Cheng Lou
Before you start: Basic familiarity with user interfaces and machine learning is helpful; the shader example assumes a little programming experience.
A GPU drawing without a scene
The opening animation visualizes matrix multiplication, but its construction is the more revealing part. Cheng Lou describes a single fragment shader running entirely on the GPU: a few hundred lines of GLSL, no imported assets, and no triangle meshes defining the pictured scene. The image comes from computation rather than a collection of modeled objects.
A small GLSL example makes that distinction tangible. This fragment shader rotates pixel coordinates with a matrix, then colors a rectangular region defined by a distance calculation. The visible shape needs no imported artwork:
glsl
#version 300 es
precision highp float;
uniform vec2 uResolution;
uniform float uTime;
out vec4 fragColor;
void main() {
vec2 p = (2.0 * gl_FragCoord.xy - uResolution)
/ uResolution.y;
float c = cos(uTime);
float s = sin(uTime);
p = mat2(c, s, -s, c) * p;
vec2 q = abs(p) - vec2(0.45, 0.2);
float distanceToBox = length(max(q, 0.0))
+ min(max(q.x, q.y), 0.0);
float coverage = 1.0 - smoothstep(-0.01, 0.01, distanceToBox);
vec3 color = mix(vec3(0.03), vec3(0.2, 0.8, 1.0), coverage);
fragColor = vec4(color, 1.0);
}
Shader art is a niche use of graphics hardware. Machine learning is a much larger departure from the GPU’s original purpose. Both suggest the opening question: what becomes possible after a technology escapes the use case it was built for?
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From individual rules to collective behavior
When Deep Blue beat Garry Kasparov and AlphaGo beat Lee Sedol, a plausible pessimistic response was that chess and Go had lost their point. If humans could no longer be the best players, why keep playing? The historical milestones need a small distinction: Deep Blue won a game in 1996 and the match in 1997; AlphaGo’s victory over Lee Sedol came in March 2016. Lou then shows a graph of professional Go decision quality whose final years rise sharply. Machine superiority need not end human improvement.
Conway’s Game of Life makes the same forecasting difficulty visible in another form. Black-and-white cells follow a few simple local rules. Larger arrangements produce moving patterns and, eventually, an implementation of Game of Life inside Game of Life. The system is Turing complete: a small rule set can support general computation when organized at sufficient scale.
An order-of-magnitude increase in quantity, quality, or performance can change which level of description is useful. Knowing the cell rules does not make the behavior of a large construction immediately obvious. Lou describes emergent behavior broadly as “mostly NP-complete”; that is his characterization, not a complexity classification established for every system here. His practical point is that prediction often requires moving up to macro patterns, heuristics, and trial and error. The work starts to resemble biology more than deduction from a compact physical law.
Second-order effects are the ripples produced by a technology’s direct consequences. Lou introduces his forecasting approach with an unattributed science-fiction aphorism: “Good sci-fi predicts cars. Great sci-fi predicts traffic.” Predicting a better machine is only the beginning; the harder question concerns the behavior that gathers around it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Who wants to do the learning?
The first question is whether the person wants an outcome or wants to acquire the ability to produce it. For chess, Lou proposes a reinforcing loop: free, always-available teachers remove dependence on a scarce local expert; more people learn the game; a more knowledgeable audience can sustain professional play. The machine’s skill helps distribute human skill rather than making it irrelevant.
The distinction is the same as going to a mental gym. A machine can practice indefinitely without giving you the benefit of practice. If your goal is your own learning, the fact that the machine performs better does not remove your reason to participate.
Drawing brings this distinction into interface design. A blank canvas is daunting for a novice. Lou imagines stroke autocompletion controlled by an assistance slider: at one end, the system does nothing; at the other, it produces the entire drawing. The interesting territory lies between those endpoints. A curved stroke on a chair could prompt the system to infer a Victorian style. A blue reflection on a face could invite a suggestion to shift toward green because the scene also contains sand.
These interventions shorten the feedback loop involved in learning color and form. Assistance can increase when the learner needs guidance and decrease as the lesson becomes internalized. In Lou’s imagined progression, successful use eventually moves the slider all the way back toward no assistance. A learning tool can succeed by making itself less necessary.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Instruments worth learning
Music offers another path beyond generating a finished artifact. One-shot song generation can satisfy a utilitarian need, but playing an instrument is a different activity. A piano has a particularly direct relationship between action and result: press a key and hear the expected sound. A theremin introduces more indirection, translating movement through space into sound. That extra distance between gesture and output can also create expressive possibilities.
Lou imagines manipulating a musical spectrogram with your fingers. Fingers cannot precisely specify all the fine structure of a sound, but an AI system with sufficient knowledge could fill in that detail—something like supersampling a coarse gesture. The performer would act more like a puppeteer, indirectly shaping an output that ordinary analog manipulation might not make available. This is a proposed instrument, not just a faster way to request a song.
That possibility leads back to conventional interfaces. Tapping controls and mastering a tool may be valuable because the person wants to learn, much as someone wants to exercise or draw. Once automation satisfies more utilitarian needs, Lou expects room for artisanal, quirky, niche interfaces used for self-fulfillment, lifestyle, or luxury. Their appeal would come from the experience of using them, rather than from beating the machine on efficiency.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A language for each recipient
The next forecasting move is to widen information bandwidth. A cluster visualization in Anthropic’s sparse-autoencoder research reminds Lou of Denis Villeneuve’s film Arrival, in which an alien language changes what humans can perceive. He pushes the association further: why stop at a shared new language? Why not a language for each person?
Standardized language compresses a large, fuzzy internal experience into a narrow communication channel. Lou takes AI having “basically solved translation” as the premise for a thought experiment, then extends translation from one national language to another person’s interpretation of the same language. If he says “I feel blue,” the intended feeling may not arrive intact. For a particular listener, “I’m feeling purple” might communicate it more faithfully. The target is preserved intent, not preserved wording.
The message need not remain text. A speech bubble could change according to the recipient’s aesthetic preferences. With sufficiently fast generation, each sentence could become a dynamic, four-dimensional artwork handed to someone in augmented reality. Lou imagines a surrounding visual cloud translated just in time for its observer, carrying more nuance than static emojis or a few Bézier curves.
At a speculative fifty-year horizon, Lou imagines verbal conflicts resolving in seconds rather than minutes or hours. The proposed mechanism is a richer, recipient-specific channel that loses less intent in transit. It depends on understanding the receiver, not merely producing more elaborate imagery.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A pencil stroke has more context than its path
After briefly pointing to the iPhone’s hardware interface, Lou turns to the iPad. In a canvas application, pressing a pencil against the screen usually draws a line. The same contact is also overloaded to select, move, or resize. Yet the user supplies more information than the application ordinarily considers. Several taps might follow a spoken question about why an area is red. A stroke might follow the observation that an object belongs somewhere else.
Without that context, moving an object becomes a sequence of interface acrobatics: find the lasso tool, select it, return to the canvas, circle the object, long-press to move it, then double-tap the pencil to return to the pen. Trying to eliminate these steps with more gesture rules can make the system harder to understand, because the rules become implicit.
Traditional gesture interpretation already considers more than geometry. It may depend on the selected tool, the object under the pencil, and a recent action that causes a tentative stroke to be reinterpreted as a tap. Lou proposes expanding that conditioning much further:
| Interpretation | Available context | Design consequence |
|---|---|---|
| Tool-driven gesture rules | Tool, target object, recent input | Explicit modes and carefully tuned exceptions |
| Proposed learned interpretation | Gesture plus speech, surrounding activity, and user behavior | Infer intent from a broader situation |
The aim is to condition a stroke on the world around it, rather than only the beginning of its Bézier path.
A slightly overlong short press illustrates the difference. A rigid threshold may trigger the wrong gesture, while a human observer would recognize the intended action almost immediately. Lou asks why a learned system could not make that correction too, including through local inference. The opportunity is to use context the person already provides.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The cost of making every agent smart
A third forecasting technique is to push a quantity or quality to an extreme, then reason about what changes. Lou begins with Smalltalk, the programming environment developed in the 1970s that helped establish object-oriented programming and influenced Objective-C. Its message-passing model invites a view of objects as active participants, potentially communicating with remote objects across a LAN or the internet.
Lou recounts Alan Kay’s biological inspiration: objects resemble cells, and each object can be understood as a full computer that can be inspected and manipulated. One physical computer might host many objects, or an object might correspond to a computer. The difficult part is not merely sending a command; it is finding the receiver that should handle it.
Such an object could, in principle, go onto the internet, perform work, and return an answer. It starts to resemble a self-directed intelligent agent. But capability has a resource cost. In Lou’s resource-budget thought experiment, sufficiently expensive agents leave room for only single- or double-digit populations. Optimizing each member’s intelligence can prevent the system from reaching the scale where quantity and collaboration produce a different kind of behavior.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Simple components can change the unit of intelligence
A multilayer perceptron offers a contrasting organization. Its graph specifies possible receivers through connectivity, and learned weights determine which connections matter most. The individual nodes are simple enough that a system can contain millions of them. The interesting capability then belongs to the aggregate.
| Organization | Individual component | Coordination emphasis |
|---|---|---|
| Capable agent swarm | A relatively expensive, self-directed process | Finding work and suitable receivers |
| Multilayer perceptron | A simple computational node | Graph connectivity and learned weights |
Lou uses this contrast to redirect attention from how impressive each agent is to what a population can do. As the population grows, the useful perspective moves outward, much as understanding civilization requires more than studying one person.
He closes the comparison with an alternate history: a few agents search Wikipedia and return snippets. In his analogy for the path actually taken, billions of simpler nodes absorb information into an aggregate that can sit in a phone, coordinated through a smart top-level process. The nodes are not literally independent web crawlers; the comparison shifts the design question from individual autonomy to collective capability.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
What if the problem is too little UI?
The final example starts with a small surprise in the Apple TV app on macOS. A description has a more button. Where should the expanded text appear? In the version Lou demonstrates, clicking it opens a large selectable text view that he identifies as a UITextView, resembling an unfinished notepad more than the surrounding interface.
Lou identifies the app as a Catalyst port of its iOS counterpart. On iOS, opening the description in another view feels relatively idiomatic; carried onto the Mac, the same interaction feels out of place. One explanation is insufficient design care. Lou offers another perspective: perhaps there are not enough suitable interfaces available.
What would it mean to increase the quantity of UI by two orders of magnitude? Instead of expecting one design to cover many different situations, a product could have a much larger collection of appropriate designs. The next problem becomes how to produce that collection and choose from it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Generate widely, curate offline
Lou’s proposed workflow begins with a twelve-column grid and an inventory of the discrete information that might appear across a view—or the entire application. AI generates thousands of possible layouts at design time. These are candidates for review, not interfaces immediately shipped to users.
The crucial contribution is semantic placement. Traditional generative algorithms can arrange boxes, but assigning a show’s title to an appropriately sized box in the right relationship to other content requires understanding what the boxes mean. The proposed pipeline combines that semantic generation with filtering:
- Generate candidate layouts from the content inventory and grid.
- Score candidates using traditional heuristics, AI-based aesthetic judgments, or both.
- Remove undesirable candidates during data generation.
- Have a designer curate the remaining collection offline.
Because this work is outside the interactive path, the algorithms can be slow and the designer can take time.
The scale matters. Lou’s target is roughly a thousand layouts rather than ten or a handful manually rearranged in Figma. The designer still participates, but the work moves beyond repeatedly shifting individual boxes in response to one more requested variation.
Rough drafts could also use diffusion, perhaps with ControlNet, to turn box arrangements into more tangible website concepts. That helps stakeholders judge whether a layout could work as an actual interface. This remains an optional design-time step in Lou’s proposal; runtime does not need to reproduce the image-generation process.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
At runtime, select rather than generate
Generating two or three designs, choosing one, and shipping it still produces a conventional application. Lou wants AI to change runtime behavior too. He frames the limitation provocatively: even AGI cannot make JavaScript faster than C++ if the web platform itself is the bottleneck. That categorical comparison is his assertion, rather than a demonstrated language benchmark. His architectural proposal is to replace some existing decision logic with a neural network where that changes what the application can do.
With a curated layout collection, runtime can use a quick decision tree or learned classifier to select an appropriate interface. Lou compares this with width-driven responsive design. Media queries already support more than width or visibility—including orientation, resolution, pointer, and hover conditions—but his proposed extension concerns richer user and task context.
The selection inputs could include several parallel kinds of information:
- Experience: Onboarding users may need a different arrangement from superusers, with layouts changing as expertise develops.
- User context: Country or age could affect which information is relevant.
- Current intent: A search query or chatbot exchange could determine which relationships deserve space.
Onboarding need not be a completely separate concept from choosing which boxes to show. It can be another situation handled by the layout selector.
Lou points to Uber and Facebook as companies already maintaining extensive contextual variations. He describes regional Uber interfaces as substantially different and attributes such adaptation to the effort of thousands of engineers, while acknowledging that business logic contributes to the competitive advantage too. His concern is that extensive bespoke logic becomes brittle and hard to inspect. A classifier over a known collection could provide a more controllable selection mechanism.
The movie-search example makes the runtime distinction concrete:
| User request | Proposed curated layout |
|---|---|
| What movies did Denis Villeneuve make? | Emphasize the movies |
| What movies did Denis Villeneuve make, and with whom? | Emphasize movies and collaborators |
The second request changes which relationships the interface should expose. In a chatbot, the surrounding conversation could supply still more selection context. The application would choose among reviewed layouts rather than invent a new screen for each request.
Lou’s napkin estimate puts the needed collection in the thousands of layouts rather than tens. AI-assisted curation is what makes that scale seem feasible to him. Once the candidates are discrete, runtime layout choice is a classification problem. Autoregressive generation and diffusion may help create the collection, but they do not have to sit in the runtime selection path.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Forecasting through invention
Second-order effects remain difficult to predict. These ways of thinking help make the uncertainty more approachable, but Lou closes by pairing them with historical study and practical work: read about what happened before, and build things that can reveal what happens next. His final line turns prediction into participation: “the best way to predict the future is to invent it.”
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
The development of Deep Blue and its 1996 and 1997 contests against Garry Kasparov.
DeepMind's account of AlphaGo, including its March 2016 victory against Lee Sedol.
Code and instructions for controlling diffusion image generation with additional spatial conditions.
The 2021 specification for adapting styles to viewport, display, and input-device characteristics.
Further reading
A study of professional Go decisions from 1950–2021 examining changes in decision quality and novel moves following superhuman AI.
Full transcript
Loading transcript…