AI Engineer World's Fair 2025

tldraw computer

Steve Ruiz· Founder & CEO, tldraw18:45

Loading slides…

Read the talk

tldraw computer: making a canvas compute

An interactive canvas becomes a place to generate software, connect multimodal instructions, and let people and models edit the same objects.

From a talk by Steve Ruiz

A canvas built from ordinary web components

What becomes possible when a whiteboard can contain a working video player or code editor, with drawing tools still available around and above it? That is the starting point for Steve Ruiz’s tour of tldraw. His work on a digital ink library, subsequently implemented in Excalidraw, led to a broader question: why wasn’t there a reusable SDK for building this kind of application? The result was tldraw, available both as a free whiteboard at tldraw.com and as an SDK documented at tldraw.dev. Its canvas is built from ordinary web components—React throughout.

A square connects by a curved arrow to a smiling star, with a dashed arrow above.Open full source frame
Shapes and arrows on the tldraw canvas.

Ruiz plays an embedded YouTube video while retaining the ability to draw over it. A CodeSandbox editor remains an editor inside the canvas; Figma and Excalidraw can live there too. He then embeds tldraw inside itself and draws into the nested canvas. This recursive example belongs to the historical demonstration: the current embed documentation blocks nested tldraw rendering in an iframe and treats pasted tldraw links as bookmarks.

The SDK also handles the less spectacular details that make a canvas feel coherent. Arrows follow the boundaries of connected shapes, and box corners retain their geometry as the boxes change. These details matter for everything that follows: an AI experiment can use interactive web content and reliable drawing behavior without rebuilding a whiteboard first.

0:170:30
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:17 · section reference included

From a wireframe to a working application

People were already drawing wireframes in tldraw. Make Real asked whether those drawings could become functioning software. With GPT-4 with Vision, the initial mechanism was straightforward: send a screenshot to the model, give it the role of a web developer receiving a designer’s low-fidelity mockup, and ask it to build the interface. The drawing supplies layout and intent that would otherwise need to be described in words.

The stop-motion example makes that intent concrete. Ruiz wants a camera feed, a way to capture pictures, a display of the captured frames, and sequential playback. He submits the drawn interface without even including its title. While that generation runs, he opens a version generated during the preceding talk: it captures frames, provides onion skinning, and produces a GIF. The functioning example is therefore a previously generated result, rather than the completion of the request just submitted onstage. Ruiz compares the input method to supplying an image in Cursor.

Returning the application to the canvas makes its output available as the next input. Ruiz draws annotations over the rendered website and uses them to request another version. An already completed revision shows a button changed to a solid fill. In the original Make Real implementation, the next request included the previous HTML alongside the annotated image: an iframe’s contents exported as a blank rectangle, so the image alone could not carry the existing application forward.

Make Real appeared at the end of 2023, and Ruiz describes it as an early route into software creation for people who could not program. Its visual loop also extends to debugging. After an application briefly flashes green, he captures the bug in a screenshot and submits it with the original source, asking for that particular problem to be fixed. He reports that the model repaired it. The canvas is doing more than collecting prompts: it holds the interface, the requested changes, and visual evidence of failure in the same workspace.

3:263:43
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:26 · section reference included

Changing an image by changing a drawing

Draw Fast applies a similar feedback loop to image generation. Ruiz tentatively identifies its mechanism as latent consistency models: the aim is to generate an image quickly enough that changes to a drawing can drive changes to the result. The server initially responds, and he begins manipulating the input. Flattening content gives him an image he can transform further, including rotating and stretching it.

Ruiz says Draw Fast can respond almost in real time under good circumstances, but the live demonstration updates intermittently. Even the uneven feedback exposes how the model interprets the canvas: sideways figures appear to become running people. Rotation changes more than composition; it can change the action inferred from the input. The public hosted demo has since ended, while the repository remains the reference for exploring the experiment.

6:507:07
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:50 · section reference included

A commercial made by connected blocks

tldraw computer turns these individual transformations into a graph. Ruiz enters text about AI Engineer, MCP observability, and a conference, then sketches a large top hat with playing cards in its brim. He connects these inputs to an instruction to write a short commercial, adds a polite “please,” and runs it. The instruction block first creates a script describing its work, then executes that script.

The resulting commercial text feeds both speech generation and image generation. Each block accepts inputs and produces outputs; arrows make those dependencies visible. Ruiz then continues the graph with a transformation that makes the commercial sad and serious, followed by another generated image. A single collection of source material can therefore support several related outputs and successive revisions of their tone.

The instruction script is small. It tells the block to inspect its inputs for guidance about the product, services, style, and other requirements; write a short commercial; and output the result. A compact TypeScript representation makes the distinction between a reusable instruction and a particular set of inputs explicit:

typescript

const commercialBlock = {
  instruction: [
    'Analyze inputs for product, services, style, and requirements.',
    'Write the text for a short commercial based on those inputs.',
    'Output the result.',
  ].join('\n'),
  inputs: [
    { kind: 'text', value: 'AI Engineer' },
    { kind: 'text', value: 'MCP observability' },
    { kind: 'text', value: 'conference' },
    { kind: 'drawing', description: 'Top hat with playing cards in the brim' },
  ],
}

The drawing description above stands in for the canvas drawing. The essential interface is that the block can reuse its instruction with new inputs, then emit data the next block can accept. The script describes the work; the model performs the transformation.

Canvas with a top-hat drawing, conference text, a commercial instruction, generated script, speech player, and image connected by arrows.Open full source frame
Connected blocks turn a hat drawing and conference text into a commercial, speech, and an image.
8:128:22
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:12 · section reference included

Working with the models already available

The project emerged from a collaboration with Google around the launch of Gemini 2. Ruiz hoped the invitation would include early access to the new models, particularly the real-time phone capabilities Google had shown. It did not: the team had to build with the models already available. Gemini 1.5 was out, and Gemini Flash offered the combination Ruiz found useful—speed, adequate quality, and multimodal input. Those existing capabilities inspired the graph experiment.

10:2810:38
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:28 · section reference included

Adding numbers, an octopus, and a camera input

Once blocks could interpret instructions and pass results onward, they could do something resembling ordinary computation. Ruiz creates an instruction to add up all its inputs and supplies 2 and 11. It returns 13. But the operation is executed by a language model rather than an arithmetic function in code. That changes what the operation can accept—and how it decides what the inputs mean.

He replaces 11 with octopus. The prompt requires the model to infer a number even from nonnumeric input, so it interprets the octopus as eight and returns 10 after adding two. Next he adds a camera input and shows four fingers; the displayed result becomes 14.

InputsInterpretationResult
2, 11Ordinary numeric inputs13
2, octopusInfer eight from the octopus10
2, octopus, camera showing fourAdd the visually inferred four14

The model supplies the interpretation as well as the calculation. An octopus does not carry a numeric type; the instruction and the model’s associations make it usable in this particular operation.

Nodes containing 2, octopus, and a camera view connect to “add up all your input,” which connects to an output showing 14.Open full source frame
Text and camera inputs feed an instruction block, with 14 displayed as the output.

This permissiveness is also useful outside arithmetic. Ruiz’s favorite personal application is taking his daughter’s drawings and turning them into pictures and stories, then passing those results through other blocks. Text, drawings, and camera images can participate in the same workflow because the model can interpret their content.

11:2211:41
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:22 · section reference included

Feedback, memory, and a third Boolean value

Ruiz had been playing Factorio while developing these ideas, and the next example resembles a small machine that keeps processing material. Its graph contains a cycle:

  1. Generate a random pop song.
  2. Append the song to a list.
  3. Feed that list back into generation so the next choice avoids previous songs.
  4. Ask whether the song is about love and route the answer.

The list carries information from one iteration into the next. The classification introduces another consequence of model execution: Ruiz jokingly gives his Boolean three values—yes, no, and maybe. The graph can preserve ambiguity instead of forcing every song into a binary category.

The cycle can continue indefinitely, consuming the credits Google supplied as it goes. It is a useful reminder that a looping diagram is also a looping workload. Ruiz says tldraw computer became popular, although less popular than Make Real, and describes it as a tool that rewards creative combinations of these small operations.

13:2313:30
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:23 · section reference included

Designing a process that can wait

Ruiz has seen people use tldraw computer for multistage prompting and decision-making analysis. From there, he imagines moving execution into an asynchronous cloud process. His example starts with a CSV of email addresses belonging to people who have engaged with a product. The process would email them, wait for replies, analyze sentiment, and branch according to the response. Before contacting someone again, it could text the operator to ask for approval.

That long-lived workflow, potentially running many branches in parallel, is a proposed direction rather than a demonstrated deployment. The canvas would serve as an interface for designing it. This fits the project’s original creative brief: a computer that works the way Ruiz imagined computers worked before he understood their internals. Put some material here, say what should happen to it, and send the result over there. The graph makes that sequence something a person can arrange and inspect.

14:3314:49
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

14:33 · section reference included

A collaborator that edits the same objects

The final AI demonstration, Teach, moves from connecting transformations to collaborating directly on the canvas. Ruiz asks for a flowchart that begins with AI and ends with engineer, incorporating existing shapes. A canvas SDK with a runtime API gives the model a way to act on the workspace. Although Ruiz describes the models’ ability as imperfect, the integration is sufficient to make the model behave like a virtual collaborator.

He then asks it to draw a cat, mentioning a pelican riding a unicycle elsewhere on the page. The cat is not generated as a bitmap in the manner of Midjourney. The model returns text with a structure that the application maps into canvas shapes. Both the person and the model can edit the resulting objects. Ruiz can correct the drawing himself, and the model can work with things he adds.

Ruiz draws an orange element beside the cat and asks it to make the cat blow out the candle. He has not separately labeled his addition as a candle; the model must interpret it in context. The backend is Claude. The result shows the cat blowing toward the now-unlit candle, with blue strokes near its mouth and a small gray smoke mark above the candle. The edit combines recognition of a human addition with changes to an existing model-created drawing.

Drawn cat beside an unlit candle, with blue horizontal strokes across its mouth and a gray mark above the candle.Open full source frame
The cat blows toward the candle, with blue strokes and a small gray smoke mark.

The result is rough, playful, and responsive enough to make the interaction compelling. Ruiz calls this “shitty but amazing,” a description he considers on brand for tldraw and an advantage over larger companies. The demonstration’s value is the shared, editable workspace: a person can add something the model did not create, and the model can continue the scene from there.

15:4316:06
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

15:43 · section reference included

The canvas as an authoring layer

Ruiz closes by inviting builders to discuss the tools that make these integrations easier. The examples extend beyond AI: Grant Kot’s Liquid Layers uses tldraw as a geometric control and authoring layer for a liquid simulation. He also points to commercial adoption by Observable; Observable Canvases is the product whose whiteboarding documentation identifies tldraw as its foundation.

A canvas can hold interactive applications, supply inputs to a model, expose the dependencies of a running graph, or provide shared objects for a person and an AI collaborator. Ruiz’s closing invitation is direct: “Please build something amazing.” The canvas and its tools are available; the remaining work is to decide what to make with them.

17:5017:59
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

17:50 · section reference included

Resources

From the talk

Updates since the talk

Transcript download
Full transcript

Loading transcript…