AI Engineer Europe 2026

Build & deploy AI-powered apps

Loading slides…

Read the talk

Build & deploy AI-powered apps

Follow video analysis, a bookshelf app, a book-to-media pipeline, and local Gemma agents through working results, failed assumptions, and live repairs.

From a talk by Paige Bailey, Guillaume Vernade and Ian Ballantyne

Before you start: Familiarity with Python, API requests, JSON schemas, and basic web application concepts will help with the implementation examples.

From scientific Python to model infrastructure

Would a company trust open-source software with business-critical work? When Paige Bailey began doing machine learning around 2009–2010, that was still a live question. Her contributions to NumPy, SciPy, Matplotlib, and scikit-learn connected scientific computing to practical engineering. With a background spanning geophysics, applied mathematics, computer science, and carbonate geology, she worked at Chevron on velocity modeling, drilling optimization, and large-scale computation using Cloudera and Spark.

Geoscience also gave her experience with CUDA before she began contributing to TensorFlow. Bailey recalls work separating CPU, GPU, and TPU execution paths, including single-node and distributed computation. Her recollection that the initial public release supported only CPUs conflicts with the TensorFlow launch announcement, which already described desktop GPU training. Open-source work led to Google, followed by a period at GitHub working on VS Code and early Copilot user-experience testing, then a return to the original PaLM 2, Gemini, and Gemma teams.

Public contributions serve two purposes: they let an engineer discover whether a team is worth joining, and they make the engineer’s work discoverable. An audience member’s project, with 700 GitHub stars, supplies a small example of that second effect.

0:150:26
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:15 · section reference included

Which constraints will survive the next model?

A harder product question follows: what happens when a model absorbs the capability around which an application was built? Bailey points to early 8,000- and 16,000-token context windows as a reason developers rushed toward vector databases. Longer contexts changed that constraint; broader multilingual support similarly reduced some demand for language-specific fine-tunes. She predicts that models may absorb more agent-framework behavior, describes a shift toward Markdown-based skills, and imagines activity listeners that create those skills automatically. These are expectations about where the platform is going, rather than reasons to assume retrieval, frameworks, or MCP have ceased to matter.

An audience member challenges that view with bioinformatics: specialized work may remain outside a general model provider’s priorities. Bailey responds with MedLM and Med-PaLM, saying that some users who previously needed medical fine-tunes now use Gemini with retrieval or custom prompts, and attributes that transition to medical training material entering the general model. The audience then raises reproducibility. Bailey acknowledges stochastic behavior and brings the discussion back to the durable application layer: an opinionated use case, developed directly with customers, can remain valuable as the underlying model improves.

4:424:58
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:42 · section reference included

A toolbox of inputs, outputs, and application surfaces

The workshop moves into a recap of demonstrations, initially framed around AI Studio and Antigravity. A computer-use agent is briefly active before Bailey disconnects the screen to avoid exposing private material. The model portfolio spans several distinct jobs:

  • Conversation and reasoning: Gemini 3.1 Flash Live, Gemini 3.1 Pro, and Gemini 3.1 Flash-Lite. Bailey reports that Augment Code chose Pro as its default for agent infrastructure because of its performance-to-cost balance.
  • Images and retrieval: Nano Banana 2 generates and edits images, with search-related capabilities. Embeddings 2.0 places video, audio, images, code, and text in a shared embedding space: a search about llamas could retrieve photographs, videos, stuffed animals, and llama sounds.
  • Generated media and worlds: Lyria 3 produces music, Genie 3 supports interactive world generation, and Veo 3.1 Lite produces video.
  • Applications and open models: AI Studio’s full-stack runtime adds databases, OAuth, and custom API keys; Gemma 4 arrives under Apache 2.0.

The useful distinction is between accepting multiple modalities and producing them. Bailey describes Gemini-family capabilities that include multimodal input alongside text, code, images, interleaved text and images, and audio-token output. The following demonstrations select particular models and tools for each task.

8:078:16
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:07 · section reference included

Turn a dinosaur video into a grounded table

AI Studio separates a model playground from Build, its application-generation surface. In the playground, Bailey selects Gemini 3.1 Flash-Lite Preview for speed and personal-account cost. Its launch pricing was $0.25 per million input tokens, with output billed separately at $1.50 per million tokens. The smaller model retains video and audio understanding, while trading away some capability relative to Pro.

The input is a long YouTube video featuring Carnotaurus. AI Studio also accepts Drive files, PDFs, local uploads, microphone recordings, and camera input, but this demonstration uses a video URL:

  1. Attach the YouTube video and select the interval from 0 to 300 seconds.
  2. Inspect the input estimate. Bailey reports approximately one sampled frame per second and approximately 30,900 input tokens for this selection.
  3. Enable Google Search grounding.
  4. Ask for a table containing timestamps, dinosaur kinds, and a fun fact about each.

The model receives visual frames, not just the video’s title or metadata. That distinction makes the request a visual-analysis task with search-supported enrichment.

The response identifies Carnotaurus, explains its name as meat-eating bull, includes Triceratops, and distinguishes Pteranodons from dinosaurs as flying reptiles. Once the prompt works, Get Code exports the selected model, video URL, offsets, and prompt into a language such as TypeScript or Python. This preserves the configuration that produced the playground result. Nearby tool settings expose structured outputs, code execution, custom function calling, and URL context for application use.

11:5112:06
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:51 · section reference included

Bounding boxes versus a verification loop

Compare mode makes the next experiment more revealing. Bailey supplies the same Lego image to Flash-Lite and Gemini 3 Flash Preview, enables code execution for both, and asks them to draw bounding boxes around green bricks using Python and display the annotated image. The tool provides a sandbox with preinstalled data-science libraries, allowing the model to write and invoke code against the supplied image. Flash-Lite returns its boxes while Flash is still working. Related requests could ask for segmentation masks, counts, orientations, or timestamped counts of objects such as Labubus in video.

Flash’s extra work turns out to be verification: it constructs a mask for green regions and checks the proposed coordinates against it.

RunObserved behaviorTradeoff in this demonstration
Flash-LiteProduces annotated boxes promptlyLess verification work shown
Flash PreviewChecks boxes against a green-region maskMore time, tool calls, and cost

Bailey describes the verified run as still costing on the order of pennies. The comparison makes tool policy part of model selection: extra checking consumes resources even when the requested output is the same. She recommends trying Flash-Lite for workloads previously served by Gemini 2.0 Flash or Gemini 2.5 Flash.

17:4218:00
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

17:42 · section reference included

Specify an account-linked bookshelf catalog

In AI Studio Build, the task expands from one inference to a full application. Shelf Scan should let a user sign in with Google, upload a bookshelf photograph, and retain a library associated with that account. The model must read visible spines, then use Google Search grounding to complete each book’s title, author, description, and genre. Recognition, enrichment, authentication, and storage are all part of the request.

Bailey switches from Pro Preview to the default build model in the hope of reducing waiting time, then starts generation. The IDE-like view exposes planning and file creation. PDFs, specifications, and Drive files can supply additional context; custom secrets can hold keys for Gemini, Supabase, or n8n. Version history supports iteration, while OAuth integrations and public or private GitHub synchronization connect the generated app to a broader development workflow. The app continues building while the workshop moves to other modalities.

21:0421:24
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

21:04 · section reference included

Change language while sharing a screen

Gemini Live adds continuous conversation around screen or camera input. Bailey describes it in terms of speech recognition, model understanding, and speech synthesis, then shares the Lego search page. Gemini responds with a description of the image results and the larger Freepik illustration preview. The response connects spoken conversation to the currently shared visual context.

AI Studio conversation showing the question “What do you see on the screen?” and a response describing Lego image results, with audio playback controls.Open full source frame
Gemini describes the Lego image search shared on screen.

An initial misunderstanding of an audience member’s language request is corrected to Spanish. Bailey first changes the system instructions and receives a Spanish response. After the audience asks for Castilian Spanish, she removes those instructions and requests the dialect within the conversation itself. The next response demonstrates that language and delivery can be steered either through configuration or through an ordinary turn. Get Code retains the selected model, settings, and tool configuration.

The camera demonstration combines perception and generation in one request: identify the number of raised fingers and compose a poem about the person. Gemini identifies two fingers, interprets the gesture as a peace sign, and produces a short poem. Screen sharing and camera feeds therefore supply context for a conversational application, rather than requiring the user to describe everything in text.

23:5324:03
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

23:53 · section reference included

Explore a world generated as pixels

After enabling Firebase for Shelf Scan, Bailey opens Project Genie. The application combines models to create an interactive experience; it is distinct from the Genie 3 world model itself. Bailey names Nano Banana, Veo, and Gemini while explaining the composition, whereas the launch description identifies Genie 3, Nano Banana Pro, and Gemini. The experience generates frames in response to actions rather than exporting a Unity or Unreal scene with conventional game assets and a physics engine.

Examples include a rover crossing a volcanic landscape, a jet ski colliding with a light, and a hurricane scene derived from a still image. Bailey describes the knocked-over light persisting if the player returns during the short exploration. For the live example, she specifies Regent’s Canal on a sunny day, dolphins in the water, and pirate flags on every boat. The player character is a pink sparkly squirrel with purple feet and a pirate hat. A generated first-frame sketch provides a checkpoint before the interactive world is created. The demonstrated public prototype limits an exploration to 60 seconds; Bailey describes Ultra access in selected regions.

While Shelf Scan generates Firestore rules in the background, the squirrel world becomes playable. Movement and viewpoint controls use the keyboard, and the space bar makes the squirrel jump. The requested boats, flags, and dolphins appear, but the squirrel walks on water and the dolphins remain still. Bailey suggests that specifying deep canal water might have helped. Those failures expose the difference between plausible generated motion and explicit physical constraints. The exploration produces a downloadable video; Bailey contrasts its pixel-based representation with what she describes as World Labs’ more asset-oriented approach.

28:5629:08
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

28:56 · section reference included

Recognition succeeds before persistence does

Back in Shelf Scan, Bailey inspects generated configuration, Firestore rules, and prompting code, grants camera permission, and signs in with Google. She uploads a WebP photograph of a cookbook shelf. The app starts searching and appears to identify about seven books, but the library does not populate as expected. She retries with a screenshot of the same image, initially testing whether the file format is involved. The next visible failure is more specific: insufficient permission to save in Firestore.

The repair process investigates validation logic and points to the image URL’s size in the Firestore rules. This does not establish WebP as the cause. The generated changes target the save path’s validation instead. While that work runs, Bailey shows sharing controls and the app gallery: Lyria music, multiplayer Neon Snake, Mandelbulb Explorer, Nano Banana image design, and a MediaPipe game controlled by hand tracking.

After the repair completes, another upload populates the catalog with dates, book types, titles, authors, and descriptions, including information not visible on the spines. Logging out and back in preserves the library. The successful application test is the complete recognition–save–reload cycle, not merely a plausible extraction response. Bailey shares access through a link and QR code, then proposes a future borrowing-request feature for friends who keep the books she lends them. That feature remains an idea; the demonstrated result is the persistent personal catalog.

35:0735:19
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

35:07 · section reference included

Choose the media model for the job

Guillaume Vernade takes over with a shortened version of his generative-media workshop, using some results prepared beforehand. His background includes the game industry and developer advocacy at Google and DeepMind. He frames multimodality as both understanding different sensory inputs and responding through different outputs. In his account, early Gemini image capabilities were restricted through post-training, with residual image refusals appearing even after later image support became available. Separately shipped image, video, and music models make individual capabilities easier to update without replacing one enormous all-purpose product. He describes a release cadence of roughly one generative-media model or capability every five days, alongside frequent broader product updates.

The practical choices differ by output:

  • Images: Nano Banana 2 provides aspect-ratio and size controls. Vernade contrasts Nano Banana Pro search grounding, used to illustrate his online footprint, with image grounding that retrieves visual references for a specific bridge or building.
  • Video: Veo 3.1 Lite is positioned for testing prompts before moving to a higher-quality model. His spoken price uses an image unit; the current pricing documentation bills Lite video with audio per second, listing $0.05 at 720p and $0.08 at 1080p. Those current rates are not an archived workshop tariff.
  • Songs: Lyria 3 offers short songs and longer songs with lyrics. An API workflow could, for example, generate a morning song about the news.
  • Continuous music: Lyria real-time keeps generating while changing prompts steer the music, much like a DJ.

These are paid-model exercises. Vernade suggests skipping video to contain cost and gives a rough €1 estimate for the lighter workshop run.

43:1143:25
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

43:11 · section reference included

Prepare a book-to-media pipeline

The notebook starts with a book, generates portraits and chapter illustrations, then extends those images into video, music, and narration. Setup includes installing the SDK, supplying an API key, and initializing a client with retry behavior. Vernade highlights retries because Nano Banana 2 is experiencing capacity contention, especially as US demand rises. He also limits the number of generated characters and chapters and warns that the paid-execution checkbox was accidentally left enabled. A separate Veo notebook, he reports, can cost about $20 to run.

The source is Kenneth Grahame’s The Wind in the Willows, downloaded from Project Gutenberg. Vernade encounters a retrieval failure while in the UK and expects a remotely hosted notebook may retrieve it successfully; the cause is not established. Project Gutenberg availability also should not be confused with worldwide permission to reuse every text.

51:2751:38
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

51:27 · section reference included

Select the API surface and retain useful context

Before processing the book, Vernade distinguishes Google’s application and API surfaces:

SurfaceMain benefitMain tradeoff
Consumer GeminiReady-made experienceLimited model, parameter, and tool visibility
Vertex AIEnterprise and regional deployment controlMore infrastructure setup
AI Studio and Developer APIFast start with an API keyFewer enterprise deployment controls

He recommends beginning with Vertex when a team already uses Google Cloud or has DevOps support. For this notebook, the Developer API provides the simpler entry point.

A three-color gauge labeled consumers, developers, and enterprise connects to Gemini, NotebookLM, Google AI Studio, Gemini Developer API, and Vertex AI.Open full source frame
Google AI products across consumers, developers, and enterprise.

The file-upload API abstracts storage-bucket setup and makes the book reusable as a prompt input. Structured output constrains responses to fields the next step can consume, avoiding conversational preambles around generated prompts. Chat mode saves and resends history so later requests retain the book’s context. An image chat similarly retains earlier images in the hope of preserving characters and style. That accumulated history is convenient, but the next demonstration exposes why it is not a sufficient reference-selection strategy by itself.

54:2054:32
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

54:20 · section reference included

Replace implicit image history with explicit references

Vernade asks for dark fantasy, with black-and-white backgrounds and colored characters. System instructions prohibit titles, borders, descriptive text, and comic panels while keeping images family-friendly. These constraints address an earlier tendency to turn portrait illustrations into book covers. An adult-only instruction left over from an earlier regional restriction is now obsolete. A separate image chat generates Mole, Water Rat, Toad, and Badger, then illustrates chapter scenes: a riverside picnic, a road scene, and a forest. In the forest illustration, Toad no longer matches the earlier portrait.

The repair changes the data contract. Each chapter now includes a name, a prompt, and an explicit list of characters. The application uses that list to attach only the relevant character reference images. The Python structure below expresses that selection step: the chapter record determines the image inputs, rather than leaving the model to locate them in an expanding conversation.

python

from pathlib import Path
from pydantic import BaseModel

class Chapter(BaseModel):
    name: str
    prompt: str
    characters: list[str]

def select_references(
    chapter: Chapter,
    portraits: dict[str, Path],
) -> list[Path]:
    missing = set(chapter.characters) - portraits.keys()
    if missing:
        raise ValueError(f"Missing character portraits: {sorted(missing)}")
    return [portraits[name] for name in dict.fromkeys(chapter.characters)]

This makes missing references visible before generation and avoids repeatedly sending unrelated portraits. Vernade argues that relying on complete history becomes increasingly fragile as the cast grows. For a larger production, he would create front, back, and side views and select the view needed for each scene. In the revised demonstration, Toad matches the intended character.

58:0958:26
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

58:09 · section reference included

Animate the scene, then score the chapter

To animate an illustration, Vernade selects the largest Veo model, supplies the latest image as the starting frame, reuses its image prompt, and requests portrait output at 720p. He says he would choose a smaller model if paying personally. The first clip includes dramatic dialogue, but a still-image prompt leaves the action after that frame underspecified. He therefore returns to the book chat with the image and asks for a continuation: Mole shivers and clutches his scarf while Water Rat draws a cutlass to protect him.

The more explicit continuation does not produce the better result in this run. Vernade judges the second clip worse and notes inconsistent additions of text across his tests, without identifying the cause. The useful distinction is still between describing a frame and specifying what happens next, but the generated clip must be inspected rather than assumed to improve with a longer prompt.

Music follows the same orchestration pattern: Gemini writes chapter-specific prompts, and generate_content sends them to Lyria. The river scene receives peaceful orchestral acoustic folk with guitar and flute; the open road gets jaunty fiddle and acoustic bass; the forest gets suspenseful staccato and pizzicato strings. Vernade plays the prepared outputs and finds their moods appropriate. He says he ran the notebook beforehand without auditioning the music, and attributes Gemini’s usefulness as a media-prompt writer partly to Gemini-generated material used in training the media models.

1:03:011:03:12
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:03:01 · section reference included

Create distinct characters with two configured voices

The narration stage uses the text-to-speech capability Vernade connects to NotebookLM’s podcast experience. Rather than assigning a separate configured voice to every character, he rewrites an excerpt as a play. Narration receives a narrator label; the other lines share a character label, with stable delivery instructions identifying recurring characters. The narrator uses Sulafat, while the other characters share Fenrir. Pace, accent, and emotional delivery create the audible distinctions. Vernade treats the automatic transcript-rewriting step as a workshop shortcut he would refine for larger-scale use.

He explicitly instructs the model to read the text, rather than supplying bare dialogue, and places performance directions in that request. The generated script repeats character-specific styles, including fast-paced, posh British speech, as speakers alternate. The boating passage between Mole and Rat then demonstrates how different the deliveries can sound despite sharing a voice identity. The same source-to-prompts pipeline could begin with an audiobook or a suitably short movie instead of a text file, using Gemini’s multimodal input capabilities.

1:07:531:08:06
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:07:53 · section reference included

Convert the notebook without hiding the workflow

Vernade next pastes the entire Python notebook into AI Studio and asks for an application implementing its book-illustration workflow. While it builds, he opens Space DJ. Its planets represent prompts; navigating toward different planets changes the continuously generated music, and autopilot lets the scene wander while the sound evolves. Returning to an earlier app build, he shows a file-upload interface and reports that generation took roughly 1,000 seconds. He says it reproduces the notebook’s workflow, but the presentation does not complete an end-to-end run establishing that equivalence.

His coding instructions aim to make generated applications easier to review:

  • Separate files by feature. If a requested feature change modifies unrelated files, the mismatch is immediately visible and deserves closer inspection.
  • Add logs around the workflow. An error message alone often omits the sequence and location needed to diagnose a failure.
  • Account for quota. A paid API key can increase the AI Studio coding quota available to a build.

These practices make the generated code and its failures observable, rather than leaving diagnosis entirely to another prompt.

1:13:401:13:44
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:13:40 · section reference included

Composition limits, service tiers, and capacity

The audience asks whether the video’s audio comes from another model and whether separate outputs can be orchestrated directly or require stitching with a tool such as FFmpeg. Vernade clarifies that Veo itself generated the demonstrated video and audio. In the demonstrated Veo 3 generation, a Lyria audio file cannot be supplied as an input soundtrack. Audio references and conversational edits to just a song’s ending are future directions he would like to see, not capabilities shown here. He advises using a dedicated music model rather than Veo for music; his explanation about limited music training is speculative.

The notebook also contains a service_tier priority setting. Vernade describes Flex as accepting longer waits for lower cost, and Priority as buying preferential handling for a live demonstration. For the workshop configuration, Vernade quotes Flex at half price and Priority at twice the price. These are his contemporaneous figures, not universal multipliers across today’s models. He recommends removing the priority setting for personal experiments, and the ongoing app demonstration still encounters insufficient capacity despite the effort to improve reliability.

Asked about open-weight media models, Vernade personally considers open image and video weights unlikely because hosted systems apply checks both to requests and to generated outputs. He sees a possible case for on-device real-time music, where local execution could reduce latency. Neither answer is presented as a release commitment.

1:18:401:18:45
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:18:40 · section reference included

Fit the model to hardware you control

Ian Ballantyne’s final portion moves from hosted generation to running models on phones, personal computers, or a cloud deployment the developer controls. He introduces Gemma 4 through the four-model lineup available in the workshop. The model card gives more precise counts than his rounded descriptions; its later expanded lineup should not be confused with the one demonstrated here.

The E2B and E4B models target phones, Raspberry Pis, and Jetson-class devices. Their effective size reflects per-layer embeddings that can remain on flash and be paged in when needed. Ballantyne describes roughly 2B and 4B effective cores, versus roughly 5B and 8B when the embeddings are included in RAM. The larger options are 26B A4B, a mixture-of-experts model, and a 31B dense model. Active parameters describe computation per token; they do not mean only the active weights need storage. Precision, quantization, context allocation, and available memory determine whether a larger model fits a laptop, desktop, or single GPU.

Thinking is configurable, and the workshop lineup supports image and video understanding; audio input is limited to the two effective models. Ballantyne describes performance in the range of models ten times larger, but supplies no task, comparator, or benchmark for that claim. His more concrete comparison is the capability mix: Gemma 2 emphasized creative writing, while Gemma 3 and Gemma 4 broadened coding, function calling, and action selection. Those agentic capabilities motivate the phone and coding demonstrations that follow.

1:24:261:24:44
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:24:26 · section reference included

Run skills and small coding loops on a phone

Google AI Edge Gallery, available for Android and iOS, provides a playground for device-local models. In the shown Pixel example, E2B selects among Agent Skills: Android Intents can invoke other apps, JavaScript can implement custom behavior, and WebViews can display interfaces. A research tracker demonstrates both inserting information and retrieving it through functions selected from natural-language requests. Other skills open maps or call services. Local inference decides which action to invoke, even when the surrounding skill uses an external API.

The phone can also host a small coding feedback loop. Ballantyne describes Python, HTML, and TypeScript tasks and shows a generated calculator that correctly produces an error for division by zero. He limits the expectation to small applications, rather than a large architectural system. He believes thinking is disabled in this example and reports that enabling it generally improves reasoning by adding a planning step before output.

1:29:421:29:52
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:29:42 · section reference included

Serve one local model to an agent workload

Gemma integrates with serving tools including Ollama, LM Studio, vLLM, and SGLang. Ballantyne loads 26B in LM Studio on an M4 Mac. For this configuration, he reports about 18 GB for the model and about 22 GB with the demonstrated context allowance. Unified memory makes that allocation available to the Mac’s compute hardware; a discrete GPU needs sufficient VRAM for the full-performance configuration he describes. He attributes 26B’s speed relative to dense 31B to its smaller active parameter set.

LM Studio Developer view with a running server indicator, a loaded model row, supported endpoints, developer logs, and model settings.Open full source frame
LM Studio shows a loaded Gemma model and a running local server.

LM Studio can expose the model through a compatible endpoint, letting an existing client use local inference. Ballantyne serves it on port 1234 and describes pointing a Chat Completions client at that server. The essential Python configuration is the endpoint plus the model identifier exposed by the local server:

python

import os
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:1234/v1",
    api_key="lm-studio",
)

response = client.chat.completions.create(
    model=os.environ["LOCAL_MODEL_ID"],
    messages=[{
        "role": "user",
        "content": "Create a self-contained SVG of a spinning planet.",
    }],
)
print(response.choices[0].message.content)

The application’s inference boundary stays familiar while the computation moves onto the developer’s machine.

The demonstration then runs one orchestrator with ten sub-agents, each assigned a different SVG, with thinking disabled for speed. The displayed throughput is combined generation across the sub-agents, not the speed of one request. The orchestrator assembles the results into a gallery, including an animated spinning planet. Ballantyne suggests the same pattern for file organization, coding, subdivided research, or data analysis; this particular workload can run without an internet connection. He expects more planning with thinking enabled to improve SVG quality, at the cost of additional work.

1:32:471:33:00
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:32:47 · section reference included

Connect OpenCode, then test the generated game

OpenCode uses the same endpoint approach. In the demonstrated setup, Ballantyne edits .config/opencode/config.json to supply the provider, schema, exposed models, endpoint parameters, and local server address. The coding harness itself does not change. When local RAM is insufficient for the desired quantization, he points to a Cloud Run deployment guide and shows—but does not execute—a beta deployment command targeting an RTX PRO 6000. Hosted demos of the two larger models provide another route for testing prompts, file and video inputs, and the Search-grounding integration he describes.

For the live coding task, 31B has already written a specification for Nebula Drift, an endless asteroid-avoidance racer. The local model receives that file in OpenCode with thinking enabled. OpenCode supplies shell, filesystem, and editing tools; the model creates a nebula_drift directory and begins writing an index file, JavaScript, and CSS. It is still making edits when Ballantyne first attempts to open the game. The page appears, but the game does not run.

A syntax error becomes the next input to the agent. Ballantyne manually copies it into the conversation, prompting the model to reread files and repair game.js. OpenCode exposes both targeted edits and whole-file rewrites. The model also has access to sub-agent behavior; Ballantyne finds 31B more reliable at choosing sub-agents, while 26B sometimes needs stronger direction. MCP servers and skills can supply additional capabilities. The repair then encounters an exact-match editing requirement and a possible model unload, so he reloads the model and explicitly requests a complete rewrite of game.js.

1:36:541:37:07
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:36:54 · section reference included

A game inside a game, and a screenshot into HTML

While Nebula Drift is being repaired, an audience member proposes a game in which players build their own game. Ballantyne asks the larger model for a specification, using the earlier spec as a formatting reference. The new project becomes Omniforge. His point is access to this workflow on hardware a developer can own, rather than claiming that specification writing exceeds frontier-model capabilities. Meanwhile, Nebula Drift receives changes to delta time and input handling, but its start button still fails. He requests regenerated files, then asks 31B to implement Omniforge in one index.html containing its CSS and JavaScript.

A third task tests visual understanding: upload a screenshot of the Gemma 4 DeepMind webpage and request a single-file HTML implementation. While that and Omniforge generate, the rebuilt Nebula Drift finally runs with a star field and movable ship. No asteroids are visible, so the racing specification remains only partially implemented. Ballantyne stops iterating on it. He also notes that the hosted demos share serving capacity and usage limits, which can make them slower than the Gemini models.

1:44:431:44:49
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:44:43 · section reference included

Map language onto robot and development tools

Open Duck extends the same action-selection mechanism to a robot simulator. E2B downloads into the browser and runs through WebGPU. It interprets a natural-language request, selects from the robot’s supported actions, and displays the intended command before the simulator performs it. The shown interface pairs conversational requests with robot movement and manual controls. Ballantyne suggests embedded execution as a possible extension; the demonstration itself runs in the simulator.

Browser-based Open Duck Mini simulator showing a turned robot beside commands including “walk forwards” and “look at me! I’m on your right!”, with action responses and manual controls.Open full source frame
The Open Duck simulator pairs robot movement with conversational commands.

Other integrations put the model into established development environments. An Android Studio example uses 26B in a chat-based application builder. The Agent Development Kit integration supplies a thinking loop and feedback system for longer-running tasks, including decisions about retrieving information. Across these examples, the host environment exposes the actions, while the model interprets requests and selects how to use them.

1:51:001:51:10
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:51:00 · section reference included

The final test is an interaction

The webpage reconstruction finishes first. Its layout and fonts resemble the uploaded screenshot closely enough to impress Ballantyne, but button destinations are not established, and an element that momentarily looks like video is not a generated video. This is a successful visual reconstruction, with interaction behavior still largely untested.

Omniforge supplies the more tangible ending. The generated page opens as an editor, accepts object placement, and runs a scene in which a triangle moves. Those interactions establish more than a convincing screenshot, even though they do not test a complete game-development environment. After the partly working racer and the closer webpage reconstruction, the miniature editor shows the value of trying the artifact itself. Ballantyne closes with the available deployment range—phones, laptops, and GPUs—and points readers toward AI Edge Gallery as a place to begin exploring.

1:52:451:52:56
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:52:45 · section reference included

Resources

From the talk

Updates since the talk

  • Gemini API pricingDocumentation

    Current model prices, billing units, and service-tier tables.

  • Gemma 4 model cardDocumentation

    Model architectures, parameter counts, modalities, and evaluations for the expanded Gemma 4 family.

Transcript download
Full transcript

Loading transcript…