AI Engineer World's Fair 2025
Magic Editor Under the Hood: Weaving Generative AI into a Billion-User App
Loading slides…
Read the talk
Magic Editor Under the Hood: Weaving Generative AI into a Billion-User App
A dark sunset portrait captures the promise of computational photography. Delivering that promise at Google Photos scale requires constrained tasks, maintained evaluations and careful control of model uncertainty.
From a talk by Kelvin Ma
Before you start: Basic familiarity with image pixels, machine-learning inference and client/server applications will help; no computer-vision background is required.
Make the photo useful after capture
Can a photo app find a restaurant receipt without making you organize your pictures first? Google Photos was built around that kind of problem: automatic backup feeds image indexing, OCR and machine-learning search. A trip can become an automatically assembled album; a receipt can become searchable. Kelvin Ma, an engineer on the editing team, reports about 1.5 billion monthly active users and hundreds of millions of edits per month across clients.
The computational photography team, formed in 2018, brought the same approach to editing. A phone may have a limited sensor, but it has substantial compute—and an editor can apply that compute to pictures from older devices or other cameras, too. Consider the sunset portrait: the background is bright, but the subject is dark. Traditional DSLR HDR combines photographs taken at different exposures, potentially requiring a tripod and manual assembly in Photoshop. Ma’s demonstration instead enhances one existing image, restoring brightness and vibrance after capture.
Google could develop this through vertical integration: Pixel hardware and Edge TPU acceleration on one side, internal computer-vision researchers on the other. In 2018, Ma recalls, developers could not browse Hugging Face for a suitable model as readily as they can today. Researchers and application engineers instead iterated together, improving the model and the interaction until the feature became useful and easy to operate.
The original stack paired native Android, iOS and web clients with a shared C++ inference library. Model inference ran on-device through TensorFlow Lite, renamed LiteRT. This local execution model shaped both the features the team could build and the operational problems it had to solve.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Start with specific photographic problems
The early editing tools each addressed a recognizable defect in a photograph:
- Background blur: Post-capture portrait segmentation lets the editor add bokeh even when the photographer lacked a lens that could produce it at capture time.
- Portrait lighting: Relighting can address a washed-out face or unwanted shadows, including changing the apparent direction of illumination.
- Background distractions: Magic Eraser identifies important subjects and unwanted background objects, removes the distractions and inpaints the exposed regions. The photographer can take the picture without waiting for tourists to leave the scene.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A mask is an output, not a guarantee
The first major feature, post-capture segmentation, used a U-Net convolutional neural network scoped to single-subject portraits. Its output separates foreground from background, with the subject represented in white on the mask. Traditional computer vision could also attempt this separation, but it required specialists to tune the system. Machine learning shifted much of that work toward assembling a dataset, defining a benchmark and training against it.
A mask gives downstream image processing a concrete interface. For background blur, the following C++ composition keeps the original pixel where the foreground mask is white and uses an already-blurred image where it is black. Intermediate mask values blend the two at boundaries:
cpp
#include <algorithm>
#include <stdexcept>
#include <vector>
struct Rgb {
float r, g, b;
};
std::vector<Rgb> compositePortrait(
const std::vector<Rgb>& original,
const std::vector<Rgb>& blurred,
const std::vector<float>& foregroundMask) {
if (original.size() != blurred.size() ||
original.size() != foregroundMask.size()) {
throw std::invalid_argument("Image and mask sizes must match");
}
std::vector<Rgb> result(original.size());
for (std::size_t i = 0; i < result.size(); ++i) {
const float a = std::clamp(foregroundMask[i], 0.0f, 1.0f);
result[i] = {
a * original[i].r + (1.0f - a) * blurred[i].r,
a * original[i].g + (1.0f - a) * blurred[i].g,
a * original[i].b + (1.0f - a) * blurred[i].b
};
}
return result;
}
This makes mask quality consequential: pixels assigned to the background receive the background treatment, even when they belong to the subject.
A model returning an output for every input simplifies the application’s control flow, but shipping the model introduces another set of concerns. Ma reports that the segmentation model was 10 MB—already too large to bundle under the team’s APK-size constraints. Downloading it after installation meant managing model availability and versions, as well as protecting the on-device model against extraction and reuse.
Benchmarks are the model’s regression tests. They need maintenance just as code tests do: the team must run them, compare successive models and ensure that the dataset still represents real-world usage. A benchmark that drifts away from actual photographs stops answering the question the product team needs answered.
The apparent advantage of always returning a result is also a failure mode. In the portrait example, the mask misses fine strands of long hair. Applying a strong blur or sharp separation then exposes imperfect edges. Ma describes correcting this after inference with traditional image understanding that follows the hair strands. The useful editing system therefore includes both a learned segmentation stage and conventional processing around it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Magic Eraser becomes a system of models
By the 2021 launch of Magic Eraser, the feature required several coordinated stages:
- Detect distracting objects.
- Segment those objects to identify the regions to remove.
- Inpaint the selected regions with replacement image content.
- Use custom on-device GL rendering to display the masks, animate their removal and reveal the inpainted areas.
The visible interaction depended on the rendering system as well as the model outputs.
Ma reports that on-device models had grown into the hundreds of megabytes by this stage. More models meant more system complexity, while more ambitious edits made failures easier to notice. In the 2021 implementation, removing a large, prominent foreground object could ask the inpainter to reconstruct more of the scene than it could handle convincingly.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Predictable inference, unpredictable fixes
These models made powerful edits easy to request, and their execution was relatively predictable. On a particular device, a model performed the same amount of computation regardless of the image, giving it consistent latency. Across Android devices, however, performance varied substantially: recent Samsung and Pixel phones could resemble some laptops in capability, while older phones could not.
Research partnerships helped the team develop capabilities over years. Engineers could bring a user need to researchers before a model was ready, follow successive improvements and keep refining the application around them. But an individual failure was rarely as straightforward to repair as a conventional software bug. Two images could look nearly identical to a person and produce sharply different model results.
The response might be to collect more data and retrain, then wait weeks or a month to discover whether the problem improved. That uncertainty collides with hardware launch schedules. In Ma’s planning example, two months before a Pixel launch might allow at best two model iterations. A launch commitment has to account for that slow, uncertain feedback loop, rather than assuming an engineer can locate a bug and immediately ship a fix.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A generative editor for recognizable memories
The excitement around DALL-E and ChatGPT in 2022–2023 changed the ambition of image editing. Ma marks the transition with a Google I/O montage repeatedly saying “AI”: larger models brought more possibilities, but also a less clearly bounded problem. That enthusiasm helped motivate Magic Editor, though it was not the only reason to build it.
The existing tools had a discoverability problem. Users needed to know when to choose Magic Eraser, when to blur the background and where to find each operation. AI could help identify the relevant edit instead of requiring the user to understand the tool collection first.
The product still needed to fit a library of personal memories. Rather than emphasizing fantastical images, Photos aimed for edits that would remain plausible when shared with friends and family. Specific prompts combined with visible selection would let users show the model what they meant and interact with it as a co-editor. Supporting more ambitious edits with the best available models also meant leaving the original on-device constraint.
That introduced servers, but it also exposed a product-definition problem: generative image editing is not a specific user need. Even within image editing, an unrestricted model can attempt many different tasks. The team had to choose which needs to serve, while making deepfake prevention and responsible use part of the requirements.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Server inference changes the operating costs
In the local-first system, users supplied inference compute and power. More people using the editor did not require the team to provision more inference servers. Moving to accelerated server compute introduced TPU and GPU capacity planning, with substantial planning time. Network conditions and data-center location also became part of the editing experience.
| Concern | On-device editing | Server-backed editing |
|---|---|---|
| Capacity | User supplies compute | Provision TPUs and GPUs |
| Latency | No inference network trip | Network, distance and queueing |
| Regression testing | Include small models in tests | Large models exceed the daily suite’s limits |
A remote user, an overloaded data center or a long geographic round trip can now delay an edit. Testing becomes harder, too: Ma says the larger models could no longer run inside the daily automated regression suite. Replaying captured responses and using a test server were possible approaches, but he presents neither as a satisfactory settled solution.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A successful demo does not define a shippable task
The old xkcd comic “Tasks” contrasts two requests that sound similarly simple: determine whether someone is in a national park, and determine whether a photograph contains a bird. The first is a geographic lookup; historically, the second could require a computer-vision research team. Models have changed that boundary. A hackathon participant can now build an idea that works in at least some cases, and a product manager can demonstrate a desired behavior with Gemini or ChatGPT.
The engineering question is how often it works. Ma uses illustrative success rates of 5%, 10%, 50% and 80% to distinguish a compelling example from a dependable capability. He rejects shipping something that works only 5% of the time; a capability that works half the time and has a credible research path toward 80% is at least worth discussing. These are hypothetical planning examples, not measured Magic Editor results.
Constraining the task also reduces the burden on the user. Ma agrees with the formulation “prompt is a bug, not a feature”: someone editing a photograph on a phone should not need to write a detailed paragraph. Prompt editing and intent extraction can translate a small amount of input into a more useful request. But the system still needs boundaries. If anything is possible and nothing is out of scope, engineers have no stable contract around which to design it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn user intent into supported edits
User research narrowed the broad editing space to concrete demands:
- Relocate: Move an object within the image seamlessly.
- Reimagine: Replace a dull gray-sky background with a more interesting scene.
- Erase: Remove an unwanted object and the visual evidence that belongs to it. In Ma’s example, the stronger eraser removes both a drink and its reflection, producing a more natural result than removing the drink alone.
Those tasks give research and product engineering something specific to optimize. Researchers can emphasize or train models for the chosen cases; application engineers can build guided interactions that make those cases easier to use reliably. For other requests, the editor can offer multiple generated alternatives. Creative editing has no single correct image, so variation that would be a defect in a factual answer can become useful choice: the user selects a result they like.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Safety has to handle ambiguous intent
Creative flexibility does not remove the need for trust and safety. Ma describes a continuing effort governed by precision and recall targets, rather than a promise that every decision will be correct. The team must prevent severe cases while setting realistic expectations with users, journalists and internal stakeholders.
Language itself complicates that work. A user might say the view from a mountain was “sick” and ask the editor to make it look that way. Does the word mean impressive, or does it refer to illness? The intended edit depends on interpreting that ambiguity correctly. Restricting the interaction can reduce uncertainty, but free-form language keeps introducing it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Use production learning to simplify the system
Ma describes AI engineering as software engineering with machine learning adding randomness to the system. The engineering responsibility is to reduce that randomness enough to produce repeatable outcomes that users value. The goal is dependable behavior, not the largest possible model.
That suggests a practical progression:
- Build evaluations, use them and make them faster to run.
- Establish that the capability provides useful value in production.
- Replace an oversized model with a smaller, faster or more efficient implementation where the evaluations support it.
The replacement might come from distillation, a different model or traditional engineering without machine learning. Each can shorten the iteration loop, giving the team more attempts to improve the product.
The next editor design extends this approach to the interaction itself. Google’s May 28, 2025 tenth-anniversary announcement introduced a rebuilt, AI-first editor; Ma describes it as newly announced at the time of the talk. Users would tap an image to surface relevant edits and adjust them. They could invoke AI as a tool, while AI could also invoke deterministic editing tools to produce a better result.
Where that compute will run remains uncertain. In his personal two-year outlook, Ma asks whether Gemini Nano has reached the capability of a previous Gemini Pro, citing something he has heard rather than a measured comparison. That possibility makes a return to on-device inference exciting to him. Alternatively, larger models—including a hypothetical Gemini 4—might keep advancing while smaller models remain behind, or progress might stall. Across those futures, fast iteration and strong benchmarks still tell the team what to change.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
The May 2025 announcement explains suggested edits, selection-based tools and the planned redesigned-editor rollout.
Google’s original preview illustrates subject relocation, background changes and filling missing image content.
The launch explanation covers distraction detection, user selection and replacement-pixel generation.
Ronneberger, Fischer and Brox introduce an encoder-decoder architecture for precise image segmentation.
The comic contrasts a straightforward geographic lookup with the historically difficult task of recognizing a bird.
Further reading
Google Research explains Ultra HDR gain maps and a lightweight model that reconstructs them after image edits.
A May 2025 implementation guide explains task-specific adaptation, evaluation and Kotlin usage for on-device text and image-description features.
Updates since the talk
Current documentation for deploying machine-learning models on devices with LiteRT.
Full transcript
Loading transcript…