AI Engineer World's Fair 2026

Agent Output Is Not UX: Rendering Layer Your LLM Pipeline Is Missing - Bala Ramdoss, Amazon Lens

Loading slides…

Read the talk

Agent Output Is Not UX: Building the Rendering Layer

A capable agent can find the right information and still leave the user doing the work. Typed UI contracts, streaming states and a backend for frontend turn that output into an usable interface.

From a talk by Bala Ramdoss

Before you start: Familiarity with API responses, frontend components and basic LLM application architecture will help.

The reservation is still unfinished

Ask an AI assistant to reserve a restaurant table, and it may return an accurate phone number, opening hours and even details about the walk-in oyster bar. The information is useful, but the reservation remains unfinished. The user still has to turn the answer into a booking.

The opening demonstration replaces that answer with a reservation card: choose a date, party size and available time, then use the booking button. The fictional Zuni Cafe mockup illustrates the difference between supplying information and providing the controls to act on it. It does not establish that a reservation was actually completed. The missing layer sits between the model’s output and the interaction a person needs. An agent may already have the tools; the product still needs to make those capabilities usable.

Chat mockup with a Zuni Cafe reservation card, date and party-size selectors, available times and a booking button. Footer says “Fictional, not real Claude.”Open full source frame
A fictional Zuni Cafe reservation card turns the request into date, party-size and time controls.

Bala Ramdoss approaches this problem from more than a decade building customer-facing applications, including six years working on Amazon Lens. Lens supports shopping with images, screenshots and barcodes to discover visually similar products. His experience with applications reaching millions of mobile devices informs the architecture here; he presents these views personally, not on behalf of his employer.

0:010:09
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:01 · section reference included

Describe the interface as data

Interfaces such as ChatGPT make the design questions visible: should an answer become cards, a carousel or a list? But presentation decisions depend on earlier architectural choices. Will the experience respond promptly? Will information arrive together or incrementally? Can the same output work across mobile app versions and device capabilities? These are delivery problems between model output and the screen, and a renderer alone cannot resolve them after the response has already been generated.

Ramdoss previously solved this layer separately for each system. Generative UI gives those bespoke implementations a shared vocabulary, and Google’s A2UI provides an open specification for one approach: the agent describes components as data instead of returning raw text or HTML, and the client renders that description with its own native widgets. The model can express what interface it intends without supplying the implementation of every control.

Slide titled “The layer has a name” with “generative UI,” an A2UI component-payload description, and a footer describing the standard as open, declarative and built for streaming.Open full source frame
Generative UI: A2UI defines the components in the payload.

A conventional API returns data and leaves presentation to the client. Generative UI gives the model some responsibility for that presentation. Ramdoss uses CopilotKit’s spectrum to distinguish three degrees of freedom:

ApproachModel’s responsibilityExample
ControlledSelect a prebuilt componentProduct card
DeclarativeCompose components from a catalogDate field, time field, submit button
Open-endedProduce a novel interfaceGenerated UI

A2UI occupies the declarative middle. Ramdoss associates the open-ended end with MCP Apps, although that protocol supports server-supplied, sandboxed HTML interfaces without requiring a model to generate them at runtime. As freedom increases, so does the client’s trust burden. His focus is the controlled and declarative approaches, which he considers the practical territory for most production mobile apps.

1:562:12
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:56 · section reference included

Design for clients you cannot promptly patch

On the web, a renderer fix can reach users quickly. Mobile developers do not control when installed applications update. Ramdoss frames the challenge at the scale of hundreds of millions of installs: an unprepared client encountering an unknown content type can crash, and that failure can persist for days or weeks among users who have not updated. This is a risk to design against, not an inevitable property of mobile rendering. The operational constraint is that you cannot rely on promptly patching every client.

That constraint shapes the whole pipeline:

  1. Build version-aware context describing the client’s capabilities.
  2. Let the model produce typed UI intent within those capabilities.
  3. Pass the intent through a backend for frontend, or BFF.
  4. Have the client renderer draw the result and fall back safely when it cannot.

The architecture divides into three connected patterns: the rendering contract defines what can be selected, streaming governs how output arrives, and the BFF sits between model output and the client.

5:065:18
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:06 · section reference included

Constrain component selection before generation

The rendering contract makes the model aware of the client it is addressing. The client should not have to examine an untyped stream of tokens and infer which interface the model meant to show. Maintain a repository mapping application versions to capabilities, then use that map when constructing model context. If a flight card first ships in app version 2.0, expose it to the model only for clients running 2.0 or later. This is an illustrative application version, not an A2UI protocol version. The model chooses the customer experience from the options that context permits.

The model-facing half of the contract separates conversational blocks—what the agent says—from UI blocks—what the client should render. It also encodes layout rules. In the flight example, one to three flights belong in a swipeable carousel; four or more belong in a vertical list. The following JSON expresses those example rules as application-level contract data:

json

{
  "blocks": ["conversation", "ui"],
  "components": [
    {
      "name": "flightCard",
      "minimumAppVersion": "2.0"
    }
  ],
  "flightLayouts": [
    {
      "minimumItems": 1,
      "maximumItems": 3,
      "layout": "carousel"
    },
    {
      "minimumItems": 4,
      "layout": "verticalList"
    }
  ]
}

The version map determines whether flightCard enters the model’s available catalog; the layout rules tell the model how to arrange the flight results.

The model never invents a component in this pattern. It selects from a fixed menu supplied by the application, ideally a handful of relevant choices. That restriction is straightforward with a small catalog, but becomes harder as features expand across more surfaces. The continuing context-engineering problem is deciding which supported components and rules to expose so that selection remains reliable.

6:356:48
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:35 · section reference included

Stream useful component states

Traditional applications often make an API call and wait for a complete response. With an LLM, generation latency is compounded by safety checks and the extra work needed to produce dynamic UI. Streaming lets the client render chunks instead of withholding the entire interface until all of that work finishes.

The flight-card demonstration progresses through three states: a skeleton establishes the component’s shape, partial content fills it in, and the complete card supplies the remaining information. The slide labels the delivery mechanism as chunks over SSE. In the illustrative progression, completing the card may take three to four seconds. That duration is an example of a tolerable progressive wait, not a measured performance guarantee.

Slide titled “Streaming into typed UI” with “Chunks over SSE” and three cards showing placeholder bars, partial flight information and a populated flight card.Open full source frame
Three card states illustrate streaming from a skeleton to populated content.

This changes what the application should optimize. Time to the first useful chunk captures when the user first receives something meaningful, while total latency describes when everything is finished. Ramdoss advocates prioritizing the former rather than judging the experience only by the latter. A spinner communicates that work is pending, but supplies neither useful content nor an explanation of progress.

8:428:58
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:42 · section reference included

Give the wait a useful interaction

Sometimes the product can make the wait interactive. Amazon Lens Live lets users focus on different things and tap an object of interest while results arrive. The user continues directing the task instead of staring at a loading indicator. This depends on the product: not every feature has an activity that remains useful while processing continues.

When the feature does not control the full screen, task-status feedback can serve a narrower purpose. Ramdoss recommends using thinking indicators sparingly: users increasingly expect to understand what is happening, not merely be told that the system is busy. His Gemini example shows glimpses of the agent’s activity as it works. Ramdoss describes personally accepting a ten-second wait when that progress is visible. This is his tolerance in the example, not a universal usability threshold; the useful mechanism is streaming states that help the user understand the work and assess the eventual output.

10:1310:26
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:13 · section reference included

Turn UI intent into actionable native components

The contract determines which UI gets selected; streaming delivers the chunks. The BFF makes those chunks meaningful to the application. Ramdoss places this responsibility within server-driven UI, where increasing server control also increases the implementation the server must own. Here, the BFF centralizes platform-specific rules, including differences between Android and iOS, so each client can make fewer presentation decisions and draw what it receives. Layout ownership is a responsibility of this proposed BFF, not a requirement of every backend-for-frontend architecture.

Formatting is only part of that work. The BFF also hydrates the output with the information needed by the rendered components and adds actions. Each rendered element carries an action payload describing what a tap does, which deep link it opens and potentially which impression metric to log. It also carries conversational context across turns so that the next response knows what came before. The output becomes part of an ongoing interaction rather than an isolated arrangement of cards.

Existing applications can adopt this layer by reusing components already in production: flight rows, product cards and other familiar units. The agent chooses among those units while the application retains its brand, information density and native feel. There is no need to introduce a separate visual identity simply because an agent now helps select the interface.

The responsibilities reinforce one another: a typed, versioned contract gives the model supported choices; streamed components make useful output and agent activity visible; and the BFF absorbs the adaptation work that would otherwise complicate every client. The restaurant example returns the architecture to its purpose. Finding the right information is only part of reserving a table. The delivery layer turns that information into an experience through which the user can finish the task.

Takeaways slide with three bullets about models choosing CX, streaming typed components rather than text, and a BFF absorbing the model while the client stays dumb and safe.Open full source frame
Takeaways: choose the customer experience, stream typed components and keep clients simple and safe.
11:1811:33
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:18 · section reference included

Resources

From the talk

  • Declarative UI descriptions, component catalogs, native rendering and streamed interface updates, with links to implementation guides.

  • An interactive explanation of controlled components, declarative composition and open-ended interfaces.

  • How MCP tools supply interactive HTML resources, communicate with hosts and operate within sandbox restrictions.

  • Amazon's overview of live camera shopping, product matching and tap-to-focus interaction in the Shopping app.

Transcript download
Full transcript

Loading transcript…