AI Engineer World's Fair 2025

How agents broke app-level infrastructure

Evan Boyle13:32

Loading slides…

Read the talk

How agents broke app-level infrastructure

As prompts become long-running workflows, compute must outlive browser connections, preserve progress, and recover from failures without making users start over.

From a talk by Evan Boyle

Before you start: Familiarity with TypeScript, HTTP APIs, and background jobs will help; the article explains the Redis Streams mechanism as it appears.

From prompts to data pipelines

What happens to the compute layer when a simple prompt grows into a data pipeline? Evan Boyle approaches that question as GenSX’s founder and CEO and an early Pulumi employee. His diagrams were generated with GPT-4o, accompanied by a warning about its spelling. The infrastructure problem starts with something equally familiar: a prompt and a few tool calls—enough, he jokes, to raise a ten-million-dollar seed round.

Turning that prototype into a reliable product means controlling more of its behavior. Instead of leaving everything to one nondeterministic interaction, developers chain tailored prompts, write evals against them, and manage the context each receives. Boyle illustrates the resulting runtime growth as thirty seconds becoming a couple of minutes. Eventually, context becomes the larger engineering problem: crawling a user’s inbox or ingesting GitHub code creates substantial additional LLM processing. Improving the agent has also turned the application into a data pipeline.

Three illustrated stages labeled Simple prompt, Build a workflow, and Process large amounts of data, connected by arrows.Open full source frame
From a simple prompt to workflows and large-scale data processing.
0:000:11
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:00 · section reference included

Longer requests, unreliable dependencies

The old request model assumed a short trip through an API and database. Boyle contrasts a conventional web response taking a few tens of milliseconds with an AI response taking a couple of seconds at best, conditional on a fast model or prompt-cache hit. He calls the latter “P1,” without defining the statistic. A delay that once might have paged the on-call engineer is now part of the expected product experience. Infrastructure designed around short requests inherits the wrong assumptions about how long work stays active.

Longer execution also leaves more opportunities to encounter dependency failures. Boyle walks through a Slack channel subscribed to provider incidents: one outage lasts roughly an hour and resolves by 10 PM, then trouble returns at 1:20 AM and remains at 4 AM. He challenges a displayed 99.9% uptime figure by pointing to yellow and red status-history bars, and reports that a Gemini outage overlapped the illustrated OpenAI outage. The example makes failover less reassuring: another provider is useful only if it is available when needed. The displayed history does not establish a comparable availability measurement across providers.

OpenAI status page with a fully operational banner above API and ChatGPT history bars containing green, yellow, and red segments.Open full source frame
OpenAI’s status page shows interruptions across its API and ChatGPT history.

Even when providers are healthy, onboarding customers or processing document batches produces bursty demand. Boyle describes high usage tiers as a spending barrier to experimentation, saying Tier 5 required thousands of dollars. That historical figure should not be used as a current requirement: the OpenAI rate-limit documentation, checked in August 2026, lists $1,000 paid for Tier 5 qualification. Qualification spending, monthly usage ceilings, and model-specific request or token throughput are separate constraints. The engineering problem remains absorbing bursts without treating every rate-limit response as a failed user operation.

1:221:38
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:22 · section reference included

The gap between a chatbot template and a workflow engine

A Next.js chatbot template makes the first version easy. Once its work lasts minutes or hours, the full-stack engineer inherits problems traditionally handled by data infrastructure. Existing tools offer several starting points:

ToolStarting point
SQSQueues around which to assemble orchestration
AirflowBatch-processing workflows
TemporalDurable execution

Boyle’s objection is partly about developer experience. As a TypeScript and full-stack developer, he would rather retain the simplicity of serverless than assemble a queue-based orchestration system. His joke about avoiding tools designed by Java engineers expresses that preference, rather than a technical evaluation of those tools.

Serverless introduces its own constraints. Boyle describes five-minute timeouts and, on some platforms, limits on outgoing HTTP requests. He also describes streaming as work bolted onto the application. These are not universal platform properties: Lambda supports execution timeouts up to fifteen minutes, and native Lambda response streaming arrived in 2023 through function URLs and its invocation API, with integration restrictions. Streaming a response and preserving a reconnectable history are different capabilities. A user refreshing a page during a long operation needs the latter, even if the platform already supports the former.

3:143:30
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:14 · section reference included

Let users start before ingestion finishes

Website onboarding makes the lifecycle problem concrete. The user supplies a URL, and an initial LLM call extracts information from that page and identifies other pages to scrape. That first interaction starts a much larger background operation. In Boyle’s example, scraping and enrichment take multiple minutes and involve hundreds of LLM calls.

The product should become usable while that work continues. Boyle presents a ten-minute onboarding wait as a risk to funnel completion, not as a measured conversion result. Moving ingestion into the background removes the wait, but creates a communication requirement: show its status inside the product. Users need to understand that more data is arriving and that the experience will improve as enrichment progresses. Background execution and visible progress are therefore parts of the same onboarding design.

4:354:51
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:35 · section reference included

A writing agent makes progress part of the product

The second product example generates a blog post about building high-performance web applications with Vercel and Next.js. It tells the user up front that the operation will take a couple of minutes. A progress panel then exposes the work: research, outline generation, and writing individual sections. Those intermediate steps give the user something meaningful to follow, but they must also survive navigation away from the page.

Generating your article message beside a stack of Research Agent and Content Writer Agent status cards describing research and individual article sections.Open full source frame
Article generation displays research and section-writing progress while the user waits.

Boyle reports that this workflow in a previous product took three to four minutes. Experimenting with workflows beyond five minutes would have required substantial changes to its serverless infrastructure, so the team delayed those experiments. They eventually migrated, but the work involved had already constrained product development. A runtime ceiling had become a limit on what the team was willing to try.

The application needs to stream both intermediate status and final content, then restore both after a refresh, as users expect from deep-research experiences. Recovery matters more as the work gets longer: Boyle describes an intermediate error potentially making a user repeat five minutes of work. That is a different cost from refreshing a quick webpage or resubmitting a form. Preserving progress is part of making the operation tolerable, not merely an observability feature for developers.

5:425:56
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:42 · section reference included

Infrastructure-aware components

The infrastructure built for that earlier product became the open-source GenSX library. Its central design couples the component model to the execution environment: the framework knows about the infrastructure, and the infrastructure knows about the framework. That relationship supports resumable streams for intermediate status and final output.

Boyle describes an unopinionated anti-framework inspired by React’s component model, applied to backend execution. The aim is to share and compose code without hiding the decisions that matter when building agents. Components are reusable, independently testable steps, described as idempotent; workflows collect those steps into a larger operation. Treating a step as idempotent still requires its implementation to tolerate repetition—especially if it changes an external system.

Slide listing an anti-framework approach, React-inspired composition, reusable components, workflows, plain functions, resumable streams, tracing, retries, caching, and durability.Open full source frame
An infrastructure-aware component model combines reusable steps, workflows, and resumable streams.

The demonstrated component uses a wrapped OpenAI SDK. The wrapper retains the familiar SDK surface and access to model parameters while adding tracing and retry tooling. The component itself remains a small function: accept a prompt and context, call the model, and return a response. It can be invoked or tested independently before becoming part of a larger workflow.

7:187:30
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

7:18 · section reference included

Compose the work, expose its boundaries

The demonstrated workflow fetches the Hacker News front page, analyzes its posts, generates reports, writes a tweet, and then edits the report. Ordinary TypeScript can express that sequence while leaving each operation independently replaceable and testable:

typescript

type Post = { title: string; url: string };
type Analysis = { summary: string; sources: Post[] };

type Steps = {
  fetchFrontPage(): Promise<Post[]>;
  analyzePosts(posts: Post[]): Promise<Analysis>;
  generateReports(analysis: Analysis): Promise<string>;
  writeTweet(report: string): Promise<string>;
  editReport(report: string): Promise<string>;
};

async function runHackerNewsWorkflow(steps: Steps) {
  const posts = await steps.fetchFrontPage();
  const analysis = await steps.analyzePosts(posts);
  const report = await steps.generateReports(analysis);
  const tweet = await steps.writeTweet(report);
  const editedReport = await steps.editReport(report);

  return { report: editedReport, tweet };
}

This expresses the composition without depending on a particular version of the GenSX API. The infrastructure-aware component model adds execution behavior around the individual calls.

Each component supplies a retry boundary and an error boundary. Tracing covers both the enclosing workflow and the components nested inside it. The framework also turns workflows into REST APIs for synchronous invocation, asynchronous invocation, and retrieval of intermediate and final output streams. The same composed operation can therefore serve an immediate request or run as a background job.

The trace view exposes nested components alongside token information, OpenAI call details, and user and system messages. This makes it possible to investigate a particular model interaction within the larger workflow. For a troublesome component, the demonstrated fluent configuration API supports retries and caching, including an exponential retry policy with a configurable retry count. The policy belongs to the problematic step rather than being applied indiscriminately to the entire workflow.

8:438:55
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:43 · section reference included

Separate execution from the API connection

The deployment platform separates the API layer from the compute layer. Rather than keeping a request handler attached to a worker for the lifetime of a job, it uses a Redis Stream as the communication path:

  1. The API invokes compute and passes it a Redis Stream ID.
  2. The executing sandbox writes status and output to that stream.
  3. The API obtains subsequent status and output through the stream rather than continued direct communication with compute.
  4. The sandbox emits heartbeats that background processes monitor. If execution dies or crashes, the system can restart it or notify the user.

The browser connection no longer has to be the channel that holds the operation together.

This separation also creates independent scaling boundaries. The API can scale with client connections while compute scales with workflow execution. Boyle describes ECS as the default compute backend, with the option to bring another compute layer. Workers are stateless with respect to stored output: they communicate with the surrounding platform and write results to Redis Streams, rather than requiring the API to retrieve output from a particular worker’s memory.

10:0110:14
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:01 · section reference included

Reconnect to stored progress

Both synchronous and asynchronous workflows write their output to Redis Streams. Because the API reads that stored output instead of reading directly from compute, a replacement browser connection can retrieve the history of status messages and generated content. Refreshing the page, navigating away, or terminating the browser’s connection to the API does not itself cancel the work.

For a concrete illustration, suppose the writing agent has stored Research complete at entry 1000-0 and Outline complete at 1001-0. The browser has displayed both and saved 1001-0 as its cursor. While it is disconnected, the worker adds Section 1 draft at 1002-0. On reconnect, reading after 1001-0 retrieves the missing draft; the earlier entries remain stored. Redis XREAD supports this last-seen-ID pattern. A refreshed client that has lost its local content instead needs to rebuild from retained history. Replaying output is distinct from resuming failed computation: retention and Redis persistence determine what remains available, while execution recovery and safe repetition require their own mechanisms.

Retrieve the draft produced while the browser was disconnected

Constructed example: The entry IDs, message text, saved browser cursor, and retained browser state are teaching values extending the writing-agent example. The comparison illustrates output replay, not recovery of failed computation.

Reconnect request — unchanged
Writing-agent stream; last-seen entry ID: 1001-0

Operation: Read stored entries after 1001-0, display the missing draft, and advance the browser cursor.

Redis entry 1000-0

Before: Disconnected browser; draft already stored
Research complete
After: Browser has replayed the missing entry · Unchanged
Research complete

Redis entry 1001-0

Before: Disconnected browser; draft already stored
Outline complete
After: Browser has replayed the missing entry · Unchanged
Outline complete

Redis entry 1002-0

Before: Disconnected browser; draft already stored
Section 1 draft
After: Browser has replayed the missing entry · Unchanged
Section 1 draft

Browser's retained research status

Before: Disconnected browser; draft already stored
Research complete
After: Browser has replayed the missing entry · Unchanged
Research complete

Browser's retained outline status

Before: Disconnected browser; draft already stored
Outline complete
After: Browser has replayed the missing entry · Unchanged
Outline complete

Browser's displayed draft

Before: Disconnected browser; draft already stored
Not present
After: Browser has replayed the missing entry · Added
Section 1 draft
Reconnection retrieves stored output without asking the worker to generate it again.
11:2311:41
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:23 · section reference included

Start small, but leave room for long-running work

There is no need to build for an hour-long workflow on day one. Start with the operation the product actually needs, while leaving room for longer execution. Coding agents and deep-research experiences motivate Boyle’s expectation that more work will become delegated: give an agent instructions, let it operate independently, and have it communicate when needed. Separating compute from the API and retaining progress in a stream allows that work to continue without requiring the user to keep a page open.

That independence also changes deployment. Safe navigation and transparent error handling are incomplete if a routine release kills active work. Boyle uses sixty-minute workflows to emphasize the need for careful worker draining and a blue-green deployment pattern. A deployment must account for workers that are still busy, not just whether replacement instances are healthy. Getting the first workflow running is easy compared with getting its lifecycle right.

Boyle closes by offering GenSX as an implementation of much of this infrastructure for developers who do not want to build it themselves. The linked repository was archived on October 2, 2025; it remains a reference for the design, rather than evidence that the hosted platform is currently available.

12:0712:19
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

12:07 · section reference included

Resources

From the talk

  • TypeScript workflow framework with composable components, tracing, and agent examples. The repository was archived in October 2025.

Updates since the talk

  • GenSX Cloud launchArticle

    Evan Boyle's subsequent launch article explains the 60-minute runtime, background invocation, storage, tracing, and durable progress streams.

  • Current usage-tier qualifications, request and token limits, response headers, and retry guidance.

Transcript download
Full transcript

Loading transcript…