AI Engineer World's Fair 2026
Emulated: The data for fully autonomous software engineers and companies
Loading slides…
Read the talk
Emulated: The data for fully autonomous software engineers and companies
Reliable infrastructure agents need environments that expose deployment failures, customer context, and cloud operations—not just the source code they must change.
From a talk by Joseph Wang and Sid Patllollu
Before you start: Familiarity with containers, distributed services, and rolling deployments will help; no reinforcement-learning implementation experience is required.
Why can an agent write the application but struggle with its infrastructure?
What would it take for an agent to operate mission-critical infrastructure with little supervision—and keep making sound architectural decisions over years? That question motivates Emulated, the data lab Joseph Wang and Sid introduce through their backgrounds in network infrastructure, distributed databases, and sandboxes. Longer autonomy requires more than completing a larger programming assignment: it requires understanding the consequences of changing a running system.
The immediate puzzle is the difference between application work and infrastructure reasoning. An agent can be proficient at the application layer while struggling with database mechanisms such as multiversion concurrency control, or MVCC, where mistakes can threaten data integrity. Sid connects this concern to a recent DynamoDB failure. That attribution needs a qualification: if he means the October 2025 disruption, AWS’s incident account identifies a DNS-management race that emptied an endpoint record, rather than database-engine MVCC corruption.
The proposed capability gap is a data gap. Joseph’s premise is that richer, higher-quality data can teach capabilities missing from current models. His stronger assertion—that adding high-quality data never causes capability regression—is a motivating thesis, not a universal guarantee established here. The practical question is what experience an agent needs in order to learn infrastructure ownership.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Expand the task from a repository to a company
The benchmark landscape provides a starting point: SWE-bench Pro, Terminal-Bench, FrontierCode, and a work named DeepSWE appear in the discussion. Joseph illustrates the prevailing task shape as 50–100 agent turns producing a pull request of a couple thousand lines. This is his characterization, not a measured result across those benchmarks. Nor does repository editing fully describe every named benchmark: Terminal-Bench includes broader terminal workflows, and FrontierCode evaluates mergeability, including scope and quality. The useful distinction is between completing a bounded assignment and owning what happens afterward.
A human team does substantial work outside the patch. Product managers talk to customers to discover what is wrong. Engineers compare approaches, test performance, and remain responsible for the infrastructure over months or years. Emulated’s response is to put the working context of software companies into containerized environments: projects, incidents, and customer conversations become part of what the agent must interpret.
The environment must also expose consequences that source code alone cannot show. Network failures between nodes, data corruption, and clock skew force the agent to reason about distributed behavior. Cluster orchestration introduces sequencing decisions; live traffic makes the operational blast radius of those decisions matter. The task becomes longer and more complex because the agent must manage the system surrounding its change, not merely produce the change itself.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Follow an etcd change into a running cluster
The concrete example is an etcd consensus cluster supporting a production service. In the environment diagram, the etcd source code occupies one small part of a larger system. The agent’s assignment extends through the surrounding organizational evidence, deployment machinery, cluster members, and monitoring.
The workflow proceeds through several dependent stages:
- Read the operational history. Tickets, projects, and postmortems describe previous failures and customer reactions. Some records are stale, so the agent must incorporate imperfect context when deciding what to change.
- Make the change and start a rolling deployment. Producing the code is an intermediate step. Deployment systems can contain conflicts or fail themselves.
- Migrate from old hardware to new hardware. Unexpected problems emerge during execution, requiring decisions based on the current system rather than only the initial plan.
- Keep the service available. Failing nodes and stale, deprecated nodes coexist with live traffic. The agent must observe and monitor the service throughout the operation, because an unsafe action can affect active users.
This is the infrastructure task end to end: historical context informs a change, and the change must survive deployment into a system that cannot simply stop serving traffic.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
How far can one sandbox go?
The etcd environment runs inside a single-node sandbox, even though it simulates a distributed cluster. It can include flapping nodes that repeatedly become unavailable, lagging learners, live traffic, and the operational problems surrounding a deployment. Requiring multiple deployments instead of one extends the task horizon without changing that basic architecture.
This fits the containerized task model Joseph associates with standard post-training workflows and Harbor. His description of the ecosystem as uniformly single-sandbox is a characterization, not a constraint established for every Harbor environment or training pipeline. The narrower concern is fidelity: deterministic simulation can introduce network failures, but the speakers argue that it does not capture everything involved in building a cloud-scale service.
To see where the boundary appears, imagine building a service at AWS, GCP, or Azure—or an infrastructure product such as Datadog, Vercel, or Supabase. The starting point may be a useful piece of software. Turning it into something customers can reliably provision and operate introduces a different class of work.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Build outward from software to a cloud service
Begin with software that serves one customer well, perhaps on a developer’s machine. For an NLB-like service, that software is a load balancer; for a Lambda-like service, it is a serverless runtime. The next requirement is somewhere to run it. Resource provisioning is the first concrete pressure on the single-node sandbox: how faithfully can that sandbox represent allocating hosts through something like EC2 or Cloud Run?
Provisioning extends beyond a host to VPCs, subnets, and security groups. Customers then need an API through which they can request the service or delete resources they no longer want. Enterprise customers add requirements around throttling, authentication, and authorization, with CloudTrail-style auditability in the AWS example. Each layer adds behavior the agent must understand and maintain; finishing the original software does not finish the service.
Updates introduce the next layer: a deployment component must roll new software out and roll it back when something goes wrong. It must manage versions and make deployments gradual enough to limit blast radius. A correct new version is not sufficient if the process of introducing it disrupts the service.
The surrounding operational machinery continues to grow:
- Health monitoring: account for network partitions when interpreting the state of the service.
- Host communication: deliver configuration changes while the system is running.
- Customer connectivity: manage DNS and certificates so customers can reach their endpoints.
- Service administration: track resources and potential fraud, then provide admin consoles, telemetry, and billing.
These are connected responsibilities of operating the same product, rather than independent programming exercises. An environment intended to teach ownership needs to make their interactions visible.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Move the environment onto real infrastructure
Beyond a certain level of complexity, the proposed next step is an environment that provisions real infrastructure: a multi-node sandbox with access to actual cloud resources. The distinction is about what the environment can do, not just how many simulated processes it contains.
| Environment | What it exposes | Boundary raised in the talk |
|---|---|---|
| Single-node simulation | Cluster behavior, failures, traffic, repeated deployments | Fidelity of cloud provisioning and service operations |
| Multi-node sandbox with real resources | Actual infrastructure and resource provisioning | New demands on post-training infrastructure |
The first can already support substantial operational work. The second is intended to extend the space of tasks into building and managing the infrastructure itself.
Changing the sandbox also changes the post-training pipeline around it. Joseph briefly suggests putting a post-training pipeline inside the sandbox, making model training itself part of the environment and opening possibilities for recursive self-improvement. These are proposed directions; the presentation does not supply a detailed pipeline design or demonstrate a self-improvement result.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Real resources still leave training constraints
The practical difficulty begins before the agent takes its first meaningful action. Joseph says initializing an entire stack like AWS Lambda can take hours. This concerns full-stack initialization, not an individual function’s cold-start latency, and no specific configuration or timing study is supplied. The engineering question is how an environment with that startup cost fits into a post-training rollout.
Resource cost and efficient management remain additional constraints. Access to real cloud resources also does not eliminate the simulation-to-reality gap: an environment still needs representative live customer traffic, and some failures appear only at sufficient scale. Real machines provide one kind of fidelity; the workloads and operating conditions determine whether the agent encounters the problems it is supposed to learn to handle.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From infrastructure ownership to broader company workflows
The first audience question asks for Emulated’s primary goal. Joseph frames it as a response to how contrived and low-fidelity many emulations of real work remain. The ambition is to make agents capable of owning complex systems and, eventually, entire companies by giving them much more faithful environments in which to learn. Full-fidelity emulation and company autonomy are the intended destination, not capabilities established by the etcd example.
The next question asks whether this work is confined to infrastructure and DevOps or extends to broader reinforcement-learning environments and digital twins. Joseph acknowledges that many valuable workflows are not infrastructure-related. Starting with infrastructure is a choice of initial domain, for two reasons. First, the founders’ own expertise helps them judge what makes specialized training data realistic and useful.
Second, infrastructure companies often have comparatively clear customer requirements, which makes company simulation easier to specify. In the Modal example, users want a GPU sandbox with low latency and low cost, and they do not want a training run to fail halfway through. Those needs give the environment a clearer problem statement than a company still searching for product-market fit.
Sid closes with the proposed path beyond that initial domain: go deep enough in infrastructure to learn how to construct demanding, realistic environments, then explore which lessons transfer to other kinds of work. The strategy is depth first, followed by broader application; transfer remains something to investigate rather than a demonstrated outcome.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
The company's overview of RL environments for production infrastructure engineering.
A benchmark of complex software engineering tasks drawn from maintained repositories.
Cognition's benchmark for code mergeability, including correctness, scope and maintainability.
Terminal-based agent benchmarks with task examples spanning engineering, administration and machine learning.
Definitions of Harbor tasks, datasets, agents and container environments.
Further reading
How a DNS management race cascaded into failures across cloud services in October 2025.
Full transcript
Loading transcript…