# Building Deterministic Infrastructure for Non-Deterministic AI Agents

Nishant Gupta, Meta

AI Engineer World's Fair 2026

[Talk page](https://ai.engineer/talks/APh1Vx0oLmQ-building-deterministic-infrastructure-non) · [Video](https://www.youtube.com/watch?v=APh1Vx0oLmQ)

## Building Deterministic Infrastructure for Non-Deterministic AI Agents

An agent that can complete a workflow still needs infrastructure that bounds its failures, controls its actions, coordinates its state, and makes recovery possible.

Familiarity with API calls, retries, shared state, and basic distributed-systems concepts will help; no specific agent framework is required.

### Can the workflow run reliably?

[0:03](https://www.youtube.com/watch?v=APh1Vx0oLmQ&t=3s)

What changes when an AI system stops answering questions and starts making decisions that affect production? Models must still reason well, but an agent also plans, calls tools, and coordinates workflows. Nishant Gupta, who introduces himself as a software engineering tech lead at Meta, locates the emerging challenge in reliability: **probabilistic decisions need infrastructure with predictable controls.**

Modern cloud infrastructure developed around short-lived requests, mostly deterministic services, known execution paths, and bounded failures. Autonomous agents strain those assumptions. They preserve state, run for extended periods, choose their next steps dynamically, and may follow different workflows for identical inputs. Gupta calls this the great mismatch. The slide contrasts stateless and stateful execution, deterministic and probabilistic behavior, request-response and multi-step workflows, and millisecond and long-running execution.

![Two columns contrast stateless with stateful, deterministic with probabilistic, request-response with multi-step workflows, and millisecond execution with long-running execution.](https://ai.engineer/api/talk-essay-media/fc63f51904ebb64df6c1c60ce0b8426bcfb6a765b062f619e60fba7f129ddf8f)

The mismatch between traditional microservices and autonomous AI agents.

A demo asks whether an agent can solve a problem, use a tool, or complete a workflow. Production asks whether it can do so repeatedly, recover from failures, operate safely, and deliver acceptable outcomes within cost and latency constraints. That shifts engineering effort below the model into orchestration, monitoring, safety, evaluation, and recovery. Task completion is the beginning of the reliability requirement, not its completion.

### How an invalid tool call becomes a compute incident

[2:06](https://www.youtube.com/watch?v=APh1Vx0oLmQ&t=126s)

Hallucinations are only one failure mode. Recursive reasoning loops, overflowing logs, retry amplification, context corruption, memory poisoning, and cost explosions can turn an incorrect model output into a system-wide problem. The distinction matters: the model makes a mistake, but infrastructure determines whether that mistake remains contained or becomes an outage.

Consider Gupta’s invalid tool-call cycle:

1. The agent sends an incorrect request to a tool.
2. The tool returns an error.
3. The agent produces a slightly different request that is still invalid.
4. The cycle repeats, consuming more compute while reasoning depth and GPU consumption rise.

Gupta describes exponential resource growth as a possible escalation of this cycle, without supplying a measured growth curve. Repetition alone does not establish that growth rate; the architectural risk is that recovery has become an uncontrolled source of additional work. A minor API error can then become a compute incident.

### Give the model proposals, not production authority

[3:08](https://www.youtube.com/watch?v=APh1Vx0oLmQ&t=188s)

**The model suggests; the platform decides.** Gupta’s strongest architectural recommendation is to keep the model from directly controlling production systems. The boundary has four distinct stages:

1. The model generates a proposal.
2. Infrastructure validates it.
3. A policy engine approves or rejects it.
4. An execution gateway enforces the decision.

Validation, approval, and execution are separate responsibilities. A valid request is not necessarily permitted, and a permission decision does not itself perform the action.

A TypeScript implementation can make that boundary explicit. Here, the proposed action restarts a named service. Model output enters as `unknown`; only a validated proposal can reach policy evaluation, and only an approved proposal reaches the gateway.

```typescript
type Proposal = {
  action: "restart_service";
  service: string;
};

type Decision =
  | { allowed: true }
  | { allowed: false; reason: string };

interface PolicyEngine {
  evaluate(proposal: Proposal): Promise<Decision>;
}

interface ExecutionGateway {
  enforceAndExecute(
    proposal: Proposal,
    approval: { allowed: true }
  ): Promise<void>;
}

function validate(input: unknown): Proposal {
  if (typeof input !== "object" || input === null) {
    throw new Error("Expected an action proposal");
  }
  const value = input as Record<string, unknown>;
  if (
    value.action !== "restart_service" ||
    typeof value.service !== "string" ||
    value.service.trim() === ""
  ) {
    throw new Error("Invalid restart proposal");
  }
  return { action: value.action, service: value.service };
}

async function handleProposal(
  input: unknown,
  policy: PolicyEngine,
  gateway: ExecutionGateway
): Promise<void> {
  const proposal = validate(input);
  const decision = await policy.evaluate(proposal);
  if (!decision.allowed) {
    throw new Error(decision.reason);
  }
  await gateway.enforceAndExecute(proposal, decision);
}
```

This example expresses the separation in application code. The gateway remains responsible for enforcing the approved operation, and production credentials belong behind that boundary rather than in the model’s control.

### A control plane needs a decision history

[3:30](https://www.youtube.com/watch?v=APh1Vx0oLmQ&t=210s)

Containers gave rise to [Kubernetes](https://kubernetes.io/docs/concepts/overview/components/), and microservices created a need for service meshes. Gupta proposes a corresponding **agentic control plane** responsible for scheduling, memory coordination, policy enforcement, evaluation, monitoring, and workload routing. His operating-system analogy describes a common layer that manages autonomous work, rather than leaving each agent to manage its own execution. He expects organizations that build this layer well to gain a competitive advantage.

That layer needs more than logs of completed actions. Traditional logs describe what happened; debugging an agent also requires reconstructing how its workflow reached that point. Traces should connect planning decisions, tool calls, memory lookups, and state transitions. The final output may reveal that something went wrong while concealing the earlier decision or retrieved state that caused it.

**Trace the observable decision sequence.** This means recording instrumented decisions and events, not assuming access to hidden model reasoning. Together, the sequence of calls, retrieved memory, and state changes provides the multiple dimensions needed to debug an autonomous workflow.

### Coordinate memory, layer safety, and route exceptions

[4:23](https://www.youtube.com/watch?v=APh1Vx0oLmQ&t=263s)

Once multiple agents share state, familiar distributed-systems failures appear: stale reads, conflicting updates, context drift, and inconsistent views. Memory becomes harder to coordinate when it is also probabilistic and retrieval based. An agent may reason from a different view of the world than another agent, so an apparent reasoning failure may actually be a consistency failure. Improving the model alone does not resolve disagreement about the underlying state.

Safety likewise cannot sit in a single component. Gupta applies defense in depth through several complementary controls:

- **Prompt-level controls:** Guide the agent’s behavior.
- **Tool permissions:** Restrict which operations it can access.
- **Policy validations:** Check whether proposed actions are allowed.
- **Human approvals:** Introduce review where judgment is needed.
- **Audit systems:** Preserve records for accountability and investigation.

Each layer addresses a different class of failure; none replaces the others.

Human involvement is not necessarily a temporary concession on the way to full autonomy. Gupta expects successful systems to remain supervised, with people serving as exception handlers: reviewing ambiguous situations, handling novel scenarios, and providing calibration signals. The design objective is to allocate human attention where it contributes the most value, rather than simply minimizing human participation.

### Inference becomes a scheduling problem

[5:36](https://www.youtube.com/watch?v=APh1Vx0oLmQ&t=336s)

Agent workloads increasingly resemble cluster scheduling problems. Demand arrives in bursts, reasoning depth is unpredictable, and workflows may run for minutes instead of milliseconds. Resource requirements also vary dramatically, making GPU efficiency, workload placement, elastic capacity management, and scheduling central concerns. Inference therefore becomes a **resource orchestration problem**, not just a model-serving problem. The slide’s jagged demand curve and stepped capacity bands bring those concerns together with resource contention and GPU limits.

![Elastic scaling graph with a jagged red demand curve and stepped blue bands, annotated with variable reasoning depth, long-running workflows, resource contention and GPU limits, and bursty demand.](https://ai.engineer/api/talk-essay-media/8775ac0c36d321515afe24d51e99fd2a8802e6c9fab761c2d28f99bb52c54e88)

Inference at scale becomes a cluster scheduling problem.

### Adapt the protections distributed systems already use

[6:03](https://www.youtube.com/watch?v=APh1Vx0oLmQ&t=363s)

These requirements do not demand an entirely new infrastructure discipline. Gupta maps established reliability patterns onto autonomous workloads:

| Established pattern | Agent infrastructure role |
| --- | --- |
| Circuit breakers | Tool isolation |
| Rate limits | Agent limits |
| Retries | Controlled recovery |
| Resource quotas | Cost governance |
| Observability | Agent tracing |

The invalid tool-call cycle illustrates why these mappings matter. A retry should be part of controlled recovery, not permission to generate work indefinitely. Tool isolation and agent limits contain failures; quotas bound resource exposure; traces make the resulting behavior explainable. The implementation changes, but the reliability purpose remains familiar.

### The differentiator moves into infrastructure

[6:27](https://www.youtube.com/watch?v=APh1Vx0oLmQ&t=387s)

Gupta describes an industry progression from prompts as the differentiator, to models, and now toward infrastructure as prompts and models become more widely available. His forecast is that the winning organizations will not necessarily have the best prompts; they will have the most reliable systems. The competitive advantage moves into the layers that make model capability dependable.

**Treat agents as distributed systems.** Their models remain stochastic, while infrastructure must make validation, permissions, execution, and recovery predictable. Observability is mandatory, and control planes become the foundation for coordinating autonomous work. Better models expand what agents can attempt; better systems determine whether those attempts can safely become production operations.

## Resources

- [Kubernetes Components](https://kubernetes.io/docs/concepts/overview/components/) — Overview of the API server, state store, scheduler and controllers that form a Kubernetes control plane.
- [Handling Overload](https://sre.google/sre-book/handling-overload/) — Google's guidance on overload protection, client throttling and retry budgets that limit request amplification.
- [Open Policy Agent](https://www.openpolicyagent.org/docs) — Policy evaluation with structured inputs and interactive Rego examples, separating decisions from application enforcement.

## Slides

### Building deterministic infrastructure for non-deterministic AI agents

Nishant Gupta frames the challenge as surrounding probabilistic AI agents with reliable infrastructure as they plan, call tools and affect production systems.

[0:00](https://www.youtube.com/watch?v=APh1Vx0oLmQ&t=0s) [Title slide with a diagram of a control enclosure surrounding a cloud-like form.](https://assets.ai.engineer/sites/aiecode2025/sha256/9d86d74316946fab38f2a5addc9285a5c024211100523feb2424d10f2b58dd84.webp)

BUILDING DETERMINISTIC INFRASTRUCTURE FOR NON-DETERMINISTIC AI AGENTS

THE EMERGING CONTROL PLANE FOR AUTONOMOUS AI SYSTEMS

DETERMINISTIC BOUNDARY [DB-500]
ENTROPY REDUCTION: 99.9%
PREDICTABILITY INDEX: 1.0
SYSTEM VARIANCE: <0.01%
CONTROL ENCLOSURE [CE-99.99%]
SIGNAL RECTIFICATION
SIGNAL SYNCHRONIZATION
STATE SYNCHRONIZATION
RIGID FRAMEWORK [RF-AI2]
CONTROL ENCLOSURE [CE-99.99%]

Diagram measurements: 50°, 29°, 28°, 16°, [illegible], 90°, 93°, 29%

Nishant Gupta
Tech Lead @ Meta

Interface text:
Nishant Gupta

### Demos optimize for capability; production demands reliability

A prompt and LLM sit above a reliability core of runtime, orchestration, evaluation, memory, monitoring, and guardrails. Gupta asks whether an agent can complete a workflow reliably 10,000 times, not just once in a demo.

[0:56](https://www.youtube.com/watch?v=APh1Vx0oLmQ&t=56s) [An isometric diagram places capabilities above a reliability core.](https://assets.ai.engineer/sites/aiecode2025/sha256/2b0756f26319e4d85db76145eb292c572e537eb93653fec9c2d0b22280ca3f55.webp)

Demos optimize for capability.
Production demands reliability.

Capabilities
Prompt
LLM

Reliability Core
Runtime
Orchestration
Evaluation
Memory
Monitoring
Guardrails

DATA
[LOAD BEARING]
[STRUCTURAL INTEGRITY: CRITICAL]
[ISOMETRIC VIEW: SYSTEM CROSS-SECTION]

Interface text:
Nishant Gupta

### Diagnostic failure tree

The failure tree traces three paths from stochastic model output: reasoning loops to deadlocks, tool hallucinations through retry storms to cost explosions, and context drift to memory poisoning.

[2:07](https://www.youtube.com/watch?v=APh1Vx0oLmQ&t=127s) [A failure tree links stochastic model output to reasoning loops, tool hallucinations and context drift, with resulting deadlocks, cost explosions and memory poisoning.](https://assets.ai.engineer/sites/aiecode2025/sha256/5e60757049dcff2e7159b985b6764b2c1ffb742d9a65ef1137101260fb114da3.webp)

Real production failures originate [illegible] the infrastructure, not the model.

Diagnostic Failure Tree

Stochastic Model Output

Logic → Recursive reasoning loops → Workflow deadlocks
Action → Tool hallucinations → Retry storms → Cost explosions
State → Context drift → Memory poisoning

Diagram measurements: 109, 150, 100, 555, 169; other small annotations [illegible]

Interface text:
Nishant Gupta

### The anatomy of an agent retry storm

An invalid tool call returns an error, but the agent responds with another invalid request. Repeated retries consume more compute, turning a minor API error into a compute incident.

[2:30](https://www.youtube.com/watch?v=APh1Vx0oLmQ&t=150s) [A sequence of invalid tool parameters and repeated errors leads to a recursive loop alongside a rising compute graph.](https://assets.ai.engineer/sites/aiecode2025/sha256/bceb38aea64c303bb295e38b18d966068cc47a4728ae89d38984cfab5bb854af.webp)

The anatomy of an agent retry storm.

Exponential GPU Compute Spike

Step 1: Agent hallucinates invalid API parameters.
Step 2: Tool returns error code.
Step 3: Agent attempts to fix error but hallucinates new invalid parameter.
Step 4: Recursive reasoning loop locks up.
LOOP RECURSION
INVALID PARAMS
Step 5: The Consequence

Step 1: Agent hallucinates invalid API parameters.
Step 2: Tool returns error code.
Step 3: Agent attempts to fix error but hallucinates new invalid parameter.
Step 4: Recursive reasoning loop locks up.
Step 5: The Consequence

Graph labels: +5559%; +4338%; +855% SPIKE
RESOURCE SATURATION
COST EXPLOSION
Y-axis: GPU Compute Units (TFLOPs)
X-axis: Time (ms)
DATA SPIKE EVENT

[CRITICAL FAILURE PATH]
[TIMELINE AXIS]

Interface text:
Nishant Gupta

### The Agent Control Plane

A proposed control plane sits between agent applications and foundation models, coordinating memory, orchestration, safety and compute. The speaker recommends letting models propose actions while infrastructure validates and enforces them.

[3:05](https://www.youtube.com/watch?v=APh1Vx0oLmQ&t=185s) [A layered architecture places a control plane between agent applications and foundational models and data APIs, with memory, orchestration, safety and compute components.](https://assets.ai.engineer/sites/aiecode2025/sha256/bd77aec437cc8ea67045d321b672b07adbf526ad50634015c9204308aa363d29.webp)

The Agent Control Plane is the new infrastructure layer.

[Control Plane Blueprint]

Agent Applications
Research
Coding
Interaction
[illegible]

Memory Coordinator
INDEX

Orchestration Engine

THE AGENT CONTROL PLANE

Safety Policy Node
VALIDATE & FILTER

Compute Scheduler
RESOURCE OPTIMIZATION

Foundational LLMs & Data APIs

[ARCHITEC[illegible]]
Agents require an operating system.

Just as Kubernetes became the control plane for orchestrating containers, a dedicated Agent Control Plane is emerging to govern the runtime execution of autonomous AI.

[ANALYSIS: HIGH]
[COST [illegible]: 10%]
[FLOW: DETERMINISTIC GOVERNANCE]
3

Interface text:
Nishant Gupta

### Multidimensional agent observability

An agent trace aligns decisions, plans, tool execution, memory access, and state changes on one timeline. Gupta argues that tracing this chain helps explain why a workflow behaved as it did, not just its final output.

[3:32](https://www.youtube.com/watch?v=APh1Vx0oLmQ&t=212s) [Agent trace timeline showing decisions, orchestration, tool execution, memory access, and state transitions.](https://assets.ai.engineer/sites/aiecode2025/sha256/d969544c0271950bf2b47822bdf87f8f3b314740523b8efdb942a44bda52ab5a.webp)

Logs are dead. Autonomous workflows require multidimensional observability.

Agent Trace Timeline
[Telemetry Sidebar]

Timeline: T+0.1s, T+0.2s, T+0.1s, T+0.5s, T+1.0s, T+1.2s

Track 1: LLM Decisions
Decision: Call Tool A
Decision: Call Tool A
Decision: Query Memory
Decision: Query Memory
Decision: Call Tool A

Track 2: Orchestration Plans
Plan: Execute Step 1
Plan: Execute Step 1
Plan: Validate Output

Track 3: Tool Calls & Execution
Tool A Executing
API Response Received

Track 4: Memory Access
Read: Context Block X
Write: New State Y

Track 5: State Transitions
State: Processing
State: Waiting for Tool
State: Updated
State: Updated

Diagram labels: [FLOW: REASONING], [DATA: ACTIVE], [FLOW: illegible]
event
Timeline: T+0.1s, T+0.1s, T+0.5s, T+1.0s, T+1.0s
[ANALYSIS: FLOW TRACING]
[DATA: CONTINUOUS STATE]
4

Interface text:
Nishant Gupta

### Shared memory and multi-agent consistency

Agents share a state store, but can encounter context drift, stale memory and conflicting facts. Probabilistic, retrieval-based memory can make these consistency problems harder to manage.

[4:21](https://www.youtube.com/watch?v=APh1Vx0oLmQ&t=261s) [Agents connect to a shared state store, with warnings for context drift, stale memory, and conflicting facts.](https://assets.ai.engineer/sites/aiecode2025/sha256/e5e8863f966f6a17a4bef061824e95be95d6c710e8a69478513b7266de4f0c23.webp)

Shared memory coordinates chaotic multi-agent consistency

Shared Memory Architecture

Agent A
Agent B
Agent C

State Store
[illegible]

Context drift
Stale memory
Conflicting facts

Interface text:
Nishant Gupta

### Layered containment boundaries

Model output passes through concentric controls, where a path is blocked. Tool permissions, policy checks, human approval and auditing provide layers that catch different classes of failure.

[4:57](https://www.youtube.com/watch?v=APh1Vx0oLmQ&t=297s) [Concentric containment layers surround model output, illustrating tool permissions, policy, human approval, and audit controls.](https://assets.ai.engineer/sites/aiecode2025/sha256/4abd9a56667cfb54808d9866972b4c2291a460f8a48080a69c0591717467dec6.webp)

Autonomous systems demand layered containment boundaries

Model Output
BLOCKED

Ring 5: Audit Layer
Ring 2: Tool Permissions
Ring 3: Policy Engine
Ring 4: Human Approval
Ring 5: Audit Layer

Interface text:
Nishant Gupta

### Human oversight as an escalation path

A decision gate routes edge cases to human review, then returns approved cases to the automated workflow. Humans handle ambiguous or novel scenarios where their attention adds value.

[5:07](https://www.youtube.com/watch?v=APh1Vx0oLmQ&t=307s) [A decision gate routes edge cases to human review before they rejoin the automated workflow.](https://assets.ai.engineer/sites/aiecode2025/sha256/0e51ed2c933bbfbd3e9c231fc707138325b0244f6144e80915fd29443215eaf7.webp)

Human oversight is an escalation path, not an operational bottleneck.

Workflow Routing Diagram

Automated Agent Tasks

Decision Gate
Risk / Confidence Check
Confidence Score: >95%
Anomaly Detection: Low

Automated Agent Tasks

Confidence Score: >95%
Anomaly Detection: Low

45°
45°

Edge Case Routing
Human Approval Node
Manual Review Required
Approved State / Resolution

[ANALYSIS: ESCALATION PATHWAYS]

Interface text:
Nishant Gupta

### Inference at scale and cluster scheduling

Bursty demand and long-running workflows can exceed available compute capacity. The speaker argues that workload placement, elastic capacity management and scheduling become critical as resource requirements vary.

[5:33](https://www.youtube.com/watch?v=APh1Vx0oLmQ&t=333s) [Graph of compute demand over time showing bursts, a sustained workload, and resource limits.](https://assets.ai.engineer/sites/aiecode2025/sha256/5f118fb4bb76e1bb034c7a3d4b78d9fe3aaef21763787251d3266d0c8e4b1775.webp)

Inference at scale fundamentally becomes a cluster scheduling problem.

Elastic Scaling Graph

Y-axis: COMPUTE UNITS (CU)
Y-axis ticks: 0, 100, 200, 300
X-axis: TIME (sec)
X-axis ticks: 0, 100, 200, 300

Variable reasoning depth
Long-running workflows
Resource Contention & GPU Limits
Bursty Demand

Interface text:
Nishant Gupta

### The Reliability Rosetta Stone

The slide maps distributed-systems reliability patterns to agent infrastructure: circuit breakers become tool isolation, retries become controlled recovery, and observability becomes agent tracing.

[6:01](https://www.youtube.com/watch?v=APh1Vx0oLmQ&t=361s) [Table mapping distributed systems patterns to agent equivalents, including tool isolation, agent limits, controlled recovery, cost governance, and agent tracing.](https://assets.ai.engineer/sites/aiecode2025/sha256/4a18da99d8a968a8fceb7c2c8bb7cc086c811e8e72c006b7ef95645819844bce.webp)

The Reliability Rosetta Stone.

Distributed Systems Pattern → Agent Equivalent

Circuit Breakers → Tool Isolation
[PATTERN: ISOLATION] → [PATTERN: ISOLATION]

Rate Limiting → Agent Limits
[LIMIT_ENFORCEMENT] → [LIMIT_ENFORCEMENT]

Retries → Controlled Recovery
[RECOVERY_LOGIC] → [RECOVERY_LOGIC]

Quotas → Cost Governance
[RESOURCE_CAPS] → [RESOURCE_CAPS]

Observability → Agent Tracing
[TRACEABILITY_SYSTEM] → [TRACEABILITY_SYSTEM]

[ANALYSIS: RELIABILITY MAPPING]

Interface text:
Nishant Gupta

### The Paradigm Shift

The conceptual graph shows prompts peaking and declining, models leveling off, and infrastructure rising steeply through 2025 and beyond. The speaker identifies infrastructure as the next differentiator.

[6:25](https://www.youtube.com/watch?v=APh1Vx0oLmQ&t=385s) [Conceptual graph showing prompts declining, models plateauing, and infrastructure rising from 2023 through 2025 and beyond.](https://assets.ai.engineer/sites/aiecode2025/sha256/330482a3ec5fb855b5ee24e03f106d25105e3172e37247597879d6065104548e.webp)

Competitive advantage has shifted fro[illegible]
prompt engineering to systems engineering.

THE PARADIGM SHIFT

PROMPTS
MODELS
INFRASTRUCTURE
Compound Growth Trajectory
Commoditization Plateau
Commoditization Plateau
Timeline: 2023, 2024, 2025+

Interface text:
Nishant Gupta

### AI Agents Are Distributed Systems

Treat agents as distributed systems: the speaker calls for deterministic infrastructure around stochastic models, with observability and control planes supporting production reliability.

[6:46](https://www.youtube.com/watch?v=APh1Vx0oLmQ&t=406s) [Closing slide calling for reliable infrastructure, observability, and agent control planes, with an emphasis on better systems.](https://assets.ai.engineer/sites/aiecode2025/sha256/587f865c9d1436ab5d0e46cbc8fb85aceacc7d159f1f4908f3d81b11c6eb0f17.webp)

AI agents are distributed systems.
Treat them accordingly.

Control Plane Blueprint

Diagram labels: Agent Execution & [illegible]; Policy Engines; Agent execution; State Management; State engine; [illegible] State
Dimensions: 8′0″; 125″; 28.6[illegible]; 28.6[illegible]

CORE TAKEAWAYS
• Models are inherently stochastic; infrastructure must be strictly.
• Reliability is no longer a model problem—it is an infrastructure.
• Multidimensional observability is mandatory, not optional.
• Agent control planes are the required runtime layer for production AI.

FINAL DIREC[illegible]

The future of AI won’t be won by better prompts.
It will be won by better systems.

Interface text: Nishant Gupta

## Full transcript

[0:03](https://www.youtube.com/watch?v=APh1Vx0oLmQ&t=3s) Hey everyone. My name is Nishant Gupta. I'm a software engineering tech lead at Meta, working on building the training and inference arc infrastructure. And today we're going to be talking about building deterministic infrastructure for non-deterministic AI agents.

[0:19](https://www.youtube.com/watch?v=APh1Vx0oLmQ&t=19s) So most of the conversations around AI over the last few years has been focused on models, bigger models, more parameters, better reasoning. But as organizations move from chatbots to autonomous agents, a different problem emerges.

[0:31](https://www.youtube.com/watch?v=APh1Vx0oLmQ&t=31s) The challenge is no longer im- intelligence. The challenge is, is reliability. At Meta and across the industry, we are seeing agents move beyond answering questions and beginning to plan, call tool calls, coordinate workflows, and make decisions that affect production systems.

[0:46](https://www.youtube.com/watch?v=APh1Vx0oLmQ&t=46s) These systems are fundamentally probabilistic. Infrastructure is not allowed to be. Today, I want to discuss this topic in more detail.

[0:58](https://www.youtube.com/watch?v=APh1Vx0oLmQ&t=58s) The modern cloud infrastructure evolved around a set of assumptions. A request-- Most of the requests are short-lived. Services are deterministic, more or less. Execution paths are known. Failures are bounded.

[1:10](https://www.youtube.com/watch?v=APh1Vx0oLmQ&t=70s) However, autonomous AI agents violate nearly every one of those assumptions. They're stateful, they're long-running. They make decisions dynamically. They make, may execute different workflows for same inputs. This is what I call the great mismatch.

[1:23](https://www.youtube.com/watch?v=APh1Vx0oLmQ&t=83s) We are trying to run autonomous systems on infrastructure that was designed for deterministic workflows.

[1:30](https://www.youtube.com/watch?v=APh1Vx0oLmQ&t=90s) This is probably the most important mind shift. Most AI demos showcase capability.

[1:36](https://www.youtube.com/watch?v=APh1Vx0oLmQ&t=96s) But can it solve a problem? Can it use a tool? Can it compete-- complete a workflow? Production systems be-- have a different objective. Can it do it reliably? Can it do it ten thousand times?

[1:47](https://www.youtube.com/watch?v=APh1Vx0oLmQ&t=107s) Hundred thousand times? Million times? Can it recover from failures? Can it operate safely? Can it do it in an ac-- at an acceptable cost with an acceptable latency, with an acceptable outcome?

[1:57](https://www.youtube.com/watch?v=APh1Vx0oLmQ&t=117s) The majority of the engineering effort moves below the model layer into orchestration, monitoring, safety, evaluation, and recovery systems.

[2:06](https://www.youtube.com/watch?v=APh1Vx0oLmQ&t=126s) When people hear AI failures, they immediately think hallucinations. In reali-in reality, hallucinations are often the least interesting failure mode. What we see instead are infrastructure failures, recursive reasoning loops, overflowed logs, retry amplification, context corruption, memory poisoning, cost explosions.

[2:25](https://www.youtube.com/watch?v=APh1Vx0oLmQ&t=145s) The model makes mistake, but however, the infrastructure turns that mistake into an outage. That's the real challenge.

[2:33](https://www.youtube.com/watch?v=APh1Vx0oLmQ&t=153s) So the as-- this slide shows a pattern that distributed system engineers will probably recognize immediately. An agent calls a tool incorrectly. The tool returns an error. Instead of recovering, the agent generates a slightly different but still invalid request.

[2:48](https://www.youtube.com/watch?v=APh1Vx0oLmQ&t=168s) The cycle repeats. Each retry consumes more compute. Reasoning depth increases. GPU consumption rises. Eventually, you get exponential resource growth. What started as a minor API error became a compute incident.

[3:01](https://www.youtube.com/watch?v=APh1Vx0oLmQ&t=181s) This is why unco-- uncontrolled retries are one of the biggest risks in agentic systems.

[3:08](https://www.youtube.com/watch?v=APh1Vx0oLmQ&t=188s) This is the architecture principle I recommend most strongly: Never let the model directly control production systems. The model should generate proposals, infrastructure validates them, policy engine approves them, execution gateway enforces them.

[3:20](https://www.youtube.com/watch?v=APh1Vx0oLmQ&t=200s) The model just suggests. The platform decides. This separation allows us to build reliable systems even when the underlying model remains probabilistic.

[3:30](https://www.youtube.com/watch?v=APh1Vx0oLmQ&t=210s) As we know, containers gave rise to Kubernetes. Microservices created service meshes. AI agents are creating something new, an agentic control plane. This layer becomes responsible for scheduling, memory coordination, policy enforcement, evaluation, monitoring, workload routing, which is very important.

[3:49](https://www.youtube.com/watch?v=APh1Vx0oLmQ&t=229s) And think of it as an operating system for autonomous AI. The organizations that build this layer will have significantly more competitive advantages. So traditional logs tell us what happened.

[3:59](https://www.youtube.com/watch?v=APh1Vx0oLmQ&t=239s) Agentic systems require understanding why it happened. We need traces to capture planning decisions, tool calls, memory lookups, state transitions. When debugging an autonomous workflow, understanding the chain of decisions and reasoning is often more important than the final output.

[4:15](https://www.youtube.com/watch?v=APh1Vx0oLmQ&t=255s) Observability becomes multi-dimensional. Without it, production debugging becomes nearly impossible.

[4:23](https://www.youtube.com/watch?v=APh1Vx0oLmQ&t=263s) So as you can see, memory is one of the most underestimated challenges in agentic architectures. Once multiple agents share state, familiar distributed system issues appear: stale reads, conflicting up- updates, context drifts, inconsistent views.

[4:37](https://www.youtube.com/watch?v=APh1Vx0oLmQ&t=277s) The challenge becomes even harder when memory itself may be probabilistic and retrieval based. Many multi-agent failures are actually consistency failures masquerading as reasoning failures.

[4:50](https://www.youtube.com/watch?v=APh1Vx0oLmQ&t=290s) Safety cannot be a single component. It must be layered. Prompt level controls, tool permissions, policy validations, human approvals, audit systems. Each of these layer catches a different class of failures, defense in depth is a well-understood security principle.

[5:05](https://www.youtube.com/watch?v=APh1Vx0oLmQ&t=305s) It applies equally well to autonomous AI systems.

[5:09](https://www.youtube.com/watch?v=APh1Vx0oLmQ&t=309s) Many people frame human involvement as temporarily-- temporary necessity. I don't think that's correct. The most successful systems are likely to remain human supervised. Humans became-- become exception handlers. They review ambiguous situations.

[5:23](https://www.youtube.com/watch?v=APh1Vx0oLmQ&t=323s) They handle novel scenarios. They provide calibration signals. The goal is not to remove humans. The goal is allocating human attention where it provides the maximum value.

[5:36](https://www.youtube.com/watch?v=APh1Vx0oLmQ&t=336s) So one of the biggest infrastructure shifts is that AI workloads increasingly resemble cluster scheduling problems. Demand is bursty. Reasoning depth is inpred-- unpredictable. Workflows may run for minutes instead of milliseconds.

[5:48](https://www.youtube.com/watch?v=APh1Vx0oLmQ&t=348s) Resource requirements vary dramatically. As a result, GPU efficiency, workload placement, elastic capacity management, and scheduling becomes critical. Inference is no longer just a model problem, it becomes a resource orchestration problem.

[6:03](https://www.youtube.com/watch?v=APh1Vx0oLmQ&t=363s) The good news is that many of these problems are not entirely new. Distributed systems have solved something similar for decades. Circuit breakers become tool isolation. Rate limits become agent limits.

[6:14](https://www.youtube.com/watch?v=APh1Vx0oLmQ&t=374s) Retries become controlled recovery. Resource quotas become cost governance. Observability becomes agent tracing. Instead of inventing entirely new infrastructure, we can adapt proven reliability patterns to autonomous systems.

[6:27](https://www.youtube.com/watch?v=APh1Vx0oLmQ&t=387s) The industry has gone through several phase, phases. Then initially, prompts were the differentiator. Then the models became the differentiator. Now both are rapid- rapidly commodit- commoditizing. The next frontier is infrastructure.

[6:38](https://www.youtube.com/watch?v=APh1Vx0oLmQ&t=398s) The organization that won't-- that wins won't necessarily have the best prompts. They'll have the most reliable systems. The competitive advantage is moving up the stack.

[6:48](https://www.youtube.com/watch?v=APh1Vx0oLmQ&t=408s) If there's one thing I want you to remember, it's this: AI agents should be treated as distributed systems. Models are stochastic. Infrastructures must be deterministic. Reliability is increasingly an infrastructure problem.

[7:01](https://www.youtube.com/watch?v=APh1Vx0oLmQ&t=421s) Observability is mandatory. Control planes are emerging as the foundation layer, and ultimately, the future of the AI, AI won't be won by better prompts. It will be won by better systems.
