Contents
  1. Execution contracts
    1. What a harness controls
    2. Name work and its attempts
  2. Preserving progress
    1. Accept work durably
    2. Save a useful recovery position
    3. Reconstruct execution safely
  3. Historical development
    1. Why recovery mechanisms coexist
  4. Dispatch and recovery
    1. Wake retained work
    2. Reject obsolete workers
    3. Bound repeated attempts
    4. Resolve uncertain effects
      1. Receiver-enforced retries
      2. Finding or counteracting the effect
  5. Limits and operator control
    1. Preserve limits across restarts
    2. Stop owned work
    3. Control long waits and approvals
      1. Binding a decision to an action
  6. Deliverables and evidence
    1. Retain and publish artifacts
    2. Enforce completion checks
    3. Expose authoritative status
  7. Verification and design choices
    1. Test interruption boundaries
    2. Choose sufficient machinery
    3. Frequently asked questions
      1. Is a transcript a checkpoint?
      2. Can work survive a closed tab or a timed-out request?
      3. Can a recovered activity create a second object?
      4. Does cancellation undo a change?
      5. What does a sandbox snapshot preserve?
      6. Does a heartbeat prove progress?
      7. Can new code resume an old run?
      8. Is the task complete when the model stops?
      9. When is a queue enough?
  8. Check understanding
  9. Open questions
  10. Selected talks
  11. References
  12. Talk library
← All topics

Harness engineering

Harness engineering is the practice of designing an AI agent’s working environment, including the tools it can use, the feedback it receives, and the checks its work must pass. The agent runtime is the software that executes tool calls, returns their results to the model, and keeps track of unfinished work. At OpenAI, one team had Codex write the code for an internal beta. Ryan Lopopolo described their division of work as “Humans steer. Agents execute.” When engineers found themselves repeating a correction in code review, they changed how the next patch would be reviewed. Lopopolo’s firsthand account

Execution contracts

What a harness controls

The OpenAI team wrote the recurring review rule into the repository. A failing test could point the patch author to the rule, or an agent reviewing the change could flag the violation. Recording the rule made it available on later patches. Surfacing it during review gave the author a chance to fix the code. A team that also wanted to block a merge would have to require a passing check. Ryan Lopopolo and Vibhu Sapra at AI Engineer Europe, review example at 38:24

Consider how Codex could run a repository check. The model proposes a command. The runtime decides whether it can dispatch that command and pauses for a client decision if approval is required. Follow the fork in the figure. On denial, the command never starts. On the permitted branch, a test can still fail. The runtime appends the actual output or error so the next model call can use the failure message to choose an edit. That return path is what makes tool use a feedback loop. Codex App Server · The Codex agent loop · Agent decisions and observations

How a repository check reaches the next model call

Example

Permission determines whether the command may start. Its output supplies evidence for the next edit.

Codex proposes tool callRun a repository check.
Runtime checks permissionPause for a client decision when required.
Execute toolRun the permitted command.
No executionThe command never starts.
Append tool resultAssociate the observation with this call.
Next model invocationCan use the check’s finding.
Codex proposes a test command. The runtime checks permission and pauses if approval is required. Denial ends dispatch. When permitted, the tool runs and the runtime appends its returned output or error for the next model invocation.
Read the diagram as text
  • Codex proposes tool call. Run a repository check.
  • Runtime checks permission. Pause for a client decision when required.
  • Execute tool. Run the permitted command.
  • No execution. The command never starts.
  • Append tool result. Associate the observation with this call.
  • Next model invocation. Can use the check’s finding.
  • Codex proposes tool call → Runtime checks permission: Tool name and arguments.
  • Runtime checks permission → Execute tool: Allowed or approval granted.
  • Runtime checks permission → No execution: Denied.
  • Execute tool → Append tool result: Actual output or error.
  • Append tool result → Next model invocation: Observation in model input.

In Anthropic’s coding experiment, the difficulty was handing a partly built application to a fresh session. The first session established a feature list. Later sessions worked on individual features, committed changes to git, and updated a progress file. An incoming session inspected prior work, started the application, and ran a basic end-to-end check before continuing. Marking a feature complete required testing that feature’s promised behavior. A shorter conversation alone had left sessions prone to declaring victory early.

Which failure needs a change?
FailureInterventionEvidence available afterward
A proposed Codex command may lack permission or may fail.Check permission before dispatch. Return the result of an allowed call.A denial or the tool’s actual output.
OpenAI reviewers repeatedly correct the same mistake.Put the rule in the repository and surface it when a patch violates it.A finding the patch author can act on.
A fresh Anthropic session loses track of unfinished features.Leave a feature list, progress file, and commits. Run a basic end-to-end check on handoff. Test each feature before marking it complete.Recorded intent, the implementation, and observed behavior.

The runtime’s execution contract states what the service promises when it accepts a task. It specifies which actions may run, what progress survives the loss of a worker, when the runtime must stop, and what it checks before reporting success. An orchestration library can help coordinate calls, while an isolation boundary restricts the files, processes, and services the executing code can reach. An evaluation harness serves a different purpose. It runs trials and grades how well the model and its working environment perform together. Execution isolation · Evals and Benchmarks

Name work and its attempts

Suppose a client submits work, loses its connection, and comes back after another worker has taken over. It needs a handle for the work it already requested. Looking up the old process would miss the replacement. Looking up the conversation could return several requests. Choose identifiers by what each participant needs to find again.

Execution identities
IdentityWhat it namesWhat changes on recovery
TaskThe caller’s requested outcomeThe same task can have more than one execution.
RunOne managed execution with a lifecycle and limitsA replacement process can continue the same run.
WorkerThe process doing the workA restarted process is a new worker.
Logical operationOne intended effect, such as creating one issueRetain its identity through repeated deliveries.
AttemptOne try at a specified unit, such as a tool callCount a new attempt each time that unit is tried.

Product terminology varies. Codex groups a conversation into a thread, a user-initiated turn, and individual input or output items. In Temporal, a business-level Workflow ID can span a retry that gets a new Run ID. A change of worker alone need not start a new Temporal run. Codex App Server · Workflow ID and Run ID

Give the execution a lifecycle a returning caller can inspect. In one design, ready work becomes running when claimed, waiting when it needs input, and ready again when the input arrives. A stop request can move it to stopping. A terminal record explains how it ended, distinguishing success, failure, denial, cancellation, deadline, or exhausted allowance as the task requires. Accept each transition in one conditional write against the expected state and version. If completion and cancellation compete, the second writer must act on the state the first one committed.

Preserving progress

Accept work durably

A durable acknowledgment promises that a replacement can discover both the task and what still needs doing. Saving a task row and then sending a queue message leaves a gap. A crash between the writes leaves a real task that no worker has been told to start. Reversing the writes lets a worker receive an instruction for a task that never committed. State at acceptance

A transactional outbox puts the task change and an outgoing instruction in the same database transaction. After commit, the service can acknowledge the caller. A separate publisher, called a relay, reads the instruction and sends it to the queue. If the relay dies before sending, its replacement finds the pending instruction. If it dies after sending but before recording delivery, the replacement may send twice. Give the instruction a stable ID and make the consumer recognize redelivery before accepting duplicate work. Transactional outbox

One commit supports two independent actions

Example

The caller can receive a durable acknowledgment while the relay is offline.

Task and instruction committedOne local transaction.
Acknowledge task
Dispatch relayReads instructions still owed.
Worker queueSeparate service.
A transaction saves the task and its dispatch instruction together. After commit the service may acknowledge the task and a relay may publish the instruction in either order. A replacement relay uses the same instruction ID on redelivery.
Read the diagram as text
  • Task and instruction committed. One local transaction.
  • Acknowledge task.
  • Dispatch relay. Reads instructions still owed.
  • Worker queue. Separate service.
  • Task and instruction committed → Acknowledge task: Commit permits acknowledgment.
  • Task and instruction committed → Dispatch relay: Pending instruction and ID.
  • Dispatch relay → Worker queue: Publish with original ID.

An authoritative record is the one an executor consults to decide what may happen next. For a long run, this includes intent, state version, accepted observations, pending operations, remaining requirements, committed allowance, and output references. A transcript can preserve everything the model said while omitting a pending dispatch. Anthropic’s Managed Agents illustrates the separation. Session events survive outside the harness process, the harness selects what to put in a model call, and a sandbox provides replaceable compute. Reconstructing the input to the model and restoring its working files are separate operations. Managed Agents architecture · Context Engineering

Save a useful recovery position

A checkpoint saves a position in an execution and the state needed there. A continuation identifies the work remaining at that position. In SnapLogic’s prototype, an agent suspends just before a tool call needing approval. Inspect the fields of the returned object. messages provides conversation history. resume_request identifies the pending call. processed carries decision information used on resumption. With only the messages, the model could discuss the proposed action again. The executor needs the call identity to release the particular action that was reviewed. Returning a serializable object makes it possible for the caller to store it durably. SnapLogic continuation record

Same interruption in both cases. The worker exits after the model chooses a tool and before dispatch.
Last committed positionWhat survivesFirst unfinished operation
Before inferenceInputs to the choiceCall the model again. It may choose different arguments.
After inferenceTool name, arguments, and pending call identityCheck whether the saved call may still run, then dispatch if permitted.
In this conceptual arrangement, replacement execution reads accepted records and retained output bytes, not the former worker’s memory. The read request and returned records share one exchange; selected context is derived input. Persistence depends on the configured storage contract.

Coverage matters as much as placement. LangGraph checkpoints can contain graph values, the next nodes to run, pending tasks, and identifiers for the thread and checkpoint. Nodes scheduled in one step can run concurrently. A successful node’s separately persisted output can spare recomputation when its peer fails. Finer boundaries add writes. Synchronous persistence waits for a write before proceeding, whereas an outstanding asynchronous write may be lost. Use a backend that survives the failures in scope, and keep the referenced files and configuration available. A disk snapshot can restore a file whose path appears in the checkpoint. Its contents cannot tell you whether a remote service subsequently changed. LangGraph checkpointers · Snapshot coverage

Reconstruct execution safely

Direct restoration loads saved state and resumes at a known position. History-based replay runs the coordination code again, supplying previously recorded results as that code asks for them. In Temporal, a unit of external work such as an inference request is called an activity. In Samuel Colvin’s demonstration, a replacement reached earlier model activities and immediately received their saved responses. It contacted the model provider when it reached a new inference. The figure applies the same mechanism to a saved tool choice. Watch which component supplies the arguments after restart. Colvin’s demonstration

Replay depends on compatible control flow. If a branch reads the wall clock again, draws a fresh random number, or queries the model directly, it can schedule a different operation. Use the runtime’s replay-aware primitives or put the observation in an activity whose result is recorded. A code edit can also break replay. For example, moving a timer before an activity can issue a timer command where the history expects the activity. In diagnostic replay, changing an input can be the point of the exercise. In recovery, it can change the meaning of unfinished work. Temporal workflow definitions · Diagnostic replay

One tool choice survives two workers

Example

History supplies the arguments during recovery. Only unfinished work needs a new call.

1 / 3 · Record the choice

The result commits before the first worker exits.

New this stepOriginal model activity
New this stepRecorded tool choiceOne saved name and argument set.
The original model activity returns a tool choice and the first worker records it before exiting. The replacement receives that same result during replay and dispatches the previously unstarted tool.
Read the diagram as text
  • Original model activity.
  • Recorded tool choice. One saved name and argument set.
  • Replacement worker. Runs compatible scheduling code.
  • Chosen tool. Dispatch had not begun at interruption.
  • Original model activity → Recorded tool choice: Record returned choice.
  • Recorded tool choice → Replacement worker: Return historical result.
  • Replacement worker → Chosen tool: Dispatch saved arguments.
  1. Record the choice. The result commits before the first worker exits. Active: Original model activity, Recorded tool choice. New: Original model activity, Recorded tool choice.
  2. Recover the choice. The replacement consumes the recorded result at the same activity boundary. Active: Original model activity, Recorded tool choice, Replacement worker. New: Replacement worker.
  3. Dispatch the tool. Execution reaches its first new external call. Active: Original model activity, Recorded tool choice, Replacement worker, Chosen tool. New: Chosen tool.

For direct restore, a new build needs to understand the checkpoint schema and what each pending action means. For replay, it also has to issue commands compatible with the stored history. Temporal supports pinning a workflow to a deployment version. Another process on that build can take over, but the operator must keep suitable workers available until the run ends. An upgrade policy trades that operating cost for compatibility work. Either way, a service can revoke access while the run is offline. Worker versioning

Historical development

Why recovery mechanisms coexist

Recovering from an incorrect computation raises a different problem from recovering data after a crash. In 1975 Brian Randell described a recovery block that saved its entry state, ran a primary implementation, and tested the result before releasing it. If the acceptance test failed or could not finish, the block restored the entry state and tried an alternative. That boundary works when the system can withhold unacceptable results. Once another party has acted on a result, restoring local variables cannot restore the earlier situation. System Structure for Software Fault Tolerance

Database recovery had to preserve committed work while transactions updated shared storage. In 1992, Mohan and colleagues described how ARIES combined write-ahead logging with checkpoints. Before a changed database page could replace its previous version on disk, the system had to save the log records needed to recover the change. A checkpoint summarized active transactions and modified pages while updates continued. After a crash, that summary helped the system find where to begin recovery in the log without requiring every changed page to have been written at checkpoint time. ARIES

For long-lived telecommunications services, a failed process also meant lost capacity. Joe Armstrong traced Erlang to a 1986 experiment with concurrent processes and described Ericsson’s 1996 Open Telecom Platform project as a stable software base for applications. A supervisor could detect an abnormal exit and restart a worker or a related group. The supervisor could thus supply a new process while a separate recovery record supplied that process with unfinished work. Armstrong’s account

Work involving people and several applications also needed an owner for the sequence of steps. The November 1994 Workflow Reference Model separated human worklists, invoked applications, process definitions, and the control data for each running instance. Amazon Simple Workflow’s February 2012 launch illustrates what a managed coordination service could take on. Activity workers performed tasks, deciders consulted history to choose subsequent work, and the service assigned tasks, retained results, and tracked timeouts. A signal could enter history while no decider was processing it. Workflow Reference Model · Amazon Simple Workflow launch

A working environment can itself be expensive to reconstruct. In January 2024 Trigger.dev proposed hosting task execution and checkpointing containers at waits, after an earlier design replayed functions using saved step results. Its September 2024 general-availability announcement described suspending a parent while its child ran. For a coding agent, the distinction is tangible. An execution history can say which tool the model chose. A machine snapshot covering memory and processes can bring back the development server that tool was using. The next user message still needs an address so it can wake the right work. January proposal · September release · Eric Allam on replay and snapshots

Dispatch and recovery

Wake retained work

A durable wait records what would make a continuation runnable, then releases the worker. A signal is an external event addressed to the execution. A durable timer records a time at which it becomes eligible to resume. Melanie Warrick demonstrated the distinction by taking an application worker offline and approving a change to an order. Temporal retained the decision. When the worker returned, the order continued. Each order had its own workflow, so one outstanding review could wait while other orders proceeded. Warrick’s demonstration

The first race occurs before anyone has waited. If a service sends a review link before creating the instance meant to receive the reply, an immediate answer may be lost. Azure Durable Task buffers early events for an existing instance and discards events for a nonexistent one. One design commits the pending wait and the instruction to notify the reviewer together. On receipt, match the event to its wait ID and revision, atomically consume that wait and record runnable work, then acknowledge the event. A duplicate reply or losing timer finds the wait already consumed. Processing the reply can still yield a denial. External events in durable orchestrations

A reply meets one pending wait

Example

Replies can arrive out of order because each addresses its own wait.

Pending waitRun, call, and revision.
Incoming replyTarget wait and decision.
Match and consume waitCheck current state in the same write.
Queue decision for processing
No new work
Compare the incoming event with the stored wait. Consume a matching eligible wait and enqueue its decision in one transaction. A duplicate or obsolete event cannot enqueue it again. The decision still needs to be checked before tool execution.
Read the diagram as text
  • Pending wait. Run, call, and revision.
  • Incoming reply. Target wait and decision.
  • Match and consume wait. Check current state in the same write.
  • Queue decision for processing.
  • No new work.
  • Pending wait → Match and consume wait: Expected event and revision.
  • Incoming reply → Match and consume wait: Event ID and target.
  • Match and consume wait → Queue decision for processing: First matching reply.
  • Match and consume wait → No new work: Consumed or obsolete wait.

Eligibility does not reserve a worker. A bounded queue absorbs a burst. If arrivals continue to exceed capacity, backpressure asks callers to slow down or defers or refuses admission. Check the remaining task deadline before dispatch. Time in the queue counts against an elapsed deadline even though the worker’s attempt timeout has yet to start. A background task may have a different deadline from the HTTP request that created it.

Reject obsolete workers

A lease grants a worker temporary authority. A heartbeat reports contact. If heartbeats stop, the process could be dead, paused, or unable to reach the coordinator. Reassignment can restore service while leaving two workers capable of sending results. Even Amazon SQS’s visibility timeout, which normally hides a received message from other consumers, allows duplicates during the visibility interval. SQS visibility timeout

Fencing lets the recipient reject a request from a former owner. In the illustrated task-row design, claiming the row atomically assigns an ownership generation. A takeover increments it. Renewal requires a current, unexpired claim. A result write must match that claim, the expected task revision, and a state that still accepts results, all in the update itself. Both workers read revision 4. Even before the replacement has changed the result, generation 7 has lost the right to write it. A separate ownership check followed later by an unconditional write would leave time for another takeover. Conditional state updates · PostgreSQL transaction isolation

Two workers return results for revision 4

Example

The task revision can match even after the writer has lost its lease.

Former workerDelayed across takeover.
Current workerHolds the valid lease.
Guarded task rowExpects generation 8 and revision 4.
Reject result
Advance revision
After takeover, the task row expects generation 8. Both results use revision 4. The write from generation 7 fails the ownership check. The current lease holder may advance the row if the revision and state also match.
Read the diagram as text
  • Former worker. Delayed across takeover.
  • Current worker. Holds the valid lease.
  • Guarded task row. Expects generation 8 and revision 4.
  • Reject result.
  • Advance revision.
  • Former worker → Guarded task row: Generation 7, revision 4.
  • Current worker → Guarded task row: Generation 8, revision 4.
  • Guarded task row → Reject result: Superseded generation.
  • Guarded task row → Advance revision: Valid claim, state and revision.

Chubby’s sequencers put lock identity and generation in a form a recipient could check. The same placement matters for tools. Rejecting a former worker’s database update still leaves any remote request it sent in flight. A receiving API needs its own authority check to reject obsolete callers. Even if it suppresses duplicate requests, it could accept the former worker’s request first. The Chubby lock service

Bound repeated attempts

A retry makes another attempt at the same logical operation. Resuming continues from a recorded position. Restarting a job can repeat several operations, including ones that succeeded. Replanning gives the agent another choice, which may require different permission. First identify which unit failed and what the response establishes about its effects. Agent recovery decisions

ObservationAppropriate next step
A repeatable read met a transient outage.Retry the read within its time and attempt allowance.
The tool rejected an invalid argument.Return the validation detail so the caller can correct it.
The current identity lacks authority.Withhold dispatch. Obtain applicable authority if the task calls for it.
A write timed out after dispatch.Keep the original operation pending and resolve its outcome or use protected redelivery.

For a safe retry, capped exponential backoff lengthens successive delays up to a maximum. Jitter varies those delays so recovering clients spread their requests over time. Count attempts at the dependency itself. If a workflow tries three times and its client library tries three times per invocation, one operation permits nine downstream calls. Choose one layer to own retry policy, including any attempts a gateway or SDK makes internally. Preserve the count and usage across restarts. Fit the next delay and attempt within the original deadline. AWS retry guidance · Model Routing and LLM Gateways

Resolve uncertain effects

Suppose a tool should create one issue. The worker records the intended creation, sends it, and exits without recording a reply. Two histories now fit the local evidence. The request might have been lost before it reached the issue service. Or the service might have created the issue and its reply was lost. Workflow replay can reuse a recorded activity result, but neither history contains one. Making the activity smaller narrows how much work can repeat. It cannot eliminate the interval between a remote commit and a local record. Temporal activity definitions · Uncertain external outcomes

The depicted effect occurred while the caller lacks confirmation. Repetition within a valid receiver contract or authoritative lookup may resolve the outcome; otherwise it remains unresolved. The two persistence regions do not share a transaction, and recovery does not rewind the effect.

Receiver-enforced retries

Idempotency means that repeating the same logical operation produces no additional effect. Retain the caller, intent, original parameters, operation key, delivery attempts, and any returned handles. For a keyed creation, the receiver must recognize the key in the caller’s scope, reject changed parameters, and coordinate recording the key with creation so concurrent deliveries cannot both create. Identical titles alone are insufficient because two intentionally distinct issues can have the same title. AWS on idempotent APIs · Tool dispatch

Two remote histories leave one local record

Example

A pending creation is compatible with either zero or one issue at the receiver.

Request lostNo issue created.
Issue createdReply never reaches durable local history.
Pending issue creationOriginal key and parameters survive.
Record original outcome
Retry original key and parameters
Retain unresolved operation
Losing the request before creation and losing the reply after creation both leave the operation pending. Conclusive lookup settles it. If lookup is inconclusive, a still-enforced operation key can permit safe retry. Otherwise the outcome remains unresolved.
Read the diagram as text
  • Request lost. No issue created.
  • Issue created. Reply never reaches durable local history.
  • Pending issue creation. Original key and parameters survive.
  • Record original outcome.
  • Retry original key and parameters.
  • Retain unresolved operation.
  • Request lost → Pending issue creation: No reply generated.
  • Issue created → Pending issue creation: No reply recorded.
  • Pending issue creation → Record original outcome: Conclusive lookup.
  • Pending issue creation → Retry original key and parameters: Inconclusive lookup, key valid.
  • Pending issue creation → Retain unresolved operation: Inconclusive lookup, no protection.

Read the receiver’s retention and error rules. Stripe, for example, stores the status and body of the first executing request for a key, including an HTTP 500. A validation failure or concurrency conflict before execution begins leaves no stored result. A repeated 500 can leave the underlying mutation indeterminate. Keys can be pruned once at least 24 hours old, after which reuse can initiate another request. A new key issued to get past a cached error can also duplicate the original intent. Stripe idempotent requests · Low-level errors

Finding or counteracting the effect

Reconciliation means consulting the system that owns the outcome. For the issue tool, a conclusive lookup by the original operation ID can settle whether the issue exists. An empty title search cannot do that reliably. If lookup is inconclusive and retry protection has expired, retain the uncertainty for investigation. If the service confirms an unwanted creation, closing the issue is a compensation, a new authorized operation that can itself fail. Garcia-Molina and Salem’s 1987 saga model explains a limit of compensation. Other participants may already have observed the committed change. Closing an issue leaves earlier notifications in recipients’ hands. Sagas

Limits and operator control

Preserve limits across restarts

Specify both the unit of a limit and who shares it. An elapsed deadline advances during queueing and review. Active compute time measures work performed. An attempt timeout bounds one try. A turn limit or tool-call limit bounds steps, and a concurrency limit restricts simultaneous work. If the allowance covers a run, a new worker or child must inherit its remaining allowance. A soft threshold can warn the operator. At a hard threshold the runtime refuses admission and records why work stopped. Activity timeouts · Choosing additional reasoning

Counting usage only when responses arrive leaves a race. Two children can both see enough remaining allowance before either reports a cost. Pydantic AI documents this limitation for its spend counter. A reservation charges pending work against admission before dispatch. In the example, each child checks and reserves in one transaction against the same ledger. Available units equal the limit less consumed units and outstanding reservations. Pydantic AI spend limits

Illustrative shared allowance of 10 units
EventConsumedReservedAvailable
Before dispatch208
Reserve 5 for child A253
Decline child B’s request for 4253
Replacement worker reads the ledger253
Settle A at 3 units and release 2505

Settle each reservation once against the original operation. TigerBeetle’s pending transfers illustrate partial settlement and release of the remainder. For an agent, a strict monetary ceiling also requires an enforceable upper bound on what a dispatched call can cost. A local timeout may occur before provider billing arrives. Keep the reservation until usage is reconciled, and budget separately for cleanup so refusing new work does not prevent an orderly stop. Two-phase transfers · Workload economics

Stop owned work

Cancellation asks the runtime to stop future and ongoing work within a specified scope. Keep its control path usable even if the current tool never returns. Persist the request, close admission, and invalidate pending dispatches and timers. Then account for work that has already escaped the queue. A disconnected client has stopped listening, a timeout ends a wait, and a pause preserves an opportunity to resume. Each leaves open the question of what is still running. Operator-facing controls

Trio gives local tasks an owner through a nursery, a scope that waits for its children to exit. Children observe cancellation at cooperative interruption points. Cleanup that needs I/O can run in a shielded scope with its own timeout. Erlang supervisors offer a different shutdown policy, requesting exit and, when configured, forcing termination after a grace period. Killing an intermediate supervisor too soon can leave its descendants alive. The evidence of shutdown is the disposition of the owned children. Trio task lifetimes · Erlang supervision · Sandbox lifecycle

Three consequences of a stop request

Example

Closing local dispatch can precede both child exit and remote resource release.

Persist stop request
Close local dispatchInvalidate pending starts.
Interrupt owned child
Local exit confirmed
Check provider jobRetain the original handle.
Cleanup outstanding
Remote release confirmed
Persisted stop intent closes admission, requests interruption of owned children, and initiates supported cancellation for a remote job. Child exit and provider status supply independent shutdown evidence. An EMR job reporting CANCELLING still has outstanding cleanup.
Read the diagram as text
  • Persist stop request.
  • Close local dispatch. Invalidate pending starts.
  • Interrupt owned child.
  • Local exit confirmed.
  • Check provider job. Retain the original handle.
  • Cleanup outstanding.
  • Remote release confirmed.
  • Persist stop request → Close local dispatch: Reject new work.
  • Persist stop request → Interrupt owned child: Propagate cancellation.
  • Interrupt owned child → Local exit confirmed: Observe child exit.
  • Persist stop request → Check provider job: Cancel by job ID, then query.
  • Check provider job → Cleanup outstanding: CANCELLING.
  • Check provider job → Remote release confirmed: CANCELLED.

A remote service has its own lifecycle. Amazon EMR Serverless uses CANCELLING while it attempts shutdown and CANCELLED after cancellation succeeds and resources are released. Azure Durable Task can terminate an orchestration while its activity functions continue. Retain job handles, query their owners, and expose outstanding cleanup. If completion races cancellation locally, use a conditional transition to accept one terminal disposition. A later provider reply may settle a pending effect without reopening that run. EMR job states · Azure instance management

Control long waits and approvals

Address an operator command to a durable run handle. Status reads accepted state. Pause requests suspension at a defined boundary, resume releases eligible work, and redirect changes pending instructions. A redirect that arrives after dispatch leaves the original operation to resolve. Cancel starts shutdown. Distinguish receipt of each command from its outcome. In Temporal, a Signal can return after server acceptance and before workflow delivery. An Update exposes acceptance and handler completion separately. A handler awaiting another operation may still be running when the main workflow reaches its return, so a workflow promising a processed decision must wait for that handler. Temporal message passing

Binding a decision to an action

An approval applies to the action the reviewer saw. If a reviewer approves publishing a report in a private folder and the destination later changes to a public one, the public release needs a new decision even though the report bytes are identical. Save the reviewed call ID, report revision, arguments, reviewer, and expiry with the continuation. Before dispatch, compare those details with the pending call and check current permissions and preconditions. For a match, consume the valid approval and release the call in one guarded transition so a duplicate reply cannot release it twice. Denial, cancellation, expiry, or a change to the action keeps that call from running. Other publications can proceed as their own reviews arrive. OWASP transaction authorization · Access governance

A decision authorizes one destination

Example

The same report and reviewer can yield a different authorization decision when the destination changes.

Reviewed publicationReport version and private destination.
Pending publicationCurrent report version, destination, and permissions.
Validate and consume decisionCheck call identity, reviewer, validity, and current scope.
Release call once
Withhold dispatch
Compare the reviewed publication with the current pending call. A permitted, unexpired first approval for the same private destination releases it once. A public destination, denial, expired authority, or consumed decision withholds dispatch.
Read the diagram as text
  • Reviewed publication. Report version and private destination.
  • Pending publication. Current report version, destination, and permissions.
  • Validate and consume decision. Check call identity, reviewer, validity, and current scope.
  • Release call once.
  • Withhold dispatch.
  • Reviewed publication → Validate and consume decision: Decision and reviewed arguments.
  • Pending publication → Validate and consume decision: Current action and authority.
  • Validate and consume decision → Release call once: Approved, valid, matching, unused.
  • Validate and consume decision → Withhold dispatch: Denied, changed, invalid, or used.

Record who owns a wait, which input would resolve it, and when to escalate. A watchdog inspecting those fields can distinguish an order legitimately awaiting a reviewer from a tool call whose worker disappeared. Bound reminders and give an expired decision an explicit disposition. In Vinoth Govindarajan’s account of an OpenClaw incident, retrying an expired approval callback preserved it across restarts and blocked later work. Govindarajan’s account · Review interfaces

Deliverables and evidence

Retain and publish artifacts

An artifact is an identifiable output retained for a later consumer, such as a report, patch, or dataset. A scratch file serves the worker creating it. A durable candidate must remain readable after that worker exits. Bound which files and how many bytes may leave the workspace, give each candidate an immutable identity, and assign an owner to retain it while consumers still depend on it. A checkpoint can then carry a reference to the output. Filesystem views and exports · Retention policy

Consider a report uploaded to Amazon S3 in parts. Before completion, persisted parts occupy storage even though consumers cannot open the assembled report. Complete the upload and confirm the readable object version. Then publish a manifest entry naming its location, digest, input versions, and producing execution. A digest identifies bytes. The other fields record provenance, allowing someone to find the inputs and run that produced them. A conditional write can guard a published reference against replacement. If registration lives in a separate database, retain enough information to finish it after a crash between assembly and registration. S3 conditional writes · SLSA provenance

When a report becomes readable and discoverable

Example

Assembly can succeed before a manifest tells consumers where to find the bytes.

1 / 3 · Stage

Uploaded fragments exist before a complete object can be opened.

New this stepReport candidateStable identity across stages.
New this stepStaged partsFragments of the report upload.
Staged parts belong to one report candidate. Completing the upload produces a readable object version. A published manifest references that version. Earlier stages remain visible as production history. The manifest supplies a usable reference before any judgment about content quality.
Read the diagram as text
  • Report candidate. Stable identity across stages.
  • Staged parts. Fragments of the report upload.
  • Readable object version. Complete report bytes.
  • Published manifest entry. Location, version, and provenance.
  • Report candidate → Staged parts: Stage candidate bytes.
  • Staged parts → Readable object version: Assemble on upload completion.
  • Published manifest entry → Readable object version: Reference exact version.
  1. Stage. Uploaded fragments exist before a complete object can be opened. Active: Report candidate, Staged parts. New: Report candidate, Staged parts.
  2. Assemble. The object is readable. A replacement can finish registration. Active: Report candidate, Staged parts, Readable object version. New: Readable object version.
  3. Publish. Consumers can resolve the manifest entry to the stored report. Active: Report candidate, Staged parts, Readable object version, Published manifest entry. New: Published manifest entry.

An interrupted upload also gives cleanup a testable outcome. S3 warns that a part already in flight can finish after the service accepts an abort request. Use ListParts to check that the part list is empty and repeat cleanup if necessary. Retaining the upload ID lets a replacement finish that obligation without disturbing another upload. AbortMultipartUpload

Enforce completion checks

A postcondition is a fact that must hold after the work. A completion check obtains evidence for it. A delivered report, for example, may have two requirements. Its contents must pass a specified check, and the destination must expose the checked version. Store a receipt naming the requirement, candidate, checker, and observation. A content pass for the new report and a destination receipt for the old one establish different facts. Delivery of the new version remains unconfirmed, so an automatic resend could duplicate it. Outcome evidence

Decisions for a required check
Check dispositionWhat the executor can do
Current passAccept this requirement within the check’s scope. Inspect any others.
Failed assertionIdentify the violated condition and allow bounded repair.
Checker unavailableWithhold success. Retry the check within limits or hand it off.
Pass for a different candidateObtain evidence for the candidate now under consideration.

Enforce success in one guarded transition that verifies run eligibility, current candidate and requirement versions, and positive receipts for every required check. Wiring in a checker alone can leave a gap. Claude Code’s TaskCompleted hook can return exit code 2 to prevent completion and send feedback to the model. Some hook failures and timeouts can be nonblocking, and the onFailure blocking option does not apply to this event. A mandatory check therefore needs an explicit pass to authorize success. If repair runs out of time, return usable partial work and identify the requirement still open. Claude Code hooks · Evaluator design

Expose authoritative status

A returning client needs a status query even if it missed every progress notification. Google’s long-running Operations protocol gives work a name and separates progress metadata, a done flag, and a terminal result or error. A wait call can return while work is still underway. For an agent run, expose the last accepted change, waiting reason, cancellation stage, remaining allowance, and unsettled effects. Each field should come from the record responsible for that claim. Google Operations protocol

One reconnect design returns a status snapshot with a version and lets the client request the accepted updates after that version. Apply the next version in order, ignore duplicates, and fetch current state again on a gap. A delayed running update then cannot overwrite a newer terminal record. Show last worker contact separately from last accepted progress. They can diverge when a worker keeps sending heartbeats while it waits on a dependency.

A trace links diagnostic records of related operations. A span describes one observed operation. Correlate spans with the stable run and operation IDs, the particular worker attempt and generation, the checkpoint it read, and the artifact and check receipts it produced. In Govindarajan’s account, a message tool returned success although the message failed to render. A successful send span helps locate where the evidence ends. An observation at the rendering client would support the stronger delivery claim. Until then the service can report send acceptance and leave rendering unconfirmed. Govindarajan on receipts · Traces and spans

Verification and design choices

Test interruption boundaries

Write down what must remain true after recovery, then interrupt the execution on either side of the event that could change it. FoundationDB’s simulation work illustrates this method with controlled timing, injected failures, and assertions about recovered state. For an agent, distinguish the number of physical calls, objects created at the receiver, and transitions accepted locally. Cornelia Davis demonstrated recovery by terminating a Temporal application worker and starting another. That experiment tests loss of worker memory with the history service still available. FoundationDB simulation testing · Davis’s demonstration

Constructed tests for a runtime promising durable acceptance and checked delivery
Interruption or raceObservation required after recovery
Stop before task commit, after commit before publish, and after publish before the relay records delivery.Every acknowledged task has recoverable dispatch. A repeated instruction admits one logical job.
Hold the saved choice and first remote creation fixed. Allow one retry. Crash before versus after recording the receiver’s reply.After recording, replay reuses the result. Before recording, inspect the actual object count. An unprotected second creation exposes the duplicate-effect bug.
Stop during a checkpoint write and then just after commit.Recovery reads a complete committed position. Count which inference calls repeat.
Deliver an approval twice, after expiry, or after changing its destination.Only a first applicable approval releases the reviewed action.
Delay a worker’s write until after takeover. Submit cancellation twice.The old write is rejected. Closed admission stays closed and children have inspectable dispositions.
Restart with a live reservation and deliver the same settlement twice.No new worker spends the reserved units. Usage settles once.
Stop during multipart upload and after assembly before publication.No complete reference points to partial bytes. Registration can finish for the original assembled candidate.
Lose the checker or replace the candidate between its pass and acceptance.Neither missing evidence nor a stale pass permits success. An accepted reference opens the checked version.

Fix the model response and control the receiver to isolate a particular race. Exercise the real receiver separately for its key retention, status lookup, and cancellation behavior. A substitute that remembers keys indefinitely will pass a test a real service can fail. Before deploying changed workflow code, replay representative histories from both open and closed executions in continuous integration. Successful replay checks compatibility for those histories. Assessing whether the agent chose a good action requires a task-quality trial with an outcome criterion. Focused tests · Integration boundaries

Choose sufficient machinery

Compare designs at the same interruption. The model has chosen a tool, the worker exits before dispatch, and the caller still wants the task completed. If making another choice is cheap and acceptable, recovery at the start of a small job may suffice. If the exact choice is part of an outstanding review, losing it invalidates the review. That one requirement changes the necessary recovery point.

ArrangementFirst recoverable positionCost and responsibility
Supervised in-process loopThe supervisor can restart the process. A choice held only in memory must be made again.The application owns timeouts, waits, and recovery of any promised work.
Persisted job with BullMQA stalled job can return to waiting and restart at job entry. Reuse requires an application record of the choice.Operate the queue, bound stalls, make redelivery safe, and persist any wait that must outlive a worker.
LangGraph with persistent checkpointsStart at a committed graph boundary. A choice in saved state can be reused. Work inside an unfinished node may repeat.Choose node boundaries and write timing, retain checkpoints, and restore referenced inputs.
Temporal with inference in an activityReconstruct control flow and consume the saved choice if the activity result reached history.The runtime retains history and waits. Operators maintain compatible code and available workers.

BullMQ’s stalled-job mechanism illustrates the operating tradeoff. A blocked Node.js event loop can miss its liveness notification even while its process is alive. Repeated stalls can eventually fail the job. For checkpointed or replayed work, account instead for retained state, growing history, and compatible workers across deployments. Measure time from interruption to useful progress and the calls repeated before reaching it. For a long review, also measure the resources retained while idle. A workspace snapshot can spare a rebuild while bringing back background processes that need cleanup. BullMQ stalled jobs · Replay and snapshot tradeoffs

Codex Web’s server-side session illustrates recovery from a browser disconnect. Recovery of a tool action interrupted by worker loss needs a separate guarantee. A schema defines the arguments a tool accepts, a protocol connects clients and servers, and a platform team may operate the shared service. The execution decision is which component retains the pending call and can account for its outcome after interruption. Structured Outputs and Tool Calling · Model Context Protocol · AI Platform Engineering

Frequently asked questions

Is a transcript a checkpoint?

Only if the transcript also records the control position and everything required to continue there. A conversation can describe a proposed tool call while omitting the identity of the pending approval. Context Engineering

Can work survive a closed tab or a timed-out request?

Yes, if an independent service owns the execution and gives the client a handle to reopen it. Codex Web supports reconnecting to server-side work. Query the run to learn its current state.

Can a recovered activity create a second object?

Yes. A receiver may create the first object before the worker loses the reply. Recover using the original operation identity and a receiver-enforced retry or conclusive status lookup. Otherwise retain the unknown outcome.

Does cancellation undo a change?

A committed change survives cancellation. If it needs reversing, verify its state, obtain authority for the compensating operation, and check the outcome of that operation too.

What does a sandbox snapshot preserve?

A disk snapshot preserves covered files. Resuming a live process also requires its execution state. External services remain authoritative for actions they received before or after capture. Sandbox lifecycle and snapshots

Does a heartbeat prove progress?

A heartbeat establishes contact at the time of the report. If it includes progress details, inspect the latest received checkpoint and the durable work it identifies. A fresh contact time can accompany the same completed segment as before.

Can new code resume an old run?

It can when it understands the saved state and produces commands compatible with any recorded history. Otherwise keep a worker on the pinned build or provide an explicit migration that preserves pending work.

Is the task complete when the model stops?

A final message ends a Codex turn and may even ask for more information. Task success requires the observations specified for the requested outcome. Outcome evidence

When is a queue enough?

A persistent queue is enough when the job can safely and affordably restart at entry and the application records any pending waits and external operations. Save a finer checkpoint if restarting would discard an approved choice or repeat expensive work. If a remote write may have succeeded before its result was saved, the application still needs to check the outcome or retry under a key the receiver enforces.

Open questions

  1. When can an operator retire a pinned build that still has approval waits? A new build may read the old record yet interpret a destination or pending command differently. Replaying histories with live waits, changed arguments, and revoked authority would test whether migration preserves the original decision.

  2. When can a run release a reservation for a provider request whose usage never arrived? Holding it indefinitely blocks useful work, while releasing it early can overspend. A provider-enforced maximum and a settlement record tied to the original request would give the runtime a defensible release condition.

  3. Which report requirements can be accepted without a human reader? An exact digest and passing structural checks can coexist with a missing answer. Comparing automated judgments with domain reviewers on those omissions would show which decisions the harness can safely automate.

Follow the curated reading path through the speakers and demonstrations behind this entry.

Explore more talks

The rest of the library, beyond the curated path. Cited talks support this entry; reviewed transcripts were processed in full. Metadata candidates have not been reviewed as sources or verified as topic members.

213 matching talks

Every catalogued talk on this subject: Developer workflows & testing →

TalkSpeakerEventYear
Aditya BhargavaAI Engineer World's Fair 20262026
Vinoth GovindarajanAI Engineer World's Fair 20262026
Why Software Factories Fail

Transcript reviewed

Dex HorthyAI Engineer World's Fair 20262026
Codex, Behind the Harness

Transcript reviewed

Dominik KundelAI Engineer World's Fair 20262026
Harnesses in AI: A Deep Dive

Cited in this entry

Tejas KumarAI Engineer Europe 20262026
AI Engineering 101

Transcript reviewed

Noah HeinAI Engineer Summit 20232023
Nico AlbaneseAI Engineer Summit 20252025
Jim BennettAI Engineer World's Fair 20252025
Chau TranAI Engineer World's Fair 20252025
Yohei NakajimaAI Engineer World's Fair 20262026
Yuval BelferAI Engineer World's Fair 20252025
Mahesh MuragAI Engineer Summit 20252025
Jon PeckAI Engineer World's Fair 20252025
The Future of MCP

Transcript reviewed

David Soria ParraAI Engineer Europe 20262026
Anton TroynikovAI Engineer Summit 20232023
Louis-François Bouchard, Omar Solano, Samridhi VaidAI Engineer World's Fair 20262026
Sally-Ann DeLuciaAI Engineer Europe 20262026
Cornelia DavisAI Engineer Code 20252025
Simon WillisonAI Engineer Summit 20232023
Fouad MatinAI Engineer World's Fair 20252025
Security Firewall for Agents

Transcript reviewed

Ryan DahlAI Engineer World's Fair 20262026
Sandipan BhaumikAI Engineer Europe 20262026
Will BrownAI Engineer World's Fair 20262026
Rishi DesaiAI Engineer World's Fair 20262026
Nick HeinerAI Engineer World's Fair 20262026
LLM Evals That Work IRL

Transcript reviewed

Aparna Dhinkaran, Aparna DhinakaranAI Engineer World's Fair 20242024
Nick Ung, Akshay SharmaAI Engineer World's Fair 20262026
Mahmoud MabroukAI Engineer Europe 20262026
Sarmad QadriAI Engineer World's Fair 20252025
Adam TerlsonAI Engineer Summit 20252025
Zhou YuAI Engineer Summit 20252025
Tom SmokerAI Engineer World's Fair 20252025
Jesse HuAI Engineer Code 20252025
Ronak MaldeAI Engineer World's Fair 20262026
Mason EggerAI Engineer World's Fair 20252025
Misha Kaletsky, Jonas TemplesteinAI Engineer Europe 20262026
Drasko ProfirovicAI Engineer World's Fair 20262026
Hamza TahirAI Engineer World's Fair 20262026
Abhishek BhardwajAI Engineer World's Fair 20262026
Brendan RappazzoAI Engineer World's Fair 20262026
KP Sawhney, Ian BallantyneAI Engineer Europe 20262026
Greg BensonAI Engineer World's Fair 20252025
Dat NgoAI Engineer Europe 20262026
Laurie VossAI Engineer Europe 20262026
Chintan Agrawal, Daniel WirjoAI Engineer World's Fair 20262026
Brian JohnAI Engineer Code 20252025
Dominik KundelAI Engineer World's Fair 20252025
Nishant GuptaAI Engineer World's Fair 20262026
Cornelia DavisAI Engineer World's Fair 20262026
Gagan Bhat, Isabella Kai HeAI Engineer World's Fair 20262026
Preeti SomalAI Engineer World's Fair 20252025
Kam LasaterAI Engineer Summit 20252025
Jeremiah LowinAI Engineer Code 20252025
Viren BaraiyaAI Engineer World's Fair 20262026
Dan FarrellyAI Engineer World's Fair 20262026
Peter WielanderAI Engineer Code 20252025
Aman SinghEvent unknownUnknown
Ben DickenAI Engineer World's Fair 20262026
Derek MeeganAI Engineer Code 2025Unknown
Qianru Lao, Lu ZhangEvent unknownUnknown
Salman MunafAI Engineer World's Fair 20262026
Wolfram RavenwolfAI Engineer World's Fair 20262026
Matt BrockmanAI Engineer World's Fair 20262026
DottaAI Engineer World's Fair 20262026
Atul RamachandranAI Engineer World's Fair 20262026
Alex Volkov, Benjamin EckelAI Engineer World's Fair 20252025
Fuad AliEvent unknownUnknown
Mike ChambersAI Engineer World's Fair 20262026
Rajiv ChandegraAI Engineer World's Fair 20262026
Soheil FeiziAI Engineer World's Fair 20262026
Sam BhagwatAI Engineer World's Fair 20262026
Vasant KearneyAI Engineer World's Fair 20262026
Stefania DrugaAI Engineer World's Fair 20262026
Yogendra MirajeAI Engineer World's Fair 20262026
Dmitry PetrovAI Engineer World's Fair 20262026
Kay MalcolmEvent unknownUnknown
Varun KrovvidiEvent unknownUnknown
Dru KnoxAI Engineer World's Fair 20262026
Laurent Gil, Žilvinas UrbonasAI Engineer Code 2025Unknown
Dominik KundelAI Engineer World's Fair 20262026
Jediah KatzEvent unknownUnknown
Nacho MartinezEvent unknownUnknown
Mithun HunsurAI Engineer Summit 20232023
Taylor Jordan SmithAI Engineer World's Fair 20252025
Max DrakeEvent unknownUnknown
James ZouAI Engineer World's Fair 20262026
Ignacio MartinezAI Engineer World's Fair 20262026
A Genius With Amnesia

Metadata candidate

Victor SavkinAI Engineer World's Fair 20262026
Barry Zhang, Mahesh MuragAI Engineer Code 20252025
Nicholas Kang, Michael AaronAI Engineer Europe 20262026
Nick Nisi, Lizzie SiegleAI Engineer World's Fair 20252025
Dax RaadAI Engineer Code 20252025
AI SDK v6

Metadata candidate

Nico AlbaneseAI Engineer Europe 20262026
Paul Klein IVAI Engineer World's Fair 20262026
Rita KozlovAI Engineer World's Fair 20252025
Stephen ChinAI Engineer Europe 20262026
Mahesh SathiamoorthyAI Engineer World's Fair 20262026
Max Kanat-AlexanderAI Engineer Code 20252025
Ornella Bahidika, Joel AllouAI Engineer World's Fair 20262026
Philipp SchmidAI Engineer World's Fair 20262026
Ara KhanAI Engineer Europe 20262026
Omri Bruchim, Tomer AstAI Engineer World's Fair 20262026
Alex CheemaAI Engineer Europe 20262026
Future-Proof Coding Agents

Metadata candidate

Bill Chen, Brian FiocaAI Engineer Code 20252025
Kenton VardaAI Engineer World's Fair 20262026
Gateways are All You Need

Metadata candidate

Karan SampathAI Engineer Europe 20262026
Hailong ZhangAI Engineer Summit 20252025
Ash Prabaker, Andrew WilsonAI Engineer Europe 20262026
Hanna Lichtenberg, Aamir ShakirAI Engineer World's Fair 20262026
Mahmoud AbdelwahabAI Engineer Code 20252025
Raymond FengAI Engineer World's Fair 20262026
Carter Abdallah, Vincent Weisser, Lucas Atkins, Chris AlexiukAI Engineer World's Fair 20262026
Mark Bain, Vasilije Markovic, Daniel Chalef, Alex GilmoreAI Engineer World's Fair 20252025
Arjun SinghAI Engineer World's Fair 20262026
Notion's Token Town

Metadata candidate

Sarah SachsAI Engineer World's Fair 20262026
Omar KhattabAI Engineer World's Fair 20252025
Antje BarthAI Engineer World's Fair 20262026
Mario ZechnerAI Engineer Europe 20262026
Nick NisiAI Engineer Europe 20262026
RL Environments at Scale

Metadata candidate

Will BrownAI Engineer Code 20252025
Louis Knight-WebbAI Engineer Europe 20262026
The New Code

Metadata candidate

Sean GroveAI Engineer World's Fair 20252025
State of Data

Metadata candidate

Sean CaiAI Engineer World's Fair 20262026
Ibragim BadertdinovAI Engineer Europe 20262026
The Agentic AI Engineer

Metadata candidate

Benedikt Sanftl, Burak Cemil ÖzafşarAI Engineer World's Fair 20262026
Natalie MeurerAI Engineer World's Fair 20262026
Addy OsmaniAI Engineer World's Fair 20262026
Justin SchroederAI Engineer World's Fair 20262026
The Log Is The Agent

Metadata candidate

Ishaan SehgalAI Engineer World's Fair 20262026
Lou BichardAI Engineer Europe 20262026
The Prompt is the Platform

Metadata candidate

Dominik, Dominik TornowAI Engineer World's Fair 20262026
Ayush BhardwajAI Engineer World's Fair 20262026
Training Agentic Reasoners

Metadata candidate

Will BrownAI Engineer World's Fair 20252025
Eugene YanAI Engineer World's Fair 20262026
Matt DaileyAI Engineer World's Fair 20262026
James LeAI Engineer World's Fair 20262026
Vision: Zero Bugs

Metadata candidate

Johann Schleier-SmithAI Engineer Code 20252025
Sai Krishna RallabandiAI Engineer World's Fair 20262026
Eugene Yan, Hamel Husain, Jason Liu, Dr Bryan Bischof, Charles Frye, Shreya ShankarAI Engineer World's Fair 20242024
Phil HetzelAI Engineer Europe 20262026
Sunil Pai, Matt CareyAI Engineer Europe 20262026
Mike ChristensenAI Engineer Europe 20262026
Talha SheikhAI Engineer Europe 20262026
Mike PhippsAI Engineer World's Fair 20262026
Alex HancockEvent unknownUnknown
Jo Kristian BergumAI Engineer World's Fair 20262026
Adi SinghAI Engineer World's Fair 20262026
Anekha SokhalEvent unknownUnknown
Dan Barrett, Raymond Weitekamp, Jose Montes de OcaEvent unknownUnknown
Noyan TokgozogluEvent unknownUnknown
Anant SrivastavaAI Engineer Code 2025Unknown
Arman VaziriAI Engineer World's Fair 20262026
Giselle van DongenEvent unknownUnknown
George HeEvent unknownUnknown
Fuad AliAI Engineer World's Fair 20262026
Thomas Wolf, Olive SongEvent unknownUnknown
Patrick DeboisAI Engineer World's Fair 20262026
Fuad AliEvent unknownUnknown
Jose PalafoxAI Engineer World's Fair 20262026
Robert BrennanAI Engineer World's Fair 20262026
Aparna DhinakaranEvent unknownUnknown
Ali-Reza Adl-TabatabaiAI Engineer World's Fair 20262026
Laurie VossAI Engineer World's Fair 2026Unknown
Jess WangEvent unknownUnknown
Hariprasad SudharshanAI Engineer NYC 2026 Online Talks2026
Krishna Prasad SrinivasanAI Engineer World's Fair 20262026
Felipe BlanesAI Engineer World's Fair 20262026
Priyanka Phatak, Gabe CemajAI Engineer World's Fair 20262026
Jack WotherspoonEvent unknownUnknown
Tejas KumarAI Engineer World's Fair 20262026
Kevin MaduraEvent unknownUnknown
Eugene CheahEvent unknownUnknown
Ivan LeoAI Engineer World's Fair 20262026
Jan CurnAI Engineer Code 2025Unknown
Greg PstruchaAI Engineer World's Fair 20262026
Jake BroekhuizenAI Engineer World's Fair 20262026
Rémi LoufAI Engineer World's Fair 20262026
Laurie VossAI Engineer World's Fair 20262026
Mikyo KingEvent unknownUnknown
Priyanka Phatak, Gabe CemajAI Engineer World's Fair 20262026
Philipp SchmidAI Engineer World's Fair 20262026
Priyanka Phatak, Gabe CemajAI Engineer World's Fair 20262026
Simran AroraAI Engineer World's Fair 20262026
Jim ClarkAI Engineer World's Fair 20262026
Al ChenEvent unknownUnknown
Zach LloydEvent unknownUnknown
Jerry LiuAI Engineer World's Fair 20262026
Benjamin ClaviéEvent unknownUnknown
Liad YosefAI Engineer World's Fair 20262026
Nnenna NdukweAI Engineer World's Fair 20262026
Mark Lummus, Navinkumar PatilAI Engineer World's Fair 20262026
Michael GrinichAI Engineer World's Fair 20262026
Zixuan LiEvent unknownUnknown
Gautam KorlamAI Engineer NYC 2026 Online Talks2026
Lakshya AgrawalAI Engineer World's Fair 20262026
Paul BakausEvent unknownUnknown
Eduardo Simon, Javier CasalAI Engineer NYC 2026 Online Talks2026
Fuad AliEvent unknownUnknown
Gabe Cemaj, Priyanka PhatakAI Engineer World's Fair 20262026
Safia AbdallaAI Engineer World's Fair 20262026
Stephanie JarmakEvent unknownUnknown
Filip MakraduliAI Engineer World's Fair 20262026
Rowan ChristmasAI Engineer Code 2025Unknown
Keegan McCallumAI Engineer World's Fair 20252025
Jason LiuAI Engineer World's Fair 20262026
Ilan BigioAI Engineer Summit 20252025
AI Engineer Summit 20252025
Tisha Chawla, Susheem KoulAI Engineer World's Fair 20262026
Zack ProserAI Engineer Europe 20262026
Philipp SchmidEvent unknownUnknown
Roberto Milev, Uday KanagalaAI Engineer World's Fair 20262026
Janie Gu, Suyeol YunEvent unknownUnknown
Andy Wong, Andrei Bocan, Shane WolfAI Engineer World's Fair 20262026

References

Coverage and source review
Processed transcripts
72 processed in full · 5 in the curated path
Automated source review
Passed
Metadata candidates
145 unreviewed; not verified topic membership
Corpus version
c76f4395d1050f81b2bfd27df900224591ef94b026792d1b65896097daa9f77c

Automated review checks source support; it is not publication approval.

A synthesis of selected conference talks and technical references. Citations link to the source material; they do not imply that every talk on this subject is included.

  1. Harness engineering: leveraging Codex in an agent-first world — Ryan Lopopolo

    https://openai.com/index/harness-engineering/

  2. Harness Engineering: How to Build Software When Humans Steer, Agents Execute — Ryan Lopopolo, Vibhu Sapra

    https://www.youtube.com/watch?v=am_oeAoUhew&t=2304s

  3. Unlocking the Codex harness — OpenAI

    https://openai.com/index/unlocking-the-codex-harness/

  4. Unrolling the Codex agent loop

    https://openai.com/index/unrolling-the-codex-agent-loop/

  5. Function calling

    https://platform.openai.com/docs/guides/function-calling

  6. Effective harnesses for long-running agents — Anthropic

    https://www.anthropic.com/engineering/effective-harnesses-for-long-running-agents

  7. Effective harnesses for long-running agents

    https://www.anthropic.com/engineering/effective-harnesses-for-long-running-agents

  8. Your Agent Didn’t Fail. Your Harness Did.

    https://www.youtube.com/watch?v=BInpv7lGp1o&t=159

  9. Building durable Agents with Workflow DevKit & AI SDK

    https://www.youtube.com/watch?v=kmV-qg4uoNI&t=2740

  10. Demystifying evals for AI agents

    https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents

  11. Google long-running Operations protocol definition

    https://raw.githubusercontent.com/googleapis/googleapis/master/google/longrunning/operations.proto

  12. Unlocking the Codex harness: how we built the App Server

    https://openai.com/index/unlocking-the-codex-harness/

  13. Workflow Id and Run Id — Temporal

    https://docs.temporal.io/workflow-execution/workflowid-runid

  14. Building Durable, Production-Ready Agents with OpenAI SDK and Temporal

    https://www.youtube.com/watch?v=k8cnVCMYmNc&t=2465

  15. Making retries safe with idempotent APIs — Amazon Builders' Library

    https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/

  16. Temporal Activity Execution

    https://docs.temporal.io/activity-execution

  17. PostgreSQL 18: Transaction Isolation

    https://www.postgresql.org/docs/current/transaction-iso.html

  18. Job run states — Amazon EMR Serverless

    https://docs.aws.amazon.com/emr/latest/EMR-Serverless-UserGuide/job-states.html

  19. Your Agent Didn’t Fail. Your Harness Did.

    https://www.youtube.com/watch?v=BInpv7lGp1o&t=608

  20. Spend — Pydantic AI Harness

    https://pydantic.dev/docs/ai/harness/spend/

  21. Transactional outbox pattern — AWS Prescriptive Guidance

    https://docs.aws.amazon.com/prescriptive-guidance/latest/cloud-design-patterns/transactional-outbox.html

  22. PostgreSQL 18: Transactions

    https://www.postgresql.org/docs/current/tutorial-transactions.html

  23. Make your own event-sourced agent harness using stream processors

    https://www.youtube.com/watch?v=vi-2nasppAg&t=649

  24. Scaling Managed Agents: Decoupling the brain from the hands

    https://www.anthropic.com/engineering/managed-agents

  25. Checkpointers — LangGraph

    https://docs.langchain.com/oss/python/langgraph/checkpointers

  26. Your Agent Didn’t Fail. Your Harness Did.

    https://www.youtube.com/watch?v=BInpv7lGp1o&t=882

  27. Agent Continuations implementation walkthrough

    https://github.com/SnapLogic/agent-continuations/blob/main/continuations.md

  28. Breaking the Chain: Agent Continuations for Resumable AI Workflows

    https://www.youtube.com/watch?v=ZB7l4uxW3Yo&t=609

  29. Thinking in LangGraph

    https://docs.langchain.com/oss/python/langgraph/thinking-in-langgraph

  30. LangGraph Functional API: deterministic resumption and idempotency

    https://docs.langchain.com/oss/python/langgraph/functional-api

  31. Persistence — LangGraph

    https://docs.langchain.com/oss/python/langgraph/persistence

  32. From fork() to Fleet: Designing an Agent Sandbox Cloud — Abhishek Bhardwaj, OpenAI

    https://www.youtube.com/watch?v=OqM67QG_Ikk&t=1815

  33. From Stateless Nightmares to Durable Agents

    https://www.youtube.com/watch?v=flf_IKnFYnE&t=294

  34. Temporal Workflow Definition

    https://docs.temporal.io/workflow-definition

  35. Worker Versioning — Temporal

    https://docs.temporal.io/production-deployment/worker-deployments/worker-versioning

  36. Building Multi-agent Systems with Finite State Machines

    https://www.youtube.com/watch?v=OD13PiXw60o&t=287

  37. The Protection of Information in Computer Systems: Basic Principles

    https://web.mit.edu/Saltzer/www/publications/protection/Basic.html

  38. System Structure for Software Fault Tolerance

    https://www.inf.pucrs.br/~zorzo/cs/system-structure-for-software.pdf

  39. ARIES: A Transaction Recovery Method Supporting Fine-Granularity Locking and Partial Rollbacks Using Write-Ahead Logging

    https://web.stanford.edu/class/cs345d-01/rl/aries.pdf

  40. Making reliable distributed systems in the presence of software errors

    https://www.cs.otago.ac.nz/coursework/cosc461/armstrong_thesis_2003.pdf

  41. The Workflow Reference Model — Workflow Management Coalition

    https://www.aiai.ed.ac.uk/project/wfmc/ARCHIVE/DOCS/refmodel/rmv1-16.html

  42. Amazon Simple Workflow – Cloud-Based Workflow Management

    https://aws.amazon.com/blogs/aws/amazon-simple-workflow-cloud-based-workflow-management/

  43. Trigger.dev v3: Durable Serverless functions. No timeouts.

    https://trigger.dev/blog/v3-announcement

  44. From beta to 3.0: Trigger.dev v3 reaches GA

    https://trigger.dev/blog/beta-to-latest-announcement

  45. Two Roads to Durable Agents: Replay vs. Snapshot — Eric Allam, Co-founder, Trigger.dev

    https://www.youtube.com/watch?v=svCnShDvgQg&t=546

  46. The Human Is an Async API — Melanie Warrick, Temporal

    https://www.youtube.com/watch?v=jc3kbZkuHTo&t=448

  47. The Human Is an Async API — Melanie Warrick, Temporal

    https://www.youtube.com/watch?v=jc3kbZkuHTo&t=827

  48. The Human Is an Async API — Melanie Warrick, Temporal

    https://www.youtube.com/watch?v=jc3kbZkuHTo&t=925

  49. Handle external events in durable orchestrations

    https://learn.microsoft.com/en-us/azure/durable-task/common/durable-task-external-events

  50. Addressing Cascading Failures — Google Site Reliability Engineering

    https://sre.google/sre-book/addressing-cascading-failures/

  51. Temporal TypeScript SDK: Activity Timeouts

    https://docs.temporal.io/develop/typescript/activities/timeouts

  52. Amazon SQS visibility timeout

    https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-visibility-timeout.html

  53. The Chubby Lock Service for Loosely-Coupled Distributed Systems

    https://research.google.com/archive/chubby-osdi06.pdf

  54. Making retries safe with idempotent APIs

    https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/

  55. Idempotent jobs — BullMQ

    https://docs.bullmq.io/patterns/idempotent-jobs

  56. AWS Well-Architected: Control and Limit Retry Calls

    https://docs.aws.amazon.com/wellarchitected/latest/reliability-pillar/rel_mitigate_interaction_failure_limit_retries.html

  57. Writing effective tools for AI agents

    https://www.anthropic.com/engineering/writing-tools-for-agents

  58. Your Agent Didn’t Fail. Your Harness Did.

    https://www.youtube.com/watch?v=BInpv7lGp1o&t=754

  59. AWS Builders' Library: timeouts, retries, backoff and jitter

    https://d1.awsstatic.com/builderslibrary/pdfs/timeouts-retries-and-backoff-with-jitter.pdf

  60. AWS Well-Architected: Set Client Timeouts

    https://docs.aws.amazon.com/wellarchitected/latest/reliability-pillar/rel_mitigate_interaction_failure_client_timeouts.html

  61. Temporal Activity Definition

    https://docs.temporal.io/activity-definition

  62. Idempotent requests

    https://docs.stripe.com/api/idempotent_requests

  63. Resolving an ambiguous payment request

    https://docs.stripe.com/error-low-level

  64. AI Agents Are Just Distributed Systems Now — Salman Munaf, TikTok

    https://www.youtube.com/watch?v=hD9-V56FNRI&t=509

  65. Sagas

    https://www.cs.cornell.edu/andru/cs711/2002fa/reading/sagas.pdf

  66. Harnesses in AI: A Deep Dive

    https://www.youtube.com/watch?v=C_GG5g38vLU&t=632

  67. AI Agents Are Just Distributed Systems Now — Salman Munaf, TikTok

    https://www.youtube.com/watch?v=hD9-V56FNRI&t=602

  68. Two-Phase Transfers — TigerBeetle

    https://docs.tigerbeetle.com/coding/two-phase-transfers/

  69. Manage Orchestration Instances in Durable Functions and Durable Task SDKs

    https://learn.microsoft.com/en-us/azure/durable-task/common/durable-task-instance-management

  70. Trio's core functionality

    https://trio.readthedocs.io/en/stable/reference-core.html

  71. Supervisor Behaviour — Erlang System Documentation

    https://www.erlang.org/doc/system/sup_princ.html

  72. gRPC lifecycle: cancellation is not rollback

    https://grpc.io/docs/what-is-grpc/core-concepts/

  73. Workflow message passing — Temporal Python SDK

    https://docs.temporal.io/develop/python/workflows/message-passing

  74. Transaction Authorization Cheat Sheet — OWASP

    https://cheatsheetseries.owasp.org/cheatsheets/Transaction_Authorization_Cheat_Sheet.html

  75. Your Agent Didn’t Fail. Your Harness Did.

    https://www.youtube.com/watch?v=BInpv7lGp1o&t=695

  76. What Does Done Even Mean? Agents and Paperclip's Liveness Model - Dotta, Paperclip

    https://www.youtube.com/watch?v=7P0elyLIxXo&t=251

  77. SLSA v1.1 Provenance

    https://slsa.dev/spec/v1.1/provenance

  78. How we built our multi-agent research system

    https://www.anthropic.com/engineering/multi-agent-research-system

  79. Your Agents Need a Save Button

    https://www.youtube.com/watch?v=bZISsg7H7DA&t=68

  80. Aborting a multipart upload — Amazon S3

    https://docs.aws.amazon.com/AmazonS3/latest/userguide/abort-mpu.html

  81. How to prevent object overwrites with conditional writes — Amazon S3

    https://docs.aws.amazon.com/AmazonS3/latest/userguide/conditional-writes.html

  82. AbortMultipartUpload — Amazon S3 API

    https://docs.aws.amazon.com/AmazonS3/latest/API/API_AbortMultipartUpload.html

  83. Your Agent Didn’t Fail. Your Harness Did.

    https://www.youtube.com/watch?v=BInpv7lGp1o&t=793

  84. What Does Done Even Mean? Agents and Paperclip's Liveness Model - Dotta, Paperclip

    https://www.youtube.com/watch?v=7P0elyLIxXo&t=11

  85. The Oracle Problem in Software Testing: A Survey

    https://philmcminn.com/publications/barr2015.pdf

  86. Hooks reference — Claude Code

    https://code.claude.com/docs/en/hooks

  87. Make your own event-sourced agent harness using stream processors

    https://www.youtube.com/watch?v=vi-2nasppAg&t=2854

  88. Temporal TypeScript Activity Timeouts

    https://docs.temporal.io/develop/typescript/activities/timeouts

  89. FoundationDB: A Distributed Key Value Store

    https://sigmodrecord.org/publications/sigmodRecord/2203/pdfs/08_fdb-zhou.pdf

  90. Testing — Temporal PHP SDK

    https://docs.temporal.io/develop/php/best-practices/testing-suite

  91. Two Roads to Durable Agents: Replay vs. Snapshot — Eric Allam, Co-founder, Trigger.dev

    https://www.youtube.com/watch?v=svCnShDvgQg&t=447

  92. Stalled — BullMQ

    https://docs.bullmq.io/guide/jobs/stalled

  93. Two Roads to Durable Agents: Replay vs. Snapshot — Eric Allam, Co-founder, Trigger.dev

    https://www.youtube.com/watch?v=svCnShDvgQg&t=349

  94. How I Learned to Stop Worrying and Love the Sandbox — Matt Brockman, E2B

    https://www.youtube.com/watch?v=fz6-NS7qpZc&t=2345

  95. Exposing Agents as MCP Servers with mcp-agent: Sarmad Qadri

    https://www.youtube.com/watch?v=uFPAtKIN-FQ&t=574

  96. Two Roads to Durable Agents: Replay vs. Snapshot — Eric Allam, Co-founder, Trigger.dev

    https://www.youtube.com/watch?v=svCnShDvgQg&t=207

  97. The Golden Age of AI Engineering

    https://www.youtube.com/watch?v=pMggiOb18tc&t=645

  98. PostgreSQL 18: Write Ahead Log

    https://www.postgresql.org/docs/18/runtime-config-wal.html