Execution contracts
What a harness controls
The OpenAI team wrote the recurring review rule into the repository. A failing test could point the patch author to the rule, or an agent reviewing the change could flag the violation. Recording the rule made it available on later patches. Surfacing it during review gave the author a chance to fix the code. A team that also wanted to block a merge would have to require a passing check. Ryan Lopopolo and Vibhu Sapra at AI Engineer Europe, review example at 38:24
Consider how Codex could run a repository check. The model proposes a command. The runtime decides whether it can dispatch that command and pauses for a client decision if approval is required. Follow the fork in the figure. On denial, the command never starts. On the permitted branch, a test can still fail. The runtime appends the actual output or error so the next model call can use the failure message to choose an edit. That return path is what makes tool use a feedback loop. Codex App Server · The Codex agent loop · Agent decisions and observations
How a repository check reaches the next model call
ExamplePermission determines whether the command may start. Its output supplies evidence for the next edit.
Read the diagram as text
- Codex proposes tool call. Run a repository check.
- Runtime checks permission. Pause for a client decision when required.
- Execute tool. Run the permitted command.
- No execution. The command never starts.
- Append tool result. Associate the observation with this call.
- Next model invocation. Can use the check’s finding.
- Codex proposes tool call → Runtime checks permission: Tool name and arguments.
- Runtime checks permission → Execute tool: Allowed or approval granted.
- Runtime checks permission → No execution: Denied.
- Execute tool → Append tool result: Actual output or error.
- Append tool result → Next model invocation: Observation in model input.
In Anthropic’s coding experiment, the difficulty was handing a partly built application to a fresh session. The first session established a feature list. Later sessions worked on individual features, committed changes to git, and updated a progress file. An incoming session inspected prior work, started the application, and ran a basic end-to-end check before continuing. Marking a feature complete required testing that feature’s promised behavior. A shorter conversation alone had left sessions prone to declaring victory early.
| Failure | Intervention | Evidence available afterward |
|---|---|---|
| A proposed Codex command may lack permission or may fail. | Check permission before dispatch. Return the result of an allowed call. | A denial or the tool’s actual output. |
| OpenAI reviewers repeatedly correct the same mistake. | Put the rule in the repository and surface it when a patch violates it. | A finding the patch author can act on. |
| A fresh Anthropic session loses track of unfinished features. | Leave a feature list, progress file, and commits. Run a basic end-to-end check on handoff. Test each feature before marking it complete. | Recorded intent, the implementation, and observed behavior. |
The runtime’s execution contract states what the service promises when it accepts a task. It specifies which actions may run, what progress survives the loss of a worker, when the runtime must stop, and what it checks before reporting success. An orchestration library can help coordinate calls, while an isolation boundary restricts the files, processes, and services the executing code can reach. An evaluation harness serves a different purpose. It runs trials and grades how well the model and its working environment perform together. Execution isolation · Evals and Benchmarks
Name work and its attempts
Suppose a client submits work, loses its connection, and comes back after another worker has taken over. It needs a handle for the work it already requested. Looking up the old process would miss the replacement. Looking up the conversation could return several requests. Choose identifiers by what each participant needs to find again.
| Identity | What it names | What changes on recovery |
|---|---|---|
| Task | The caller’s requested outcome | The same task can have more than one execution. |
| Run | One managed execution with a lifecycle and limits | A replacement process can continue the same run. |
| Worker | The process doing the work | A restarted process is a new worker. |
| Logical operation | One intended effect, such as creating one issue | Retain its identity through repeated deliveries. |
| Attempt | One try at a specified unit, such as a tool call | Count a new attempt each time that unit is tried. |
Product terminology varies. Codex groups a conversation into a thread, a user-initiated turn, and individual input or output items. In Temporal, a business-level Workflow ID can span a retry that gets a new Run ID. A change of worker alone need not start a new Temporal run. Codex App Server · Workflow ID and Run ID
Give the execution a lifecycle a returning caller can inspect. In one design, ready work becomes running when claimed, waiting when it needs input, and ready again when the input arrives. A stop request can move it to stopping. A terminal record explains how it ended, distinguishing success, failure, denial, cancellation, deadline, or exhausted allowance as the task requires. Accept each transition in one conditional write against the expected state and version. If completion and cancellation compete, the second writer must act on the state the first one committed.
Preserving progress
Accept work durably
A durable acknowledgment promises that a replacement can discover both the task and what still needs doing. Saving a task row and then sending a queue message leaves a gap. A crash between the writes leaves a real task that no worker has been told to start. Reversing the writes lets a worker receive an instruction for a task that never committed. State at acceptance
A transactional outbox puts the task change and an outgoing instruction in the same database transaction. After commit, the service can acknowledge the caller. A separate publisher, called a relay, reads the instruction and sends it to the queue. If the relay dies before sending, its replacement finds the pending instruction. If it dies after sending but before recording delivery, the replacement may send twice. Give the instruction a stable ID and make the consumer recognize redelivery before accepting duplicate work. Transactional outbox
One commit supports two independent actions
ExampleThe caller can receive a durable acknowledgment while the relay is offline.
Read the diagram as text
- Task and instruction committed. One local transaction.
- Acknowledge task.
- Dispatch relay. Reads instructions still owed.
- Worker queue. Separate service.
- Task and instruction committed → Acknowledge task: Commit permits acknowledgment.
- Task and instruction committed → Dispatch relay: Pending instruction and ID.
- Dispatch relay → Worker queue: Publish with original ID.
An authoritative record is the one an executor consults to decide what may happen next. For a long run, this includes intent, state version, accepted observations, pending operations, remaining requirements, committed allowance, and output references. A transcript can preserve everything the model said while omitting a pending dispatch. Anthropic’s Managed Agents illustrates the separation. Session events survive outside the harness process, the harness selects what to put in a model call, and a sandbox provides replaceable compute. Reconstructing the input to the model and restoring its working files are separate operations. Managed Agents architecture · Context Engineering
Save a useful recovery position
A checkpoint saves a position in an execution and the state needed there. A continuation identifies the work remaining at that position. In SnapLogic’s prototype, an agent suspends just before a tool call needing approval. Inspect the fields of the returned object. messages provides conversation history. resume_request identifies the pending call. processed carries decision information used on resumption. With only the messages, the model could discuss the proposed action again. The executor needs the call identity to release the particular action that was reviewed. Returning a serializable object makes it possible for the caller to store it durably. SnapLogic continuation record
| Last committed position | What survives | First unfinished operation |
|---|---|---|
| Before inference | Inputs to the choice | Call the model again. It may choose different arguments. |
| After inference | Tool name, arguments, and pending call identity | Check whether the saved call may still run, then dispatch if permitted. |
Coverage matters as much as placement. LangGraph checkpoints can contain graph values, the next nodes to run, pending tasks, and identifiers for the thread and checkpoint. Nodes scheduled in one step can run concurrently. A successful node’s separately persisted output can spare recomputation when its peer fails. Finer boundaries add writes. Synchronous persistence waits for a write before proceeding, whereas an outstanding asynchronous write may be lost. Use a backend that survives the failures in scope, and keep the referenced files and configuration available. A disk snapshot can restore a file whose path appears in the checkpoint. Its contents cannot tell you whether a remote service subsequently changed. LangGraph checkpointers · Snapshot coverage
Reconstruct execution safely
Direct restoration loads saved state and resumes at a known position. History-based replay runs the coordination code again, supplying previously recorded results as that code asks for them. In Temporal, a unit of external work such as an inference request is called an activity. In Samuel Colvin’s demonstration, a replacement reached earlier model activities and immediately received their saved responses. It contacted the model provider when it reached a new inference. The figure applies the same mechanism to a saved tool choice. Watch which component supplies the arguments after restart. Colvin’s demonstration
Replay depends on compatible control flow. If a branch reads the wall clock again, draws a fresh random number, or queries the model directly, it can schedule a different operation. Use the runtime’s replay-aware primitives or put the observation in an activity whose result is recorded. A code edit can also break replay. For example, moving a timer before an activity can issue a timer command where the history expects the activity. In diagnostic replay, changing an input can be the point of the exercise. In recovery, it can change the meaning of unfinished work. Temporal workflow definitions · Diagnostic replay
One tool choice survives two workers
ExampleHistory supplies the arguments during recovery. Only unfinished work needs a new call.
The result commits before the first worker exits.
Read the diagram as text
- Original model activity.
- Recorded tool choice. One saved name and argument set.
- Replacement worker. Runs compatible scheduling code.
- Chosen tool. Dispatch had not begun at interruption.
- Original model activity → Recorded tool choice: Record returned choice.
- Recorded tool choice → Replacement worker: Return historical result.
- Replacement worker → Chosen tool: Dispatch saved arguments.
- Record the choice. The result commits before the first worker exits. Active: Original model activity, Recorded tool choice. New: Original model activity, Recorded tool choice.
- Recover the choice. The replacement consumes the recorded result at the same activity boundary. Active: Original model activity, Recorded tool choice, Replacement worker. New: Replacement worker.
- Dispatch the tool. Execution reaches its first new external call. Active: Original model activity, Recorded tool choice, Replacement worker, Chosen tool. New: Chosen tool.
For direct restore, a new build needs to understand the checkpoint schema and what each pending action means. For replay, it also has to issue commands compatible with the stored history. Temporal supports pinning a workflow to a deployment version. Another process on that build can take over, but the operator must keep suitable workers available until the run ends. An upgrade policy trades that operating cost for compatibility work. Either way, a service can revoke access while the run is offline. Worker versioning
Historical development
Why recovery mechanisms coexist
Recovering from an incorrect computation raises a different problem from recovering data after a crash. In 1975 Brian Randell described a recovery block that saved its entry state, ran a primary implementation, and tested the result before releasing it. If the acceptance test failed or could not finish, the block restored the entry state and tried an alternative. That boundary works when the system can withhold unacceptable results. Once another party has acted on a result, restoring local variables cannot restore the earlier situation. System Structure for Software Fault Tolerance
Database recovery had to preserve committed work while transactions updated shared storage. In 1992, Mohan and colleagues described how ARIES combined write-ahead logging with checkpoints. Before a changed database page could replace its previous version on disk, the system had to save the log records needed to recover the change. A checkpoint summarized active transactions and modified pages while updates continued. After a crash, that summary helped the system find where to begin recovery in the log without requiring every changed page to have been written at checkpoint time. ARIES
For long-lived telecommunications services, a failed process also meant lost capacity. Joe Armstrong traced Erlang to a 1986 experiment with concurrent processes and described Ericsson’s 1996 Open Telecom Platform project as a stable software base for applications. A supervisor could detect an abnormal exit and restart a worker or a related group. The supervisor could thus supply a new process while a separate recovery record supplied that process with unfinished work. Armstrong’s account
Work involving people and several applications also needed an owner for the sequence of steps. The November 1994 Workflow Reference Model separated human worklists, invoked applications, process definitions, and the control data for each running instance. Amazon Simple Workflow’s February 2012 launch illustrates what a managed coordination service could take on. Activity workers performed tasks, deciders consulted history to choose subsequent work, and the service assigned tasks, retained results, and tracked timeouts. A signal could enter history while no decider was processing it. Workflow Reference Model · Amazon Simple Workflow launch
A working environment can itself be expensive to reconstruct. In January 2024 Trigger.dev proposed hosting task execution and checkpointing containers at waits, after an earlier design replayed functions using saved step results. Its September 2024 general-availability announcement described suspending a parent while its child ran. For a coding agent, the distinction is tangible. An execution history can say which tool the model chose. A machine snapshot covering memory and processes can bring back the development server that tool was using. The next user message still needs an address so it can wake the right work. January proposal · September release · Eric Allam on replay and snapshots
Dispatch and recovery
Wake retained work
A durable wait records what would make a continuation runnable, then releases the worker. A signal is an external event addressed to the execution. A durable timer records a time at which it becomes eligible to resume. Melanie Warrick demonstrated the distinction by taking an application worker offline and approving a change to an order. Temporal retained the decision. When the worker returned, the order continued. Each order had its own workflow, so one outstanding review could wait while other orders proceeded. Warrick’s demonstration
The first race occurs before anyone has waited. If a service sends a review link before creating the instance meant to receive the reply, an immediate answer may be lost. Azure Durable Task buffers early events for an existing instance and discards events for a nonexistent one. One design commits the pending wait and the instruction to notify the reviewer together. On receipt, match the event to its wait ID and revision, atomically consume that wait and record runnable work, then acknowledge the event. A duplicate reply or losing timer finds the wait already consumed. Processing the reply can still yield a denial. External events in durable orchestrations
A reply meets one pending wait
ExampleReplies can arrive out of order because each addresses its own wait.
Read the diagram as text
- Pending wait. Run, call, and revision.
- Incoming reply. Target wait and decision.
- Match and consume wait. Check current state in the same write.
- Queue decision for processing.
- No new work.
- Pending wait → Match and consume wait: Expected event and revision.
- Incoming reply → Match and consume wait: Event ID and target.
- Match and consume wait → Queue decision for processing: First matching reply.
- Match and consume wait → No new work: Consumed or obsolete wait.
Eligibility does not reserve a worker. A bounded queue absorbs a burst. If arrivals continue to exceed capacity, backpressure asks callers to slow down or defers or refuses admission. Check the remaining task deadline before dispatch. Time in the queue counts against an elapsed deadline even though the worker’s attempt timeout has yet to start. A background task may have a different deadline from the HTTP request that created it.
Reject obsolete workers
A lease grants a worker temporary authority. A heartbeat reports contact. If heartbeats stop, the process could be dead, paused, or unable to reach the coordinator. Reassignment can restore service while leaving two workers capable of sending results. Even Amazon SQS’s visibility timeout, which normally hides a received message from other consumers, allows duplicates during the visibility interval. SQS visibility timeout
Fencing lets the recipient reject a request from a former owner. In the illustrated task-row design, claiming the row atomically assigns an ownership generation. A takeover increments it. Renewal requires a current, unexpired claim. A result write must match that claim, the expected task revision, and a state that still accepts results, all in the update itself. Both workers read revision 4. Even before the replacement has changed the result, generation 7 has lost the right to write it. A separate ownership check followed later by an unconditional write would leave time for another takeover. Conditional state updates · PostgreSQL transaction isolation
Two workers return results for revision 4
ExampleThe task revision can match even after the writer has lost its lease.
Read the diagram as text
- Former worker. Delayed across takeover.
- Current worker. Holds the valid lease.
- Guarded task row. Expects generation 8 and revision 4.
- Reject result.
- Advance revision.
- Former worker → Guarded task row: Generation 7, revision 4.
- Current worker → Guarded task row: Generation 8, revision 4.
- Guarded task row → Reject result: Superseded generation.
- Guarded task row → Advance revision: Valid claim, state and revision.
Chubby’s sequencers put lock identity and generation in a form a recipient could check. The same placement matters for tools. Rejecting a former worker’s database update still leaves any remote request it sent in flight. A receiving API needs its own authority check to reject obsolete callers. Even if it suppresses duplicate requests, it could accept the former worker’s request first. The Chubby lock service
Bound repeated attempts
A retry makes another attempt at the same logical operation. Resuming continues from a recorded position. Restarting a job can repeat several operations, including ones that succeeded. Replanning gives the agent another choice, which may require different permission. First identify which unit failed and what the response establishes about its effects. Agent recovery decisions
| Observation | Appropriate next step |
|---|---|
| A repeatable read met a transient outage. | Retry the read within its time and attempt allowance. |
| The tool rejected an invalid argument. | Return the validation detail so the caller can correct it. |
| The current identity lacks authority. | Withhold dispatch. Obtain applicable authority if the task calls for it. |
| A write timed out after dispatch. | Keep the original operation pending and resolve its outcome or use protected redelivery. |
For a safe retry, capped exponential backoff lengthens successive delays up to a maximum. Jitter varies those delays so recovering clients spread their requests over time. Count attempts at the dependency itself. If a workflow tries three times and its client library tries three times per invocation, one operation permits nine downstream calls. Choose one layer to own retry policy, including any attempts a gateway or SDK makes internally. Preserve the count and usage across restarts. Fit the next delay and attempt within the original deadline. AWS retry guidance · Model Routing and LLM Gateways
Resolve uncertain effects
Suppose a tool should create one issue. The worker records the intended creation, sends it, and exits without recording a reply. Two histories now fit the local evidence. The request might have been lost before it reached the issue service. Or the service might have created the issue and its reply was lost. Workflow replay can reuse a recorded activity result, but neither history contains one. Making the activity smaller narrows how much work can repeat. It cannot eliminate the interval between a remote commit and a local record. Temporal activity definitions · Uncertain external outcomes
Receiver-enforced retries
Idempotency means that repeating the same logical operation produces no additional effect. Retain the caller, intent, original parameters, operation key, delivery attempts, and any returned handles. For a keyed creation, the receiver must recognize the key in the caller’s scope, reject changed parameters, and coordinate recording the key with creation so concurrent deliveries cannot both create. Identical titles alone are insufficient because two intentionally distinct issues can have the same title. AWS on idempotent APIs · Tool dispatch
Two remote histories leave one local record
ExampleA pending creation is compatible with either zero or one issue at the receiver.
Read the diagram as text
- Request lost. No issue created.
- Issue created. Reply never reaches durable local history.
- Pending issue creation. Original key and parameters survive.
- Record original outcome.
- Retry original key and parameters.
- Retain unresolved operation.
- Request lost → Pending issue creation: No reply generated.
- Issue created → Pending issue creation: No reply recorded.
- Pending issue creation → Record original outcome: Conclusive lookup.
- Pending issue creation → Retry original key and parameters: Inconclusive lookup, key valid.
- Pending issue creation → Retain unresolved operation: Inconclusive lookup, no protection.
Read the receiver’s retention and error rules. Stripe, for example, stores the status and body of the first executing request for a key, including an HTTP 500. A validation failure or concurrency conflict before execution begins leaves no stored result. A repeated 500 can leave the underlying mutation indeterminate. Keys can be pruned once at least 24 hours old, after which reuse can initiate another request. A new key issued to get past a cached error can also duplicate the original intent. Stripe idempotent requests · Low-level errors
Finding or counteracting the effect
Reconciliation means consulting the system that owns the outcome. For the issue tool, a conclusive lookup by the original operation ID can settle whether the issue exists. An empty title search cannot do that reliably. If lookup is inconclusive and retry protection has expired, retain the uncertainty for investigation. If the service confirms an unwanted creation, closing the issue is a compensation, a new authorized operation that can itself fail. Garcia-Molina and Salem’s 1987 saga model explains a limit of compensation. Other participants may already have observed the committed change. Closing an issue leaves earlier notifications in recipients’ hands. Sagas
Limits and operator control
Preserve limits across restarts
Specify both the unit of a limit and who shares it. An elapsed deadline advances during queueing and review. Active compute time measures work performed. An attempt timeout bounds one try. A turn limit or tool-call limit bounds steps, and a concurrency limit restricts simultaneous work. If the allowance covers a run, a new worker or child must inherit its remaining allowance. A soft threshold can warn the operator. At a hard threshold the runtime refuses admission and records why work stopped. Activity timeouts · Choosing additional reasoning
Counting usage only when responses arrive leaves a race. Two children can both see enough remaining allowance before either reports a cost. Pydantic AI documents this limitation for its spend counter. A reservation charges pending work against admission before dispatch. In the example, each child checks and reserves in one transaction against the same ledger. Available units equal the limit less consumed units and outstanding reservations. Pydantic AI spend limits
| Event | Consumed | Reserved | Available |
|---|---|---|---|
| Before dispatch | 2 | 0 | 8 |
| Reserve 5 for child A | 2 | 5 | 3 |
| Decline child B’s request for 4 | 2 | 5 | 3 |
| Replacement worker reads the ledger | 2 | 5 | 3 |
| Settle A at 3 units and release 2 | 5 | 0 | 5 |
Settle each reservation once against the original operation. TigerBeetle’s pending transfers illustrate partial settlement and release of the remainder. For an agent, a strict monetary ceiling also requires an enforceable upper bound on what a dispatched call can cost. A local timeout may occur before provider billing arrives. Keep the reservation until usage is reconciled, and budget separately for cleanup so refusing new work does not prevent an orderly stop. Two-phase transfers · Workload economics
Stop owned work
Cancellation asks the runtime to stop future and ongoing work within a specified scope. Keep its control path usable even if the current tool never returns. Persist the request, close admission, and invalidate pending dispatches and timers. Then account for work that has already escaped the queue. A disconnected client has stopped listening, a timeout ends a wait, and a pause preserves an opportunity to resume. Each leaves open the question of what is still running. Operator-facing controls
Trio gives local tasks an owner through a nursery, a scope that waits for its children to exit. Children observe cancellation at cooperative interruption points. Cleanup that needs I/O can run in a shielded scope with its own timeout. Erlang supervisors offer a different shutdown policy, requesting exit and, when configured, forcing termination after a grace period. Killing an intermediate supervisor too soon can leave its descendants alive. The evidence of shutdown is the disposition of the owned children. Trio task lifetimes · Erlang supervision · Sandbox lifecycle
Three consequences of a stop request
ExampleClosing local dispatch can precede both child exit and remote resource release.
Read the diagram as text
- Persist stop request.
- Close local dispatch. Invalidate pending starts.
- Interrupt owned child.
- Local exit confirmed.
- Check provider job. Retain the original handle.
- Cleanup outstanding.
- Remote release confirmed.
- Persist stop request → Close local dispatch: Reject new work.
- Persist stop request → Interrupt owned child: Propagate cancellation.
- Interrupt owned child → Local exit confirmed: Observe child exit.
- Persist stop request → Check provider job: Cancel by job ID, then query.
- Check provider job → Cleanup outstanding: CANCELLING.
- Check provider job → Remote release confirmed: CANCELLED.
A remote service has its own lifecycle. Amazon EMR Serverless uses CANCELLING while it attempts shutdown and CANCELLED after cancellation succeeds and resources are released. Azure Durable Task can terminate an orchestration while its activity functions continue. Retain job handles, query their owners, and expose outstanding cleanup. If completion races cancellation locally, use a conditional transition to accept one terminal disposition. A later provider reply may settle a pending effect without reopening that run. EMR job states · Azure instance management
Control long waits and approvals
Address an operator command to a durable run handle. Status reads accepted state. Pause requests suspension at a defined boundary, resume releases eligible work, and redirect changes pending instructions. A redirect that arrives after dispatch leaves the original operation to resolve. Cancel starts shutdown. Distinguish receipt of each command from its outcome. In Temporal, a Signal can return after server acceptance and before workflow delivery. An Update exposes acceptance and handler completion separately. A handler awaiting another operation may still be running when the main workflow reaches its return, so a workflow promising a processed decision must wait for that handler. Temporal message passing
Binding a decision to an action
An approval applies to the action the reviewer saw. If a reviewer approves publishing a report in a private folder and the destination later changes to a public one, the public release needs a new decision even though the report bytes are identical. Save the reviewed call ID, report revision, arguments, reviewer, and expiry with the continuation. Before dispatch, compare those details with the pending call and check current permissions and preconditions. For a match, consume the valid approval and release the call in one guarded transition so a duplicate reply cannot release it twice. Denial, cancellation, expiry, or a change to the action keeps that call from running. Other publications can proceed as their own reviews arrive. OWASP transaction authorization · Access governance
A decision authorizes one destination
ExampleThe same report and reviewer can yield a different authorization decision when the destination changes.
Read the diagram as text
- Reviewed publication. Report version and private destination.
- Pending publication. Current report version, destination, and permissions.
- Validate and consume decision. Check call identity, reviewer, validity, and current scope.
- Release call once.
- Withhold dispatch.
- Reviewed publication → Validate and consume decision: Decision and reviewed arguments.
- Pending publication → Validate and consume decision: Current action and authority.
- Validate and consume decision → Release call once: Approved, valid, matching, unused.
- Validate and consume decision → Withhold dispatch: Denied, changed, invalid, or used.
Record who owns a wait, which input would resolve it, and when to escalate. A watchdog inspecting those fields can distinguish an order legitimately awaiting a reviewer from a tool call whose worker disappeared. Bound reminders and give an expired decision an explicit disposition. In Vinoth Govindarajan’s account of an OpenClaw incident, retrying an expired approval callback preserved it across restarts and blocked later work. Govindarajan’s account · Review interfaces
Deliverables and evidence
Retain and publish artifacts
An artifact is an identifiable output retained for a later consumer, such as a report, patch, or dataset. A scratch file serves the worker creating it. A durable candidate must remain readable after that worker exits. Bound which files and how many bytes may leave the workspace, give each candidate an immutable identity, and assign an owner to retain it while consumers still depend on it. A checkpoint can then carry a reference to the output. Filesystem views and exports · Retention policy
Consider a report uploaded to Amazon S3 in parts. Before completion, persisted parts occupy storage even though consumers cannot open the assembled report. Complete the upload and confirm the readable object version. Then publish a manifest entry naming its location, digest, input versions, and producing execution. A digest identifies bytes. The other fields record provenance, allowing someone to find the inputs and run that produced them. A conditional write can guard a published reference against replacement. If registration lives in a separate database, retain enough information to finish it after a crash between assembly and registration. S3 conditional writes · SLSA provenance
When a report becomes readable and discoverable
ExampleAssembly can succeed before a manifest tells consumers where to find the bytes.
Uploaded fragments exist before a complete object can be opened.
Read the diagram as text
- Report candidate. Stable identity across stages.
- Staged parts. Fragments of the report upload.
- Readable object version. Complete report bytes.
- Published manifest entry. Location, version, and provenance.
- Report candidate → Staged parts: Stage candidate bytes.
- Staged parts → Readable object version: Assemble on upload completion.
- Published manifest entry → Readable object version: Reference exact version.
- Stage. Uploaded fragments exist before a complete object can be opened. Active: Report candidate, Staged parts. New: Report candidate, Staged parts.
- Assemble. The object is readable. A replacement can finish registration. Active: Report candidate, Staged parts, Readable object version. New: Readable object version.
- Publish. Consumers can resolve the manifest entry to the stored report. Active: Report candidate, Staged parts, Readable object version, Published manifest entry. New: Published manifest entry.
An interrupted upload also gives cleanup a testable outcome. S3 warns that a part already in flight can finish after the service accepts an abort request. Use ListParts to check that the part list is empty and repeat cleanup if necessary. Retaining the upload ID lets a replacement finish that obligation without disturbing another upload. AbortMultipartUpload
Enforce completion checks
A postcondition is a fact that must hold after the work. A completion check obtains evidence for it. A delivered report, for example, may have two requirements. Its contents must pass a specified check, and the destination must expose the checked version. Store a receipt naming the requirement, candidate, checker, and observation. A content pass for the new report and a destination receipt for the old one establish different facts. Delivery of the new version remains unconfirmed, so an automatic resend could duplicate it. Outcome evidence
| Check disposition | What the executor can do |
|---|---|
| Current pass | Accept this requirement within the check’s scope. Inspect any others. |
| Failed assertion | Identify the violated condition and allow bounded repair. |
| Checker unavailable | Withhold success. Retry the check within limits or hand it off. |
| Pass for a different candidate | Obtain evidence for the candidate now under consideration. |
Enforce success in one guarded transition that verifies run eligibility, current candidate and requirement versions, and positive receipts for every required check. Wiring in a checker alone can leave a gap. Claude Code’s TaskCompleted hook can return exit code 2 to prevent completion and send feedback to the model. Some hook failures and timeouts can be nonblocking, and the onFailure blocking option does not apply to this event. A mandatory check therefore needs an explicit pass to authorize success. If repair runs out of time, return usable partial work and identify the requirement still open. Claude Code hooks · Evaluator design
Expose authoritative status
A returning client needs a status query even if it missed every progress notification. Google’s long-running Operations protocol gives work a name and separates progress metadata, a done flag, and a terminal result or error. A wait call can return while work is still underway. For an agent run, expose the last accepted change, waiting reason, cancellation stage, remaining allowance, and unsettled effects. Each field should come from the record responsible for that claim. Google Operations protocol
One reconnect design returns a status snapshot with a version and lets the client request the accepted updates after that version. Apply the next version in order, ignore duplicates, and fetch current state again on a gap. A delayed running update then cannot overwrite a newer terminal record. Show last worker contact separately from last accepted progress. They can diverge when a worker keeps sending heartbeats while it waits on a dependency.
A trace links diagnostic records of related operations. A span describes one observed operation. Correlate spans with the stable run and operation IDs, the particular worker attempt and generation, the checkpoint it read, and the artifact and check receipts it produced. In Govindarajan’s account, a message tool returned success although the message failed to render. A successful send span helps locate where the evidence ends. An observation at the rendering client would support the stronger delivery claim. Until then the service can report send acceptance and leave rendering unconfirmed. Govindarajan on receipts · Traces and spans
Verification and design choices
Test interruption boundaries
Write down what must remain true after recovery, then interrupt the execution on either side of the event that could change it. FoundationDB’s simulation work illustrates this method with controlled timing, injected failures, and assertions about recovered state. For an agent, distinguish the number of physical calls, objects created at the receiver, and transitions accepted locally. Cornelia Davis demonstrated recovery by terminating a Temporal application worker and starting another. That experiment tests loss of worker memory with the history service still available. FoundationDB simulation testing · Davis’s demonstration
| Interruption or race | Observation required after recovery |
|---|---|
| Stop before task commit, after commit before publish, and after publish before the relay records delivery. | Every acknowledged task has recoverable dispatch. A repeated instruction admits one logical job. |
| Hold the saved choice and first remote creation fixed. Allow one retry. Crash before versus after recording the receiver’s reply. | After recording, replay reuses the result. Before recording, inspect the actual object count. An unprotected second creation exposes the duplicate-effect bug. |
| Stop during a checkpoint write and then just after commit. | Recovery reads a complete committed position. Count which inference calls repeat. |
| Deliver an approval twice, after expiry, or after changing its destination. | Only a first applicable approval releases the reviewed action. |
| Delay a worker’s write until after takeover. Submit cancellation twice. | The old write is rejected. Closed admission stays closed and children have inspectable dispositions. |
| Restart with a live reservation and deliver the same settlement twice. | No new worker spends the reserved units. Usage settles once. |
| Stop during multipart upload and after assembly before publication. | No complete reference points to partial bytes. Registration can finish for the original assembled candidate. |
| Lose the checker or replace the candidate between its pass and acceptance. | Neither missing evidence nor a stale pass permits success. An accepted reference opens the checked version. |
Fix the model response and control the receiver to isolate a particular race. Exercise the real receiver separately for its key retention, status lookup, and cancellation behavior. A substitute that remembers keys indefinitely will pass a test a real service can fail. Before deploying changed workflow code, replay representative histories from both open and closed executions in continuous integration. Successful replay checks compatibility for those histories. Assessing whether the agent chose a good action requires a task-quality trial with an outcome criterion. Focused tests · Integration boundaries
Choose sufficient machinery
Compare designs at the same interruption. The model has chosen a tool, the worker exits before dispatch, and the caller still wants the task completed. If making another choice is cheap and acceptable, recovery at the start of a small job may suffice. If the exact choice is part of an outstanding review, losing it invalidates the review. That one requirement changes the necessary recovery point.
| Arrangement | First recoverable position | Cost and responsibility |
|---|---|---|
| Supervised in-process loop | The supervisor can restart the process. A choice held only in memory must be made again. | The application owns timeouts, waits, and recovery of any promised work. |
| Persisted job with BullMQ | A stalled job can return to waiting and restart at job entry. Reuse requires an application record of the choice. | Operate the queue, bound stalls, make redelivery safe, and persist any wait that must outlive a worker. |
| LangGraph with persistent checkpoints | Start at a committed graph boundary. A choice in saved state can be reused. Work inside an unfinished node may repeat. | Choose node boundaries and write timing, retain checkpoints, and restore referenced inputs. |
| Temporal with inference in an activity | Reconstruct control flow and consume the saved choice if the activity result reached history. | The runtime retains history and waits. Operators maintain compatible code and available workers. |
BullMQ’s stalled-job mechanism illustrates the operating tradeoff. A blocked Node.js event loop can miss its liveness notification even while its process is alive. Repeated stalls can eventually fail the job. For checkpointed or replayed work, account instead for retained state, growing history, and compatible workers across deployments. Measure time from interruption to useful progress and the calls repeated before reaching it. For a long review, also measure the resources retained while idle. A workspace snapshot can spare a rebuild while bringing back background processes that need cleanup. BullMQ stalled jobs · Replay and snapshot tradeoffs
Codex Web’s server-side session illustrates recovery from a browser disconnect. Recovery of a tool action interrupted by worker loss needs a separate guarantee. A schema defines the arguments a tool accepts, a protocol connects clients and servers, and a platform team may operate the shared service. The execution decision is which component retains the pending call and can account for its outcome after interruption. Structured Outputs and Tool Calling · Model Context Protocol · AI Platform Engineering
Frequently asked questions
Is a transcript a checkpoint?
Only if the transcript also records the control position and everything required to continue there. A conversation can describe a proposed tool call while omitting the identity of the pending approval. Context Engineering
Can work survive a closed tab or a timed-out request?
Yes, if an independent service owns the execution and gives the client a handle to reopen it. Codex Web supports reconnecting to server-side work. Query the run to learn its current state.
Can a recovered activity create a second object?
Yes. A receiver may create the first object before the worker loses the reply. Recover using the original operation identity and a receiver-enforced retry or conclusive status lookup. Otherwise retain the unknown outcome.
Does cancellation undo a change?
A committed change survives cancellation. If it needs reversing, verify its state, obtain authority for the compensating operation, and check the outcome of that operation too.
What does a sandbox snapshot preserve?
A disk snapshot preserves covered files. Resuming a live process also requires its execution state. External services remain authoritative for actions they received before or after capture. Sandbox lifecycle and snapshots
Does a heartbeat prove progress?
A heartbeat establishes contact at the time of the report. If it includes progress details, inspect the latest received checkpoint and the durable work it identifies. A fresh contact time can accompany the same completed segment as before.
Can new code resume an old run?
It can when it understands the saved state and produces commands compatible with any recorded history. Otherwise keep a worker on the pinned build or provide an explicit migration that preserves pending work.
Is the task complete when the model stops?
A final message ends a Codex turn and may even ask for more information. Task success requires the observations specified for the requested outcome. Outcome evidence
When is a queue enough?
A persistent queue is enough when the job can safely and affordably restart at entry and the application records any pending waits and external operations. Save a finer checkpoint if restarting would discard an approved choice or repeat expensive work. If a remote write may have succeeded before its result was saved, the application still needs to check the outcome or retry under a key the receiver enforces.
Open questions
When can an operator retire a pinned build that still has approval waits? A new build may read the old record yet interpret a destination or pending command differently. Replaying histories with live waits, changed arguments, and revoked authority would test whether migration preserves the original decision.
When can a run release a reservation for a provider request whose usage never arrived? Holding it indefinitely blocks useful work, while releasing it early can overspend. A provider-enforced maximum and a settlement record tied to the original request would give the runtime a defensible release condition.
Which report requirements can be accepted without a human reader? An exact digest and passing structural checks can coexist with a missing answer. Comparing automated judgments with domain reviewers on those omissions would show which decisions the harness can safely automate.

























































































































































































































