# We Trust AI Agents That Break Every Day - Shy Ruparel, Temporal

Shy Ruparel, Temporal

AI Engineer New York 2026 · 2026-10-13

[Talk page](https://ai.engineer/talks/mJ7q8CuVmbw-we-trust-ai-agents-that-break-every) · [Video](https://www.youtube.com/watch?v=mJ7q8CuVmbw)

## We Trust AI Agents That Break Every Day

Shy Ruparel connects Temporal’s agent-adoption survey to a $2,500 payout demonstration: completed model work survives worker crashes because execution history lives outside the process, while the workflow retains control of money, approvals and recovery.

![Recording frame at 5:48: We Trust AI Agents That Break Every Day](https://ai.engineer/api/talk-essay-media/ad2865f497970eec73578266eb67c8c446af85020002f9699345c5e5da1db0f1)

We Trust AI Agents That Break Every Day — recording at 5:48

### Financial work outlives the process running it

[0:12](https://www.youtube.com/watch?v=mJ7q8CuVmbw&t=12s)

A payment can leave funds reserved in a bank even after the application handling it has disappeared. Temporal developer advocate Shy Ruparel described that mismatch at AI Engineer New York on October 13, 2026: financial work crosses systems and people, and its consequences outlast any one running process. Adding an agent puts model decisions into an already long-running coordination problem.

![Recording frame at 1:57: Financial work outlives the process running it](https://ai.engineer/api/talk-essay-media/6aa2801cbaf5fb4d37a804551e30dbb58b168984a7782b6d3effc147b4352de6)

Financial work outlives the process running it — recording at 1:57

Loan origination, know-your-customer and anti-money-laundering checks, fraud review and payouts can span days, multiple services and several people. A useful model response is only one contribution. The surrounding application still has to remember which actions succeeded, wait for decisions and finish the transaction without losing track of money.

Temporal’s second State of Development survey supplies the talk’s organizing evidence. It covered 554 engineers and engineering leaders actively using AI agents. Of those respondents, 48.4% had at least 11 years of industry experience, and roughly another third had six to ten years. The sample included engineering, AI, data, platform and IT roles; about two-thirds were in the United States, with the remaining third in the UK and EMEA. These are reports from an agent-using professional sample, rather than a measure of adoption across all engineers.

### Daily use does not mean production deployment

[3:03](https://www.youtube.com/watch?v=mJ7q8CuVmbw&t=183s)

The first gap separates using agents from shipping systems that depend on them. Daily use rose from 47.3% in the survey’s 2025 edition to 80.8% in the edition presented here. Yet only 49.1% reported running agents in production or treating them as core to how they ship. The other group was still experimenting or mainly using assistants.

![Recording frame at 4:13: Daily use does not mean production deployment](https://ai.engineer/api/talk-essay-media/4902e441e53443a2624a3936a3d7d75c5cc6bfacdeac3ef098abf81745f50d91)

Daily use does not mean production deployment — recording at 4:13

Those percentages describe different levels of commitment. An engineer can use an assistant every day without making an agent responsible for a customer-facing process. Production adds consequences: a failed run may interrupt work, leave external state behind or require someone to recover it. The survey establishes the adoption gap; the rest of the talk asks what engineering work sits inside it.

### High trust coexists with frequent failures

[4:33](https://www.youtube.com/watch?v=mJ7q8CuVmbw&t=273s)

The second gap is more surprising. Fully 85.5% of respondents trusted agent output at least somewhat, with roughly a quarter trusting it almost completely. Ruparel places himself among the skeptics, which makes the result particularly striking to him. But confidence in an answer does not tell us whether the system delivering it can finish its work.

How often do those systems encounter issues? The issue-frequency chart is the useful comparison: 41.1% of respondents reported issues daily or more often, including 16.4% reporting them hourly. Ruparel describes the daily-or-worse results as pink bars on the survey chart. Their prominence explains the tension in the title: people can trust agent output while still encountering failures every day.

Experience did not remove the problem. Teams describing themselves as successful were reported to be 1.7 times as likely to encounter issues in the hourly-or-daily category, while the report found no correlation between agent count and issue frequency. Ruparel offers a possible explanation: successful teams may look harder and therefore find more failures. That is a hypothesis, not a causal finding from the survey.

The resulting design goal is to contain the cost of failure rather than assume competence will eliminate it. A separate 40% of teams identified security as the blocker to autonomous agents. That remains outside this talk’s recovery demonstration: preserving execution progress does not, by itself, answer whether an agent should have permission to act.

### Enterprise readiness changes the shipping clock

[7:33](https://www.youtube.com/watch?v=mJ7q8CuVmbw&t=453s)

The third gap appears between a prototype and enterprise deployment. Across the survey, 51.3% reported getting an AI prototype production-ready in hours or faster. At organizations with more than 5,000 employees, about one in five reported taking months; elsewhere, about one in twenty did. The same broad tool category is operating on very different delivery clocks.

![Recording frame at 8:08: Enterprise readiness changes the shipping clock](https://ai.engineer/api/talk-essay-media/2d0489c8fdd0c0aabef6aab7d98c10979e4fdf6b942328f04b601d1a7e56e4d9)

Enterprise readiness changes the shipping clock — recording at 8:08

Banks, insurers and payment companies need approvals, audit trails and confidence that a payment will not disappear between services. Ruparel’s interpretation is that meeting those requirements accounts for some of the enterprise delay. The survey does not break down the causes, so it cannot establish how much time goes to reliability work versus other organizational constraints. It does explain why a fast prototype is an incomplete measure of readiness.

### A trace can explain a failure without repairing it

[9:03](https://www.youtube.com/watch?v=mJ7q8CuVmbw&t=543s)

The fourth gap gets closest to the implementation problem: seeing a failure and recovering from it are different capabilities. The leading productivity blockers concerned execution state and debugging, rather than model capability:

- **Tracking state across services:** 35.7% named it as a top blocker.
- **Isolating and debugging issues:** 30.9% identified this problem.
- **Redoing work after crashes or restarts:** 22.2% reported losing progress.

A separate survey from the previous year found that 27% called their state management fragile. Ruparel explicitly cautions against treating the two surveys as a trend line: the surveys asked different questions. Still, the current responses make the operational concern concrete. Teams have to know what happened across services, not just what the model said.

YouTube was the most common reported destination for help when an agent broke—a result Ruparel finds bewildering. His interpretation is that people seek an explanation elsewhere when their own system cannot supply one. Traces, logs and LLM observability tools can reveal a failure, but visibility alone does not restore a lost approval or reconcile reserved funds.

Only 13% of respondents were very confident in their ability to observe and debug AI workflows at scale. The gap becomes a person reading a trace at two in the morning and repairing state by hand. The payout demonstration asks a more useful question: can the execution record itself carry enough information to recover, instead of merely helping someone investigate?

### What a crash leaves behind in a payout

[10:58](https://www.youtube.com/watch?v=mJ7q8CuVmbw&t=658s)

The example is a $2,500 payout for a synthetic customer, implemented as a real Temporal workflow with a model running locally on the laptop. Its nine stages compress a larger financial process: validate the request, reserve funds, obtain an FX quote, investigate suspicious transaction details and obtain a human decision. The agent can assess whether a transaction warrants escalation and prepare a brief; the human makes the approval decision in the demonstrated path.

![Recording frame at 12:16: What a crash leaves behind in a payout](https://ai.engineer/api/talk-essay-media/dd8bb4a7b9c45fabaeacf3f148f430c8e43279dd49e9730e360caaa172553031)

What a crash leaves behind in a payout — recording at 12:16

The crash is deliberately placed after useful work has happened. Funds have been reserved in another system, and the agent has made tool calls and produced material for review. Stopping the worker simulates failures such as a timeout, deployment error or exhausted memory. The important distinction is where state lives: killing application memory does not undo the bank’s reservation.

In the memory-only design Ruparel contrasts with Temporal, the application could lose the brief and its knowledge of completed steps while the funds remain reserved. Regenerating the brief repeats model work; repairing the transaction requires persistent state tables, retry logic, recovery code and reconciliation jobs. Those mechanisms are necessary work somewhere in the system, but writing them separately for each business process distracts from the payout logic itself.

### Keep execution history outside the worker

[12:58](https://www.youtube.com/watch?v=mJ7q8CuVmbw&t=778s)

Temporal’s answer is durable execution: the code should be able to finish even if the process running it does not. Progress is recorded in an event history held by the Temporal Service, outside the worker. A replacement worker can use that history to reconstruct workflow state and continue from recorded progress.

![Recording frame at 13:54: Keep execution history outside the worker](https://ai.engineer/api/talk-essay-media/d16540f5f9cf2ff76a0c0991296f2ea7aad36bf280636fd5b09081dc772b2d30)

Keep execution history outside the worker — recording at 13:54

Three kinds of work have different roles:

- **Workflow code decides what happens next.** It is deterministic, allowing replay to rebuild the same state from the recorded history.
- **Activities perform nondeterministic or side-effecting work.** Model calls, tool calls and bank API calls run as activities. Choosing the investigation, looking up customer information and drafting the brief each have separately recorded results.
- **Signals bring external decisions into the history.** Human approval or rejection is recorded as a signal, including when no worker is running. Processing can resume when a worker becomes available.

What survives the disappearance of a worker? The relationship below separates completed work, persistent history and later execution. Activity results and human decisions both enter the service’s history; a replacement worker reads that history rather than asking the model to recreate completed work. The history is the bridge between two processes that do not share memory.

There is an important limit to the token-saving claim. Completed activities with recorded results are retained, but an activity still in flight when the worker dies may be retried. The demonstration therefore supports reuse of completed model work, not a blanket promise that every external call executes exactly once. That distinction matters when a call costs money or changes another system.

Ruparel connects this to the survey’s finding that eight in ten respondents said cost limited their agent use. Recovering a workflow without regenerating its completed investigation avoids paying again for that work. It also preserves the earlier result, instead of substituting a fresh model response during recovery.

Completed work survives through the service’s history
Arrows show recorded information and its use during recovery. Human approval can reach the history while the original worker is stopped; a replacement worker later reconstructs state and continues.
- Completed AI activities — Investigation choices, customer lookups and the drafted brief have recorded results before the crash.
- Human approval or rejection — An external decision can arrive even when no worker is running.
- Temporal Service event history — Stores activity results and signals independently of the worker process.
- Replacement worker — Replays workflow code against recorded history to rebuild state.
- Payout continues — Uses completed results and the recorded decision without restarting the investigation.
- Completed AI activities → Temporal Service event history (Records completed results)
- Human approval or rejection → Temporal Service event history (Records a signal during downtime)
- Temporal Service event history → Replacement worker (Supplies history for replay)
- Replacement worker → Payout continues (Resumes using reconstructed state)

### Approve the payout with no worker running

[15:05](https://www.youtube.com/watch?v=mJ7q8CuVmbw&t=905s)

Which inputs make this transaction worth investigating? Ruparel sets the payout to $2,500, marks the recipient as new, identifies the amount as unusual for the customer and adds an unresolved account alert to the customer record. These are deliberately suspicious inputs, chosen to push the agent toward an investigation rather than leave escalation to chance.

Starting the job creates the workflow. The agent examines the customer profile and payout information, then drafts a brief for human review. Approve and Reject controls are available, but before making the decision, Ruparel stops the worker. With no workers running, he approves the transaction. The approval is still logged as a signal.

The intended next step is to start a worker, let the retry policy handle the timeout and resume from the last successful event. The live restart takes longer than expected; Ruparel refreshes and switches to a backup recording. The subsequent recovery walkthrough is therefore a recorded example, rather than an uninterrupted live completion.

The backup example repeats the approval during downtime and resumes the worker. It also introduces a second interruption while the payout waits for the bank to approve or mark the transaction complete. The bank-related signal is recorded during that downtime too, and a new worker picks up the waiting workflow. The two interruptions test separate waits in the same business process: one for a person, another for an external financial system.

What distinguishes recovery from starting over? Ruparel reads the resulting event history from top to bottom: each AI activity shows one attempt and one successful execution, while the approval and bank-related events arrived during periods with no workers. Replacement workers continued the payout without repeating the completed investigation or spending its tokens again. The record connects the cost claim to specific activity attempts, rather than merely showing that the final payout advanced.

![Recording frame at 17:52: Approve the payout with no worker running](https://ai.engineer/api/talk-essay-media/65b660f74eacc7c98ef9aa3fb81610674415b59e29626a2a8373c58d0569cd9c)

Approve the payout with no worker running — recording at 17:52

### The agent advises; the workflow owns the money

[18:32](https://www.youtube.com/watch?v=mJ7q8CuVmbw&t=1112s)

The closing code discussion explains what the example moved out of application-specific recovery logic. Workflow state appears as ordinary fields and variables; replay rebuilds it after a crash. Ruparel did not build a separate database or cache for that state—the persistence sits in Temporal’s execution machinery. Completed model and tool results are retained with the workflow.

Retries are declared as a policy rather than implemented as a hand-written loop around each call. In this example, the configured policy applies to the transaction’s calls. Long waits also remain part of the workflow: the human-approval timer could be much longer without changing the basic waiting code. Ruparel uses a year-long wait as an illustration of the execution model, not as a duration exercised by the demo.

The most consequential separation is between advice and transaction control. How does the application keep an unavailable model from taking the payout down with it? The agent has a bounded opportunity to investigate and produce a brief; the workflow owns money movement and state. In this design, model failure can leave the payout continuing without the brief. That is a deliberate fallback choice with a narrower outcome than uninterrupted investigation: execution can proceed, but the AI’s review material is absent.

![Recording frame at 19:25: The agent advises; the workflow owns the money](https://ai.engineer/api/talk-essay-media/5e455b816984e80f31f43792288a8f531037441e93c3dfd249b967cac5daad19)

The agent advises; the workflow owns the money — recording at 19:25

The same separation clarifies debugging. The history inspected in the UI is the history the worker replays. An engineer does not have to reconcile one observational account with a separately maintained recovery record before resuming execution. This is the concrete sense in which one record closes the seeing-versus-recovering gap: it both explains completed progress and supplies the information used to rebuild it.

Ruparel groups the demonstration’s benefits into seven concerns: state, debugging, cost, retries, approval, long-running work and crash recovery. He also names OpenAI’s Codex and Replit’s agents as production uses of Temporal. Those examples position the mechanism beyond this payout, but the demonstrated result remains specific: recorded work and external decisions survived worker replacement. The survey’s adoption, enterprise-readiness and security challenges are broader than that one recovery result.

One final extension brings output quality into the activity model. If application logic rejects a model response and treats it as an activity failure, Temporal can apply the retry policy to that failure too. The talk does not provide the acceptance check, so retrying cannot substitute for deciding what counts as a good answer. It does give that decision an execution mechanism: retain accepted results, retry rejected attempts under policy and let the workflow—not the model’s current process—carry the transaction forward.

## Key takeaways

- Temporal’s survey separates daily agent use, reported by 80.8% of respondents, from production use or agents being core to shipping, reported by 49.1%. Using an assistant does not exercise the same operational requirements as depending on an agent.
- High trust in agent output can coexist with frequent system issues: 85.5% trusted output at least somewhat, while 41.1% reported issues daily or more often. Recovery has to be designed even when users like the answers.
- Deterministic workflow code controls progression; activities contain model calls and external side effects; signals record outside decisions. Their shared event history lives in the Temporal Service rather than in worker memory.
- The payout example retained completed AI activities and accepted approval during worker downtime. Completed results were reused, but an activity interrupted in flight could still be retried.
- The agent’s investigation is advisory and bounded; the workflow owns payout state and money movement. Continuing without a brief is an explicit fallback, while rejecting bad model output still requires an application-defined acceptance check.

## Resources

- [What is durable execution?](https://temporal.io/blog/what-is-durable-execution) — A Temporal explainer for following up on the distinction between a short-lived process and an execution that retains progress through failures.
- [Temporal quickstarts](https://docs.temporal.io/quickstarts) — A practical starting point for experimenting with workflows and activities after understanding the payout example’s execution model.

## Related talks

- [Don't Let the LLM Drive - Ornella Bahidika & Joel Allou, Microsoft](https://ai.engineer/talks/m24UKZomm7k-dont-let-llm-drive) — Develops the same separation between bounded model work and application-controlled progression, using a voice tutor whose state machine governs lessons and validates transitions.
- [Building Self-Learning Loops for Your Agent — Fuad Ali, Arize AI](https://ai.engineer/talks/fnuNVFgLjHA-building-self-learning-loops-your-agent-fuad) — Complements crash recovery with behavioral repair: tracing a shopping assistant’s faulty price filter, proposing a code change and evaluating the fix against collected failures.
- [The Lethal Trifecta Is Already on Your Laptops — Michael Patterson, Coder](https://ai.engineer/talks/abw_m-DRi1k-lethal-trifecta-is-already-on-your-laptops) — Examines the security concerns left outside the payout demonstration, including untrusted content, private data, external communication and deliberately scoped execution permissions.

## Slides

### State of the Agent

[0:15](https://www.youtube.com/watch?v=mJ7q8CuVmbw&t=15s) [Shy Ruparel introduces a survey of reliability and observability for production agents.](https://assets.ai.engineer/sites/aiecode2025/sha256/187f733fdb114c0222f41c6f6144264f4721b2a1c2c7623ce0fc56c2a8b440a8.webp)

Temporal
State of the Agent
Reliability, Observability, and the Gap Between Them
Shy Ruparel
Sr. Developer Advocate, Temporal
AI Engineer NYC 2026

### AI in financial services

[1:04](https://www.youtube.com/watch?v=mJ7q8CuVmbw&t=64s) [A large 94% figure highlights the share of surveyed financial-services leaders who call AI core to their operations.](https://assets.ai.engineer/sites/aiecode2025/sha256/451abce80d97f904b9fc94d32dec952194d9367f3c6a0148b8e1771a63ee165f.webp)

94%
of financial services leaders say AI is core to their
business operations.
SS&C Blue Prism, "The Trillion Dollar Transformation" (2025, n=377 financial services professionals)
5

### The cost of reliability issues

[1:16](https://www.youtube.com/watch?v=mJ7q8CuVmbw&t=76s) [A 62% figure highlights teams reporting measurable losses in time or revenue.](https://assets.ai.engineer/sites/aiecode2025/sha256/a6fda483abf14adb8d5e8f6227e52811b3fa35f418808c7810bf3595c134a379.webp)

62%
of teams lose measurable time or revenue to AI
reliability issues.
AI Production Stack Report 2025 (Temporal, 153 respondents)

### Who answered the survey

[2:10](https://www.youtube.com/watch?v=mJ7q8CuVmbw&t=130s) [The slide introduces a survey of 554 engineers and engineering leaders.](https://assets.ai.engineer/sites/aiecode2025/sha256/f542270f9cdccf4830b222fc973f20ea4eb08d0489dcbec23dac4a8be046d681.webp)

STATE OF DEVELOPMENT 2026
554 engineers and engineering
leaders. How is it actually going?
Temporal survey, fielded by Qualtrics, April 29 - May 25, 2026.
10

### Four gaps

[3:04](https://www.youtube.com/watch?v=mJ7q8CuVmbw&t=184s) [A list names four gaps between expectations for agents and production experience.](https://assets.ai.engineer/sites/aiecode2025/sha256/73fb54b01b3b5f3274c405207b33029e2a4175feecf75482dc2853f798461634.webp)

01
02
03
04
Four gaps.
Usage vs. production
Trust vs. reality
Prototype vs. enterprise
Seeing vs. recovering

### Daily agent use

[3:21](https://www.youtube.com/watch?v=mJ7q8CuVmbw&t=201s) [Two bars compare how often respondents used agents a year ago and at the time of the survey.](https://assets.ai.engineer/sites/aiecode2025/sha256/b8909ad574cf35b44bf3ca747469fb56b57b397b2b328c38ff4ef0397d0c9250.webp)

Daily agent use, a year ago vs. now
A year ago: 47.3%
Now: 80.8%
State of Development 2026 • Chart 1.2 • “daily or more” = continuously + hourly + daily

### Using agents and shipping them

[4:12](https://www.youtube.com/watch?v=mJ7q8CuVmbw&t=252s) [The slide places daily agent use beside the share of respondents shipping them.](https://assets.ai.engineer/sites/aiecode2025/sha256/119ed867583eac615503ce659d2ac572a9e32c455ddc97d9edbe496590014a44.webp)

Gap 1 • Usage vs. production
80.8% use agents daily or more
49.1% ship them
State of Development 2026 • Charts 1.1, 1.2

### Trust and daily failures

[5:47](https://www.youtube.com/watch?v=mJ7q8CuVmbw&t=347s) [Two percentages compare trust in agent output with the frequency of production issues.](https://assets.ai.engineer/sites/aiecode2025/sha256/df0b35ab850d80ffff90201601a3c97aeb60d502d10f089927a5bec596326984.webp)

Gap 2 • Trust vs. reality
85.5% trust agent output
41.1% hit issues daily
State of Development 2026 • Charts 2.4, 2.6

### From prototype to production

[8:07](https://www.youtube.com/watch?v=mJ7q8CuVmbw&t=487s) [The slide contrasts fast releases with the longer timelines reported by large companies.](https://assets.ai.engineer/sites/aiecode2025/sha256/f1fa2b438a0cf436520e89208bacb9db92be18c16fdf862ce0267209a8767d45.webp)

Gap 3 • Prototype vs. enterprise
51.3% ship in hours or faster
18.5% of 5,001+ companies take months
State of Development 2026 • Chart 2.2, Section 2.2

### What limits team productivity

[9:03](https://www.youtube.com/watch?v=mJ7q8CuVmbw&t=543s) [A horizontal bar chart ranks reported obstacles, with tracking state across services at the top.](https://assets.ai.engineer/sites/aiecode2025/sha256/8e42f3be30761c2a0b3bcb1c0ab20eb64c37530332bd5612ff91d9d8d52abca3.webp)

What limits your team's productivity?
Tracking state across services
Isolating & debugging issues
Managing token & compute costs
Code unrelated to business logic
Retries & timeouts code
Human approvals in workflows
Long-running multi-step workflows
Security & compliance
Lost progress from crashes
Organizational buy-in
Too much work for team size
No production-grade infra
No time to think and plan
State of Development 2026 • Chart 3.1• multiple answers allowed
35.7%
30.9%
30.7%
29.4%
28.9%
27.6%
27.4%
24.4%
22.2%
20.0%
17.3%
17.1%
11.0%
42

### Debugging at scale

[10:26](https://www.youtube.com/watch?v=mJ7q8CuVmbw&t=626s) [Two survey figures compare confidence in debugging AI at scale with lost progress after crashes.](https://assets.ai.engineer/sites/aiecode2025/sha256/81e40eb3126b6d2701401783c2225ca6140a26f1606e6953e64be77789d948cb.webp)

Gap 4 • Seeing vs. recovering
13% very confident they can debug AI at scale
22.2% lose progress to crashes
AI Production Stack Report 2025 (153 teams) • State of Development 2026 • Chart 3.1 • different surveys

### A $2,500 payout

[10:52](https://www.youtube.com/watch?v=mJ7q8CuVmbw&t=652s) [The example introduces a synthetic customer payment that uses an LLM inside a workflow.](https://assets.ai.engineer/sites/aiecode2025/sha256/755cbc318f41835ed86df07a8100b6f16f58d415fe144a3100f37429762c4e39.webp)

A PAYOUT
Let's move $2,500.
Synthetic customer. Real Temporal workflow. Local LLM on this laptop.
52

### Steps in the payout workflow

[11:52](https://www.youtube.com/watch?v=mJ7q8CuVmbw&t=712s) [The payout checklist includes reserving funds, checking an exchange-rate quote and waiting for human approval.](https://assets.ai.engineer/sites/aiecode2025/sha256/ebc38958a193f94502082ca27e0af5b47dc465f43c89eafb304db172b19ee70f.webp)

A payout workflow with an agent in it
Validate the request
Reserve the funds
Check the FX quote
AI: decide what to investigate
AI: look up customer evidence
AI: draft a brief for the approver
Wait for a human to approve
Submit to the bank rail
Confirm and complete

### When the payout process stops

[12:19](https://www.youtube.com/watch?v=mJ7q8CuVmbw&t=739s) [The slide lists what happens to reserved funds, the approval brief and the cost of completed model calls.](https://assets.ai.engineer/sites/aiecode2025/sha256/088709c90bd0d0da674167528b86ba0e197599fa2c34f42b2127dc2f9a256105.webp)

Funds reserved with no payout. A brief the
approver never sees. LLM tokens spent for nothing.
58

### Durable execution

[13:09](https://www.youtube.com/watch?v=mJ7q8CuVmbw&t=789s) [A definition explains how workflow code can finish after the process that started it has stopped.](https://assets.ai.engineer/sites/aiecode2025/sha256/7305534abedcc9ab1962105b46569ee72e6a707007215ba77514a325ab612dd5.webp)

DURABLE EXECUTION
Code that runs to completion,
even when the process doesn't.
61

### Every step in the event history

[13:26](https://www.youtube.com/watch?v=mJ7q8CuVmbw&t=806s) [A timeline records workflow steps and model calls within the same execution.](https://assets.ai.engineer/sites/aiecode2025/sha256/77f2446dbda4a463e4bf0df2f874bcd0535b5a5343b583b22a63ef3c523671d5.webp)

Every step, recorded. Including the AI.
Timeline
DraftAiBrief • Draft AI payout brief
LookupRecentPayouts • Look up synthetic customer evidence
LookupCustomerProfile • Look up synthetic customer evidence
ChooseInvestigation • AI chooses customer investigation tools
ValidateFxQuote • Check FX quote validity
ReserveFunds • Reserve funds in the ledger
ValidatePayout • Validate payout request

### Replaying completed steps

[14:26](https://www.youtube.com/watch?v=mJ7q8CuVmbw&t=866s) [A diagram shows two workers connected through a shared event history.](https://assets.ai.engineer/sites/aiecode2025/sha256/c98c2535f91221d280758d771bf8d3d08a39e209b90e725dbd4b5093f7954297.webp)

Replay: finished steps are read back, not re-run
Worker A: killed mid-workflow
Event history
ValidatePayout → result
ReserveFunds → result
ValidateFxQuote → result
ChooseInvestigation • LLM
LookupCustomerProfile
LookupRecentPayouts
DraftAiBrief • LLM
Timer: wait for approval
Signal: approve
Worker B: results injected from history
0 LLM calls repeated
SubmitToRail

### Where the demo helps

[18:18](https://www.youtube.com/watch?v=mJ7q8CuVmbw&t=1098s) [Seven bars are highlighted in the chart of problems reported by surveyed engineering teams.](https://assets.ai.engineer/sites/aiecode2025/sha256/b811647ccd6787a8db050ffdfc28c8f1e501340fb1b91203e8902322ddcbdd8c.webp)

That demo just handled seven of them
Tracking state across services
Isolating & debugging issues
Managing token & compute costs
Code unrelated to business logic
Retries & timeouts code
Human approvals in workflows
Long-running multi-step workflows
Security & compliance
Lost progress from crashes
Organizational buy-in
Too much work for team size
No production-grade infra
No time to think and plan
State of Development 2026 • Chart 3.1

### Recording model results

[18:46](https://www.youtube.com/watch?v=mJ7q8CuVmbw&t=1126s) [A code example treats model calls as activities whose results are stored in workflow history.](https://assets.ai.engineer/sites/aiecode2025/sha256/8e54083656f6b5024815e975375ce6929a3cc5c81bb05a419b79ac0db46b4ceb.webp)

• MANAGING COSTS • 30.7%
LLM calls are Activities. Their results live in history.
val choice = aiInvestigation-chooseInvestigation(request)
choice. tools. forEach { tool →> lookup (tool) }
aiBrief = ai.draftAiBrief(evidence).brief
// On replay, each result comes from history: zero tokens.
// LLM
M tool calls
// LLM
State of Development 2026 • Chart 3.1 • code: PayoutWorkflowlmpl.kt (simplified)
80

### Retry policy

[19:00](https://www.youtube.com/watch?v=mJ7q8CuVmbw&t=1140s) [The code example configures retries as a policy on an activity.](https://assets.ai.engineer/sites/aiecode2025/sha256/d0b40836b36a04c50b2e6243c8340853da7fd60b05a8fa799a6aa00428906941.webp)

• RETRIES & TIMEOUTS • 28.9%
Retries are a policy, not a loop.
RetryOptions {
setInitialInterval (Duration. ofSeconds (1))
setBackoffCoefficient (1.1)
setMaximumInterval (Duration.ofSeconds (20))
setMaximumAttempts (10)
State of Development 2026 • Chart 3.1 • code: PayoutWorkflowlmpl.kt (simplified)
81

### Long-running timers

[19:35](https://www.youtube.com/watch?v=mJ7q8CuVmbw&t=1175s) [A timer can wait seconds or days without a polling loop.](https://assets.ai.engineer/sites/aiecode2025/sha256/cf3dec8a312d823428400f0e0105dc728dd63660fa1f6b005eca4c77517da6ff.webp)

V LONG-RUNNING WORKFLOWS • 27.4%
That timer could be 60 seconds or 60 days.
Same code. No thread held open. No polling loop. No cron job.
State of Development 2026 • Chart 3.1
84

### The trace is the recovery

[19:49](https://www.youtube.com/watch?v=mJ7q8CuVmbw&t=1189s) [Side-by-side flow diagrams compare diagnosing an agent after failure with recovering from the workflow history.](https://assets.ai.engineer/sites/aiecode2025/sha256/9027a0440a81f26e90b46a2566befa5ddc707b8db7aa7cc14ff8a2f688f2d6cd.webp)

v Debugging • 30.9%: the trace is the recovery
TWO RECORDS
ONE RECORD
Your agent
Your workflow
Event History
Logs & traces
observability
read by humans
State in a DB
recovery
read by code
They drift. A human closes the gap.
Temporal Ul
you observe it
Worker replay
it recovers from it
The trace is the recovery.
State of Development 2026 • Chart 3.1
86

### Survey source

[21:06](https://www.youtube.com/watch?v=mJ7q8CuVmbw&t=1266s) [The closing slide gives the URL for the survey behind the figures.](https://assets.ai.engineer/sites/aiecode2025/sha256/bf00101e04d309bc53f27b039404947c46f6571aa1f8d316cfc14cdb5f235f8c.webp)

Every number in this talk:
temporal.io/reports/state-of-development-2026
Come find us at the Temporal booth.

## Full transcript

[0:12](https://www.youtube.com/watch?v=mJ7q8CuVmbw&t=12s) All right. Hi, everyone. I'm Shy. I am a developer advocate at Temporal. Uh, I cannot see anyone in the room, so if you boo me or, like, wanna throw tomatoes or something, I, I will have no idea. Um... All right. Oh, clickers are difficult. There we go.

[0:33](https://www.youtube.com/watch?v=mJ7q8CuVmbw&t=33s) I guess I'm not using the clicker today. So, uh, agents, they are everywhere. I'm guessing if you are in this room, you are probably building an agent or shipping an agent. Does that sound about right? Yeah? Um, so last year KPMC, uh, conducted a survey, and they figured out that, uh, we have seen $3 trillion in terms of corporate productivity gained in 2025, and that's from their, uh, agentic AI report. Uh, and from that three trillion, it's really fin- uh, financial services that are

[1:03](https://www.youtube.com/watch?v=mJ7q8CuVmbw&t=63s) leading the way here. So 94% of fintech are saying that AI is core to how their businesses operate, and they've decided that AI isn't just a side project, but, but it's like, it's part of the business now, like, this is how we operate. Uh, and 62% of those teams lose measurable time, uh, or revenue to reliability issues. So they haven't deci- you know, deciding that AI is, uh, that something we're doing isn't necessarily the same as shipping it. There's definitely still a gap between

[1:33](https://www.youtube.com/watch?v=mJ7q8CuVmbw&t=93s) those two things there. Now, to break it down, uh, part of this is the shape of the work. Finance runs on long-running workflows, not, not on individual models. Um, you know, for some examples of their process, you can think of loan origination, KYC and AML, fraud review, uh, paying people. Each of these processes tan- spans multiple days, multiple systems, and usually multiple people. Uh, so there's many, many steps. There's many, many systems, and there's real money involved,

[2:03](https://www.youtube.com/watch?v=mJ7q8CuVmbw&t=123s) so they can't mess anything up. And now imagine that we're gonna go ahead and put an agent into the middle of that process. So, uh, at Temporal, we wanted to understand how things are actually going, so we conducted a survey. This is our second year running this survey, the State of Development Survey. Uh, we surveyed 554 engineers and engineering leaders who are actively using AI agents right now. Every number on this slide deck is either from that report, uh, or it'll be qualified in the bottom left from the

[2:33](https://www.youtube.com/watch?v=mJ7q8CuVmbw&t=153s) report that I'm pulling it from. So to start things off, we can talk-- take a look at the demographic of that survey. 48.4% have 11 plus years of experience working in industry, so we're not talking about hobbyists. And then the remai-- another third of that segment also has six to 10 years. So none of these people are hobbyists. They're all professionals working, um, every day. Um, it's a mix of engineers, AI engineers, data engineers, platform teams, IT leaders. Two-thirds of them were from the US. A third of them were from the UK and

[3:03](https://www.youtube.com/watch?v=mJ7q8CuVmbw&t=183s) EMEA. Uh, and what we found, uh, is four gaps. I promised you all that this talk was about gaps in the title. Uh, and the data shows that there are four major gaps, and I wanted to walk through each of them and then close with kind of one big idea. So let's go into gap one, and that's the gap between, uh, usage versus production. So daily agent use from our 2025 version of this survey w- had us at about 47.3% of the folks that we talked to were using agents daily. Uh, today, uh,

[3:33](https://www.youtube.com/watch?v=mJ7q8CuVmbw&t=213s) nearly a year later, we've doubled that to 80.8%, uh, of users are using agents in their day-to-day. Um, now, using agents isn't necessarily the same thing as shipping agents. Uh, and so if we take a look at those teams again and ask them what they're doing, about half of them are running those agents in production, or they say that those agents are core to how they ship. And then the other half of that is still experimenting or still only really using assistants. So we've got

[4:03](https://www.youtube.com/watch?v=mJ7q8CuVmbw&t=243s) 49.1% of the folks that we talked to, half of the folks that we talked to are running agents in production or saying that those agents are core to how, how things ship. Um, so that's a huge gap, right? Like, that's a 30% gap between 80% of the people are using agents every day and 50%, 49.1% are actually shipping them and using them in production, so far fewer people in production than actually using them daily. A- and something seems to keep breaking on the way to production, and let's talk about what that is, and

[4:33](https://www.youtube.com/watch?v=mJ7q8CuVmbw&t=273s) that brings us to gap number two, which is trust versus reality. Now, one of the things we asked folks in this survey was, how much do you actually trust your agent output? And trust is really high, actually. I'm really impressed by this number. Uh, a quarter of the folks that we, we surveyed trusted the agent output almost completely. Most people trusted it somewhat, and almost no one disrupt it-- distrusted it. I did not get asked to, to participate in this survey because I'm in that complete somewhat distrust category myself. So I'm actually really surprised by this

[5:03](https://www.youtube.com/watch?v=mJ7q8CuVmbw&t=303s) number from that survey. Um, yeah, 85.5% trust agent output at least somewhat, uh, and, and that's gonna be an important number. Let's come back to that 85.5 in a little bit. So we also asked, how often do your agents have issues? Um, the pink bars are daily or worse, and that's a lot of pink for a number that, uh, 85% of people trust. That's a lot more pink than I would expect for an 85% trust rating. Uh,

[5:32](https://www.youtube.com/watch?v=mJ7q8CuVmbw&t=332s) 41.1% of those folks hit those agent issues daily or more, so that's four in 10 of the people that we talked to see issues every day, and then 16.4 hit them every hour. So one in six of the people that we surveyed were getting an agent issue every hour. And then that brings us back to that trust, that 85.5% gap versus the 41.1%, uh, system breaking. Uh, so,  so we trust the output and the system, but it still breaks every day. Uh, both of those things are true at the same time.

[6:02](https://www.youtube.com/watch?v=mJ7q8CuVmbw&t=362s) And getting good at agents doesn't seem to, to make this go away. You can't skill your way out of this. Um, we were seeing teams that call themselves successful were actually 1.7 more times likely to hit issues continuously, so be in that hourly or daily category. Uh, and it's not because they've run more agents, right? The report found that agent count and issue frequency aren't actually correlated in any way. I'm guessing, and the team at Temporal are guessing, that maybe those folks are just looking harder

[6:32](https://www.youtube.com/watch?v=mJ7q8CuVmbw&t=392s) for issues, and so they're finding those issues some more. That's definitely something we need to dig into a little more when we do next year's version of this survey. But the important, I guess, takeaway is that even with how much you trust your agents, failure isn't going away. Uh, you have to account for it. You have to plan for it. So the goal isn't zero failures, right, when we're using agents in production. It's just how do we make it so these failures don't cost us anything? Now, a really interesting stat is saying that 40%, uh, of the teams that we talked to says security is truly the thing that

[7:02](https://www.youtube.com/watch?v=mJ7q8CuVmbw&t=422s) is blocking them from autonomous agents, and that's its own talk. Mason, uh, from the Temporal team and Kim from the Keycard team actually gave a talk about this yesterday on the workshop day, so I'm gonna defer to the recording of that talk, uh, rather than going to it myself, um, as well. And let's jump into that third gap, the prototype, uh, that we- we're probably all seeing and working on last year versus the enterprise deployment. You know, customers are actually touching it. That prototype to production-ready code system. And most teams going from an AI prototype

[7:33](https://www.youtube.com/watch?v=mJ7q8CuVmbw&t=453s) wanna go to production-ready code, uh, really fast. We're seeing it in, like, really low numbers. Uh, 51.3%, uh, are able to get a- an AI prototype to production ready in hours or faster. Uh, that's like startup speed, right? That's traditional startup speed. Uh, this falls apart though if you work somewhere big. Uh, so if you work somewhere, uh, five thousand plus employees, it tends to take months, uh, and the data breaks down at that point. The biggest companies, about one in five says it takes months.

[8:03](https://www.youtube.com/watch?v=mJ7q8CuVmbw&t=483s) Everywhere else, it's about one in twenty saying that it takes months to get an AI agent deployed into production. So big companies, uh,  are shipping at very different speeds. We're using the same tools, uh, at very different speeds. Half are shipping in hours, and then at these large companies, we're doing it one in five, taking five months. Um, big companies, regulated work, uh, we're talking about banks, we're talking about insurers, we're talking about payment companies. I'm guessing there's a lot of those in the room right now, and those companies all need something really--

[8:32](https://www.youtube.com/watch?v=mJ7q8CuVmbw&t=512s) need a bunch of things that are really important, right? They need approvals. They need audit trails. They need systems that will not lose a payment. My read is that's where the five months are going, into the auditability and the trust of, of all of those things. And the survey doesn't break that down, right? But it's not bureaucracy for its own sake, or at least I hope it's not. I hope it's, it's bureaucracy for something productive and, like, getting to that reliability bar. Which brings us to that last reli-reliability gap, uh, seeing versus recovering, and this is one of the ones that was in the title, right?

[9:03](https://www.youtube.com/watch?v=mJ7q8CuVmbw&t=543s) So we asked, "What limits your team's productivity?" Number one is tracking state, and number two is debugging. The models aren't in the top two, uh, at all. Um, 35%-- .7% were saying, uh, state, uh, tracking state across services is their top blocker, and then 30.9% says that it's isolating and debugi-debugging issues. 22.2% are lo-- uh, of that survey are losing progress and redoing work because of crashes and restarts. So that's one in five losing work. And then

[9:32](https://www.youtube.com/watch?v=mJ7q8CuVmbw&t=572s) last year, in a different survey, um, 27% called their state management fragile. This year it's the number one blocker. Uh, different surveys, different questions, um, so it's not a trend line, but the story is consistent. State got more important as agents got more real. And one of the wildest responses that we got was when, when an agent breaks, where you go for help, and the number one answer to that being YouTube is kind of wild to me. I don't really code with YouTube, and so this is very

[10:02](https://www.youtube.com/watch?v=mJ7q8CuVmbw&t=602s) different from me, uh, and the way that I build things. Um, but that's what debugging looks like when your system can't tell you what happened, is you end up going to YouTube and trying to find someone that can. Observability right now just tells you your agent failed. Most teams have that, right? Traces, logs, LLM observability tools. But reliability means that failure doesn't cost you anything.

[10:28](https://www.youtube.com/watch?v=mJ7q8CuVmbw&t=628s) Uh, only 13% of the people that we talked to were very confident that they can observe and debug AI workflows at scale, and then 22% of the people that we talked to lose progress because of crashes. And so the gap between seeing, uh, and recovering is a person reading a trace at two AM in the morning and then fixing their state by hand. And that's not realistic, right? Like, that's not a realistic thing to expect, uh, is someone fixing things at two AM in the morning. So I wanna show a quick little demo, right, of what this looks like, and I'm gonna move twenty-five

[10:58](https://www.youtube.com/watch?v=mJ7q8CuVmbw&t=658s) thousand-- twenty-five hundred dollars. This is a synthetic customer, but it's a real temporal workflow. It's a real LLM running on this laptop. I have Qwen three six B, uh, nine bi- nine B running on this laptop right now. Uh, and I'll show you a demo of what we're about to go through. But basically, we're running a payout workflow. So this is gonna be a nine-stage workflow, uh, which is gonna be much shorter than I think it would be in, like, in production. Uh, money is gonna move. Uh, the money is gonna look kind of weird. Uh, and so an agent is gonna jump in and is gonna do a fraud

[11:28](https://www.youtube.com/watch?v=mJ7q8CuVmbw&t=688s) investigation. The agent is gonna just take a look at the transaction and decide, do we need to escalate this to a human being, or is this good to just wave it through? Uh, and then we're gonna have a human in the loop that actually makes the decision and decides what to-- uh, whether or not to make the final call of this transaction getting approved or not. We're gonna validate the request, reserve the funds, check in with the FX quote, and those funds will be reserved, so the real state will exist in another system. And then I'm gonna go ahead and kill-- uh, sorry.  And we're gonna have that agent do the

[11:58](https://www.youtube.com/watch?v=mJ7q8CuVmbw&t=718s) real work, do those tool calls, uh, and then I'm gonna kill the process. Um, so I'm gonna just tell the worker to die. I'm gonna stop the worker, uh, and so our process will die, uh, and that could be for any reason, right? Like, it could be a timeout, it could be a deploy error, we could run out of memory. I'm gonna just intentionally kill it to simulate a worker crash, and we'll see what we lose. Um- What we would lose traditionally is that the funds would be reserved with no payouts, uh, and a brief would get produced that never actually makes its way to the approver. So we're gonna have to re-spend our

[12:28](https://www.youtube.com/watch?v=mJ7q8CuVmbw&t=748s) LLM tokens traditionally to regenerate that AI, uh, draft or summary for the check. Um, in that tech- in, like, a traditional stack, all of that is gonna live in process memory. And when that worker crashes, it's all gone. And so the reserve funds are still reserved, it's just not in our system. It's in the bank system. So traditionally, we'd have to write recovery code, we'd have to build state tables, we'd have to build retry logic, we'd have to create reconciliation jobs, and that's all code that's unrelated to the core business logic that I'm building to just do a payment transaction,

[12:58](https://www.youtube.com/watch?v=mJ7q8CuVmbw&t=778s) and that's something that 30% of our respondents complained about, is that they're building all of this code unrelated to bus- business logic. Or you could use a platform that remembers you, and, uh, that's where Durable Execution comes in. So Durable Execution is a temporal concept that code essentially should run to completion even when the process doesn't. We know that crashes happen, but with Temporal, state survives, progress survives, uh, and every step is recorded in an event history that looks like this. And so we've also recorded

[13:28](https://www.youtube.com/watch?v=mJ7q8CuVmbw&t=808s) the AI in this state. So this is a timeline that'll- we'll see in a second live that's a run from r- the payout demo that we're gonna run in a second. Uh, each of those steps, uh, each of those AI steps, choosing the investigation, the tool calls that the agentic loop is gonna make, uh, for the lookup of the customer and then drafting the brief, each of those is its own individual activity with a recorded result, and the yellow bar is gonna be our, our essentially activity waiting for a human approval. Um, the workflow code is deciding what's next, and it's

[13:58](https://www.youtube.com/watch?v=mJ7q8CuVmbw&t=838s) deterministic. Anything non-deterministic or side effecting like the LLM is gonna be run in an activity. So a tool call or a bank API call is gonna be an activity. And then the person when they approve or reject the transaction is gonna be an event too. It's a signal, is what we call them in Temporal, which is, like, recorded in history, uh, like everything else is recorded in the Temporal history. Even though the worker's not running, that signal is still gonna get captured, uh, and get processed. So when the worker dies, another worker is able to resume from the last previous success. Uh,

[14:28](https://www.youtube.com/watch?v=mJ7q8CuVmbw&t=868s) yeah. And the history lives in the Temporal Service, uh, and we're not running our LLM tool calls multiple times, so we're not wasting tokens, uh, on multiple tool calls. We're gonna retain the agent's work, uh, and not redo it. Uh, if the call was mid-flight while the worker died, we would retry that activity, but everything before that is, uh, is kept. Um, and that matters. Eight in 10 people say that cost limits how they use their agents, or being able to hold onto the agent's memory, uh, is really helpful. Rerunning an agent after every crash is a lot of money that you're

[14:58](https://www.youtube.com/watch?v=mJ7q8CuVmbw&t=898s) burning right there. Uh, so let's go into the quick demo and...

[15:05](https://www.youtube.com/watch?v=mJ7q8CuVmbw&t=905s) Yeah. So this is my, uh, payment transfer. So, uh, I'm gonna go ahead and set it to $2,500. I'm gonna say that it's a new recipient. I'm gonna say it's an unusual amount for this customer. And just to make sure that, uh, our AI flags it as something that needs an investigation, we're gonna go ahead and set the customer record to say that there's an unresolved account alert. I'm gonna go ahead and start this job, which is gonna trigger this new workflow, and you can see down here in the timeline, all of those nine steps that we talked about are happening. Um, so we're gonna reserve our funds, validate, uh, the AI is doing an agentic

[15:35](https://www.youtube.com/watch?v=mJ7q8CuVmbw&t=935s) loop and deciding, okay, we're gonna look at the customer profile, we're gonna look at the payouts, and then we're gonna draft a breach-- uh, a brief for a human being to review. So I've got an Approve or Reject button that I can click right here, but before I click them, uh, like I said, I am going to intentionally kill my worker. So I'm just gonna hit Stop, and you can see that there are no workers running at this point. So I'm gonna go ahead and approve my transaction, and we can see the approval gets logged as a signal. Now, traditionally, that would be something that just gets lost. Um,

[16:05](https://www.youtube.com/watch?v=mJ7q8CuVmbw&t=965s) we'll get a timeout in a second, but that's not really a big deal, uh, 'cause we're gonna have a retry policy that'll handle this automatically. So as soon as I turn on my worker, it's just gonna pick up that signal and resume the behavior and continue on from that last successful event. So we'll start a worker. This will take a second.

[16:24](https://www.youtube.com/watch?v=mJ7q8CuVmbw&t=984s) Of course it takes a little longer  when I'm demoing it live. Usually, it's instantaneous. Um, let me just go ahead and give it a refresh.

[16:34](https://www.youtube.com/watch?v=mJ7q8CuVmbw&t=994s) You know what? I have a video for just this scenario, so we can go ahead and just skip straight forward in the video and see what it's like, uh, if the video plays.

[16:48](https://www.youtube.com/watch?v=mJ7q8CuVmbw&t=1008s) Yeah. Let me see if I can skip to the middle of this.

[16:53](https://www.youtube.com/watch?v=mJ7q8CuVmbw&t=1013s) Oh. Oh, it's in the other tab. What am I doing?

[17:02](https://www.youtube.com/watch?v=mJ7q8CuVmbw&t=1022s) So I'm gonna approve while the worker is dead, uh, and then I'm gonna go ahead and resume the worker right below. Uh, and it should go ahead and just, uh, pick things up and, like, resume, uh, resume. So the agent is gonna investigate, write the brief, uh, and then I give the approval, and then, you know, I can go ahead and crash the worker again while we're waiting for the bank to approve the transaction or mark that it was completed. Again, the signal logs, and then whenever I turn the worker back on, it's gonna pick it up and rerun again. So, so this is what happened according to the Event

[17:32](https://www.youtube.com/watch?v=mJ7q8CuVmbw&t=1052s) History, and let me full screen that again for y'all. Oh. There we go. Um, so reading from top to bottom, each of our AI activities ran once, they attempted once, they worked once, and they didn't run again. We didn't rerun that workflow. We didn't respend those tokens. We just ran it once, and so we got that one-time cost. Uh, our approval signal came through while there were no sig- while there were no Workers running. Uh, and then our bank submission also, you know, came through while no Workers were running, and both of those things happened on

[18:02](https://www.youtube.com/watch?v=mJ7q8CuVmbw&t=1082s) new Workers. We stopped the Workers each time, uh, and then we got new Workers that just picked things up. So zero Workers, uh, the payout waited, uh, the approval landed, and we didn't have to rerun anything from the beginning. So what are the full list, uh, of what  those engineering leaders are saying, uh, slow them down? Um, our demo just handled seven of the top, uh, use cases right there that they gave us for problems. Um, so state, debugging, cost, retries, approval, long-running work, and crash recovery. Um, and we can take a look at the code for each

[18:32](https://www.youtube.com/watch?v=mJ7q8CuVmbw&t=1112s) of them. So state management in Temporal is the number one blocker... Or sorry, was the number one blocker in that survey. In a workflow, state is just fields. It's plain variables. There's no database that I built. There's no cache. After a crash, the replay just rebuilt them exactly. Uh, our LLM calls, uh, would cost us normally for multiple runs. In this run, the agent, we maintained the state, so we only ran it once. The tool run was held, was sustained, so we didn't need to spend any new tokens. Retries

[19:01](https://www.youtube.com/watch?v=mJ7q8CuVmbw&t=1141s) themselves were a policy, not a loop. We declared the retry policy, uh, in our, uh, configuration once, and then it applied to every single call in that transaction. There's no loop anywhere in that code. Um, the, uh... This is probably the most important slide for the room. So the agent itself gave us advice. The workflow itself owned the money and the state, and the AI had a really bounded, uh, attempt to, to jump in, and if the model had failed, the payout would have continued

[19:31](https://www.youtube.com/watch?v=mJ7q8CuVmbw&t=1171s) without the brief. A model outage wouldn't have degraded anything. Um, we had our human approval, uh, and we could have set that timer for as long as we wanted, uh, and it would have been the same code. Um, we could have waited for a year and resumed that Worker, and it would have been fine. And then I didn't write any crash recovery code. That was all built into Temporal. Um, so debugging. Uh, in most stacks, observability and recovery are two separate recor- records that drift apart, and then a human brings them together. Here, we were able to have the history that we debugged in the

[20:01](https://www.youtube.com/watch?v=mJ7q8CuVmbw&t=1201s) UI be the same history that our Worker replays, which closed gap four. So we had seven blockers, and we solved them with one workflow. And this isn't just a demo, so there are people using this stuff in production for real right now. OpenAI's Codex runs on Temporal. Uh, the coding agent that you use a lot of the day is running on Temporal. Uh, Replit runs their Replit agents on Temporal. Their engineering team says it's a bad user experience when the agent gets super far into something and then hits a catastrophic error, and that's just what we ha- watched happen, or

[20:31](https://www.youtube.com/watch?v=mJ7q8CuVmbw&t=1231s) watched not happen. Um, and I'm gonna skip through this because I know I'm running out of time. Uh, so from the AI engineer's side, you write the prompt, you run it, and the out- if the output's bad, you throw it away, and then Temporal retries it for you. Um, bad model output is just an activity failure, so same similar retry policy, uh, as well. Um, so back to the four gaps. One record, uh, closed them all. Um, and just to skip through what those four gaps were, uh, again,  uh, and re-summarize that, uh, that yeah,

[21:01](https://www.youtube.com/watch?v=mJ7q8CuVmbw&t=1261s) reliability and observability, uh, were just one record. And so every number of this talk that I, I, um, mentioned is in that temporal.io report State of Development, and I'll leave that up screen if anyone wants to take a photo and come check it out. Uh, and there's a bunch of us that'll be at the Temporal booth running this demo live if anyone wants to come see it or say hi. So, um, yeah, that's, that's my talk. Let's hope.

[21:31](https://www.youtube.com/watch?v=mJ7q8CuVmbw&t=1291s) Yeah.
