# Tribal Dungeons of Global Shipping: AI Agents at Global Scale — Dmitry Buykin, Maersk

Dmitry Buykin, Maersk

AI Engineer World's Fair 2026

[Talk page](https://ai.engineer/talks/dQ-_i1tZiws-tribal-dungeons-global-shipping-ai-agents) · [Video](https://www.youtube.com/watch?v=dQ-_i1tZiws)

## Tribal Dungeons of Global Shipping: AI Agents at Global Scale

Dmitry Buykin explains why the difficult part of production agents is not the agent loop, but translating local operational knowledge into executable procedures, constraining consequential actions, and turning expert corrections into reusable software.

### The expensive long tail begins when systems disagree

[0:13](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=13s)

A shipment may look like one workflow on paper, but Buykin describes it as an orchestration of many parallel state machines. The happy path holds only while the participating systems remain coherent. When one step cannot complete, the job becomes exception work: an expert must coordinate across several incomplete systems to recover the process.

Many companies have already automated the easy majority. What remains is a long tail containing more exceptions than the existing systems were designed to handle, which makes that tail disproportionately expensive. This is the production reality behind Buykin’s contrast with tidy agent demonstrations: the agent is entering an environment where failures cross system boundaries and local exceptions dominate the remaining work.

Standard operating procedures, or SOPs, are supposed to preserve those pathways so an expert and a model can interpret the work consistently. Getting there requires more than natural-language instructions. Buykin names stable intent detection, safe tool calls, legacy-backend integration, and expert evaluation as parts of the gap between knowing a process and executing it safely.

He calls the inaccessible knowledge **“tribal dungeons”**: the organization knows how to handle the case, but has not represented that knowledge in a form an agent can execute. The operational consequence is blunt. If an organization cannot represent a process—including its exceptional paths—it cannot safely delegate that process to an agent.

### A screenshot sequence is not an agent procedure

[2:49](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=169s)

Buykin’s concrete example is a legacy SOP made from screenshots arranged in order. That is a useful record of what a person sees and clicks, but it leaves the execution rules implicit. A person may infer which record is being edited, whether the current state permits the action, how to recognize success, and what to do when the screen differs. An agent cannot safely rely on those unstated assumptions.

An agent-ready SOP must make the missing execution contract explicit:

- **Preconditions:** what must already be true before work starts.
- **Decisions:** which branch applies to the current case.
- **Identifiers:** which shipment, record, or other entity the action concerns.
- **Backend calls:** which operation performs the work.
- **Validation:** how the system checks the resulting state.
- **Recovery:** what happens when execution fails or returns an unexpected result.
- **Evidence:** what establishes that the operation actually succeeded.

Buykin summarizes the division of labor as experts owning the **what** and agents owning the **how**. The hard work lies in translating and negotiating between those sides until their assumed “common sense” becomes explicit. Once a recurring exception has been understood, it can become a guardrail for future executions rather than remaining knowledge available only through expert intervention.

The resulting architecture has three parts: an SOP corpus that stores process memory, an execution runtime, and subject-matter-expert feedback capture. Buykin’s main architectural judgment is that the agent loop is only one component. The more complex system is the refinement loop that surrounds it and continually updates what the runtime knows how to do.

Country variation makes the corpus much larger than a single generic workflow. The same operational step may mean something different—or be described differently—under different local conditions. Buykin gives the corpus-to-runtime proportion as **20 to 1**, but does not define whether that ratio measures code, storage, engineering effort, or another unit. The defensible point is the relative one: localized process memory outweighs the runtime that executes it.

The production system is larger than the agent loop
Execution depends on localized process memory and a feedback path that turns expert findings into future behavior.
- SOP corpus — Company process memory adapted to country-specific conditions.
- Execution runtime — Runs the represented procedure against operational systems.
- SME feedback capture — Collects expert findings needed to refine procedures and controls.
- SOP corpus → Execution runtime (supplies executable procedures)
- Execution runtime → SME feedback capture (produces cases and failures for review)
- SME feedback capture → SOP corpus (refines process memory)

### Production scale is limited by backends and expert attention

[4:42](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=282s)

Buykin reports more than **200 production instances**, with individual latencies ranging from a few minutes to as much as **10 minutes**. He attributes much of that delay to dependencies on legacy systems. Running many instances concurrently does not make each case fast: the agent still has to wait for the backends participating in the workflow.

Expert time becomes another bottleneck. The subject-matter-expert bench therefore triages and clusters failures before handing them back to engineering. Clustering turns a stream of individual incidents into groups that can be prioritized, while the underlying trace preserves the details needed to diagnose a particular case.

The trace is shared evidence: an expert and an engineer can inspect the same execution and agree on what happened. That shared view matters because a verbal correction is not yet a production improvement. Buykin counts a correction only when it becomes an executable change that alters system behavior.

The team verifies changes by replaying real examples with writes disabled, protecting production systems while checking whether behavior improves. This closes the loop between diagnosis and verification: inspect an observed case, implement a change, replay the case without production side effects, and compare the resulting behavior.

The refinement loop around the agent
A correction becomes valuable only after it changes executable behavior and survives replay against a real case.
- Production trace — Records what happened in a real execution.
- SME triage — Clusters failures into actionable groups.
- Executable change — Converts expert judgment into changed system behavior.
- Read-only replay — Replays real examples with writes disabled and checks improvement.
- Updated production behavior — Returns a verified change to the operating system.
- Production trace → SME triage (supplies evidence)
- SME triage → Executable change (identifies actionable correction)
- Executable change → Read-only replay (is tested safely)
- Read-only replay → Updated production behavior (verifies improved behavior)
- Updated production behavior → Production trace (generates new execution evidence)

### Accuracy accumulates through small corrections

[6:13](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=373s)

Buykin argues that both vibe coding and specification-driven development reach a limit in this system. A specification can define intended behavior, but it cannot anticipate every country-specific exception or interaction with incomplete legacy systems. Beyond that point, reliability comes from established engineering practice applied repeatedly at production scale.

Accuracy was not designed completely in an initial diagram. Buykin reports more than **100,000 corrections over nine months** of development, each addressing some observed gap. The figure shows the scale of refinement, but the talk does not define the counting unit or provide a corresponding accuracy rate. It should not be read as proof of a particular success percentage.

Heat maps turn thousands of traces into priorities. Each cell groups tracked scenarios so experts and engineers can focus on the same failure class and decide where work would be most beneficial. Buykin says the work represented by a single block can require roughly **one to two months** from the whole team, including engineers and AI agents. A compact red cell can therefore hide a substantial investigation and implementation effort.

This changes the meaning of “the agent failed.” It is not a sufficient diagnosis or the end of the process. It is the starting signal for an investigation that must connect the observed failure to a specific fix. Without that mapping, a failure dashboard records pain but does not improve the system.

### Production needs a cage, not a request to be careful

[8:22](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=502s)

Buykin draws a sharp distinction between discovery and production. Discovery benefits from freedom because the agent needs room to explore possible methods. Production needs a cage: a harness whose job is to make avoidable mistakes impossible, rather than merely giving the model more room to act.

“Please be careful” cannot enforce a safety property. Buykin maps different failure mechanisms to different controls:

- **Wrong workflow → classifier evaluation:** test and improve the routing mechanism that selected the procedure.
- **Wrong write → write gate:** block an unsafe mutation before it reaches the backend.
- **Wrong assumption → SME review:** send domain judgment back to a qualified expert.

The goal is preventive control: remove the unsafe path instead of hoping the model declines to take it.

Critical paths retain review and approval. Engineering then concentrates on safe handoffs and a trustworthy trail of actions and decisions. The agent may propose or prepare work, but consequential execution remains bounded by controls outside the model and, where necessary, by human approval.

Buykin presents the methodology—not the agent itself—as the project’s main outcome. Its five moves are to make work representable, make execution bounded, make every agent’s behavior observable, make correction cheap, and make improvement compound. These moves form a sequence: explicit procedures enable controlled execution; traces expose failures; cheaper corrections let the organization repeat the loop; repeated improvements can then accumulate.

### Successful sequences become composite tools

[9:41](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=581s)

Compounding improvement means folding successful behavior back into code. Buykin describes an AI-native operation as a system that learns what works and turns those lessons into new composite tools adapted to the surrounding applications and people. The lasting asset, in his framing, is the adaptive architecture rather than any rented model capability.

The team aggregates repeatable step sequences and successful scenarios into larger reusable tools that other agents can call. This reduces repeated improvisation: once a sequence has been observed, corrected, and proven useful, later agents can invoke the packaged behavior instead of reconstructing every step. Buykin says this makes rollout across hundreds of countries possible in one go, though the talk does not document a completed rollout to that many countries.

The closing integration decision is deliberately system-specific. Buykin says the team does not use **MCP** for this system because the connected systems produce responses the team considers too bloated. Instead, it distills those responses and tunes tools for agents through function calling, seeking tighter control over the information returned and the software’s task processing.

That is not a comparative benchmark or a general proof that MCP is unsuitable. It is an architectural tradeoff for this production environment: the team prefers tailored function-call interfaces when they offer greater control over tool responses and execution quality. The final point fits the rest of the talk. Production reliability comes from shaping the environment around the model, not from emotional attachment to a protocol or tool.

## Key takeaways

- A human-readable screenshot sequence is not an executable SOP. Agents need explicit preconditions, branches, identifiers, backend operations, validation, recovery, and evidence of success.
- The agent loop is only one component. The larger system combines localized process memory, bounded execution, expert feedback, traces, and replay-based verification.
- A correction becomes a production fix only when it changes executable behavior and is checked against real examples with writes disabled.
- Production safety requires mechanism-specific controls: evaluate bad routing, gate unsafe writes, and retain expert review for unsupported assumptions and critical paths.
- Improvement compounds when successful, corrected sequences become reusable composite tools instead of being rediscovered by every agent.

## Resources

- [Tribal Dungeons of Global Shipping — official AI Engineer talk page](https://ai.engineer/talks/dQ-_i1tZiws-youtube-0fc1d9ac5cf725071cd1) — The official recording page includes the video, timestamped transcript, chapter navigation, and a concise reading version of the presentation.
- [Dmitry Buykin — AIE World’s Fair speaker page](https://aiewf.app/speakers/dmitry-buykin) — Conference profile and session information for Dmitry Buykin, identified there as Applied AI Lead and Staff Software Engineer at Maersk.
- [Tribal Dungeons of Global Shipping — conference session page](https://aiewf.app/sessions/s0403?trail=W1siRG1pdHJ5IEJ1eWtpbiIsIi9zcGVha2Vycy9kbWl0cnktYnV5a2luIl1d) — The conference session description situates the talk within AI-native enterprise operations and summarizes its focus on SOPs-as-code, evaluation, replay, guardrails, and expert refinement.

## Related talks

- [12-Factor Agents: Patterns of reliable LLM applications](https://ai.engineer/talks/8kMaTybvDUw-12-factor-agents) — A complementary production-oriented account of keeping agent applications modular, controlling orchestration, and retaining humans in consequential workflows.
- [Agents Need Feature Flags](https://ai.engineer/talks/zU4EagB311U-agents-need-feature-flags) — Extends Buykin’s production-cage argument with segmented rollout, canaries, rollback, kill switches, and independent controls over agent behavior.
- [Continual Learning for AI Agents: From Failures to Durable Improvements - Soheil Feizi, RELAI](https://ai.engineer/talks/2IxD9OB3XuQ-continual-learning-ai-agents-from-failures-durable) — Develops the same failure-to-improvement loop through replayable environments, regression checks, and reviewable changes across model, harness, and memory layers.

## Slides

### Tribal Dungeons of Global Shipping

Dmitry Buykin introduces production work turning scattered shipping knowledge into executable SOPs for agents, with evaluations and experts in the loop.

[0:20](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=20s) [Title slide introducing executable SOPs, agents, evaluations, and experts for global shipping.](https://assets.ai.engineer/sites/aiecode2025/sha256/e23988f3aafbe19f547068319c10578c0a5b4eae487f2442be393db2de251137.webp)

PRACTITIONER REPORT · AI ENGINEER

Tribal Dungeons of Global Shipping
How I Learned to Stop Worrying and Love the SOPs

PDFs + spreadsheets + portals + legacy systems + emails + experts
↓
executable SOPs + agents + evals + experts

### Container ship at port

A container ship sits alongside port cranes and a tugboat as Buykin introduces his production work supporting global shipping operations.

[0:34](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=34s) [A container ship navigates a port alongside cranes and a tugboat.](https://assets.ai.engineer/sites/aiecode2025/sha256/60c443d71fd27a6b86c72d430f344e26040f02dad38877be9a16d6a48377ca44.webp)

ALL THE WAY
TO ZERO

### Global logistics is a coordination problem

One shipment coordinates parallel state machines for booking, transport, customs and billing. When one drifts, the mismatch creates exception work.

[0:46](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=46s) [Maersk scale figures sit beside a list of state machines involved in one shipment.](https://assets.ai.engineer/sites/aiecode2025/sha256/e1dbc66ef3ea6257dab8a437e2518c29250cb0b341462668c037f13230d4a068.webp)

Global logistics is a coordination problem

Maersk global scale:
700+ container vessels
60 terminals · 35 countries
130 countries covered
8.8m+ sqm warehousing · 500+ sites
flightpaths · inland · ocean · terminals · warehousing

One shipment
├─ Booking state machine
├─ Vessel / voyage state machine
├─ Equipment state machine
├─ Documentation state machine
├─ Customs state machine
├─ Trucks state machine
├─ Terminal state machine
├─ Harbor crane state machine
└─ Billing state machine

The hard part: keeping logistics state machines coherent when reality changes.

### The business reality: the long tail tax

An illustrative 80/20 split shows an automated majority and a costly tail of exceptions handled across incomplete systems. The split is a rule of thumb, not an audited metric.

[1:03](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=63s) [An illustrative 80/20 bar contrasts automated work with semi-automated exceptions across incomplete systems.](https://assets.ai.engineer/sites/aiecode2025/sha256/732b53925b69f138ed56604d1998ba5fbc282cce87e210c388d0c3faccdcb1a5.webp)

The business reality: the long tail tax

The clean 80% is already system-driven. The hard 20% is semi-automated.

80% automated
20% semi

portals + Excel + PDFs + emails + screenshots + local rules + specialist judgment

The edge cases are semi-automated across incomplete systems.

80/20 is a rule-of-thumb shape, not an audited metric.

### The good-luck automated process

A 70-year-old mobile harbor crane, in service 10 years ago, illustrates the process with worn cabin levers and a corroded switchboard.

[1:20](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=80s) [Photographs show an old harbor crane operator cabin and switch control board.](https://assets.ai.engineer/sites/aiecode2025/sha256/66845a14aba9e9e57a7d19638dea781c3700e90f441ceca127c820cf6cb73125.webp)

The good-luck automated process

OPERATORS CABIN

SWITCH CONTROL BOARD
HYD PUMP
OIL COOLER
HYD PUMP
2ND START
SERVO PUMP
1ST START
[illegible]
[illegible]

Disclaimer: This 70-year-old mobile harbor crane was in service 10 years ago, used as an illustration.

### The exception starts when state machines disagree

A release is blocked while shipment and terminal are ready and billing is correct; the document is not ready and customs is pending. Buykin says these exceptions require expert orchestration across systems.

[1:40](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=100s) [Shipment, document, terminal, customs, and billing states are shown with a blocked release.](https://assets.ai.engineer/sites/aiecode2025/sha256/67a21d07a7292f5d60f411fbe73e562efacb96961729424f31540d1b41da0544.webp)

The exception starts when state machines disagree

A logistics process may depend on:
shipment state
 document state
equipment state
customs state
terminal state
inland state
billing state
contract / service commitments

shipment  ✓ ready
document  X not ready
terminal  ✓ ready
customs   X pending
billing   ✓ correct
release   X blocked

The system must know which workflow is valid, which facts are verified, and when to stop.

### The new glue layer: executable SOPs

An executable SOP guides a request from intent and verified backend facts to a guarded action. A trace and expert evaluation feed corrections into the next SOP version.

[1:58](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=118s) [Workflow from request and intent through verified facts, guarded action, tracing and expert evaluation.](https://assets.ai.engineer/sites/aiecode2025/sha256/6ad90ccc8de52df5621f84768e0ad214d41456a2c8f588c7e97b898e56674663.webp)

The new glue layer: executable SOPs

request arrives
↓ intent + identifiers
↓ SOP-guided workflow
↓ backend facts (verified)
↓ guarded action
↓ trace
↓ SME evaluation → next SOP version

Identify intent · load the right SOP · verify facts · constrain action · trace · feed corrections back.

SOP = Standard Operating Procedure:
step-by-step instructions for consistent, compliant execution in regulated industries.

### Prototype thinking breaks in production

The prototype stops after a tool call. Production adds country, scenario and identifiers, followed by dynamic tools, guardrails, evidence and evaluation.

[2:14](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=134s) [A simple prototype flow is contrasted with a production flow that includes context, dynamic tools, guardrails, evidence and evaluation.](https://assets.ai.engineer/sites/aiecode2025/sha256/2b2be9837d5054e9b9b87210fe9fd2b31b57f9d287e1696e0c356c45bc431d0d.webp)

Prototype thinking breaks in production

Prototype
request → LLM → tool call → done

Production
request → country → scenario → identifiers →
→ dynamic tools → guardrails → evidence → eval

A prototype can hide the enterprise. Production cannot.

### What is a tribal dungeon?

A tribal dungeon scatters process knowledge across people, documents, systems and exceptions. An agent cannot safely execute a process the organization cannot represent.

[2:30](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=150s) [A tribal dungeon is process knowledge scattered across people, documents, systems and exceptions rather than represented as executable knowledge.](https://assets.ai.engineer/sites/aiecode2025/sha256/484f99dcc29e528b26e3a719e8d855073503178d09fe6d433121acb5018656fd.webp)

What is a tribal dungeon?

A process that exists, but not as executable knowledge.

It lives across:
people
documents
screenshots
legacy systems
local variants
exception rules

The agent cannot safely execute a process the organization cannot represent.

### Legacy SOP vs Agent SOP

Legacy SOPs document what people see and click. Agent SOPs add preconditions, decisions, backend calls, validation, recovery, and evidence of execution.

[2:46](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=166s) [Legacy screenshot-based instructions are compared with agent SOPs containing preconditions, decisions, validation, recovery and evidence.](https://assets.ai.engineer/sites/aiecode2025/sha256/45116e58603e49ca6a39cc9fea16d53eb8134a3d0fb6de9ca27e1a4d16c4ff75.webp)

Legacy SOP vs Agent SOP

Legacy SOP
Screenshots and comments:
• Open screen
• Click tab
• Copy value
• Ask colleague
• Send reply

Agent SOP
• Preconditions
• Decision tree
• Verified identifiers
• Distilled backend calls + validation
• Recovery path
• Evidence trail

A screenshot is not a process.
A process is decisions, actions, validation, recovery, evidence.

### SME WHAT vs Agent HOW

Experts define the business rules; agents turn them into execution mechanisms. For example, an exception becomes a guardrail. Buykin says much of the work lies in translating and negotiating between the two.

[3:20](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=200s) [Table maps expert-owned business truth, including decisions and approval boundaries, to agent execution mechanisms such as workflows and guardrails.](https://assets.ai.engineer/sites/aiecode2025/sha256/d70254ee84f4c00803d3651c06b936ef2d80926d92168a23da89ead2a46ec7b1.webp)

SME WHAT vs Agent HOW

SME owns WHAT (business truth) | Agent owns HOW (executable contract)
Lifecycle | Branching / Tool contract
Decision | Data mapping / Workflows
Exception | Guardrail
Template | Structured output
Approval boundary | Write gate / Dynamic tools

The hard work is balance between the two.

### The three-component architecture

The SOP corpus feeds the agent runtime, then the SME bench clusters traces, evaluates, corrects and replays. A refinement loop returns to the corpus.

[3:36](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=216s) [SOP corpus, agent runtime and SME bench form a flow with a refinement loop back to the corpus.](https://assets.ai.engineer/sites/aiecode2025/sha256/a86c9b20980a2243aa835f87b5ad2d78e8727c3de2706c2c26ffa1b20523e080.webp)

The three-component architecture

1 · SOP Corpus
executable process memory

2 · Agent Runtime
classifier + execution loop + traces + dynamic tools

3 · SME Bench
clustering traces + evaluation + correction + replay

SOP Corpus → Agent Runtime → SME Bench
↑                             │
└────── refinement loop ──────┘

The agent loop is not the system.
The refinement loop around the agent is the system.

### SOPs-as-code trumps prompt engineering

The slide contrasts one big instruction with versioned, country-aware SOPs that are replayed, reviewed and measured. It lists a 20:1 SOP corpus-to-runtime-code ratio, 110 countries, 25+ languages and ~10K process intents / scenarios.

[3:56](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=236s) [Compares a single prompt with managed SOPs-as-code and summarizes corpus scale and process memory.](https://assets.ai.engineer/sites/aiecode2025/sha256/153b6857c50b201c90e98fa25e9787e409c68aed078c58c7b2991d4018fb8632.webp)

SOPs-as-code trumps prompt engineering

Prompt engineering
One big instruction.

SOPs-as-code
versioned
country-aware
tool-scoped
linted
replayed
reviewed
deployed
measured

20 : 1
SOP corpus : runtime code

110
countries

25+
languages

~10K
process intents / scenarios

The SOPs are the asset: a living knowledge base, adapted to the company's processes, that becomes its process memory.

### SOPs-as-code beats prompt engineering

Country-aware SOPs capture process variations and are versioned, reviewed and deployed. The slide reports a 20:1 SOP corpus-to-runtime-code ratio and coverage of 110 countries.

[4:02](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=242s) [Compares prompt engineering with managed SOPs-as-code and summarizes corpus scale and process memory.](https://assets.ai.engineer/sites/aiecode2025/sha256/43b6ba6b40390188be611541be71af67efd3834a78e22ee685fbf97c0155f563.webp)

SOPs-as-code beats prompt engineering

Prompt engineering
One big instruction.

SOPs-as-code
versioned
country-aware
tool-scoped
linted
replayed
reviewed
deployed
measured

20 : 1
SOP corpus : runtime code

110
countries

25+
languages

~10K
process intents / scenarios

The SOPs are the asset: a living knowledge base, adapted to the company's processes, that becomes its process memory.

### Request latency and container instances

Production monitoring pairs latency percentiles with container counts; one selected point shows 225 active instances. Buykin says latency ranges from a few minutes to 10 minutes, citing legacy-system dependencies.

[4:40](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=280s) [Charts show request latency percentiles and container instance counts, including a selected point with 225 active instances.](https://assets.ai.engineer/sites/aiecode2025/sha256/9b836f1b1bbf39a060498851d1c0f4490870e3eed28f8b5e3012da5816da3a6e.webp)

End-to-end request latency
Y-axis ticks: 0, 5min, 10min
Timeline: UTC+2; 25 Jun 2026; 2 Jul 2026
Legend: 50%, 95%, 99%

Container instance count
Y-axis ticks: 0, 200, 400
Timeline: UTC+2; 25 Jun 2026; 2 Jul 2026
Legend: active, idle
Selected point: 25 Jun 2026, 12:00:00; active 225; idle 0

Interface text: Event markers 12, 2, 5, 8, 6, 4, 4

### Evaluation UX: from noise to patterns

Expert time is the bottleneck. The evaluation workflow clusters failed traces for expert correction, then turns the findings into SOP change requests; the table shows three suspected workflow gaps.

[4:54](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=294s) [Failed traces are clustered for expert correction and SOP changes, with a table of example workflow gaps.](https://assets.ai.engineer/sites/aiecode2025/sha256/395965d0dac9039c50c75f50c5a8a30c8e6f1048ef8681f364396a11db44f765.webp)

Evaluation UX: from noise to patterns

failed / rejected traces → clusters → SME correction → SOP change request

Scope | Workflow | Cases | Suspected gap
Region A | Docs | 38 | Missing gate
Region B | Release | 24 | Bad template
Region C | Invoice | 17 | Tool gap

Group similar failures. Make the correction actionable, not just visible.

### Evaluation UX: the trace is the evidence object

A trace gives an expert and an engineer shared evidence to review a case. The example records verified facts, a gated action and a result requiring approval.

[5:26](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=326s) [Lists trace fields and shows a case with verified facts, a gated action and an approval requirement.](https://assets.ai.engineer/sites/aiecode2025/sha256/6cc5bbf777b08399d4c6bf151ca4548042d6251431135c8013ba55a87779cf7a.webp)

Evaluation UX — the trace is the evidence object

A trace carries:
intent
routing
SOP branch
facts verified
tools called
decision
output
failure reason

Case trace
intent:      document update
classifier:  workflow A
SOP branch:  readiness check
facts:       verified
action:      gated
result:      needs approval

### Evaluation UX: correction to executable change

The correction replaces a generic response with conditions on state and country rules. Matching requests trigger a readiness check and specific response; other cases use a fallback template.

[5:37](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=337s) [A generic response rule is replaced by conditions, a readiness check and a fallback template.](https://assets.ai.engineer/sites/aiecode2025/sha256/9fe003ae440440aadfefcccc5f9befdf99cc660696c532d30b61bd35637b8d88.webp)

Evaluation UX: correction → executable change

Before
IF request mentions X
   → generic response

After
IF request mentions X
   AND state = Y
   AND country rule = Z
THEN run readiness check → specific response
ELSE use fallback template

### Replay result

Replay compares three changed outcomes against an unchanged control. Writes are disabled to protect production systems while behavior is checked.

[5:49](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=349s) [Four replay examples show three changed outcomes and one unchanged control.](https://assets.ai.engineer/sites/aiecode2025/sha256/4e17f38e1a75dbf0b9d28e5cb98544a4024a0c175a9bfd091237ec224dca8533.webp)

Replay result

Real examples · writes disabled · behaviour diffed

Example A  rejected → accepted
Example B  loop → stopped safely
Example C  wrong route → classifier NONE
Control example  unchanged

Not vibes. Not one model score. Replay real examples before every change.

### The cognitive effort ratio

The project’s effort ratio assigns 10 to AI coding, 20 to specs and SOP refinement, and 80 to evaluation, replay, testing and trace review. Buykin frames the last category as where the real work starts.

[6:10](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=370s) [Bars label AI coding as 10, specs and SOP refinement as 20, and evaluation, replay, testing and trace review as 80.](https://assets.ai.engineer/sites/aiecode2025/sha256/085eb00d2db0adb4010b827cc328d973236047bea9830bde02c52ab3742f9c27.webp)

The cognitive effort ratio

AI coding · 10
specs / SOP refinement · 20
evaluation / replay / testing / trace review · 80
TDD / BDD / Linters / Pairwise Testing / Data Analysis / Simulations / SME Feedback Loops

AI coding made it viable.
The work is everything around the code.

### Production quality came from iteration

The team made over 100,000 corrections in nine months, cycling through observation, diagnosis, changes, replay, deployment and measurement to improve coverage and accuracy.

[7:08](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=428s) [The slide cites over 100,000 incremental corrections and shows an observe, diagnose, change, replay, deploy and measure cycle.](https://assets.ai.engineer/sites/aiecode2025/sha256/b41fc64242afe6f1191dbf5490734c0fb28209ebb7d1d10deb6f1ced3b265225.webp)

Production quality came from iteration

100K+
incremental corrections

observe → diagnose → change → replay → deploy → measure

Coverage and accuracy are earned, not designed.

### Process-specific quality signals map to fixes

The team investigates agent failures by signal: routing misses point to the classifier, tool failures to contract mismatches, and generic-path failures to country exceptions.

[8:05](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=485s) [Eight examples connect failure signals, including routing misses and tool failures, to specific fixes.](https://assets.ai.engineer/sites/aiecode2025/sha256/c39afecf1da0e9591a08ec83e58317f577e29579fa4b77828271bcbcea87eb36.webp)

Process-specific quality signals map to fixes

routing misses → classifier
unsupported requests → SOP gap
failed tool → tool contract mismatch
loop breaker hits → mixed scenarios
generic path failed → country exception
template issues → template fix
write-gate triggered → unsupported path
expert rejections → approval boundary

"Agent failed" is where the investigation starts. Each root cause maps to an actionable fix.

### Make dumb mistakes structurally impossible

Each failure maps to a preventive measure: verify shipment identifiers, gate writes, break loops, and replay SOP edits before deployment. The goal is to eliminate unsafe paths.

[8:34](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=514s) [Table pairs operational failures with structural prevention measures.](https://assets.ai.engineer/sites/aiecode2025/sha256/3973eefe3c8dade31111df166f43ca0f3f873c0d1fb6bfdb1ff4ce613ae9605e.webp)

Make dumb mistakes structurally impossible

Failure | Structural prevention
Wrong shipment | verify identifiers
Wrong workflow | classifier eval
Wrong tool | tool allowlist
Wrong write | write gate
Wrong loop | loop breaker
Wrong SOP edit | replay before deploy
Wrong assumption | SME review

A preventive measure eliminates the unsafe path.

### Critical paths stay supervised

Agent discovery passes through runtime gates and specialist review before approved action executes. A stored trace feeds SOP improvement.

[8:53](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=533s) [Flow from agent discovery through gates, specialist review, approved execution, trace storage and SOP improvement.](https://assets.ai.engineer/sites/aiecode2025/sha256/e395679207ce1fa927b81e5bc0ade4f7639329393363bb7fd1aa892a41c14d1c.webp)

Critical paths stay supervised

agent discovers → runtime gates → specialist reviews
→ approved action executes → trace stored → SOP improves

The engineering question: safe handoffs and evidence trails.

### Outcome: a methodology, not a bot

After 100K iterations, the methodology turns tribal knowledge into executable SOPs, bounds and traces execution, and uses feedback and replay to improve workflows.

[9:05](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=545s) [Five principles cover representable work, bounded execution, observable behavior, cheap correction and compounding improvement.](https://assets.ai.engineer/sites/aiecode2025/sha256/9cdb0da74db662bad86fa21884ebdca4bb87204c358b9bf27d27f12911086544.webp)

Outcome: a methodology, not a bot

After 100K iterations, the pattern became clear:

1 · Make work representable: collect data & tribal knowledge → executable SOPs
2 · Make execution bounded: tools → scopes → gates → approvals
3 · Make behavior observable: every run produces a trace
4 · Make correction cheap: SME feedback → SOP diff → replay
5 · Make improvement compound: daily refinements → reusable workflows as tools

It applies anywhere an enterprise has long-tail exceptions across systems, documents, and local rules.

## Full transcript

[0:01](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=1s) [music]

[0:13](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=13s) Hello everyone. Um, this is a

[0:15](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=15s) practitioner report uh from real

[0:17](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=17s) production work. So, let's get into it.

[0:21](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=21s) Um, I'll skip the generic uh yet another

[0:24](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=24s) loop agent intro. This is about the hard

[0:26](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=26s) part most agent demos skip.

[0:30](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=30s) and about turning messy operational

[0:32](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=32s) knowledge into something an agent can

[0:35](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=35s) execute safely.

[0:37](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=37s) This comes from real work uh in my

[0:39](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=39s) company I'm working for supporting

[0:42](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=42s) global shipping operations and grounded

[0:44](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=44s) in production.

[0:48](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=48s) On paper it's uh one workflow usually

[0:50](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=50s) but uh in reality every shipment is an

[0:53](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=53s) orchestration of many parallel state

[0:55](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=55s) machines. While they agree the happy

[0:59](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=59s) paths work the moment one drifts you get

[1:02](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=62s) exception work.

[1:06](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=66s) The easy majority is already automated

[1:09](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=69s) in many companies. What's left is the

[1:12](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=72s) long tail and more exceptions than

[1:15](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=75s) system built uh to handle them. That

[1:18](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=78s) tail is uh the expensive part.

[1:23](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=83s) And then there's my favorite category.

[1:25](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=85s) And it comes with a special uh plate

[1:29](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=89s) here. See for EI builder dreams and

[1:33](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=93s) their laptops. This what you can find

[1:36](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=96s) outside of AI bubble in San Francisco.

[1:42](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=102s) The signal process uh depends on many

[1:44](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=104s) systems being coherent at once. If any

[1:47](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=107s) step uh can't complete the happy path

[1:50](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=110s) breaks and then it takes expert uh

[1:53](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=113s) archist expert orchestration across uh

[1:56](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=116s) multiple incomplete systems.

[2:00](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=120s) All these uh variations um path pathways

[2:04](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=124s) should be captured in SOPs. SOPs is a

[2:07](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=127s) standard operating procedure common and

[2:09](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=129s) regulated industries. So an expert and

[2:11](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=131s) the model read them uh the same way.

[2:16](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=136s) That gap is the hard part. Stable intent

[2:20](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=140s) detection tool calls you can guarantee

[2:23](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=143s) are safe integrating with legacy back

[2:26](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=146s) ends and results evaluated with experts.

[2:32](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=152s) Uh I call this uh tribal dungeons. Uh

[2:35](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=155s) the knowledge exists but not in a form

[2:38](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=158s) uh agent can execute and you can safely

[2:41](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=161s) run a process. You can't safely run a

[2:44](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=164s) process. The organization cannot

[2:45](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=165s) represent

[2:49](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=169s) standard legacy SOPs [clears throat]

[2:51](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=171s) bunch of bunch of screenshots organized

[2:55](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=175s) in sequence and but screenshots not uh a

[2:58](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=178s) process. A legacy SOPs explain what a

[3:01](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=181s) person sees and clicks. And an agent SOP

[3:05](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=185s) needs a more complex uh setup,

[3:08](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=188s) preconditions, uh decisions,

[3:10](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=190s) identifiers, back end calls, validation,

[3:13](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=193s) recovery, and evidence of uh successful

[3:16](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=196s) execution.

[3:21](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=201s) Experts own the what, agents own the

[3:24](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=204s) how. And exception becomes a guardrail.

[3:28](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=208s) Most of the effort is the translation

[3:30](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=210s) and negotiation between them to align on

[3:34](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=214s) common sense.

[3:38](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=218s) Three parts here um in this architecture

[3:41](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=221s) it's SOP memory uh organized as SOP

[3:44](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=224s) corpus

[3:46](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=226s) execution runtime and theme feedback

[3:48](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=228s) capture. The agent loop is not the

[3:51](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=231s) system. The refining loop around the

[3:53](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=233s) agent is the system

[3:56](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=236s) and it's the most complex part. Oh,

[3:58](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=238s) sorry SAP is okay. It's this slide for

[4:02](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=242s) UK. This is correct one. So and it's

[4:06](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=246s) good illustration why the the same thing

[4:09](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=249s) is means different and uh describing

[4:12](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=252s) differently in different countries and

[4:15](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=255s) it's creating a lot of variations

[4:16](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=256s) between each country

[4:19](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=259s) and that corpus is a asset the company

[4:22](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=262s) company's process memory uh modified and

[4:25](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=265s) aligned with every country um conditions

[4:30](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=270s) and far bigger than than than runtime

[4:32](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=272s) you could see the proportion 20 to1

[4:36](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=276s) So and this is concurrently operating

[4:38](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=278s) system

[4:41](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=281s) and this is the scale we run in

[4:43](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=283s) production today

[4:46](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=286s) over 200 instances and spikes and

[4:50](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=290s) latencies deviates from few minutes to

[4:53](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=293s) up to 10 minutes.

[4:56](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=296s) Um and mainly yeah the mainly main

[5:00](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=300s) reason for it that u we depending on

[5:02](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=302s) many legacy system which is uh so cannot

[5:05](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=305s) be faster than agent loop itself.

[5:11](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=311s) Expert time is the bottleneck. So the

[5:15](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=315s) theme bench uh does the triage for us.

[5:19](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=319s) It clusters the failures and hands back

[5:22](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=322s) something you can act on. Not just look

[5:24](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=324s) at look at it.

[5:28](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=328s) The trace is the shared evidence that

[5:31](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=331s) lets an expert and an engineer review

[5:34](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=334s) the same case and agree on what

[5:36](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=336s) happened.

[5:39](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=339s) A correction only counts when it becomes

[5:41](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=341s) an executable change. And that's the

[5:45](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=345s) line between an opinion and a production

[5:47](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=347s) fix.

[5:52](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=352s) And and this is where quality comes

[5:55](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=355s) from. not from vibes uh not from a

[5:59](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=359s) bigger model from replaying real

[6:01](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=361s) examples with u disabled rights to uh

[6:06](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=366s) protect the production systems and

[6:08](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=368s) checking whether behavior improved.

[6:13](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=373s) You can see here on the

[6:16](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=376s) uh cognitive proportion u or this effort

[6:19](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=379s) ratio uh between each activity in our

[6:21](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=381s) project. So usually uh pipe coding ends

[6:25](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=385s) here.

[6:27](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=387s) Here there ends um

[6:30](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=390s) specdriven development because it cannot

[6:33](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=393s) uh grow improve accuracy more than this

[6:37](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=397s) stage on this scale. And this is uh

[6:40](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=400s) where the real work starts. Nothing

[6:44](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=404s) exotic. It's engineering common

[6:46](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=406s) engineering sense applied at scale.

[6:50](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=410s) So if uh you don't know all this uh

[6:53](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=413s) terminology which developed over lastuh

[6:57](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=417s) 30 years in software development

[6:58](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=418s) argument to check because this is what

[7:01](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=421s) every AI agent uh AI coding agent should

[7:03](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=423s) know uh to help you develop reliable

[7:07](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=427s) production systems

[7:12](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=432s) and accuracy it's uh wasn't designed uh

[7:15](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=435s) in one diagram up front it was earned

[7:19](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=439s) one small correction at the time at the

[7:21](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=441s) scale you see here. So we have over

[7:24](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=444s) 100,000

[7:26](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=446s) corrections over last 9 months in the

[7:29](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=449s) system when we developing it

[7:33](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=453s) [clears throat]

[7:35](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=455s) and this um heat maps uh turned

[7:39](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=459s) thousands of traces into priorities. is

[7:41](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=461s) how we keep experts and engineers uh

[7:44](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=464s) looking at the same problems and

[7:46](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=466s) prioritize where the the most beneficial

[7:49](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=469s) work for them. Every cell is a group of

[7:53](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=473s) tracked scenarios we have and uh usually

[7:57](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=477s) to turn one block in red it's around one

[8:02](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=482s) two months of force for the whole team

[8:06](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=486s) whole team of engineers and also AI

[8:09](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=489s) agents

[8:11](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=491s) um the agent failed is uh where the

[8:13](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=493s) investigation starts not where it ends

[8:16](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=496s) each failure maps to a specific uh fix

[8:22](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=502s) discovery needs agent freedom and

[8:25](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=505s) production needs a cage. Uh the harness

[8:28](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=508s) isn't there to give the agent more room.

[8:30](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=510s) It's there to make the dumb mistakes

[8:32](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=512s) impossible.

[8:36](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=516s) So on this scale please be careful is

[8:38](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=518s) not a guard guard. Uh if we have wrong

[8:41](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=521s) workflow then classifier eval. If it's

[8:44](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=524s) wrong right then right gate. If it's

[8:47](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=527s) wrong assumption then it's a mere view.

[8:49](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=529s) A preventive measure eliminates the

[8:51](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=531s) unsafe path

[8:55](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=535s) on critical paths. U review and approval

[8:58](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=538s) stay in the loop. The engine engineering

[9:00](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=540s) focus is uh to build safe hands offs and

[9:03](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=543s) a trail you can trust.

[9:08](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=548s) The real outcome

[9:11](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=551s) uh wasn't the agent in the system. It

[9:13](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=553s) was the [clears throat] methodology we

[9:15](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=555s) built around it. If you want the

[9:18](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=558s) blueprint, then it's uh these five

[9:20](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=560s) moves. Make work representable. Make exe

[9:24](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=564s) execution bounded. Make behavior

[9:26](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=566s) observable for every agent and make

[9:29](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=569s) correction cheap. And last thing is make

[9:32](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=572s) improvement compound. So gradually

[9:35](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=575s) systematically improve the quality of

[9:37](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=577s) the system.

[9:41](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=581s) AI native um operation is more than

[9:45](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=585s) agents in workflow. It's a system that

[9:48](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=588s) learns from what works and fold folds it

[9:51](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=591s) back into code as new composite tools

[9:54](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=594s) adapting to the applications and the

[9:56](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=596s) people around it. The best AI models um

[10:00](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=600s) oriented intelligence for us. The

[10:03](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=603s) adaptive architecture we built is the

[10:05](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=605s) asset, the final asset

[10:10](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=610s) and

[10:12](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=612s) we aggregating all um repeatable

[10:16](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=616s) sequences of steps successful scenarios

[10:18](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=618s) and uh merging them into bigger tools

[10:22](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=622s) which uh combine the disproven scenarios

[10:25](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=625s) into the reusable snippets by other

[10:27](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=627s) agents. So and then um it's possible to

[10:31](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=631s) roll out them not only for one country

[10:33](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=633s) but for hundreds country in one go.

[10:38](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=638s) So this is um um all for the talk and

[10:43](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=643s) little time for questions and I'll be

[10:44](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=644s) around afterwards. And the final

[10:47](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=647s) reminder you know if you you know if you

[10:50](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=650s) are AI builder if you emotionally

[10:54](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=654s) attached to tools not MCPS

[10:57](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=657s) we're not using MCPS because uh for us

[10:59](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=659s) it's uh always not the best choice. So

[11:02](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=662s) because all all systems usually really

[11:05](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=665s) bloated and we have to distill responses

[11:08](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=668s) and uh tune the tools through function

[11:12](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=672s) calling uh to our agents then we can

[11:15](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=675s) control quality of um our software and

[11:19](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=679s) ensure that uh it's correctly

[11:22](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=682s) processing assigned tasks.

[11:27](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=687s) Thank you. Any questions?

[11:33](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=693s) Okay, then um thanks for your attent u

[11:37](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=697s) attention. Then I will be around so you

[11:39](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=699s) can ask me questions if you want.

[11:44](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=704s) [applause]

[12:00](https://www.youtube.com/watch?v=dQ-_i1tZiws&t=720s) >> [music]
