AI Engineer World's Fair 2026
Tribal Dungeons of Global Shipping: AI Agents at Global Scale — Dmitry Buykin, Maersk
Loading slides…
Read the talk
Tribal Dungeons of Global Shipping: AI Agents at Global Scale
Dmitry Buykin explains why the difficult part of production agents is not the agent loop, but translating local operational knowledge into executable procedures, constraining consequential actions, and turning expert corrections into reusable software.
From a talk by Dmitry Buykin
At a glance
Ideas worth remembering
A human-readable screenshot sequence is not an executable SOP. Agents need explicit preconditions, branches, identifiers, backend operations, validation, recovery, and evidence of success.
The agent loop is only one component. The larger system combines localized process memory, bounded execution, expert feedback, traces, and replay-based verification.
A correction becomes a production fix only when it changes executable behavior and is checked against real examples with writes disabled.
Production safety requires mechanism-specific controls: evaluate bad routing, gate unsafe writes, and retain expert review for unsupported assumptions and critical paths.
Improvement compounds when successful, corrected sequences become reusable composite tools instead of being rediscovered by every agent.
The expensive long tail begins when systems disagree
A shipment may look like one workflow on paper, but Buykin describes it as an orchestration of many parallel state machines. The happy path holds only while the participating systems remain coherent. When one step cannot complete, the job becomes exception work: an expert must coordinate across several incomplete systems to recover the process.
Many companies have already automated the easy majority. What remains is a long tail containing more exceptions than the existing systems were designed to handle, which makes that tail disproportionately expensive. This is the production reality behind Buykin’s contrast with tidy agent demonstrations: the agent is entering an environment where failures cross system boundaries and local exceptions dominate the remaining work.
Standard operating procedures, or SOPs, are supposed to preserve those pathways so an expert and a model can interpret the work consistently. Getting there requires more than natural-language instructions. Buykin names stable intent detection, safe tool calls, legacy-backend integration, and expert evaluation as parts of the gap between knowing a process and executing it safely.
He calls the inaccessible knowledge “tribal dungeons”: the organization knows how to handle the case, but has not represented that knowledge in a form an agent can execute. The operational consequence is blunt. If an organization cannot represent a process—including its exceptional paths—it cannot safely delegate that process to an agent.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A screenshot sequence is not an agent procedure
Buykin’s concrete example is a legacy SOP made from screenshots arranged in order. That is a useful record of what a person sees and clicks, but it leaves the execution rules implicit. A person may infer which record is being edited, whether the current state permits the action, how to recognize success, and what to do when the screen differs. An agent cannot safely rely on those unstated assumptions.
An agent-ready SOP must make the missing execution contract explicit:
- Preconditions: what must already be true before work starts.
- Decisions: which branch applies to the current case.
- Identifiers: which shipment, record, or other entity the action concerns.
- Backend calls: which operation performs the work.
- Validation: how the system checks the resulting state.
- Recovery: what happens when execution fails or returns an unexpected result.
- Evidence: what establishes that the operation actually succeeded.
Buykin summarizes the division of labor as experts owning the what and agents owning the how. The hard work lies in translating and negotiating between those sides until their assumed “common sense” becomes explicit. Once a recurring exception has been understood, it can become a guardrail for future executions rather than remaining knowledge available only through expert intervention.
The resulting architecture has three parts: an SOP corpus that stores process memory, an execution runtime, and subject-matter-expert feedback capture. Buykin’s main architectural judgment is that the agent loop is only one component. The more complex system is the refinement loop that surrounds it and continually updates what the runtime knows how to do.
Country variation makes the corpus much larger than a single generic workflow. The same operational step may mean something different—or be described differently—under different local conditions. Buykin gives the corpus-to-runtime proportion as 20 to 1, but does not define whether that ratio measures code, storage, engineering effort, or another unit. The defensible point is the relative one: localized process memory outweighs the runtime that executes it.
The production system is larger than the agent loop
Read the diagram as text
- SOP corpus. knowledge — Company process memory adapted to country-specific conditions.
- Execution runtime. execution — Runs the represented procedure against operational systems.
- SME feedback capture. refinement — Collects expert findings needed to refine procedures and controls.
- SOP corpus → Execution runtime: supplies executable procedures.
- Execution runtime → SME feedback capture: produces cases and failures for review.
- SME feedback capture → SOP corpus: refines process memory.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Production scale is limited by backends and expert attention
Buykin reports more than 200 production instances, with individual latencies ranging from a few minutes to as much as 10 minutes. He attributes much of that delay to dependencies on legacy systems. Running many instances concurrently does not make each case fast: the agent still has to wait for the backends participating in the workflow.
Expert time becomes another bottleneck. The subject-matter-expert bench therefore triages and clusters failures before handing them back to engineering. Clustering turns a stream of individual incidents into groups that can be prioritized, while the underlying trace preserves the details needed to diagnose a particular case.
The trace is shared evidence: an expert and an engineer can inspect the same execution and agree on what happened. That shared view matters because a verbal correction is not yet a production improvement. Buykin counts a correction only when it becomes an executable change that alters system behavior.
The team verifies changes by replaying real examples with writes disabled, protecting production systems while checking whether behavior improves. This closes the loop between diagnosis and verification: inspect an observed case, implement a change, replay the case without production side effects, and compare the resulting behavior.
The refinement loop around the agent
Read the diagram as text
- Production trace. Records what happened in a real execution.
- SME triage. Clusters failures into actionable groups.
- Executable change. Converts expert judgment into changed system behavior.
- Read-only replay. Replays real examples with writes disabled and checks improvement.
- Updated production behavior. Returns a verified change to the operating system.
- Production trace → SME triage: supplies evidence.
- SME triage → Executable change: identifies actionable correction.
- Executable change → Read-only replay: is tested safely.
- Read-only replay → Updated production behavior: verifies improved behavior.
- Updated production behavior → Production trace: generates new execution evidence.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Accuracy accumulates through small corrections
Buykin argues that both vibe coding and specification-driven development reach a limit in this system. A specification can define intended behavior, but it cannot anticipate every country-specific exception or interaction with incomplete legacy systems. Beyond that point, reliability comes from established engineering practice applied repeatedly at production scale.
Accuracy was not designed completely in an initial diagram. Buykin reports more than 100,000 corrections over nine months of development, each addressing some observed gap. The figure shows the scale of refinement, but the talk does not define the counting unit or provide a corresponding accuracy rate. It should not be read as proof of a particular success percentage.
Heat maps turn thousands of traces into priorities. Each cell groups tracked scenarios so experts and engineers can focus on the same failure class and decide where work would be most beneficial. Buykin says the work represented by a single block can require roughly one to two months from the whole team, including engineers and AI agents. A compact red cell can therefore hide a substantial investigation and implementation effort.
This changes the meaning of “the agent failed.” It is not a sufficient diagnosis or the end of the process. It is the starting signal for an investigation that must connect the observed failure to a specific fix. Without that mapping, a failure dashboard records pain but does not improve the system.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Production needs a cage, not a request to be careful
Buykin draws a sharp distinction between discovery and production. Discovery benefits from freedom because the agent needs room to explore possible methods. Production needs a cage: a harness whose job is to make avoidable mistakes impossible, rather than merely giving the model more room to act.
“Please be careful” cannot enforce a safety property. Buykin maps different failure mechanisms to different controls:
- Wrong workflow → classifier evaluation: test and improve the routing mechanism that selected the procedure.
- Wrong write → write gate: block an unsafe mutation before it reaches the backend.
- Wrong assumption → SME review: send domain judgment back to a qualified expert.
The goal is preventive control: remove the unsafe path instead of hoping the model declines to take it.
Critical paths retain review and approval. Engineering then concentrates on safe handoffs and a trustworthy trail of actions and decisions. The agent may propose or prepare work, but consequential execution remains bounded by controls outside the model and, where necessary, by human approval.
Buykin presents the methodology—not the agent itself—as the project’s main outcome. Its five moves are to make work representable, make execution bounded, make every agent’s behavior observable, make correction cheap, and make improvement compound. These moves form a sequence: explicit procedures enable controlled execution; traces expose failures; cheaper corrections let the organization repeat the loop; repeated improvements can then accumulate.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Successful sequences become composite tools
Compounding improvement means folding successful behavior back into code. Buykin describes an AI-native operation as a system that learns what works and turns those lessons into new composite tools adapted to the surrounding applications and people. The lasting asset, in his framing, is the adaptive architecture rather than any rented model capability.
The team aggregates repeatable step sequences and successful scenarios into larger reusable tools that other agents can call. This reduces repeated improvisation: once a sequence has been observed, corrected, and proven useful, later agents can invoke the packaged behavior instead of reconstructing every step. Buykin says this makes rollout across hundreds of countries possible in one go, though the talk does not document a completed rollout to that many countries.
The closing integration decision is deliberately system-specific. Buykin says the team does not use MCP for this system because the connected systems produce responses the team considers too bloated. Instead, it distills those responses and tunes tools for agents through function calling, seeking tighter control over the information returned and the software’s task processing.
That is not a comparative benchmark or a general proof that MCP is unsuitable. It is an architectural tradeoff for this production environment: the team prefers tailored function-call interfaces when they offer greater control over tool responses and execution quality. The final point fits the rest of the talk. Production reliability comes from shaping the environment around the model, not from emotional attachment to a protocol or tool.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
The official recording page includes the video, timestamped transcript, chapter navigation, and a concise reading version of the presentation.
Further reading
Conference profile and session information for Dmitry Buykin, identified there as Applied AI Lead and Staff Software Engineer at Maersk.
The conference session description situates the talk within AI-native enterprise operations and summarizes its focus on SOPs-as-code, evaluation, replay, guardrails, and expert refinement.
Related talks
- 12-Factor Agents: Patterns of reliable LLM applications
A complementary production-oriented account of keeping agent applications modular, controlling orchestration, and retaining humans in consequential workflows.
- Agents Need Feature Flags
Extends Buykin’s production-cage argument with segmented rollout, canaries, rollback, kill switches, and independent controls over agent behavior.
- Continual Learning for AI Agents: From Failures to Durable Improvements - Soheil Feizi, RELAI
Develops the same failure-to-improvement loop through replayable environments, regression checks, and reviewable changes across model, harness, and memory layers.
Full transcript
Loading transcript…