AI Engineer World's Fair 2026

The Missing Layer After Launch

Raphael Kalandadze· Co-Founder & CTO, Wandero AI19:33

Loading slides…

Read the talk

The Missing Layer After Launch

A working demo does not tell you whether an agent helps real users. Production agents need a loop that turns traces, session analysis, and browser checks into reviewed improvements.

From a talk by Raphael Kalandadze

Before you start: Familiarity with agent tool calls, execution traces, and pull-request review will help you follow the operational workflows.

The demo works. What happens in production?

You built an agent, launched it, and watched it work well in a demo. Now hundreds or thousands of real conversations arrive every day. How do you know whether it is working? How do you understand the system’s health, improve it, and find problems you do not yet know to look for? The demo establishes that the agent can succeed. Operating the product requires understanding what happens across actual use.

Raphael Kalandadze calls this the missing layer after launch. Models make it possible to build products quickly, sometimes in days or weeks, but shipping starts the work of maintaining control over their behavior. The feedback loop is part of the product: it supplies the evidence needed to improve the system every day. Monitoring, logs, diagnosis, and fixes are familiar software practices; agents make each harder because their behavior is much less constrained.

Slide headed “After you launch” lists “Monitor it?”, “Understand it?”, “Improve it?”, and “Find the holes?”Open full source frame
After launch: monitor, understand, improve, and find the holes.
0:000:09
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:00 · section reference included

You cannot enumerate the conversations

An agent does not expose only a few buttons and predefined flows. Coding agents such as Claude Code and Codex illustrate the breadth of the task surface: users bring instructions, and the system attempts to handle them. You cannot write every possible conversation in advance. After launch, that breadth can make you lose the feel for your own system—whether it is getting better, getting worse, or behaving differently from what you intended.

Unit tests, regular expressions, rule-based checks, and scripted customer conversations still help. Kalandadze’s team tried all of them. They cover a slice of the problem, while customers continually introduce conversations outside that slice. Model-driven execution also varies: the same input can take a different path, and a small input change can redirect an entire trajectory. Pre-launch testing therefore cannot exhaust the behavior that production will reveal.

1:501:59
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:50 · section reference included

Recovery can hide the defect

Consider a long-running task that encounters a problem halfway through. The agent struggles, tries another tool call, finds a workaround, and eventually finishes. No red alert appears. The dashboard looks healthy, yet the trajectory contains an early warning: a defect remains, and this execution happened to recover from it. Reliability requires inspecting those recoveries, not merely accepting the final status.

A related problem is premature completion. Kalandadze points to Anthropic’s observation that agents sometimes mark features complete without adequately checking whether they work. The testing discussion in Effective harnesses for long-running agents describes this failure in long-running web development. Reported completion needs evidence from the resulting system, rather than confidence in the agent’s own declaration.

Slide titled “The failure hides itself” gives examples of Claude marking features complete without checking and agents reporting success despite contrary system state.Open full source frame
The failure hides itself: reported completion can conflict with actual results.
3:443:55
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:44 · section reference included

A completed itinerary can still be wrong

Long tasks widen the investigative surface further. They can involve extensive tool use, intermediate summarizations, sub-agents, generated code, terminal commands, external services, and third-party libraries. Each introduces behavior that the final response alone cannot explain. Kalandadze describes a scale of hundreds or thousands of tools, without distinguishing separate tools from repeated invocations. The operational point is the size and variety of the execution path.

In the team’s travel use case, a user asks for an itinerary. The agent runs the flow and builds a trip, but books a different service and makes mistakes calculating the price. Technically, the workflow completes; the user’s task fails. That distinction gives production analysis a concrete target: compare what the user requested with what the system actually produced.

For example, the service and price checks can become small regression tests once the failure is understood. This TypeScript example uses an illustrative itinerary with prices represented in cents:

typescript

type Itinerary = {
  serviceId: string;
  lineItemsCents: number[];
  totalCents: number;
};

function checkItinerary(
  requestedServiceId: string,
  itinerary: Itinerary,
): string[] {
  const problems: string[] = [];
  if (itinerary.serviceId !== requestedServiceId) {
    problems.push("Booked service differs from requested service");
  }
  const expectedTotal = itinerary.lineItemsCents.reduce(
    (sum, price) => sum + price,
    0,
  );
  if (itinerary.totalCents !== expectedTotal) {
    problems.push("Total differs from the sum of line items");
  }
  return problems;
}

const problems = checkItinerary("airport-transfer", {
  serviceId: "city-tour",
  lineItemsCents: [4000, 2500],
  totalCents: 7000,
});
// Both the service mismatch and the incorrect total are flagged.

These checks capture two known failures; they do not determine whether the whole trip is useful. Production teaches you which assertions were missing in the first place.

4:384:51
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:38 · section reference included

Turn logs into a reviewed fix

Logs are the starting evidence for understanding what happened. Their structured information is well suited to agents that can write scripts to filter large volumes of records. But filtering is only the beginning. The investigator must distinguish a real bug from noise, follow the relevant trace or trajectory, and separate the symptom from its root cause. Operating an agent becomes an agent task itself because that work requires reasoning across evidence.

The repair loop follows a concrete sequence:

  1. Give a diagnostic agent traces, trajectories, and access to the codebase.
  2. Have it investigate the problem and submit a pull request.
  3. Give a separate review agent fresh context so it can examine the proposed fix from another angle.
  4. Have the reviewer run focused tests, criticize and score the change, and request revisions or close the PR when appropriate.
  5. Hand the reviewed change to a human where the workflow retains human review.

Separating diagnosis from review matters because the fixing agent is eager to produce a patch. A reviewer with fresh context can challenge both the explanation and the proposed solution.

The loop needs calibration before the team can trust it. Kalandadze reports that a PR can be ready in half an hour once this loop is in place. That is time to a proposed fix, not a measured time to merge or deployment. He presents the implementation as his team’s practice, rather than an optimal design: one flow handles rapid diagnosis and repair, while another provides a wider view of system health.

5:475:57
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:47 · section reference included

Make recurring investigations actionable

The log-monitoring agent runs on a schedule, with hourly or fifteen-minute intervals offered as options. It receives logs, trajectories, and codebase access, then investigates questions such as whether a user ended up stuck. Depending on what it finds, it can produce a PR or send a Slack alert. Critical problems need immediate attention; other findings can wait.

The output must make the investigation easy to understand:

  • Pull request: a short explanation of the problem, supporting metadata, and diagrams or tables that explain the change. Kalandadze mentions Mermaid diagrams, ASCII tables, and optional HTML artifacts for a quick visual understanding.
  • Slack notification: a concise signal that distinguishes a critical problem from a heads-up warning someone can investigate later.

These artifacts carry the diagnostic work forward. The human should be able to understand why a change exists without reconstructing the entire investigation.

9:079:21
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:07 · section reference included

Close the loop before removing the human

The review agent criticizes and scores the proposed change before handing it to a human. That creates a throughput challenge: Kalandadze reports that the PR and review agents send ten times as many PRs per day as the three-person team. This is a reported comparison of PR volume, without acceptance-rate or task-complexity data; it does not establish a tenfold increase in useful fixes.

Clear descriptions and supporting artifacts help manage that volume. Kalandadze says he can often understand a problem in a few minutes, which makes continued human participation workable for the team. Removing the human is a possible future choice. His practical ordering is to close the loop first, then address the human bottleneck if it becomes the limiting factor.

The review examples show summaries, diagrams, risks, edge cases, and requests for changes. Fixing and reviewing can repeat until no further changes are needed, at which point the human enters the loop. The completed review slide shows a ready verdict tied to a root-cause fix and a test that reproduces the problem—a more useful handoff than an unsupported assertion that the patch looks good.

Slide headed “PR-review — the root-cause fix” shows a dark review report and a footer reading “Verdict: ready — root cause, with a test that reproduces it.”Open full source frame
Review verdict: ready—root cause, with a test that reproduces it.
10:3110:45
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:31 · section reference included

Look beyond the latest incident

In the team’s actual setup, the local monitoring agent runs hourly and examines the preceding hour of logs. That makes it useful for local problems, but gives it little basis for understanding the system as a whole. A separate session analyzer supplies the wider view by inspecting conversations and scoring their outcomes.

The analyzer aims to examine every conversation through multiple sub-agents. It consumes substantial tokens, but can connect observations across sessions: recurring patterns, clusters of problems, and possible causes. It also tracks tool calls, sub-agent use, and summarizations, while producing insights and extracting entities. The purpose is to restore a sense of system health that individual incident investigations cannot provide.

12:2212:39
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

12:22 · section reference included

From system health to affected sessions

Kalandadze demonstrates a custom dashboard built around the questions he wants to investigate, despite the availability of commercial alternatives. It starts with system health, the number of analyzed sessions, cost, average score, success rate, and trends. The most valuable layer for him is the AI insights: findings that connect observations rather than merely display counters. Each critical insight explains what happened, why it matters, the root cause, the affected sessions, and a recommended fix.

The demonstration uses anonymized data derived from real conversations. Its views move from aggregates to individual evidence:

ViewInformation shown
DistributionSession scores and per-company session counts or costs
Conversation analysisSentiment and extracted entities
Tool analysisSuccess rates, rejected calls, and rejection reasons
Session detailRanking, score, runs, messages, tool calls, and summarizations

Detailed explanations accompany individual sessions so an investigator can understand the problem behind a score. The talk does not define the scoring rubric or success-rate denominator, so those labels should be read as this system’s analytical outputs rather than standardized measures.

The dashboard’s main job is oversight across hundreds or thousands of conversations, rather than fixing a particular bug. Kalandadze suggests running this broader analysis once or twice a week. That slower cadence complements the fast repair loop: one catches local problems promptly, while the other reveals patterns and changes in overall behavior.

14:1014:16
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

14:10 · section reference included

Check what the customer actually sees

Both monitoring and session analysis primarily look through logs, code, or recorded conversations. They need a complementary customer perspective. A computer-use agent opens the browser, logs in, and simulates customer interaction to find UI problems that backend evidence may not reveal. Kalandadze names Codex as one tool the team sometimes uses to drive this work.

Generic browser interaction is slow in his experience. The team therefore built a skill that knows its website and DOM, which he reports makes the workflow faster. The agent can open the site, log in, open a session, send messages, and inspect both behavior and appearance, including the resulting artifacts. This adds direct evidence of the user experience, though it remains token-intensive.

16:2216:37
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

16:22 · section reference included

Connect the evidence into a meta harness

These agents need the evidence a human investigator would need: logs, trajectories, metrics, database state, and the live UI. A computer-use agent that discovers a visible problem should be able to follow it into the execution trajectory and inspect the database to understand what happened. Otherwise, it can describe the symptom without establishing its cause.

Kalandadze calls the connected system a meta harness. The diagram brings the live product’s evidence sources into an internal harness containing log monitoring, PR review, session analysis, and computer-use QA, with Slack carrying notifications. Connecting those capabilities lets a finding lead to investigation and a proposed fix grounded in the actual failure.

Diagram flows from “Your live product” through five evidence sources into an internal harness containing log-monitor, PR-review, session-analyzer, and QA/computer-use, then to Slack labeled “The nervous system.”Open full source frame
Logs, trajectories, metrics, database, and live UI connect the product to an internal harness and Slack.

The durable advantage is the operational system around the model. At minimum, it observes the product and helps the team understand its behavior. Ideally, it closes the loop with proposed fixes and notifications that accelerate improvement. Other teams can use the same model, agent, or harness; the internal system that discovers your product’s problems and turns production experience into better behavior is what keeps improving after launch.

17:4317:59
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

17:43 · section reference included

Resources

From the talk

Updates since the talk

  • Wandero's expanded written account covers monitoring, PR review, session analysis, customer usage and codebase maintenance.

Transcript download
Full transcript

Loading transcript…