# The Missing Layer After Launch

Raphael Kalandadze, Wandero AI

AI Engineer World's Fair 2026

[Talk page](https://ai.engineer/talks/kZsf_Sfm7RU-missing-layer-after-launch) · [Video](https://www.youtube.com/watch?v=kZsf_Sfm7RU)

## The Missing Layer After Launch

A working demo does not tell you whether an agent helps real users. Production agents need a loop that turns traces, session analysis, and browser checks into reviewed improvements.

Familiarity with agent tool calls, execution traces, and pull-request review will help you follow the operational workflows.

### The demo works. What happens in production?

[0:00](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=0s)

You built an agent, launched it, and watched it work well in a demo. Now hundreds or thousands of real conversations arrive every day. How do you know whether it is working? How do you understand the system’s health, improve it, and find problems you do not yet know to look for? The demo establishes that the agent can succeed. Operating the product requires understanding what happens across actual use.

Raphael Kalandadze calls this the missing layer after launch. Models make it possible to build products quickly, sometimes in days or weeks, but shipping starts the work of maintaining control over their behavior. **The feedback loop is part of the product:** it supplies the evidence needed to improve the system every day. Monitoring, logs, diagnosis, and fixes are familiar software practices; agents make each harder because their behavior is much less constrained.

![Slide headed “After you launch” lists “Monitor it?”, “Understand it?”, “Improve it?”, and “Find the holes?”](https://ai.engineer/api/talk-essay-media/3d810cb931edec3ac95f136a11e4ff39ff215dfdeeb92b9854d1c104d99636e5)

After launch: monitor, understand, improve, and find the holes.

### You cannot enumerate the conversations

[1:50](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=110s)

An agent does not expose only a few buttons and predefined flows. Coding agents such as **Claude Code** and **Codex** illustrate the breadth of the task surface: users bring instructions, and the system attempts to handle them. You cannot write every possible conversation in advance. After launch, that breadth can make you lose the feel for your own system—whether it is getting better, getting worse, or behaving differently from what you intended.

Unit tests, regular expressions, rule-based checks, and scripted customer conversations still help. Kalandadze’s team tried all of them. They cover a slice of the problem, while customers continually introduce conversations outside that slice. Model-driven execution also varies: the same input can take a different path, and a small input change can redirect an entire trajectory. Pre-launch testing therefore cannot exhaust the behavior that production will reveal.

### Recovery can hide the defect

[3:44](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=224s)

Consider a long-running task that encounters a problem halfway through. The agent struggles, tries another tool call, finds a workaround, and eventually finishes. No red alert appears. The dashboard looks healthy, yet the trajectory contains an early warning: a defect remains, and this execution happened to recover from it. Reliability requires inspecting those recoveries, not merely accepting the final status.

A related problem is premature completion. Kalandadze points to Anthropic’s observation that agents sometimes mark features complete without adequately checking whether they work. The testing discussion in [Effective harnesses for long-running agents](https://www.anthropic.com/engineering/effective-harnesses-for-long-running-agents) describes this failure in long-running web development. **Reported completion needs evidence from the resulting system**, rather than confidence in the agent’s own declaration.

![Slide titled “The failure hides itself” gives examples of Claude marking features complete without checking and agents reporting success despite contrary system state.](https://ai.engineer/api/talk-essay-media/5b94d4834cba93a8d28500c7c18e4e79bd3c6db80cb728eda3fe5af82c3c98a7)

The failure hides itself: reported completion can conflict with actual results.

### A completed itinerary can still be wrong

[4:38](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=278s)

Long tasks widen the investigative surface further. They can involve extensive tool use, intermediate summarizations, sub-agents, generated code, terminal commands, external services, and third-party libraries. Each introduces behavior that the final response alone cannot explain. Kalandadze describes a scale of hundreds or thousands of tools, without distinguishing separate tools from repeated invocations. The operational point is the size and variety of the execution path.

In the team’s travel use case, a user asks for an itinerary. The agent runs the flow and builds a trip, but books a different service and makes mistakes calculating the price. Technically, the workflow completes; the user’s task fails. That distinction gives production analysis a concrete target: compare what the user requested with what the system actually produced.

For example, the service and price checks can become small regression tests once the failure is understood. This TypeScript example uses an illustrative itinerary with prices represented in cents:

```typescript
type Itinerary = {
  serviceId: string;
  lineItemsCents: number[];
  totalCents: number;
};

function checkItinerary(
  requestedServiceId: string,
  itinerary: Itinerary,
): string[] {
  const problems: string[] = [];
  if (itinerary.serviceId !== requestedServiceId) {
    problems.push("Booked service differs from requested service");
  }
  const expectedTotal = itinerary.lineItemsCents.reduce(
    (sum, price) => sum + price,
    0,
  );
  if (itinerary.totalCents !== expectedTotal) {
    problems.push("Total differs from the sum of line items");
  }
  return problems;
}

const problems = checkItinerary("airport-transfer", {
  serviceId: "city-tour",
  lineItemsCents: [4000, 2500],
  totalCents: 7000,
});
// Both the service mismatch and the incorrect total are flagged.
```

These checks capture two known failures; they do not determine whether the whole trip is useful. Production teaches you which assertions were missing in the first place.

### Turn logs into a reviewed fix

[5:47](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=347s)

Logs are the starting evidence for understanding what happened. Their structured information is well suited to agents that can write scripts to filter large volumes of records. But filtering is only the beginning. The investigator must distinguish a real bug from noise, follow the relevant trace or trajectory, and separate the symptom from its root cause. Operating an agent becomes an agent task itself because that work requires reasoning across evidence.

The repair loop follows a concrete sequence:

1. Give a diagnostic agent traces, trajectories, and access to the codebase.
2. Have it investigate the problem and submit a pull request.
3. Give a separate review agent fresh context so it can examine the proposed fix from another angle.
4. Have the reviewer run focused tests, criticize and score the change, and request revisions or close the PR when appropriate.
5. Hand the reviewed change to a human where the workflow retains human review.

Separating diagnosis from review matters because the fixing agent is eager to produce a patch. A reviewer with fresh context can challenge both the explanation and the proposed solution.

The loop needs calibration before the team can trust it. Kalandadze reports that a PR can be ready in half an hour once this loop is in place. That is time to a proposed fix, not a measured time to merge or deployment. He presents the implementation as his team’s practice, rather than an optimal design: one flow handles rapid diagnosis and repair, while another provides a wider view of system health.

### Make recurring investigations actionable

[9:07](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=547s)

The log-monitoring agent runs on a schedule, with hourly or fifteen-minute intervals offered as options. It receives logs, trajectories, and codebase access, then investigates questions such as whether a user ended up stuck. Depending on what it finds, it can produce a PR or send a Slack alert. Critical problems need immediate attention; other findings can wait.

The output must make the investigation easy to understand:

- **Pull request:** a short explanation of the problem, supporting metadata, and diagrams or tables that explain the change. Kalandadze mentions Mermaid diagrams, ASCII tables, and optional HTML artifacts for a quick visual understanding.
- **Slack notification:** a concise signal that distinguishes a critical problem from a heads-up warning someone can investigate later.

These artifacts carry the diagnostic work forward. The human should be able to understand why a change exists without reconstructing the entire investigation.

### Close the loop before removing the human

[10:31](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=631s)

The review agent criticizes and scores the proposed change before handing it to a human. That creates a throughput challenge: Kalandadze reports that the PR and review agents send ten times as many PRs per day as the three-person team. This is a reported comparison of PR volume, without acceptance-rate or task-complexity data; it does not establish a tenfold increase in useful fixes.

Clear descriptions and supporting artifacts help manage that volume. Kalandadze says he can often understand a problem in a few minutes, which makes continued human participation workable for the team. Removing the human is a possible future choice. His practical ordering is to close the loop first, then address the human bottleneck if it becomes the limiting factor.

The review examples show summaries, diagrams, risks, edge cases, and requests for changes. Fixing and reviewing can repeat until no further changes are needed, at which point the human enters the loop. The completed review slide shows a ready verdict tied to a root-cause fix and a test that reproduces the problem—a more useful handoff than an unsupported assertion that the patch looks good.

![Slide headed “PR-review — the root-cause fix” shows a dark review report and a footer reading “Verdict: ready — root cause, with a test that reproduces it.”](https://ai.engineer/api/talk-essay-media/667545348b0cbf47d2cebb8dfa9a723451b63f6f6ad3a09355b6ceddfdb5ff0a)

Review verdict: ready—root cause, with a test that reproduces it.

### Look beyond the latest incident

[12:22](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=742s)

In the team’s actual setup, the local monitoring agent runs hourly and examines the preceding hour of logs. That makes it useful for local problems, but gives it little basis for understanding the system as a whole. A separate session analyzer supplies the wider view by inspecting conversations and scoring their outcomes.

The analyzer aims to examine every conversation through multiple sub-agents. It consumes substantial tokens, but can connect observations across sessions: recurring patterns, clusters of problems, and possible causes. It also tracks tool calls, sub-agent use, and summarizations, while producing insights and extracting entities. The purpose is to restore a sense of system health that individual incident investigations cannot provide.

### From system health to affected sessions

[14:10](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=850s)

Kalandadze demonstrates a custom dashboard built around the questions he wants to investigate, despite the availability of commercial alternatives. It starts with system health, the number of analyzed sessions, cost, average score, success rate, and trends. The most valuable layer for him is the AI insights: findings that connect observations rather than merely display counters. Each critical insight explains what happened, why it matters, the root cause, the affected sessions, and a recommended fix.

The demonstration uses anonymized data derived from real conversations. Its views move from aggregates to individual evidence:

| View | Information shown |
| --- | --- |
| Distribution | Session scores and per-company session counts or costs |
| Conversation analysis | Sentiment and extracted entities |
| Tool analysis | Success rates, rejected calls, and rejection reasons |
| Session detail | Ranking, score, runs, messages, tool calls, and summarizations |

Detailed explanations accompany individual sessions so an investigator can understand the problem behind a score. The talk does not define the scoring rubric or success-rate denominator, so those labels should be read as this system’s analytical outputs rather than standardized measures.

The dashboard’s main job is oversight across hundreds or thousands of conversations, rather than fixing a particular bug. Kalandadze suggests running this broader analysis once or twice a week. That slower cadence complements the fast repair loop: one catches local problems promptly, while the other reveals patterns and changes in overall behavior.

### Check what the customer actually sees

[16:22](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=982s)

Both monitoring and session analysis primarily look through logs, code, or recorded conversations. They need a complementary customer perspective. A computer-use agent opens the browser, logs in, and simulates customer interaction to find UI problems that backend evidence may not reveal. Kalandadze names Codex as one tool the team sometimes uses to drive this work.

Generic browser interaction is slow in his experience. The team therefore built a skill that knows its website and DOM, which he reports makes the workflow faster. The agent can open the site, log in, open a session, send messages, and inspect both behavior and appearance, including the resulting artifacts. This adds direct evidence of the user experience, though it remains token-intensive.

### Connect the evidence into a meta harness

[17:43](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=1063s)

These agents need the evidence a human investigator would need: logs, trajectories, metrics, database state, and the live UI. A computer-use agent that discovers a visible problem should be able to follow it into the execution trajectory and inspect the database to understand what happened. Otherwise, it can describe the symptom without establishing its cause.

Kalandadze calls the connected system a **meta harness**. The diagram brings the live product’s evidence sources into an internal harness containing log monitoring, PR review, session analysis, and computer-use QA, with Slack carrying notifications. Connecting those capabilities lets a finding lead to investigation and a proposed fix grounded in the actual failure.

![Diagram flows from “Your live product” through five evidence sources into an internal harness containing log-monitor, PR-review, session-analyzer, and QA/computer-use, then to Slack labeled “The nervous system.”](https://ai.engineer/api/talk-essay-media/a3170f65d74435520cbe55c02fd71f91dec0d83b424ca3ec4a54f53e4c789d41)

Logs, trajectories, metrics, database, and live UI connect the product to an internal harness and Slack.

The durable advantage is the operational system around the model. At minimum, it observes the product and helps the team understand its behavior. Ideally, it closes the loop with proposed fixes and notifications that accelerate improvement. Other teams can use the same model, agent, or harness; the internal system that discovers your product’s problems and turns production experience into better behavior is what keeps improving after launch.

## Resources

- [Effective harnesses for long-running agents](https://www.anthropic.com/engineering/effective-harnesses-for-long-running-agents) — Anthropic explains persistent progress records, feature lists and browser testing for coding agents working across multiple context windows.
- [The Missing Layer After Launch](https://www.wandero.ai/blog/the-missing-layer-after-launch) — Wandero's expanded written account covers monitoring, PR review, session analysis, customer usage and codebase maintenance.

## Slides

### The Missing Layer After Launch

After launching an agent, the speaker asks how to monitor its health across real conversations and find gaps that need improvement.

[0:00](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=0s) [Title slide stating that important work starts after shipping.](https://assets.ai.engineer/sites/aiecode2025/sha256/fd376787e5745209242f576ce21f9ebef43fe297da57c4e3ca0a4fc5b9d2ea70.webp)

The Missing Layer
After Launch

Everything important starts after you ship.

AI ENGINEER WORLD'S FAIR 2026

### We shipped it in 3 weeks

The team reached its first product in 3 weeks, with ~300K lines and ~$35K spent. The speaker calls shipping the easy part: a tight feedback loop helps improve the product after launch.

[0:40](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=40s) [Three metrics: three weeks to first product, approximately 300,000 lines, and approximately $35,000 spent.](https://assets.ai.engineer/sites/aiecode2025/sha256/8f3090e744e448064a1eb06733305ab470ea55ae44d3fbe52fd019bd598cf094.webp)

THE EASY PART

We shipped it in 3 weeks.

3 wks
TO FIRST PRODUCT

~300K
LINES

~$35K
SPENT

Shipping is fast now. That was the easy part.

### Almost anything

An agent can take on far more tasks than a conventional app, expanding what can break. That makes it harder to monitor behavior, detect problems and test flows before launch.

[1:24](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=84s) [The slide contrasts a conventional app's limited features with an agent's broad range of tasks and potential failures.](https://assets.ai.engineer/sites/aiecode2025/sha256/472ccda8e19f3fde683a3dbe5bcaf2c71b65ead951f10e10ea88f5360ca32867.webp)

WHY IT'S HARD

Not a few features.
Almost anything.

A normal app has 5 features and 3 buttons. An agent can do whatever you ask.

If it can do almost anything, almost anything can break.

### How do you know it's healthy?

Thousands of conversations and almost infinite tasks make a product agent hard to monitor, understand, and improve. Tests and simulated conversations cover only a slice of the problem.

[2:22](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=142s) [A product agent is surrounded by questions about monitoring, understanding, improving it, and finding its holes.](https://assets.ai.engineer/sites/aiecode2025/sha256/1e16d3b2560b8a75f205af52d3d2762cdf791a4f099b09fca470bc3801894842.webp)

THE REAL PROBLEM

How do you even know it’s healthy?

thousands of conversations · almost infinite tasks

Monitor it?
Understand it?
PRODUCT AGENT
Improve it?
Find the holes?

You lose the feel for your own system.

You lose the feel for your own system. Getting it back is the goal.

### Agent behavior in production

Rule-based checks and simulated conversations cover only a slice of agent behavior. Customers do different things, so the hard work continues after the agent goes live.

[3:16](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=196s) [An article screenshot says you do not know what an agent will do until it is in production; the slide says the hard part starts after shipping.](https://assets.ai.engineer/sites/aiecode2025/sha256/f48bb122ad5ecc444f59533a78c41120ce9fa400dba4dcd5b74d7a1e3ecd7c2c.webp)

YOU'RE NOT THE ONLY ONE

The hard part starts after you ship — you don't know what your agent does until it's live.

You don’t know what your agent will do until it’s in production

Repeat
2. Annotation Queues
Let you review and label traces
3. Datasets
Incorporate examples for testing
4. Experiments
Test whether fixes improve behavior
5. Online Evals
Validate the fix in production

When you ship traditional software to production, you have a good sense of what to expect. Users click buttons, fill out forms, navigate through

Interface text:
Article
Harrison Chase
@hwchase17
44 · 79 · 407 · 157K

### This is a new problem

LLMs can take different paths from the same input, making agent behavior difficult to anticipate before production.

[3:23](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=203s) [Slide introducing a new problem, with the word new highlighted in blue.](https://assets.ai.engineer/sites/aiecode2025/sha256/ad0a8b67138abbf94eb5ef74061ae77800706ced67357e68fb3274089e3305db.webp)

WHAT CHANGED

This is a new problem.

### Same input, different path

LLMs can take different paths from the same input, and small changes can alter the trajectory. The range of possible tasks cannot be fully listed and pre-tested.

[3:28](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=208s) [Slide says the same input can take different paths and that endless tasks cannot all be pre-tested.](https://assets.ai.engineer/sites/aiecode2025/sha256/703dcf6086f6c1f8c4a774dc8a3d0c19ae491485c1ba5caf8d2251229d34aa86.webp)

01 · NON-DETERMINISTIC

Same input, different path.
Endless tasks.

You can't list it — so you can't pre-test it.

### The failure hides itself

Agents can work around problems without triggering alerts. The slide also cites features marked complete without checks and success reports contradicted by system state.

[3:42](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=222s) [Slide describes agents marking features complete without verification and reporting success despite contrary system state.](https://assets.ai.engineer/sites/aiecode2025/sha256/7958247862f7e7e66eaad6d1bc4911139bc5f88d1227ade74d092c0cb905c57e.webp)

02 · INVISIBLE FAILURE

The failure hides itself.

Claude marked features "complete" — without checking they actually worked.
"EFFECTIVE HARNESSES FOR LONG-RUNNING AGENTS," ANTHROPIC

Agents reported success while the system state said the opposite.
NORTHEASTERN RED-TEAM

Nothing crashes. Nothing turns red. Nothing knows.

### One task, hundreds of tools

A long-running task can use hundreds of tools, from writing code and running a terminal to calling other companies’ services. Each tool adds another possible way to fail quietly.

[4:36](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=276s) [Slide says agents write code, run terminals and call external services, with each tool creating another way to fail.](https://assets.ai.engineer/sites/aiecode2025/sha256/06924cd9421cdce8037b658167267756eded9aed75546fbb7779c77e3f11b5c3.webp)

03 · HUGE TOOL SURFACE

One task.
Hundreds of tools.

It writes code, runs the terminal, calls other companies' services.

Every tool is a new, quiet way to fail.

### Finished is not helped

An agent can complete an itinerary flow while booking a different service and miscalculating prices, leaving the user unhappy despite a technically successful response.

[5:06](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=306s) [Slide says a technically successful response can still fail the task.](https://assets.ai.engineer/sites/aiecode2025/sha256/0a5f4c6093305fdfddba8682cb3bd123ea3a2a0a63cb41317921df3beae01cb1.webp)

04 · DONE ≠ HAPPY

"Finished" is not "helped."

A "technically successful" response can still fail the task.
HARRISON CHASE · LANGCHAIN

An agent can succeed and still be wrong.

### Unit tests catch a sliver

Production reveals what unit tests should cover: an itinerary agent can complete its flow while booking a different service or miscalculating prices.

[5:36](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=336s) [Slide says unit tests catch a sliver and production reveals what to test.](https://assets.ai.engineer/sites/aiecode2025/sha256/9f7f6fcdf7899ec643f299d0528a5c0dea0c1cab80f3aa8bb7ead6e9db28182a.webp)

05 · TESTS DON'T FIT

Unit tests catch a sliver.

Production is where you learn what to test for.

### Operating an agent is an agent problem

The speaker puts agents on operations because interpreting logs requires reasoning: distinguishing bugs from noise, tracing causes, and separating symptoms from root causes.

[5:45](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=345s) [Operating an agent requires reasoning about noise, causes and symptoms.](https://assets.ai.engineer/sites/aiecode2025/sha256/d353bcc587a4cd0523ece6ceb67b7166b5f339d09b0510ef9f212ceae68e15e7.webp)

THE SHIFT

Operating an agent is itself
an agent problem.

"Noise or bug?" · "Trace the cause" · "Root cause or symptom?" — all reasoning tasks.

So I put agents on the operations.

### The closed loop

Agents trace production sessions through logs and code to diagnose problems and draft pull requests. A separate agent reviews the PR before human review and merge; the dashboard feeds the next fix back into production.

[6:48](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=408s) [A loop connects production sessions, logs, diagnosis, pull requests, agent and human review, and the next fix.](https://assets.ai.engineer/sites/aiecode2025/sha256/cc2d8ea562808541baf1fdac1c1d71e02ab929a90364e76b531f02c246a667e9.webp)

THE CLOSED LOOP

Production sessions → Logs & traces → Trajectory → Code path → Diagnosis

Pull request (log-monitor) → PR review (agent) → Human review + merge → Dashboard → Next fix → Production sessions

iterate until ready

The agents watch and draft. The human decides.

### The fastest loop

The speaker says the loop helps reveal problems quickly and can have a pull request ready in half an hour.

[8:12](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=492s) [The speaker describes the loop as the fastest they have had, with near-real-time improvement.](https://assets.ai.engineer/sites/aiecode2025/sha256/2c37e252e506cfaaa8d029a425439675c3cd513d68a137d59833227df0f6b7b1.webp)

THE PAYOFF

The fastest loop
we've ever had.

You feel the system improve in near-real-time.

### Four operating agents

Three reactive agents detect and fix issues, review PRs, and assess session health. Proactive QA using computer use is on the roadmap.

[8:36](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=516s) [The slide lists log monitoring, PR review, session analysis and planned proactive QA.](https://assets.ai.engineer/sites/aiecode2025/sha256/e93276d3e7ab763b51b0536d39aa2a260fda1e9b4e107c3a6c57f516a55e642e.webp)

HOW I HANDLE IT

Four operating agents.

01 log-monitor — detect & fix fast (reactive)
02 PR-review — gate the fix (reactive)
03 session-analyzer — feel the health (reactive)
04 QA / computer-use — go test it (proactive · roadmap)

One example — not a recipe.

### Log-monitor agent

The log-monitor filters recent logs, investigates trajectories and code, and asks whether the user is stuck. It can open a pull request with evidence or send a Slack alert.

[9:05](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=545s) [The log-monitor reads operational data, detects and analyzes issues, decides whether users are stuck, and can create a fix or alert.](https://assets.ai.engineer/sites/aiecode2025/sha256/bcd3190c6fdb7ab3e19da082ddfc8ed3789cb7e2e8ba8d8c05841bc900acd1be.webp)

AGENT 01 · LOG-MONITOR

What it can read
Production logs
Agent trajectories
Codebase
Database (read-only)
Traces

the full signal surface — so it can really explore

log-monitor agent

DETECT
pull last hour
filter noise
ignore-list

ANALYZE
pull trajectory
clone repo
read code

DECIDE
does the user end up stuck?

FIX
write patch
open PR

Pull request
description
mermaid flow
evidence
labels

Slack alert
severity
PR link

### PR review agent workflow

The review agent checks the branch, runs tests and asks whether the fix addresses the root cause. Its readiness verdict leads to human review and merge, with a feedback loop back to the pull request.

[10:29](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=629s) [A review agent checks a pull request and sends its verdict to a human for review and merge.](https://assets.ai.engineer/sites/aiecode2025/sha256/628968f40f08d46406c0b916394c21599951901bf6ae8c2687273c7da4f81952.webp)

AGENT 02 · PR-REVIEW

Pull request

Review agent
(different angle)
checkout the branch
run focused tests
typecheck + lint
root cause, or symptom?

verdict: READY?

Human
review + merge

### Review catches a symptom patch

The review agent requests changes: a broad exception handler returns 200 while silently dropping messages, leaving the underlying race unaddressed.

[11:50](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=710s) [A review marks a proposed fix as needing changes because it hides the bug.](https://assets.ai.engineer/sites/aiecode2025/sha256/215f2eaf77394b9458de0fe39dc19e7d918d226757633620c717256fd3d0533a.webp)

PR-REVIEW — CAUGHT A SYMPTOM PATCH

repo: backend
pr: "#1"
risk: high
decision: needs-changes
checks: partial

Review Summary
Area | Result
What it is | Wraps the inbound-message webhook handler in a broad try/except so a null field stops crashing.
Why needed | The webhook was throwing on messages that arrive mid-state-change.
How it works | Any exception in the handler is caught, logged at WARNING, and the request returns 200.
Main risk | The crash is silenced, not fixed — messages that hit the bug are now dropped without a trace.
Merge call | Needs changes — this patches the symptom, not the root cause.

What Looks Good
• Correctly identifies the right handler and the right failure window.
• Adds a regression test for the normal (non-racing) path.

Flow / Diagram
message arrives → handler reads conversation field
  (field briefly null during a state change)
       |
  try/except swallows the error → returns 200 → message SILENTLY DROPPED
       |
  real bug (unguarded read during transition) is still live

Risks / What Can Break
Risk | Impact | Evidence
Swallowed exception | Messages lost with no error surfaced | broad except Exception returns 200 in the handler
Happy-path-only test | Bug recurs undetected | test never exercises the mid-state-change race
Masks a data path | Conversation history gaps | dropped messages never persisted

Verdict: needs changes — the bug is hidden, not fixed.

Interface text:
RRaphaell commented 1 hour ago
Author

### Review details: needs changes

The review flags a webhook patch that catches exceptions and returns 200, silently dropping affected messages. Its regression test covers the normal path but misses the state-change race.

[12:01](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=721s) [An enlarged review explains that caught exceptions can silently drop messages and identifies gaps in testing.](https://assets.ai.engineer/sites/aiecode2025/sha256/de07c2fdeef15b8beac0d1356c4ce0bbfca8c25b3f6156567df0252066a19cfc.webp)

repo: backend
pr: "#1"
risk: high
decision: needs-changes
checks: partial

Review Summary
Area | Result
What it is | Wraps the inbound-message webhook handler in a broad try/except so a null field stops crashing.
Why needed | The webhook was throwing on messages that arrive mid-state-change.
How it works | Any exception in the handler is caught, logged at WARNING, and the request returns 200.
Main risk | The crash is silenced, not fixed — messages that hit the bug are now dropped without a trace.
Merge call | Needs changes — this patches the symptom, not the root cause.

What Looks Good
• Correctly identifies the right handler and the right failure window.
• Adds a regression test for the normal (non-racing) path.

Flow / Diagram
message arrives → handler reads conversation field
  (field briefly null during a state change)
      ↓
  try/except swallows the error → returns 200 → message SILENTLY DROPPED
      ↓
  real bug (unguarded read during transition) is still live

Risks / What Can Break
Risk | Impact | Evidence
Swallowed exception | Messages lost with no error surfaced | broad except Exception returns 200 in the handler
Happy-path-only test | Bug recurs undetected | test never exercises the mid-state-change race
Masks a data path | Conversation history gaps | dropped messages never persisted

Interface text:
RRaphaell commented 1 hour ago
Author

### Review of the root-cause fix

The review marks the backend fix ready: it guards a briefly null field and adds a test that reproduces the race. The speaker says a human enters the loop once the code is ready.

[12:04](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=724s) [A review marks the root-cause fix ready and notes a test that reproduces the failure.](https://assets.ai.engineer/sites/aiecode2025/sha256/59b4fff2f90b880b53034024ed9ab6aa6cfc4caaf986a31b50d86f7ea124e911.webp)

PR-REVIEW — THE ROOT-CAUSE FIX

repo: backend
pr: "#2"
risk: low
decision: ready
checks: passed

Review Summary
Area | Result
What it is | Guards the inbound-message webhook against a field that is briefly null during a conversation state change.
Why needed | The handler assumed the field was always populated; it crashed when a message arrived mid-transition.
How it works | Reads the field defensively, resolves the conversation through the transition, and only then processes the message.
Main risk | Low — change is scoped to the one read path; behavior unchanged when the field is present.
Merge call | Ready — fixes the root cause with a test that reproduces the race.

What Looks Good
• Fixes the actual cause (the unguarded read), not the symptom.
• Adds a test that reproduces a message arriving during the state change — the exact failing trajectory.
• No scope creep; only the affected handler changed.
• Error path now resolves the conversation correctly instead of dropping the message.

Risks / What Can Break
Risk | Impact | Evidence
Other readers of the same field | Low | grep shows this handler is the only mid-transition reader

Required Changes
No blocking changes requested.

Checks
Check | Result
focused webhook tests | passed (incl. new race test)
typecheck / lint | passed

Final Recommendation
Ready to merge. Root-cause fix, scoped change, regression test reproduces the original failing trajectory.

Verdict: ready — root cause, with a test that reproduces it.

Interface text:
[illegible] commented 1 hour ago
Author

### Trace analysis at scale

Dev Shah describes teams recording 100k+ traces a day that humans cannot summarize at scale. The speaker wants a system-wide view beyond the previous agent’s one-hour log window.

[12:19](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=739s) [A quotation describes the difficulty of analyzing 100,000 traces a day.](https://assets.ai.engineer/sites/aiecode2025/sha256/038ec1ac4a2d337c83653481f4da4e0896ba08286df294384bd2abf7ccbc367e.webp)

YOU'RE NOT THE ONLY ONE

Visibility is the easiest piece. The hard part is analyzing and understanding what you're observing. I've spoken to teams recording 100k+ traces every single day. What are they doing with those traces? Literally nothing. Because it's impossible to read and summarize 100,000 traces at any human scale.

- Dev Shah

100k traces a day — and no human can read them. That's the job of the next agent.

### Session analyzer workflow

The session analyzer checks each conversation for outcomes, quality, cost and issues, then groups recurring issues into patterns and reports system health.

[13:11](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=791s) [A diagram shows sessions analyzed individually, combined into patterns, and summarized as a health score.](https://assets.ai.engineer/sites/aiecode2025/sha256/536d4c5d8cd5b3e3f47702d3a5db1ab7b1236db9ab1e615f7c510275f32a7889.webp)

AGENT 03 · SESSION-ANALYZER

sessions
session s-9c1d
session s-3b7e
session s-7a02

session-analyzer
1 · analyse EVERY session
outcome
quality
cost · issues

2 · collapse into patterns
the same issue across many sessions = ONE pattern

feel the pulse
77 / 100
health score
84% success · cost trend

[illegible]erior hit ~95% on the model, then got to 99% by building this kind of loop over production.
[illegible]OPHER LOVEJOY · ANTERIOR

### AI insights and score distribution

The dashboard connects session trends to AI-identified patterns: stale inventory appears in 9 sessions, and unsupported client-facing claims in 5. Below, score distribution and organization metrics break down the results.

[14:42](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=882s) [The dashboard lists recurring issues and shows the beginning of score distribution and organization metrics.](https://assets.ai.engineer/sites/aiecode2025/sha256/03de9c4798f109a87ccc6308aa7c3f0f4cac8cb6e68031ccf62f916acd6ea9cf.webp)

Sessions Analyzed
97/97
Total sessions in period

Total Cost
$216.67
Avg $2.23/session

Average Score
7.7/10
Across all sessions

Success Rate
84.5%
Completed successfully

Partial
15.5%
Needs attention

Failed
0%
Did not complete

Needs Review
1
Flagged for review

Score Trend (avg: 7.7)
Legend: 1, 16, 31+ sessions
Y-axis ticks: 0, 2, 4, 6, 8, 10
Timeline: 06-08, 06-09, 06-10, 06-11

Sessions per Day (avg: 24.3/day, total: 97)
Y-axis ticks: 0, 10, 20, 30, 38
Timeline: 06-08, 06-09, 06-10, 06-11

Cost per Day (avg: $54.17/day, total: $216.67)
Y-axis ticks: $0, $55, $110, $165, $202
Timeline: 06-08, 06-09, 06-10, 06-11

AI Insights
4 high
11 insights
All (11) | Patterns (8) | Orgs | Training (3) | Actions

HIGH · PATTERN · Priority 2
Stale inventory data
Records contradict operational reality → rejected drafts and price conflicts.
9 sessions

HIGH · PATTERN · Priority 1
Ungrounded client-facing claims
Agent asserts product or booking facts its sources don't support, instead of hedging.
Drags down: Northwind Travel Co.
5 sessions

HIGH · TRAINING GAP
Hedge instead of guessing
5 occurrences
5 sessions

HIGH · PATTERN · Priority 3
Safety-guard bypass
Agent hand-rolled a raw write to the booking system, routing around the safe wrapper.
1 sessions

MEDIUM · PATTERN · Priority 5
8

Score Distribution
9–10: 17
[illegible]: 67
[illegible]: 13
[illegible]: 0
[illegible]: 0

Organizations
Organization | Sessions | Avg Score | Total Cost
Azure Coast DMC | 25 | 7.7 | $216.67
Northwind Travel Co. | 20 | 7.1 | -
Verde Valley Tours | 16 | 7.9 | -
Solara Journeys | 11 | 7.8 | -
Highland & Harbor Travel | 8 | 7.9 | -
Maris Hospitality | 6 | 8.5

### Recurring issue details

An issue detail panel links conflicting records to rejected drafts and price conflicts across four agencies. It lists nine affected sessions and recommends reconciling inventory and applying user-supplied corrections.

[14:51](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=891s) [A detail panel describes an inventory issue affecting nine sessions and recommends reconciling inventory.](https://assets.ai.engineer/sites/aiecode2025/sha256/3554bdb8fbd4c695e33f8983e0f92f0953ac854342b5c3b9c72c1cdc5ecb4768.webp)

9 sessions affected

What is this?
An issue appearing repeatedly across multiple sessions, indicating a single root cause.

Records contradict operational reality → rejected drafts and price conflicts. It showed up in 9 sessions, so it is systemic — not a one-off. Price conflicts and rejected drafts across 4 agencies.

Why it matters
Price conflicts and rejected drafts across 4 agencies.

Root Cause
Records contradict operational reality → rejected drafts and price conflicts.

Recommended Fix
Reconcile inventory; apply user-supplied corrections.

Affected Sessions (9)
s-087  s-074  [illegible]  s-088  s-082  s-029  s-010  s-003  s-005

## Full transcript

[0:00](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=0s) All right. So you built an agent, you launched it, everything works pretty well in a demo, everyone is happy. But now let me ask you a few simple questions.

[0:09](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=9s) So how do you know if it's actually working out there? How do you watch across hundreds or thousands of real conversations every day? How do you feel or understand the health of the system?

[0:20](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=20s) How do you make it better? How do you find the holes that you don't know are there yet? And that's the thing, right? So most of the talks about the agents end at the moment when you ship.

[0:29](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=29s) So we build it, it work, the end. But I think the shipping is the moment when the real work begins. And somehow only a few peoples are talking about that, and I'm calling it the missing layer.

[0:40](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=40s) So let's dive into it. So that's the world we are living. So you can create the whole product. You can create a whole startup in a couple of days, in a couple of weeks.

[0:49](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=49s) You can write hundred thousands of lines of code. You can spend a lot of tokens. And to be honest, it's the easiest part today with the help of the latest models.

[0:58](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=58s) But I think the shipping is the moment when the real work begins because you need to close the loop as soon as possible. So after you launch, you need to have some control and understanding of the system.

[1:09](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=69s) And, uh, from my experience, the loop is, uh, at least as important as the product itself, sometimes even more, because the tight feedback is the one that helps you to make the product better every single day.

[1:22](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=82s) And that's the missing layer, and that's what the rest of this talk is all about. So what happens after you launch? And this is not something surprising. We had the same questions in the classical old software.

[1:33](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=93s) You need to monitor what is happening. You need to understand how it behaves. You need to have some logs to detect the problems and fix them. Uh, and for agentic systems, each one of those are even harder and sometimes, uh, they turn into something genuinely new.

[1:50](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=110s) So let's talk about why this is hard and why this is hard now. So the agent is not a normal software, right? You, you don't have a few features and several buttons.

[1:59](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=119s) You don't have a, uh, predefined flow that you can test before you go to the, to the live. Uh, and the coverage is endless. So think about like Claude Code or Codex.

[2:10](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=130s) They can do a giant range of stuff, whatever the user needs, and, uh, most of the agents do the same, right? So you give the instructions, and they can handle it.

[2:18](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=138s) And you cannot write all the conversations, uh, in advance. So th-this leads to the deepest part of the problem. The part that keeps me up all night, which is you lose the feel for your own system.

[2:31](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=151s) So after you, uh, build the product, you need to have some kind of understanding. Does it get better or worse? So you need to monitor, understand what is happening.

[2:41](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=161s) And the problem is that this normal-- the normal safety nets don't save you here. Uh, and believe me, we try, uh, a few stuff or we, we build a unit test.

[2:52](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=172s) We have some regex, some rule-based checks. We even create some scripts to simulate the customer conversation. Uh, and, uh, yeah, it helps in some ways. Uh, but at the end of the day, uh, it is like only a one slice of the whole problem

[3:08](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=188s) because, uh, customers always do something different. Uh, you cannot write it all down. They are too many, and they are all different. And the horizon for it in the same way.

[3:20](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=200s) So you don't know what your agent will do until it is in the production.

[3:25](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=205s) So why this is a new problem? So as you know, LLMs are not, uh, deterministic. Uh, the same input can have a different path. Even slight modification on the input can cause a different trajectory, and the coverage is endless.

[3:39](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=219s) You cannot list it all down. You cannot pretest until you go into production.

[3:44](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=224s) The second and the scariest one is the failure hides itself. So let's think about what happens when the agent is running for a long time. It struggle in the middle of a task.

[3:55](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=235s) Uh, it has some problems, but it was lucky. Uh, it was recovered. It finds some workarounds, uh, try some other tool calls or whatever, and you did not get any red alerts, any problems on the dashboard.

[4:08](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=248s) Everything looks fine. But you know, this is a early warning for you. This is a problem that, uh, that is hidden, uh, and that lives in your code base.

[4:18](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=258s) And you need to fix it as soon as possible because if you're talking about the reliable agents, uh, it shouldn't be dependent on the luck, right? And, uh, also as Anthropic mentioned in the blog post, sometimes the agent loves to make-- mark the feature as complete, uh, without checking if they actually worked.

[4:38](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=278s) The next one is, uh, the tool calling, right? So the tool surface is huge. So long-running task needs a hundred of tools, even thousands of tools. Um, sometimes it have, uh, several summarization in the middle.

[4:51](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=291s) It use some sub-agent, writes a lot of code. It use a terminal for sure. It calls some other company services, third-party libraries, and, uh, they, they work with different way, uh, all the time.

[5:04](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=304s) So actually, you don't know what you're looking for. So sometimes finished does not mean it is helpful for the user. Maybe there wasn't any-- there wasn't any problems. So agent finished, everything looks good.

[5:16](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=316s) The answer was successful. But what happens, like for example, for our use case, sometimes, uh, user asks to build the itinerary, agent run the flow, it builds a trip, but it book a different service.

[5:30](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=330s) It made a lot of mistakes in calculating the price, so user is not happy. So technically is successful, but still fail in the task. And as I mentioned, the unit tests don't save you here and production is the place when you learn what you need to, uh, what you need to test on the first place.

[5:47](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=347s) All right, so what happens after you launch the product, right? So you need to monitor, you need to understand and improve the system. Uh, and what's the source of the truth, right?

[5:57](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=357s) This is the logs. Everyone has the logs. Everyone loves the logs. You have structured information. And, uh, you know, those machines are the best to understand and explore the logs much better than any human, much faster.

[6:10](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=370s) They can write some scripts to filter the giant of walls. Um, and this is the most obvious way that you can hand it to the agent. Uh, but as soon as you start working on that, maybe you build an agent or skill or whatever, you, you quickly understand this is not as easy as you imagine because it

[6:30](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=390s) needs a lot of reasoning. You need to understand the problem to, uh, differentiate if, if it's a real bug or a noise, to deep dive in its trace or trajectory, understand if it's a, a symptom or a root cause.

[6:43](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=403s) So actually, you'll find out that the operat-operating an agent itself is an agent problems.

[6:50](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=410s) So this is a loop like end-to-end. So you have a traces, you have trajectories, you give the agent to access the code base. It diagnose the problem, understand what was happening, and send the PR.

[7:02](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=422s) And then you can have a skill or sub-agent or, or whatever that controls the PR, uh, because you know, most of the agents are pretty eager to send the PR.

[7:11](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=431s) They love to fix the problems. We prefer to have a separate agent, which has a, like, fresh context. It try to check, um, mm-hmm, the PR on different angle, on different view, run the, um, uh, the focus test.

[7:27](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=447s) And, uh, it is not biased of the problem itself. It tries to criticize, score the PR, and, uh, sometimes it, uh, requires the changes. Maybe it close the PR directly and help us to, to filter those problems.

[7:41](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=461s) Uh, and, uh, mm, then you have the human-in-the-loop. Sometimes you don't. Uh, and we can talk about this later, but you know, this is the most obvious thing that you can hand it to the agent, which is-- which seems like, uh, pretty obvious for most of the people.

[7:57](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=477s) And, um, a lot of teams use the same practice, but I think people don't appreciate how important it is. And you need s-- you need to spend some time on that.

[8:07](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=487s) You need to calibrate. You need to make it reliable, to trust the loop. And this is the, the fastest loop that we ever had. So,

[8:16](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=496s) you know, after you build this simple, uh, simple system, you, uh, already have the feel and understanding how it behaves, what is happening. You detect the problems and look for fixes as soon as possible.

[8:30](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=510s) And you can have the PR in-- PR ready in half an hour, and you can, uh, easily understand what is happening. So let me walk you through, uh, how I'm handling this.

[8:40](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=520s) Maybe this isn't optimal, but I think this, this will help you to, to get some point. So actually we have a two main flow. The first one is the, the fastest loop that helps you to, to detect and, uh, fix the problems as soon as possible.

[8:56](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=536s) And another one is like more on a zoom out that hel-- that helps you to have, uh, on a high-level understanding, and it helps you to have hand on the pulse.

[9:07](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=547s) So the first one is a log monitoring agent, the one that I already mentioned. So you have trajectories, you have logs, you have the access to a code base, and it runs every hour or, um, uh, every few, uh, fifteen minutes.

[9:21](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=561s) And it try to understand the problem. It deep dives in the, in the logs, uh, understand does the us-user end up stuck and, uh, send the PR or sometimes send the Slack alert.

[9:33](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=573s) And this is, this is pretty, uh, important because sometimes the problem is, uh, uh, so critical, so we need to, mm, fix them as soon as possible. And, uh, yeah, it works pretty well.

[9:48](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=588s) So this is the one example how the PR can look like. So you have a nice description, a short explanation what is happening. You have some metadata. You have nice diagrams, Mermaid or ASCII tables.

[10:00](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=600s) Maybe you will have some HTML artifact that helps you to give you a glance of the problem and quickly, mm, understand what is this PR for.

[10:12](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=612s) So this is an example how a Slack notification can look like. It helps you to quickly, uh, give you a feedback, detect if there are some, uh, critical problems.

[10:21](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=621s) Sometimes it just, it is just like heads up, so you know there are some warnings, there are some problems, so you need to check them, uh, when you have some time.

[10:31](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=631s) And as I mentioned, the review agent is, uh, pretty critical because it try to check the problem in different angle. It always try to criticize the problem, score the problem, and, uh, as soon as it's ready, it send the PR to the human.

[10:45](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=645s) And sometimes people talk about that if you need to have the human in the loop because you know, still human in the bottleneck in this case. Uh, in our case, the, the, the PR agent and the review agent send ten times more PR than three of us, uh, every day.

[11:01](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=661s) So you need to have some clean system how we're gonna handle this to not be a bottleneck in this system. But from my experience, as I mentioned, this PR and the review agent helps me to have a n-nice description, uh, some artifacts that helps me to quickly understand the problem.

[11:20](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=680s) Uh, may-- and maybe I spend a few minutes to at least understand what is happening. So I think at this time it's okay, but maybe we will remove it in the future.

[11:30](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=690s) And yeah, people are talking about that. So you need to... You ne-- you don't need to be a bottleneck in this problem. Some of them prefer, uh, to remove the humans in this loop.

[11:40](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=700s) But you know the trend is that you need to close the loop first. So let's make the problem when you are the bottleneck, and then you can remove yourself pretty easily, I think.

[11:52](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=712s) So this is, uh, some examples how the review agent feedback can look like. As you know, uh, it requires some changes. You have some summary. You have some diagrams, explanation where there are some risks, some edge cases.

[12:05](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=725s) And sometimes, uh, after a few iterations or they go in loop, uh, where the k- where the code is ready, so we don't no- we don't need to, uh, have any more changes.

[12:16](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=736s) So after that, the human will, uh, jump, uh, into the loop.

[12:22](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=742s) For the previous agent, as I mentioned, that is especially good when you want to fix some local problems. Uh, for our case, it runs every hour. It checks only a one-hour window of the logs, and, uh, it doesn't have any, uh, high-level understanding of the problem.

[12:39](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=759s) So we find-- found out that we need to have another system that helps us to, to get, uh, to get, uh, mm, like more on a high level explanation, some kind of a health of the system.

[12:51](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=771s) And, you know, visibility is the easiest piece. Um, before the agent view was impossible, so you cannot, uh, deep dive or summarize hundreds of conversations. But right now we can have a system that helps you, uh, to, to score every conversations, understand what is happening, and give you a high level zoom out of the system.

[13:12](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=792s) So this is the session analyzer, and the main goal is to just give me the score of the health of the system, right? So we try to check every, every conversation, run through a lot of sub-agents.

[13:24](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=804s) It spend a lot of tokens. Uh, but yeah, it helps us to detect some patterns, connecting the dots and, uh, try to understand some high-level patterns. And, uh, yeah, it also detects some, uh, mm, uh, like cluster problems.

[13:42](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=822s) What is the cause? Uh, how many tool calls are used? How many sub-agents? Uh, how many summary happened or whatever. But also it gives some AI insights, some entities, so we have an understanding of the problem.

[13:55](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=835s) So, right. As I mentioned, one of the main problem is that you lose the control after you launch the production. So you need to have the system that helps you to control at least to have some understanding what is happening, how it looks like, is it healthy or what or not.

[14:10](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=850s) So, um, [lip smack] actually let me, let me show you,

[14:16](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=856s) uh, one of the example how, how it looks like. So actually we build it ourselves. So there are a lot of other, um, companies and tools that provide the same kind of system, but [lip smack] I prefer to build it myself because I know what I'm interesting for, what I'm looking for.

[14:33](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=873s) So as you can h-- see, we have the health of the system, we have number of sessions that was analyzed, the cost, we have the average score, some success rate.

[14:41](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=881s) We have some trends, we have the AI insights, and this is the most important part. When the AI or when the agent try to connect the dots, find the patterns.

[14:50](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=890s) If there are some critical ones, you have a description for each of them. What is this? Why it matters? What are the root cause? All the sessions, uh, affected, and some recommendation fix.

[15:02](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=902s) So right now this is just, uh, anonymized data, but it comes from a, from a real conversations. So we have a score distribution. Uh, each company, um, number of sessions or the cost.

[15:15](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=915s) We have, uh, the sentiment analysis, some entities. We have analytics about tool calls, all the success rate, how many rejected and why. And you have, uh, detailed, uh, analysis of the each session.

[15:28](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=928s) So it ranks each one of them, scores, that is, um, how many times it was running, how many messages, how many tool calls, how many summary. And you have a detailed explanation for each of them.

[15:40](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=940s) What was the problem? It is something that you need to session and, uh, believe me, it helps a lot and it helps you to check and watch across hundreds of, uh, conversations.

[15:52](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=952s) So the, the main goal of this dashboard and the system is not to fix some, uh, um, uh, specific problems or bug. This is more on a high level that helps you to watch across hundreds or thousands of real conversations.

[16:07](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=967s) So you can run it once in a week, two times in a week, or something like that. All right, so the next one is that, mm, so the problem of the previous, uh, agent that I mentioned is like the first one helps you to detect the specific problems.

[16:22](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=982s) The second one is to give you a high-level understanding, but both of them work from a angle of the logs or a, uh, code or a session itself. But you need to have some kind of user perspective, right?

[16:37](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=997s) So that's why we have some computer use agent that helps you to, uh, to open the browser, uh, to log in and try to simulate the customer itself, because sometimes you have some problems in the UI.

[16:49](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=1009s) You need to, um, um, check if everything looks good. Uh, and uh, yeah, sometimes code and logs don't help you, uh, in this way. So this is, uh, example how it looks like.

[17:03](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=1023s) So you, you use Codex, uh, sometimes. So it use, um, uh, the browser itself.

[17:10](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=1030s) Uh, actually it is pretty slow. Uh, and we try to build the specific skill that know our, uh, website, our DOM, how it looks like, and that is much faster.

[17:22](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=1042s) And it can, uh, open the website, log in, open the session, send the messages, check what is happening, uh, and also check, uh, how it looks like, what are, what are the some artifacts.

[17:34](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=1054s) And uh, yeah, it works pretty good. But yeah, you, you, you need to know that it, it, it will spend a lot of, uh, lot of tokens.

[17:43](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=1063s) And also, uh, for all of those problems, you need to remember that, uh, you need to give access to all kind of tools, uh, that is needed. So you need to have a log, trajectories and metrics, database, UI as, uh, so as the humans need, right?

[17:59](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=1079s) So you need to give all context, all possibilities to understand what is happening. For example, for the computer use agents, when detect some problems, it, uh, should be able to, uh, analyze the trajectories, check the, uh, database to understand what happened.

[18:16](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=1096s) And you know, that's why I'm calling it the meta harness, the whole system when everything is connected and, uh, the PR or the answer from the agents are, uh, depend on the, depend on the real problem and they're not guessing what is happening.

[18:33](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=1113s) So the moat, uh, is not the model alone. And you need to build, uh, the agent or a system or a harness around it, uh, which, which, uh, watch itself, understand, improve, and help you at least to, uh, at least to monitor what is happening.

[18:50](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=1130s) But also in the ideal case, to close the loop, send, uh, automatic PR notifications wherever and, uh, help you to speed up the process. So shipping is the easiest part today.

[19:03](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=1143s) Uh, if you want to ru-- If you want to build a production agent, you need to close the loop first because somehow people are not talking about how important it is, uh, what happens after you launch.

[19:13](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=1153s) So everyone can have the same model, everyone can have the same agent or harness. But you need to have some internal system that helps you in this process, uh, detect the problems, uh, and give you a sense what is happening and helps to make the product better.

[19:31](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=1171s) Thank you.
