AI Engineer World's Fair 2025

Will Agent Evaluation via MCP Stabilize Agent Networks?

Loading slides…

Read the talk

Will Agent Evaluation via MCP Stabilize Agent Networks?

A marketing-message revision and a hotel reservation agent show how evaluator discovery, scores and explanations can become a feedback loop inside an agent.

From a talk by Ari Heljakka

Before you start: Familiarity with LLM agents and tool calling is helpful; no prior experience with MCP or Root Signals is required.

How do you know an agent is making progress?

When an agent cannot finish a task in one shot, how do you know its next action moves it closer to the goal? The question becomes harder for a network of agents: you cannot observe everything, the environment changes, and advance testing cannot cover every situation. Ari Heljakka, introducing himself as CEO of Root Signals, starts with the prospect of stable agent swarms handling broad knowledge work. Stability is the condition that makes that prospect interesting—and the part that remains difficult.

Slide listing four challenges to agent swarm stabilization beside an illustration of dogs scattered around an office.Open full source frame
Obstacles to agent swarm stabilization: limited observability, communication constraints, dynamic environments, and uncertain progress.

Evaluations offer a way to measure progress, but attaching an evaluation stack is only the beginning. The feedback process must improve both the agent and its evaluators. Checks need systematic use, maintenance and adjustment as the actual business requirements become clearer. Otherwise, adding more measurements does little to make the workflow more dependable.

0:030:19
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:03 · section reference included

Evaluate what the agent understands and what it does

An agent has more than one surface to evaluate. Its representation of the world can be wrong even when its actions look reasonable; its actions can be inappropriate even when it has the facts right. Heljakka separates these concerns into two broad groups:

Evaluation surfaceQuestions to ask
RepresentationDoes it model reality accurately, communicate it correctly and ground claims in truth?
BehaviorDoes it infer the right goals, make progress and select appropriate tools?

Comprehensive coverage would address both groups. A practical starting point is a smaller set of checks that can be applied consistently, rather than an exhaustive evaluation design that never becomes part of the workflow.

2:222:43
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:22 · section reference included

Turn hotel requirements into a feedback loop

A hotel reservation agent makes the evaluation requirements concrete. It needs checks for policy adherence to that hotel's reservation rules, accuracy of its answers to guests, and appropriate behavior overall. The demonstration uses evaluators configured in Root Signals, but the operational requirement applies to any platform: you need to see what each evaluator does, maintain the collection and improve it systematically when dozens of checks run repeatedly.

The stabilization loop turns those checks into information the agent can use:

  1. The agent attempts a task and produces an output.
  2. An evaluation engine assesses that output.
  3. The engine returns a numeric score and an explanation of what went wrong or well.
  4. The agent uses that feedback to adjust its work.

The score supplies a measurement; the explanation supplies material for a revision. Model Context Protocol, or MCP, connects the agent to the evaluation engine. The corrective information comes from the evaluators reached through that connection.

Stabilization Loop slide with five steps and an illustration connecting an agent dog to an evaluation engine through circular arrows.Open full source frame
The stabilization loop sends agent output to an evaluation engine via MCP and returns scores and explanations for revision.
3:293:47
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:29 · section reference included

Discover a judge, then revise a marketing message

The first experiment uses text rather than agent code: “Root Signals MCP Server is awesome, lets you access evals nicely.” Before trying to improve that sentence, the client needs to discover which evaluations are available. In Cursor, Heljakka asks MCP to list his judges. In Root Signals terminology, a judge is a collection of evaluators; individual evaluators can also be listed directly.

The Root Signals MCP server has already been attached to Cursor before the demonstration. It exposes evaluators Heljakka configured earlier, along with general-purpose evaluators. The discovery request finds a marketing message quality judge, providing a relevant evaluation target without requiring him to navigate the evaluator collection manually.

He then asks: “Optimize the message by using the marketing message optimization judge.” Cursor goes back through MCP to find the requested judge and score the sentence. It identifies issues and proposes an improved version based on that feedback, then runs a final evaluation. This is the feedback loop in an inspectable form: discover, evaluate, revise and evaluate again.

The interface does not immediately expose all the scores. Heljakka points out that the user can ask to see them; the displayed dimensions include persuasiveness, quality of writing and engagingness. The final displayed scores are between zero and one. The demonstration shows feedback and a proposed revision, but does not establish a numerical before-and-after improvement.

Cursor editor showing green additions and a red deletion in a marketing message, with evaluation feedback and four suggested improvements in the right panel.Open full source frame
A proposed marketing-message revision appears beside evaluation feedback and further improvement suggestions.

This manual interaction makes visible the process that could run inside an agent. There are two ways to choose its checks:

  • Prescribed evaluation: tell the agent exactly which evaluators or judges to run.
  • Agent-selected evaluation: let the agent discover the collection and choose relevant checks itself.

The next demonstration moves that choice from a Cursor conversation into a running agent.

5:125:22
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:12 · section reference included

Run the hotel agent with evaluation off, then on

The second experiment uses a hotel reservation agent built with Pydantic AI. Cursor now serves as the environment for viewing the code; it is no longer the MCP client conducting the conversation. Heljakka runs the agent separately in a console.

The agent represents Shire Hotel, next door to ACME Hotel. Its business policy prohibits recommending ACME and preferably avoids mentioning it at all. The test request asks about Shire while hinting at interest in the neighboring hotel. Heljakka describes this as an artificial example: it is designed to expose a precise conflict between responding helpfully to the guest and following the hotel's policy.

With MCP evaluation disabled, the agent politely explains that rooms will be available later—but also calls ACME Hotel a great option. That is the unwanted behavior. Heljakka returns to the configuration, sets the flag controlling MCP evaluation to true, and reruns the same console command.

During the enabled run, the console shows evaluator discovery and repeated calls through the MCP server. Multiple evaluators run, with some called more than once. The resulting response no longer mentions ACME, and the trace shows that the agent invoked the Hotel Shire booking policy evaluator without being explicitly told to select that particular evaluator.

Demonstrated runEvaluation activityResponse concerning ACME
MCP evaluation disabledNo MCP evaluationRecommends ACME as a great option
MCP evaluation enabledDiscovers and invokes evaluatorsDoes not mention ACME

The relevant distinction is selection of the particular hotel evaluator, not an agent spontaneously deciding to evaluate without any guidance. Heljakka presents relevant evaluator selection as usually possible, while allowing specific checks to be enforced when required. The observed result demonstrates correction in this hotel example; it does not establish reliable stabilization of an agent network or swarm. Extending the same feedback pattern to more complex agents remains the proposed next step.

9:039:24
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:03 · section reference included

Make evaluator maintenance part of agent development

Putting this approach into practice starts with the evaluation platform. It needs to support diverse checks and their lifecycle: creating, maintaining and optimizing evaluators as well as the agent. Repeated evaluation has an operating cost, so improving the evaluation stack is also an economic concern. A growing collection of checks must remain useful enough to justify running it.

The adoption sequence follows the demonstrations. First, run evaluations manually, outside the agent's live workflow, as with the marketing message. Inspect how the evaluators behave, what their feedback reveals and where they fail. Then attach them to the agent through MCP. Greater control, transparency and dynamic self-correction are the intended benefits of that sequence, rather than measured guarantees.

Take-aways slide recommending a two-tier platform or library, maintaining dozens of evaluators, manual offline MCP evaluations, and MCP with principled evaluations for agent development.Open full source frame
Takeaways: maintain and optimize evaluators, run MCP evaluations manually offline, and combine MCP with principled evaluations.

Heljakka closes by describing the Root Signals MCP server as freely available and anticipating similar integrations from other evaluation platforms. Free server availability does not make repeated evaluation execution free. The useful capability is a connection through which an agent can receive maintained, task-relevant feedback and act on it.

12:2612:51
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

12:26 · section reference included

Resources

From the talk

Updates since the talk

Transcript download
Full transcript

Loading transcript…