AI Engineer World's Fair 2025
Will Agent Evaluation via MCP Stabilize Agent Networks?
Loading slides…
Read the talk
Will Agent Evaluation via MCP Stabilize Agent Networks?
A marketing-message revision and a hotel reservation agent show how evaluator discovery, scores and explanations can become a feedback loop inside an agent.
From a talk by Ari Heljakka
Before you start: Familiarity with LLM agents and tool calling is helpful; no prior experience with MCP or Root Signals is required.
How do you know an agent is making progress?
When an agent cannot finish a task in one shot, how do you know its next action moves it closer to the goal? The question becomes harder for a network of agents: you cannot observe everything, the environment changes, and advance testing cannot cover every situation. Ari Heljakka, introducing himself as CEO of Root Signals, starts with the prospect of stable agent swarms handling broad knowledge work. Stability is the condition that makes that prospect interesting—and the part that remains difficult.
Evaluations offer a way to measure progress, but attaching an evaluation stack is only the beginning. The feedback process must improve both the agent and its evaluators. Checks need systematic use, maintenance and adjustment as the actual business requirements become clearer. Otherwise, adding more measurements does little to make the workflow more dependable.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Evaluate what the agent understands and what it does
An agent has more than one surface to evaluate. Its representation of the world can be wrong even when its actions look reasonable; its actions can be inappropriate even when it has the facts right. Heljakka separates these concerns into two broad groups:
| Evaluation surface | Questions to ask |
|---|---|
| Representation | Does it model reality accurately, communicate it correctly and ground claims in truth? |
| Behavior | Does it infer the right goals, make progress and select appropriate tools? |
Comprehensive coverage would address both groups. A practical starting point is a smaller set of checks that can be applied consistently, rather than an exhaustive evaluation design that never becomes part of the workflow.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn hotel requirements into a feedback loop
A hotel reservation agent makes the evaluation requirements concrete. It needs checks for policy adherence to that hotel's reservation rules, accuracy of its answers to guests, and appropriate behavior overall. The demonstration uses evaluators configured in Root Signals, but the operational requirement applies to any platform: you need to see what each evaluator does, maintain the collection and improve it systematically when dozens of checks run repeatedly.
The stabilization loop turns those checks into information the agent can use:
- The agent attempts a task and produces an output.
- An evaluation engine assesses that output.
- The engine returns a numeric score and an explanation of what went wrong or well.
- The agent uses that feedback to adjust its work.
The score supplies a measurement; the explanation supplies material for a revision. Model Context Protocol, or MCP, connects the agent to the evaluation engine. The corrective information comes from the evaluators reached through that connection.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Discover a judge, then revise a marketing message
The first experiment uses text rather than agent code: “Root Signals MCP Server is awesome, lets you access evals nicely.” Before trying to improve that sentence, the client needs to discover which evaluations are available. In Cursor, Heljakka asks MCP to list his judges. In Root Signals terminology, a judge is a collection of evaluators; individual evaluators can also be listed directly.
The Root Signals MCP server has already been attached to Cursor before the demonstration. It exposes evaluators Heljakka configured earlier, along with general-purpose evaluators. The discovery request finds a marketing message quality judge, providing a relevant evaluation target without requiring him to navigate the evaluator collection manually.
He then asks: “Optimize the message by using the marketing message optimization judge.” Cursor goes back through MCP to find the requested judge and score the sentence. It identifies issues and proposes an improved version based on that feedback, then runs a final evaluation. This is the feedback loop in an inspectable form: discover, evaluate, revise and evaluate again.
The interface does not immediately expose all the scores. Heljakka points out that the user can ask to see them; the displayed dimensions include persuasiveness, quality of writing and engagingness. The final displayed scores are between zero and one. The demonstration shows feedback and a proposed revision, but does not establish a numerical before-and-after improvement.
This manual interaction makes visible the process that could run inside an agent. There are two ways to choose its checks:
- Prescribed evaluation: tell the agent exactly which evaluators or judges to run.
- Agent-selected evaluation: let the agent discover the collection and choose relevant checks itself.
The next demonstration moves that choice from a Cursor conversation into a running agent.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Run the hotel agent with evaluation off, then on
The second experiment uses a hotel reservation agent built with Pydantic AI. Cursor now serves as the environment for viewing the code; it is no longer the MCP client conducting the conversation. Heljakka runs the agent separately in a console.
The agent represents Shire Hotel, next door to ACME Hotel. Its business policy prohibits recommending ACME and preferably avoids mentioning it at all. The test request asks about Shire while hinting at interest in the neighboring hotel. Heljakka describes this as an artificial example: it is designed to expose a precise conflict between responding helpfully to the guest and following the hotel's policy.
With MCP evaluation disabled, the agent politely explains that rooms will be available later—but also calls ACME Hotel a great option. That is the unwanted behavior. Heljakka returns to the configuration, sets the flag controlling MCP evaluation to true, and reruns the same console command.
During the enabled run, the console shows evaluator discovery and repeated calls through the MCP server. Multiple evaluators run, with some called more than once. The resulting response no longer mentions ACME, and the trace shows that the agent invoked the Hotel Shire booking policy evaluator without being explicitly told to select that particular evaluator.
| Demonstrated run | Evaluation activity | Response concerning ACME |
|---|---|---|
| MCP evaluation disabled | No MCP evaluation | Recommends ACME as a great option |
| MCP evaluation enabled | Discovers and invokes evaluators | Does not mention ACME |
The relevant distinction is selection of the particular hotel evaluator, not an agent spontaneously deciding to evaluate without any guidance. Heljakka presents relevant evaluator selection as usually possible, while allowing specific checks to be enforced when required. The observed result demonstrates correction in this hotel example; it does not establish reliable stabilization of an agent network or swarm. Extending the same feedback pattern to more complex agents remains the proposed next step.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make evaluator maintenance part of agent development
Putting this approach into practice starts with the evaluation platform. It needs to support diverse checks and their lifecycle: creating, maintaining and optimizing evaluators as well as the agent. Repeated evaluation has an operating cost, so improving the evaluation stack is also an economic concern. A growing collection of checks must remain useful enough to justify running it.
The adoption sequence follows the demonstrations. First, run evaluations manually, outside the agent's live workflow, as with the marketing message. Inspect how the evaluators behave, what their feedback reveals and where they fail. Then attach them to the agent through MCP. Greater control, transparency and dynamic self-correction are the intended benefits of that sequence, rather than measured guarantees.
Heljakka closes by describing the Root Signals MCP server as freely available and anticipating similar integrations from other evaluation platforms. Free server availability does not make repeated evaluation execution free. The useful capability is a connection through which an agent can receive maintained, task-relevant feedback and act on it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Evaluator and judge tools for MCP clients, now preserved in an archived repository under the Scorable name.
Further reading
A Pydantic AI hotel agent with competitor restrictions, evaluator acceptance criteria and MCP integration.
- MCP tools specificationDocumentation
The March 2025 specification for exposing and invoking tools through MCP.
Updates since the talk
Current setup and lifecycle guidance for connecting Pydantic AI agents to local and remote MCP servers.
Full transcript
Loading transcript…