AI Engineer Europe 2026

Bending a Public MCP Server Without Breaking It — Nimrod Hauser, Baz

Loading slides…

Read the talk

Bending a Public MCP Server Without Breaking It

A browser-based Spec Reviewer shows how tool filtering, better descriptions, path checks, evidence capture, and deterministic login change an agent’s behavior.

From a talk by Nimrod Hauser

Before you start: Familiarity with Python async functions, tool-calling agents, and MCP clients will help you follow the wrappers and execution flow.

Why can adding tools make an agent worse?

A third-party tool should make an agent more capable. What happens when it instead makes the application less reliable? Nimrod Hauser opens with an MCP server apparently on fire—a running repair metaphor for an integration that needs more than an unmodified tool list. At Baz, he works on AI code reviewers and other agentic features for developers and product managers. He introduces himself as a founding engineer there since 2023, with roughly twenty years in backend and data, including Salesforce and startups. The practical question is how to adapt someone else’s tools to the application using them.

Agenda slide with three topic boxes beside an illustration of robots gathered in a workshop.Open full source frame
Third-party tool problems, five integration practices, and the Spec Reviewer walkthrough.

An agentic tool is a callable function paired with a description. The function performs the operation; the description tells the agent when and how to request it. That distinction applies whether the integration comes from an MCP server, a library, or copied code. The worked example uses Playwright MCP: browser integration code whose descriptions must accommodate many different applications.

Generic tools can produce unexpected actions, incorrect results, or unnecessarily awkward execution. They can also expose security problems. In a multi-tenant application, an agent may not understand which folders, databases, or schemas belong to which customer. Giving it a broadly capable integration without enforcing those boundaries can allow one client’s data to reach another. Application-specific constraints must survive the tool call, even when the tool’s author knows nothing about the application’s architecture.

0:000:21
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:00 · section reference included

From a ticket and design to browser evidence

Baz’s Spec Reviewer compares requirements with implementation. Its workflow begins by reading tickets from systems such as Jira or Linear and examining visual designs from Figma. It then starts Playwright MCP, opens the application for the branch under review, and checks whether the implementation satisfies the requirements. The output is a verdict supported by screenshot evidence, automating repetitive validation work that would otherwise fall to a product manager. Here, opening a branch’s application is the browser-inspection step; the demonstration does not show provisioning that application’s environment.

The Python toy project keeps that workflow small. It defines a screenshot output directory, configures one Playwright MCP server, and initializes an MCP client. A base class exposes get_tools; successive implementations of that method supply increasingly customized tool sets. After loading tools, main calls login_to_baz. Only then does it create the LLM and agent, supply system and human messages, and invoke the review. The reason for keeping login outside the agent comes later.

VS Code showing the main function with client initialization, reviewer selection, tool loading, and authentication calls above terminal output.Open full source frame
The demo initializes its MCP client, selects a reviewer, and loads tools.

The system message assigns a meticulous QA role and a concrete procedure:

  1. Read and understand the ticket.
  2. Navigate through the application and inspect the implementation.
  3. Return a pass or fail verdict with specific observations.
  4. Reference screenshots as evidence.

The human message supplies images of the ticket and its accompanying design. Using a ticket screenshot is a shortcut in this toy project, not the real product’s ticket-ingestion mechanism. The task is to find the Spec Reviewer configuration drawer inside Baz’s Agents tab and check that it resembles the supplied design.

4:515:05
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:51 · section reference included

The baseline mistakes failed navigation for a failed requirement

Version v0 loads the server’s tools through LangChain’s MCP adapters, using load_mcp_tools without modification. At the baseline breakpoint, the demo exposes 21 tools. These cover browser operations such as closing or resizing a page, handling dialogs, uploading files, filling forms, and pressing keys. Their descriptions are correspondingly broad: browser_resize says “Resize the browser window,” while browser_close says “Close the page.” Playwright cannot know the particular workflow this reviewer needs to complete.

The browser starts, the deterministic login completes, and the agent takes over on Baz’s Changes homepage. It needs to find the Agents tab. Instead, the run ends with a failed verdict: the requirement is supposedly unimplemented because the agent could not navigate to an apparently invented Baz.co/spec-reviewer address. Its observation is about a failed navigation attempt, not the requested drawer. The result also references a 404 screenshot, but inspection of the screenshots folder shows that the evidence was not saved properly. The baseline has failed both to locate the feature and to preserve usable support for its verdict.

The repair proceeds through five interventions: curate the available tools, wrap their descriptions, enforce deterministic guardrails, compose a task-specific tool, and call selected functions directly outside the agent loop. The first changes shape the agent’s context; the others give application code more control over execution. All reuse the existing third-party integration code. The useful boundary is whichever division of guidance and enforcement produces the required application behavior.

10:2610:38
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:26 · section reference included

v1: Remove choices the reviewer does not need

The v1 implementation changes get_tools by adding a tool-name exclusion list and filtering the loaded tools with a Python list comprehension. For this reviewer, Hauser excludes operations such as resizing the browser, dragging elements, and autonomously running code inside the page. Those capabilities may be useful elsewhere; they are unnecessary choices for this workflow. The filtering mechanism is small enough to keep explicit:

python

from langchain_mcp_adapters.tools import load_mcp_tools

async def get_curated_tools(session, excluded_names: set[str]):
    tools = await load_mcp_tools(session)
    return [tool for tool in tools if tool.name not in excluded_names]

The application supplies the exclusion set based on the tools its task actually needs.

At the curated breakpoint, the debugger shows 16 tools. This reduces the inventory presented to the agent and the number of operations it must choose among. It does not imply that every improvement should shorten the context: the next version deliberately adds more guidance to the tools that remain.

VS Code paused after loading tools, with the expanded tool list and a length of 16 in the variables pane.Open full source frame
The debugger shows a filtered list containing 16 tools.
15:5516:08
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

15:55 · section reference included

v2: Teach the agent how to inspect before acting

Version v2 introduces ToolWrapper.wrap_playwright_tools. Alongside the tool names, it maintains a dictionary of enhanced descriptions. These descriptions encode experience with the reviewer’s particular navigation task, including which tool should precede another.

The crucial distinction is between an accessibility snapshot and a visual screenshot:

Tool outputWhat it providesRole in the review
Accessibility snapshotText describing page controls, including buttons and menu itemsUnderstand the page before interacting
Visual screenshotAn image of the rendered pageInspect appearance and preserve evidence

Despite its name, browser_snapshot produces the accessibility representation. Hauser finds that this gives the agent a useful understanding of the current page. The replacement descriptions tell it to take that snapshot before hovering or clicking, and to prefer it over an image when trying to understand the page’s controls.

The wrapper iterates over the original tools, filters unwanted entries, looks up each remaining tool’s enhanced description, and creates a replacement tool. Its callable still invokes the original function. The execution stays the same; the model-facing instructions change. At the breakpoint, the descriptions are visibly longer, including the instruction attached to browser_click to inspect the page first. The version also filters additional tools, so the smaller inventory and richer descriptions operate together.

18:0618:30
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

18:06 · section reference included

v3: Reject an invalid screenshot destination before execution

Descriptions guide decisions, but sensitive operations need enforcement. An agent can ignore instructions even when those instructions are present. Hauser invokes needle-in-a-haystack behavior and Lost in the Middle as reasons to be cautious about relying on context alone; they do not diagnose the earlier navigation failure. For this demonstration, the enforceable rule is simple: visual screenshots must be saved under the configured output directory.

Version v3 retains filtering and enhanced descriptions, then adds PathValidation inside create_playwright_tool_wrapper. Before forwarding a screenshot request, the wrapper extracts its proposed destination and checks whether it lies under the screenshots root. The historical demonstration checks path or filename; current Playwright MCP documents filename for browser_take_screenshot. The transferable mechanism is the check before invocation, rather than an assumption that both argument names belong to today’s schema.

For a concrete illustration, let the permitted directory be /work/screenshots and the proposed filename be ../outside.png. Resolving that destination produces /work/outside.png, outside the permitted directory. The wrapper should reject it before calling the screenshot function:

python

from pathlib import Path


def validate_destination(root: Path, filename: str) -> Path:
    root = root.resolve()
    candidate = Path(filename)
    destination = (
        candidate if candidate.is_absolute() else root / candidate
    ).resolve()
    if not destination.is_relative_to(root):
        raise PermissionError(f"Save screenshots under {root}")
    return destination


async def save_screenshot(original_tool, root: Path, filename: str):
    try:
        destination = validate_destination(root, filename)
    except PermissionError as error:
        return f"Access denied: {error}. Provide a valid filename."
    return await original_tool.ainvoke({"filename": str(destination)})

An invalid request never reaches the original tool. The internal validation error becomes an agent-facing response that names the allowed destination and asks for corrected arguments, rather than terminating the whole review. A retry with a filename such as drawer.png can then pass validation; the initial rejection does not itself execute that retry. Hauser expects this feedback to help the agent recover. This is a bounded destination guardrail, not a complete filesystem or tenant-isolation boundary.

A rejected destination never reaches the screenshot tool

Constructed example: The paths /work/screenshots, ../outside.png, and /work/outside.png, and the exact response text, are teaching values shared with the adjacent Python example; they are not values shown in the recording.

Proposed screenshot argument — unchanged
filename = ../outside.png

Operation: Resolve the proposed destination against /work/screenshots, check containment, and reject the request before invocation.

Allowed directory

Before: Request pending
/work/screenshots
After: Validation rejected · Unchanged
/work/screenshots

Requested filename

Before: Request pending
../outside.png
After: Validation rejected · Unchanged
../outside.png

Resolved destination

Before: Request pending
Not present
After: Validation rejected · Added
/work/outside.png

Underlying screenshot call

Before: Request pending
Not invoked; awaiting validation
After: Validation rejected · Changed
Not invoked; destination is outside the allowed directory

Agent-facing response

Before: Request pending
Not present
After: Validation rejected · Added
Access denied: Save screenshots under /work/screenshots. Provide a valid filename.
The wrapper blocks the write while leaving the agent able to propose a corrected filename.
22:5323:09
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

22:53 · section reference included

v4: Give evidence capture its own tool

A screenshot taken while exploring the application and a screenshot saved as final review evidence use much the same browser functionality, but serve different purposes. Version v4 gives evidence capture a separate tool. Its distinct description allows the agent to select it at the end of the review, while leaving general screenshot capture available for other uses. A dedicated wrapper also creates a place for purpose-specific actions before the underlying screenshot call.

The implementation keeps ToolWrapper and PathValidation, completes the existing filtering and wrapping loop, and then injects the evidence tool. In this demo it adds the new tool only if the underlying screenshot tool has not been filtered out. That is an explicit dependency policy chosen for this application.

The new description aligns with the prompt’s final evidence-taking step. It reserves the tool for evidence capture and instructs the agent to identify the relevant ticket and include its number in the saved image’s filename. That gives the agent both a selection rule and a naming convention. The same underlying operation can therefore support two different task contracts:

  • General screenshots: Capture an image when the review needs one during exploration.
  • Evidence screenshots: Capture support for the final verdict and name it so it can be associated with the ticket.

The wrapper can also host additional deterministic checks or preparation specific to evidence capture. In the demonstrated description, ticket-aware naming is the concrete distinction.

VS Code showing an evidence screenshot description with usage restrictions, ticket identification steps, and screenshot naming instructions.Open full source frame
The evidence screenshot tool specifies when to use it and how to name saved images.
28:4528:54
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

28:45 · section reference included

Call mandatory setup functions directly

The fifth practice returns to login_to_baz, which has been present since the baseline. A tool’s callable remains useful without asking an agent to interpret its description. In main.py, the application obtains the MCP tools and runs login before creating and invoking the agent. This keeps a mandatory setup step under deterministic control.

Authentication is a good candidate because every Spec Reviewer run needs it, while different client systems can have different login mechanisms and secrets. Hauser reports subpar results when leaving this work to the agent. The workflow therefore reuses the MCP functions directly, without delegating the choice or sequence of login actions.

The toy implementation reads JWT tokens from environment variables, selects the browser tools it needs, injects the tokens into browser local storage, and clicks Login. Hauser explicitly distinguishes this shortcut from the real product’s authentication. Once the browser is authenticated, control passes to the agent to carry out the review. The application avoids spending the agent’s reasoning and context on a setup sequence it already knows must happen.

32:5333:06
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

32:53 · section reference included

The rerun finds the drawer and saves evidence

With the tool changes in place, Hauser removes the breakpoint and reruns the same ticket-and-design task. The JWT login runs first, and the browser reaches Changes. The agent then needs to navigate to Agents and find the configuration drawer. Hauser notes latency and lag in the browser display, so the visible navigation is not a complete account of each action. The run finishes and reports a pass: the drawer is present and contains the required sections.

VS Code displaying a drawer image above terminal output labeled Agent result and Status: PASS.Open full source frame
The completed run reports PASS beneath a configuration drawer reference image.

The result includes reasoning and a screenshot whose filename contains the ticket identifier. On opening the evidence, Hauser points out that the supplied design was in dark mode while the inspected application was in light mode. The toy reviewer found the drawer and its expected contents; that does not establish pixel-perfect matching. Hauser says the real product supports finer visual verification of padding, margins, and styling. The recording presents a failed baseline and a successful improved run, without a repeated-trial reliability measurement or pixel-level accuracy benchmark.

The final design trade-off is between useful context and necessary control. Removing irrelevant tools reduces the agent’s choices; lengthening the remaining descriptions can make those choices better informed. Deterministic checks enforce constraints at execution time, while direct calls remove work that does not need agentic decision-making. The appropriate balance depends on the task, whether the integration comes from MCP, a library, or another source. The goal is to shape the available functions into the interface the application actually needs.

36:3836:50
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

36:38 · section reference included

Resources

From the talk

Updates since the talk

  • July 2026 announcement of branch-specific application sandboxes and a limited technical preview of private review in customer Kubernetes environments.

Transcript download
Full transcript

Loading transcript…