AI Engineer Europe 2026
Bending a Public MCP Server Without Breaking It — Nimrod Hauser, Baz
Loading slides…
Read the talk
Bending a Public MCP Server Without Breaking It
A browser-based Spec Reviewer shows how tool filtering, better descriptions, path checks, evidence capture, and deterministic login change an agent’s behavior.
From a talk by Nimrod Hauser
Before you start: Familiarity with Python async functions, tool-calling agents, and MCP clients will help you follow the wrappers and execution flow.
Why can adding tools make an agent worse?
A third-party tool should make an agent more capable. What happens when it instead makes the application less reliable? Nimrod Hauser opens with an MCP server apparently on fire—a running repair metaphor for an integration that needs more than an unmodified tool list. At Baz, he works on AI code reviewers and other agentic features for developers and product managers. He introduces himself as a founding engineer there since 2023, with roughly twenty years in backend and data, including Salesforce and startups. The practical question is how to adapt someone else’s tools to the application using them.
An agentic tool is a callable function paired with a description. The function performs the operation; the description tells the agent when and how to request it. That distinction applies whether the integration comes from an MCP server, a library, or copied code. The worked example uses Playwright MCP: browser integration code whose descriptions must accommodate many different applications.
Generic tools can produce unexpected actions, incorrect results, or unnecessarily awkward execution. They can also expose security problems. In a multi-tenant application, an agent may not understand which folders, databases, or schemas belong to which customer. Giving it a broadly capable integration without enforcing those boundaries can allow one client’s data to reach another. Application-specific constraints must survive the tool call, even when the tool’s author knows nothing about the application’s architecture.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From a ticket and design to browser evidence
Baz’s Spec Reviewer compares requirements with implementation. Its workflow begins by reading tickets from systems such as Jira or Linear and examining visual designs from Figma. It then starts Playwright MCP, opens the application for the branch under review, and checks whether the implementation satisfies the requirements. The output is a verdict supported by screenshot evidence, automating repetitive validation work that would otherwise fall to a product manager. Here, opening a branch’s application is the browser-inspection step; the demonstration does not show provisioning that application’s environment.
The Python toy project keeps that workflow small. It defines a screenshot output directory, configures one Playwright MCP server, and initializes an MCP client. A base class exposes get_tools; successive implementations of that method supply increasingly customized tool sets. After loading tools, main calls login_to_baz. Only then does it create the LLM and agent, supply system and human messages, and invoke the review. The reason for keeping login outside the agent comes later.
The system message assigns a meticulous QA role and a concrete procedure:
- Read and understand the ticket.
- Navigate through the application and inspect the implementation.
- Return a pass or fail verdict with specific observations.
- Reference screenshots as evidence.
The human message supplies images of the ticket and its accompanying design. Using a ticket screenshot is a shortcut in this toy project, not the real product’s ticket-ingestion mechanism. The task is to find the Spec Reviewer configuration drawer inside Baz’s Agents tab and check that it resembles the supplied design.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The baseline mistakes failed navigation for a failed requirement
Version v0 loads the server’s tools through LangChain’s MCP adapters, using load_mcp_tools without modification. At the baseline breakpoint, the demo exposes 21 tools. These cover browser operations such as closing or resizing a page, handling dialogs, uploading files, filling forms, and pressing keys. Their descriptions are correspondingly broad: browser_resize says “Resize the browser window,” while browser_close says “Close the page.” Playwright cannot know the particular workflow this reviewer needs to complete.
The browser starts, the deterministic login completes, and the agent takes over on Baz’s Changes homepage. It needs to find the Agents tab. Instead, the run ends with a failed verdict: the requirement is supposedly unimplemented because the agent could not navigate to an apparently invented Baz.co/spec-reviewer address. Its observation is about a failed navigation attempt, not the requested drawer. The result also references a 404 screenshot, but inspection of the screenshots folder shows that the evidence was not saved properly. The baseline has failed both to locate the feature and to preserve usable support for its verdict.
The repair proceeds through five interventions: curate the available tools, wrap their descriptions, enforce deterministic guardrails, compose a task-specific tool, and call selected functions directly outside the agent loop. The first changes shape the agent’s context; the others give application code more control over execution. All reuse the existing third-party integration code. The useful boundary is whichever division of guidance and enforcement produces the required application behavior.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
v1: Remove choices the reviewer does not need
The v1 implementation changes get_tools by adding a tool-name exclusion list and filtering the loaded tools with a Python list comprehension. For this reviewer, Hauser excludes operations such as resizing the browser, dragging elements, and autonomously running code inside the page. Those capabilities may be useful elsewhere; they are unnecessary choices for this workflow. The filtering mechanism is small enough to keep explicit:
python
from langchain_mcp_adapters.tools import load_mcp_tools
async def get_curated_tools(session, excluded_names: set[str]):
tools = await load_mcp_tools(session)
return [tool for tool in tools if tool.name not in excluded_names]
The application supplies the exclusion set based on the tools its task actually needs.
At the curated breakpoint, the debugger shows 16 tools. This reduces the inventory presented to the agent and the number of operations it must choose among. It does not imply that every improvement should shorten the context: the next version deliberately adds more guidance to the tools that remain.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
v2: Teach the agent how to inspect before acting
Version v2 introduces ToolWrapper.wrap_playwright_tools. Alongside the tool names, it maintains a dictionary of enhanced descriptions. These descriptions encode experience with the reviewer’s particular navigation task, including which tool should precede another.
The crucial distinction is between an accessibility snapshot and a visual screenshot:
| Tool output | What it provides | Role in the review |
|---|---|---|
| Accessibility snapshot | Text describing page controls, including buttons and menu items | Understand the page before interacting |
| Visual screenshot | An image of the rendered page | Inspect appearance and preserve evidence |
Despite its name, browser_snapshot produces the accessibility representation. Hauser finds that this gives the agent a useful understanding of the current page. The replacement descriptions tell it to take that snapshot before hovering or clicking, and to prefer it over an image when trying to understand the page’s controls.
The wrapper iterates over the original tools, filters unwanted entries, looks up each remaining tool’s enhanced description, and creates a replacement tool. Its callable still invokes the original function. The execution stays the same; the model-facing instructions change. At the breakpoint, the descriptions are visibly longer, including the instruction attached to browser_click to inspect the page first. The version also filters additional tools, so the smaller inventory and richer descriptions operate together.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
v3: Reject an invalid screenshot destination before execution
Descriptions guide decisions, but sensitive operations need enforcement. An agent can ignore instructions even when those instructions are present. Hauser invokes needle-in-a-haystack behavior and Lost in the Middle as reasons to be cautious about relying on context alone; they do not diagnose the earlier navigation failure. For this demonstration, the enforceable rule is simple: visual screenshots must be saved under the configured output directory.
Version v3 retains filtering and enhanced descriptions, then adds PathValidation inside create_playwright_tool_wrapper. Before forwarding a screenshot request, the wrapper extracts its proposed destination and checks whether it lies under the screenshots root. The historical demonstration checks path or filename; current Playwright MCP documents filename for browser_take_screenshot. The transferable mechanism is the check before invocation, rather than an assumption that both argument names belong to today’s schema.
For a concrete illustration, let the permitted directory be /work/screenshots and the proposed filename be ../outside.png. Resolving that destination produces /work/outside.png, outside the permitted directory. The wrapper should reject it before calling the screenshot function:
python
from pathlib import Path
def validate_destination(root: Path, filename: str) -> Path:
root = root.resolve()
candidate = Path(filename)
destination = (
candidate if candidate.is_absolute() else root / candidate
).resolve()
if not destination.is_relative_to(root):
raise PermissionError(f"Save screenshots under {root}")
return destination
async def save_screenshot(original_tool, root: Path, filename: str):
try:
destination = validate_destination(root, filename)
except PermissionError as error:
return f"Access denied: {error}. Provide a valid filename."
return await original_tool.ainvoke({"filename": str(destination)})
An invalid request never reaches the original tool. The internal validation error becomes an agent-facing response that names the allowed destination and asks for corrected arguments, rather than terminating the whole review. A retry with a filename such as drawer.png can then pass validation; the initial rejection does not itself execute that retry. Hauser expects this feedback to help the agent recover. This is a bounded destination guardrail, not a complete filesystem or tenant-isolation boundary.
A rejected destination never reaches the screenshot tool
Constructed example: The paths /work/screenshots, ../outside.png, and /work/outside.png, and the exact response text, are teaching values shared with the adjacent Python example; they are not values shown in the recording.
filename = ../outside.png
Operation: Resolve the proposed destination against /work/screenshots, check containment, and reject the request before invocation.
Allowed directory
/work/screenshots
/work/screenshots
Requested filename
../outside.png
../outside.png
Resolved destination
Not present
/work/outside.png
Underlying screenshot call
Not invoked; awaiting validation
Not invoked; destination is outside the allowed directory
Agent-facing response
Not present
Access denied: Save screenshots under /work/screenshots. Provide a valid filename.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
v4: Give evidence capture its own tool
A screenshot taken while exploring the application and a screenshot saved as final review evidence use much the same browser functionality, but serve different purposes. Version v4 gives evidence capture a separate tool. Its distinct description allows the agent to select it at the end of the review, while leaving general screenshot capture available for other uses. A dedicated wrapper also creates a place for purpose-specific actions before the underlying screenshot call.
The implementation keeps ToolWrapper and PathValidation, completes the existing filtering and wrapping loop, and then injects the evidence tool. In this demo it adds the new tool only if the underlying screenshot tool has not been filtered out. That is an explicit dependency policy chosen for this application.
The new description aligns with the prompt’s final evidence-taking step. It reserves the tool for evidence capture and instructs the agent to identify the relevant ticket and include its number in the saved image’s filename. That gives the agent both a selection rule and a naming convention. The same underlying operation can therefore support two different task contracts:
- General screenshots: Capture an image when the review needs one during exploration.
- Evidence screenshots: Capture support for the final verdict and name it so it can be associated with the ticket.
The wrapper can also host additional deterministic checks or preparation specific to evidence capture. In the demonstrated description, ticket-aware naming is the concrete distinction.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Call mandatory setup functions directly
The fifth practice returns to login_to_baz, which has been present since the baseline. A tool’s callable remains useful without asking an agent to interpret its description. In main.py, the application obtains the MCP tools and runs login before creating and invoking the agent. This keeps a mandatory setup step under deterministic control.
Authentication is a good candidate because every Spec Reviewer run needs it, while different client systems can have different login mechanisms and secrets. Hauser reports subpar results when leaving this work to the agent. The workflow therefore reuses the MCP functions directly, without delegating the choice or sequence of login actions.
The toy implementation reads JWT tokens from environment variables, selects the browser tools it needs, injects the tokens into browser local storage, and clicks Login. Hauser explicitly distinguishes this shortcut from the real product’s authentication. Once the browser is authenticated, control passes to the agent to carry out the review. The application avoids spending the agent’s reasoning and context on a setup sequence it already knows must happen.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The rerun finds the drawer and saves evidence
With the tool changes in place, Hauser removes the breakpoint and reruns the same ticket-and-design task. The JWT login runs first, and the browser reaches Changes. The agent then needs to navigate to Agents and find the configuration drawer. Hauser notes latency and lag in the browser display, so the visible navigation is not a complete account of each action. The run finishes and reports a pass: the drawer is present and contains the required sections.
The result includes reasoning and a screenshot whose filename contains the ticket identifier. On opening the evidence, Hauser points out that the supplied design was in dark mode while the inspected application was in light mode. The toy reviewer found the drawer and its expected contents; that does not establish pixel-perfect matching. Hauser says the real product supports finer visual verification of padding, margins, and styling. The recording presents a failed baseline and a successful improved run, without a repeated-trial reliability measurement or pixel-level accuracy benchmark.
The final design trade-off is between useful context and necessary control. Removing irrelevant tools reduces the agent’s choices; lengthening the remaining descriptions can make those choices better informed. Deterministic checks enforce constraints at execution time, while direct calls remove work that does not need agentic decision-making. The appropriate balance depends on the task, whether the integration comes from MCP, a library, or another source. The goal is to shape the available functions into the interface the application actually needs.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Official browser automation MCP server, with installation instructions, accessibility snapshots, screenshots, and configuration documentation.
Python adapter library with examples for loading MCP tools into LangChain agents and managing MCP sessions.
Further reading
Baz’s product introduction explains how design references and ticket requirements inform implementation reviews and visual findings.
Original research on how the position of relevant information affects long-context question answering and key-value retrieval.
Updates since the talk
July 2026 announcement of branch-specific application sandboxes and a limited technical preview of private review in customer Kubernetes environments.
Full transcript
Loading transcript…