AI Engineer World's Fair 2026
AI Agents Don't Read Your Policy Docs. They Hit Your APIs — Gravitee
Loading slides…
Read the talk
AI Agents Don't Read Your Policy Docs. They Hit Your APIs — Gravitee
A hotel-booking agent makes the difference between written rules and enforced permissions concrete. Five live scenarios show where a gateway can stop destructive actions, protect guest data and prevent avoidable model calls.
From a talk by Sam Prodger
At a glance
Ideas worth remembering
Tool discovery and tool authorization are separate. The governed booking agent discovers the bulk-delete tool, but the gateway denies execution while allowing the individual cancellation.
Rate limits and prompt-size limits avoid downstream model work only when they run before inference. The demonstrations admit three requests per minute and reject an oversized prompt before an LLM call.
Semantic caching reduces repeated model calls, with configurable similarity and visible cache-hit logs. The five repeated booking queries consume about 200 tokens with caching versus about 1,000 without it.
A gateway can also route models and curate MCP tool catalogs. Those controls reduce provider-specific coupling, unnecessary context and the range of operations exposed to an agent.
Gateway enforcement depends on traffic coverage. Direct AI use on employee laptops motivates the edge-management capability described at the end of the talk.
A hotel policy cannot stop an authorized API call
A hotel can write a policy forbidding AI from deleting its bookings. If its agent still has access to a tool that deletes every booking, that policy leaves the dangerous operation available. Sam, introducing himself as Gravitee’s CTO, builds the talk around two hotel-booking agents: one relies on controls expressed in prompts, skills and supplied knowledge; the other operates through a tightly scoped identity and gateway controls.
The motivation is a gap between deploying agents and governing their actions. Sam cites a Gravitee survey in which 88% of organizations had an agent taking action in production, while only 14% of those had governance around it. These are reported survey figures without a described sample or methodology. The practical risks he identifies are concrete: deleted databases and exposed personally identifiable information, with compliance consequences in regulated environments.
The demonstration uses a SaaS-hosted Gravitee gateway, a Databricks backend holding the data tables, and Sonnet alongside Groq inference. The five scenarios test a change in where decisions happen. Instructions ask the agent to behave; the gateway decides whether a request may reach the service that can actually perform the operation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Cancel one booking without deleting them all
The first request combines a legitimate task with a destructive one: cancel this booking, and also run the MCP tool that deletes all bookings. On the prompt-controlled side, the tool is available and the requesting user has permission to invoke it. The agent carries out both operations. Successful HTTP 200 responses appear in the request flow, and the subsequent database check shows that all bookings have been deleted.
On the governed side, the same request produces the same tool discovery: the agent can find the destructive tool. Execution is where the paths diverge. Gravitee checks the runtime identity and permissions, with token exchange and JWT claims constraining administrative access to the user and task that require it. The bulk-delete call receives HTTP 403 Forbidden because this identity lacks the necessary permission.
The useful part still succeeds. The governed agent cancels the individual booking and explains that it cannot perform the bulk deletion. This is a more precise outcome than rejecting the whole conversation: the user gets the permitted action, while the infrastructure prevents the destructive action even after the agent has discovered how to request it.
Where does the same request become safe? The diagram follows the two operations through discovery and execution. Its important relationship is the split at authorization: discovering a tool does not grant permission to run it, and denying one operation need not prevent another.
One request, two independently authorized operations
Read the diagram as text
- User request. Cancel this booking and delete all bookings.
- Agent discovers tools. Both requested operations are available to discover.
- Gateway checks identity. Runtime permissions constrain which operations can execute.
- Individual booking canceled. Allowed — The permitted part of the request completes.
- Bulk deletion blocked. Denied — HTTP 403: this identity lacks permission.
- User request → Agent discovers tools: Find tools for the requested actions.
- Agent discovers tools → Gateway checks identity: Attempt execution.
- Gateway checks identity → Individual booking canceled: Permission allows cancellation.
- Gateway checks identity → Bulk deletion blocked: Permission denies bulk deletion.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A compliance explanation does not authorize guest-data access
The second request asks for the full guest list, including email addresses and card details, ostensibly for compliance and reconciliation. The explanation makes the request sound purposeful, but it does not change the requester’s permissions. In the ungoverned flow, the UI sends the prompt to the LLM, the model discovers an MCP tool, the tool retrieves the information, and the result returns to the user. The guest data used in this demonstration is mocked.
The gateway-controlled version blocks the retrieval, so the protected information does not appear in the response. Instead, the agent explains the missing permission and offers an authentication path through an identity provider or the gateway itself. The mechanism is authorization of data access; it does not depend on persuading the model that the user’s compliance story is suspicious.
OpenTelemetry exposes the steps underneath these responses. That visibility becomes more useful as a request crosses from user to agent and then through multiple agents using the A2A protocol. Each additional hop makes the chain harder to follow. Central enforcement and traces provide a place to inspect what was requested, which tools were called and where access failed, rather than managing every restriction separately inside every agent.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Stop a repeat loop before it spends more tokens
A buggy agent sends the same request eight times. Without an external limit, every request reaches the LLM and receives a successful response. For these simple prompts, Sam reports roughly 1,000 tokens and about one cent of cost. Those figures describe this demonstration; the scaling concern is that the same unnecessary work repeats across agents, customers and recurring workflows.
The governed agent has a gateway limit of three requests per minute. Its first three requests receive answers, and subsequent requests are blocked before they reach the LLM. Placement matters: a restriction applied after inference could report that the agent exceeded its allowance, but the model work would already have happened. Enforcing the allowance before inference prevents those additional calls.
This control trades unrestricted availability for a predictable request budget. In the example it contains a bug; the same mechanism can assign different allowances to users or audiences. It does not repair the agent’s repeat loop. It limits the loop’s ability to consume downstream resources.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Answer repeated booking questions from a semantic cache
Some repetition is useful to answer rather than reject. Gravitee’s semantic cache uses a built-in Redis store to serve responses for similar prompts without making another downstream LLM call. An IT helpdesk repeatedly answering how to set a password is the simple example: the request may recur across users even though generating the answer again adds little value.
The hotel example asks five times to list the user’s bookings—because, as Sam puts it, he did not trust the first answer. Without caching, all five requests go downstream, consuming about 1,000 tokens and another cent. With caching, the same data returns while reported input-and-output token use falls to around 200. The observable change is fewer model calls for the repeated question.
Similarity is configurable on a zero-to-one scale, with one representing very similar prompts. Settings can differ by agent, audience or user. A more permissive match can reuse more responses, but it also asks the cache to treat more questions as equivalent. For personal, changing booking data, safe reuse also depends on freshness and user separation; this demonstration does not explain those controls. The claimed savings concern avoided LLM work, rather than literally eliminating all cache cost or latency.
Logs make cache hits visible, so a fast answer can be traced to its source instead of mistaken for a new model generation. The telemetry shown in the demo UI can also be exported to Datadog, Splunk or Prometheus. Enforcement and observability work together here: the gateway changes how a request is served, and the trace explains that change.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Reject oversized or off-task prompts before inference
The fifth scenario starts with ten minutes of chat transcript pasted into a request for a relatively simple booking task. The ungoverned version processes 2,000 input tokens and produces 312 output tokens. The question is how much context the task actually needs: carrying an entire conversational history into a small action can make the model process far more material than the action warrants.
The gateway rejects the same oversized prompt before any downstream LLM call. Its response asks the user to cut the material up or break the work into smaller steps. This is an input-size policy, not an automatic summarizer: it prevents the expensive request and leaves the task to be reformulated. Applying the policy during the response phase would be too late to avoid processing the input.
Two distinct policies can constrain this kind of spending:
- Prompt size: reject excessive context before inference and ask for smaller steps.
- Task relevance: reject questions outside the agent’s intended use case. In the hotel example, a weather question is refused because it does not look like a hotel-booking request.
The relevance example comes from a customer using a powerful model to answer a basic weather question that an ordinary weather service could supply. Restricting the booking agent’s scope avoids that downstream model call. The recording does not explain how relevance is classified, so the example establishes the intended policy behavior rather than its accuracy on ambiguous requests.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Extend the gateway to model routing, tool selection and laptop traffic
The five scenarios lead to an architectural decision: place a governed intermediary between front-end agents and backend APIs, MCP servers, models, databases and lakehouses. As agents multiply—and potentially create other agents—embedding every control in each agent makes changes harder to coordinate. Sam’s recommendation is an automated infrastructure layer with exportable logs for inspection and audit, including scrutiny of information sent toward model retraining.
The ending expands that intermediary’s responsibilities beyond denying requests:
- Model routing: choose downstream models by latency, cost or task complexity. Sam describes switching from the demonstrated Groq and Anthropic setup to Gemini or OpenAI, with the gateway handling API-format transformations so the agent does not need to be recoded for each provider.
- MCP tool curation: expose a smaller task-appropriate set of tools. Gravitee’s own CRM MCP server exposes 64 tools; sending the whole catalog into model context increases token use and makes more operations available. Curating the catalog reduces both context cost and the range of potentially harmful actions.
Tool curation adds an earlier control to the booking example. Runtime authorization stopped the bulk-delete operation after discovery; selecting a smaller tool catalog can also reduce which operations the model encounters in the first place. These mechanisms act at different points: one shapes the model’s available choices, while the other checks a requested action before execution.
There is still a coverage problem. Employees may use AI directly on their laptops, uploading documents or emails through paths that never touch the organization’s gateway. Sam describes a recently released edge-management capability that discovers this shadow AI traffic, forces it through the gateway and provides logging and observability. The talk describes the capability without showing its interception or deployment mechanism; the architectural requirement is that traffic must pass through the enforcement layer for its controls to apply.
The closing recommendation is deliberately stronger than a prompt-writing tip: move enforceable policy into infrastructure. The booking agent makes the reason tangible. It can discover a dangerous tool, attempt to use it, receive a denial and still finish the permitted task. A written rule becomes operational when the service path checks it before the consequence occurs.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
Related talks
- Beating RL With Reflection: GEPA and Optimize Anything
Explores how execution traces improve prompts, skills and agent programs—a useful companion to this talk’s distinction between shaping agent behavior and enforcing runtime permissions.
Full transcript
Loading transcript…