AI Engineer Summit 2025
Balancing Innovation with Security & Safety
Loading slides…
Read the talk
Balancing Innovation with Security & Safety
An agent’s permissions must survive every task and tool call. Security evaluations, runtime enforcement and live monitoring make that requirement testable and enforceable.
From a talk by Don Bosco Durai
Before you start: Familiarity with LLM tool calls, service credentials and basic application access control will help you follow the architecture and examples.
Where does one component’s access end?
When an agent delegates work to a task or tool, what keeps each component within its own permissions? Don Bosco Durai approaches that question from data governance: he introduces himself as Privacera’s co-founder and CTO and a creator and PMC member of Apache Ranger. Ranger’s governance work extends into cloud services through integrations such as Amazon EMR, Google Cloud Dataproc and Azure HDInsight. The agent problem brings that concern about access into an application that can decide its own next steps.
The architecture starts with four moving parts:
| Component | Responsibility |
|---|---|
| Agent | Reasons, constructs a workflow and invokes tasks or tools |
| Task | Performs a specific action, possibly using an LLM, retrieval or tools |
| Tool | Calls functions that reach internet data, databases or service APIs |
| Memory | Supplies context shared among agents, tasks and tools |
An application can contain multiple agents, each with multiple tasks and tools. In the diagram, the agent, tasks and tools sit inside the pink VPC region, with connections to an LLM, RAG, an API and a database. Those connections identify the places where context leaves a component and where an action reaches another system.
Durai characterizes most agent frameworks at the time of the talk as running agents, tasks and tools in a single process. A database tool needs credentials; an API tool needs a token. If those credentials are available within the shared process, another tool can potentially access them. The service credentials in his example carry super-admin privileges. That is an overprivileged configuration, not an inherent property of service accounts, but it makes unintended access especially consequential.
The exposure extends beyond credentials. Prompts used by agents and tasks also reside in the process, where third-party libraries may access them. Calls to an inadequately secured LLM add another exploitation surface. Meanwhile, the agent chooses a workflow according to the task it receives, so the developer cannot simply enumerate one fixed sequence of operations. Shared access and nondeterministic workflow selection combine to create security behavior that is harder to anticipate than a predefined application path.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A working agent can still be undeployable
These architectural weaknesses produce several different kinds of failure. Missing access controls can expose sensitive or confidential information. An unreliable model can return an incorrect result, and a changed prompt can alter behavior even when the underlying application code remains intact. Security and trust therefore depend on more than whether the agent completes its intended task.
Governance adds a separate production barrier. Durai recounts a conversation that Tuesday with an unnamed customer he describes as one of the top three credit bureaus. The customer had built many agents but was struggling to put them into production. It treated an agent much like a human user: onboarding, training and regulatory obligations had to be addressed before access was granted.
In the customer’s example, California residents’ data could not be used for marketing without the relevant consent. European data and other regional requirements introduced further restrictions on who could access information. The marketing restriction is the customer’s stated requirement, rather than a universal description of California law. The engineering consequence is concrete: an agent must respect the permitted purpose and scope of data use, not merely demonstrate that it can retrieve the data. Without those controls, the customer could not release its agents.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Evaluation, enforcement and observability
No single control covers this entire problem. Durai organizes the work into three complementary layers, beginning with a release decision. Security and safety evaluations must supplement tests of response quality and hallucination. Their purpose is to produce a risk assessment that determines whether an agent can enter production. An externally supplied agent should pass the same gate as one developed internally.
Enforcement supplies the controls that evaluations test. If the implementation cannot reliably enforce a restriction, its evaluation should fail and prevent release. Observability then covers behavior that development and initial testing did not encounter.
| Layer | Operational question |
|---|---|
| Evaluation | Is the agent’s risk acceptable for deployment? |
| Enforcement | Are permissions and safety constraints applied during execution? |
| Observability | What is happening in real use, and when must someone respond? |
The layers depend on one another: testing establishes confidence in controls, while monitoring reveals where that confidence needs to be revisited.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Test permissions and failure behavior before release
Agent security evaluations can build on established software release practices. Test coverage checks whether important behavior has been exercised. Docker image scans and dependency scans identify vulnerabilities and CVEs. Durai describes remediating medium, high and critical risks before production, alongside penetration testing for vulnerabilities such as cross-site scripting. Agents still need those checks; their model-driven behavior adds further cases to test.
Start with use cases and ground truth that define a regression baseline. Then changing a prompt, library, framework or LLM becomes a testable change rather than an assumption that behavior will remain acceptable. Durai also calls for checks on third-party models for poisoning and vulnerabilities, and minimum vulnerability criteria for libraries. Prompt-injection tests should exercise the application’s blocking controls; protection cannot be assumed merely because an LLM is involved.
Data-leakage evaluations need to distinguish users who ask the same question but have different permissions. Consider an HR agent that retrieves salary and benefits information:
| Requester | Requested record | Expected permission |
|---|---|---|
| Employee | Their own salary and benefits | Allowed |
| Employee | Another employee’s salary and benefits | Denied |
| HR administrator | Another employee’s salary and benefits | Allowed only if their role permits it |
A successful answer is not sufficient evidence of correctness. The test must establish whether that requester was entitled to the answer. It should also probe whether a malicious user can exploit a loophole to obtain a record that ordinary use would deny.
Two further evaluation areas extend beyond reading data:
- Unauthorized actions: An agent that changes a system must act for the right person or persona. Tests need to cover permission to perform the change, not just permission to inspect its inputs.
- Runaway agents: A bad user prompt, a problematic task prompt or an inability to resolve the task can send an agent into a tight loop. Those failure scenarios belong in preproduction testing.
Together, vulnerability checks, leakage tests, action tests and loop tests inform the risk score used for release. Durai supplies the categories and the decision they support, but does not prescribe a numerical scoring formula.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Constrain both the agent and the person it represents
The risk score depends on the quality of runtime enforcement. With libraries potentially able to access shared process state, sensitive backend access calls for a zero-trust approach: a component’s presence inside the application is not enough reason to trust its request. Prompt-injection detection and moderation remain relevant, but Durai’s enterprise enforcement discussion concentrates on authentication and authorization.
A request travels from user to agent, from agent to task, and from task to tool before reaching a service API or database. Authentication establishes the identity behind that request and prevents impersonation. Authorization determines what the established identity may access or do. These are separate requirements; a correctly authenticated request can still ask for something forbidden.
There are also two permission scopes to preserve. The agent has its own role and must not exceed it. When it acts on behalf of a person, that person’s permissions must also constrain the operation. Delegation must not turn a restricted user into a privileged backend caller. An agent’s ability to connect to a database or API does not entitle the user to everything reachable through that connection.
Approvals provide another boundary. In a traditional leave-request workflow, a manager approves the request through a predefined process. Durai proposes that a separately designed approval agent could check requests and results automatically. Its authority would be bounded by thresholds: work within the allowed range can proceed automatically, while a request beyond that range brings a human into the loop. This is a proposed approval pattern, not a demonstrated implementation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Carry identity all the way to the operation
Authentication at the application entrance is insufficient. The user’s identity must travel through every task and tool call to the final data access or API operation. That last boundary needs enough context to apply the correct policy. Otherwise, a backend may see only a powerful service account and lose the distinction between the employee and the HR administrator.
A TypeScript expression of that boundary can keep the HR example explicit. Here, trusted authentication supplies userId and roles, and trusted agent configuration supplies agentPermissions. The tool checks both permission scopes before asking the store for a record:
typescript
type RequestContext = Readonly<{
userId: string;
roles: readonly string[];
agentPermissions: readonly string[];
}>;
type Benefits = Readonly<{
employeeId: string;
salary: number;
benefits: readonly string[];
}>;
interface BenefitsStore {
read(employeeId: string): Promise<Benefits>;
}
async function readBenefitsTool(
context: RequestContext,
employeeId: string,
store: BenefitsStore,
): Promise<Benefits> {
const agentAllowed =
context.agentPermissions.includes("benefits:read");
const userAllowed =
context.userId === employeeId ||
context.roles.includes("hr-admin");
if (!agentAllowed || !userAllowed) {
throw new Error("Forbidden");
}
return store.read(employeeId);
}
async function benefitsTask(
context: RequestContext,
employeeId: string,
store: BenefitsStore,
): Promise<Benefits> {
return readBenefitsTool(context, employeeId, store);
}
The task preserves identity; the tool applies policy where the read occurs. An employee asking for another employee’s record reaches the denial before store.read. This application check is meaningful only if the runtime also prevents components from bypassing the controlled access path. Passing a context object alone does not isolate libraries sharing a process.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Monitor behavior that testing did not anticipate
Release testing does not freeze an agent’s behavior. Models change rapidly, agent frameworks evolve and third-party libraries can start behaving differently. Traditional applications also need monitoring and vulnerability management, but the agent adds a changing interpretation of user intent to those existing concerns.
Even an unchanged deployment may encounter language that its tests never covered. A sunny-day test might ask to apply for leave using one familiar phrase, while a real employee expresses the same intent very differently. Monitoring must reveal how responses change with those inputs. It should also track personal and confidential information sent out, so an unexpected increase in exposure can trigger action.
At sufficient request volume, reviewing every interaction is impractical. Define failure metrics and an acceptable tolerance, then alert when the observed failure rate exceeds that tolerance. An increase may come from a misbehaving agent or from a malicious user trying to compromise the application. The alert identifies a need to investigate; the failure count alone does not establish the cause.
Durai then looks toward behavioral anomaly detection, using established user behavior analytics as the analogy. Instead of only counting known failures, such a system would assess whether an agent remains within accepted operating boundaries. He explicitly describes agent anomaly detection as short of the capability he envisions at the time of the talk. The intended outcome is a security process that supplies near-real-time evidence of how the agent is behaving in live use.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A production security process and an open-source invitation
Production readiness therefore extends beyond a one-time evaluation result. Risk assessment governs both deployment of an internally built agent and adoption of a third-party one. Guardrails, enforcement and a secure sandbox constrain execution after admission. Real-time or near-real-time observability then makes anomalies visible soon enough to adjust the system. Sandboxing matters particularly for the opening architecture: permission checks cannot provide the whole security boundary if untrusted components can reach around them.
Durai closes by naming PAIG, introduced in the recording as the newly open-sourced PAIG.ai safety and security solution. Security and compliance span more than one company can solve, so his invitation is to design partners and contributors who can help develop it. He invites interested readers to contact him by email or on LinkedIn. The practical ambition is to make agent security a shared engineering discipline, with controls and evidence that let useful agents reach production.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Source code, examples and contribution information for PAIG's GenAI security, safety and observability framework.
Overview of centralized authorization policies and access auditing across Hadoop services.
Further reading
An architecture for applying user-specific Ranger policies and retaining audit logs across temporary Dataproc clusters.
Practical guidance on least privilege, credential access and isolation of untrusted workloads.
- California Consumer Privacy Act FAQsDocumentation
California's consumer-facing explanation of privacy rights, including opting out of personal-information sale or sharing.
Full transcript
Loading transcript…