Contents
  1. Model the system under attack
    1. Assets, attackers, and successful attacks
      1. Assets and losses
    2. Entry points and trust boundaries
      1. Questions for every crossing
  2. Established principles, new interpreter
    1. Security foundations across four shifts
    2. When content behaves like instruction
      1. One mechanism, two entry points
      2. Why marking is not mediation
  3. From influence to effects
    1. The exfiltration path
    2. A valid request is not an authorized action
      1. Separate claims at the action boundary
    3. Delegated authority and accumulated harm
      1. Authority patterns
  4. Bound reach and protect assets
    1. Isolation defines the blast radius
    2. Sensitive data leaves more than answers
      1. Name the threatened representation
    3. Models and artifacts are supply-chain assets
      1. Loading crosses a boundary
  5. Place and test defenses
    1. Mediate every outbound sink
    2. Controls belong on attack paths
      1. Independent enforcement
    3. Adversarial tests need observable effects
      1. Check effects, not rhetoric
      2. Security test record
  6. Detect, respond, and decide
    1. Evidence from proposal to effect
    2. Contain, reconcile, and restore
    3. Make a bounded security decision
      1. Decision dimensions
  7. Check understanding
  8. Open questions
  9. Selected talks
  10. References
  11. Talk library
← All topics

AI security

An AI application is secure only to the extent that attacker-controlled influence cannot violate the confidentiality, integrity, or availability of its assets. The analysis must also account for delegated authority—the capability to cause effects—and economic value when judging impact. That question reaches beyond the model. Retrieved documents, tool descriptions, model artifacts, caches, renderers, credentials, execution environments, and external services all participate in the result.

The central engineering move is to follow a complete attack path. Identify what the attacker controls, where it enters, which privileged component interprets it, what asset or authority becomes reachable, and what observable consequence would count as success. Prompts and classifiers can reduce model-level failures, but dependable boundaries are enforced by application and infrastructure components the model cannot bypass or modify.

I. Model the system under attack

Assets, attackers, and successful attacks

Security begins with an asset: something whose loss or misuse matters. Confidentiality, integrity, and availability are security objectives for those assets. Private records and credentials require confidentiality; training data, model artifacts, policy decisions, and business records require integrity; serving capacity and tool dependencies require availability. An agent may also hold delegated authority—the capability to send messages, alter accounts, deploy code, or spend money. A hosted model's behavior can have economic value even when its weights remain inaccessible, making its loss or misuse consequential.

A threat model describes more than a bad output. It states the attacker, required capability, entry point, affected asset, existing controls, and adverse consequence. Relevant actors include malicious users, authors of retrieved content, compromised suppliers or integrations, insiders, and users attempting to cross tenant boundaries. The same observed response may be harmless in one architecture and a breach in another: mentioning a salary is different from retrieving another employee's actual record.

Assets and losses

An asset inventory should distinguish security objectives, delegated capabilities, and measures of impact instead of compressing unlike considerations into one unsupported score.
AssetSecurity objective or other concernExample loss
Customer recordConfidentiality and integrityUnauthorized disclosure or modification
Tool credentialDelegated capabilityActions performed outside the represented user's permission
Model or checkpointIntegrity; economic value affects impactSubstitution, malicious loading, or behavioral extraction
Service capacityAvailability; economic cost affects impactRunaway calls consume shared capacity or budget
Decision recordIntegrity and attributionA consequential action cannot be tied to its actor or evidence

Entry points and trust boundaries

A trust boundary is a crossing between components governed by different identities, permissions, operators, or validation assumptions. Extend the processing inventory from Privacy and Data Governance: for every message, document, tool result, cache entry, and outbound request, record its origin, who may influence it, which identity carries it, what validation occurred, and which assets become reachable after the crossing.

The attack surface includes direct requests, uploads, webpages, retrieved records, long-term memory, tool descriptions, tool results, model and provider APIs, generated files, renderers, logs, caches, and plugins. An input can be structurally valid and relevant to the task while still being adversarial. Likewise, a component inside one cloud account is not automatically trusted by every tenant or authorized for every resource.

Questions for every crossing

  • Origin — Who created or can modify the value?
  • Identity — Which user, tenant, workload, or service principal is represented?
  • Authority — What data, tools, destinations, and effects become reachable?
  • Enforcement — Which component independently validates and authorizes the operation?
  • Evidence — Which record would distinguish a proposal, denial, accepted request, and completed effect?
One application architecture: private reads, protected writes and outbound disclosure each cross independent enforcement. A model proposal grants no permission; a sandbox does not authorize business operations.

II. Established principles, new interpreter

Security foundations across four shifts

AI security did not begin with prompt injection. Saltzer and Schroeder's 1975 protection principles articulated fail-safe defaults, complete mediation, and least privilege: deny absent permission, check authority on every protected access, and grant only the powers required for the task. A reference monitor sharpens the placement requirement: the enforcement mechanism must always be invoked, resist tampering, and remain small enough for meaningful analysis and testing.

Learned models added new attack surfaces. Goodfellow, Shlens, and Szegedy's 2015 adversarial-example work showed that ordinary predictive accuracy and resistance to deliberately chosen inputs are different claims. Tramer and collaborators demonstrated model extraction through prediction APIs in 2016. BadNets, released in 2017, showed how an attacker-controlled training process could preserve ordinary validation behavior while introducing trigger-dependent behavior. These lines of work addressed input manipulation, model confidentiality, and training integrity; none replaced access control around the application.

Instruction-following applications changed the interface again. Simon Willison proposed the name prompt injection in September 2022 for attacks in which untrusted text redirects an application's instructions. In 2023, Greshake and collaborators demonstrated indirect injection through material retrieved by integrated applications. Tool-using agents then connected model interpretation to delegated operations. The old rule still applies: protected effects require enforcement outside the fallible interpreter.

Protection principles meet new attack surfaces

  1. 1975Saltzer and SchroederCheck every protected access, deny absent permission, and grant only necessary authority.Sources & context

    Contributors: Jerome Saltzer and Michael Schroeder

    What changed: The Protection of Information in Computer Systems supplies the chapter’s enforcement foundation: a model’s proposal cannot establish permission to affect a protected resource.

  2. 2015Explaining and Harnessing Adversarial ExamplesOrdinary predictive accuracy does not establish resistance to deliberately chosen inputs.Sources & context

    Contributors: Goodfellow, Shlens, and Szegedy

    What changed: Small input changes can accumulate into large output changes. Adversarial training improves tested robustness without establishing immunity; attacks can also transfer across models.

  3. 2016Stealing Machine Learning Models via Prediction APIsQuery access can expose enough information to reproduce valuable model behavior.Sources & context

    Contributors: Tramer and collaborators

    What changed: The protected asset can be the prediction function even when weights remain inaccessible. Removing confidence outputs reduces the efficiency of some attacks but does not eliminate the studied label-only extraction attacks.

  4. September 12, 2022Prompt InjectionWillison names the application risk of untrusted text redirecting instructions.Sources & context

    Contributors: Simon Willison

    What changed: The translation example makes the boundary failure concrete: text supplied as task data changes the requested output. This milestone marks the naming and framing, not a claim of first discovery.

  5. 2023Not What You’ve Signed Up ForRetrieved material can redirect an application without changing the legitimate user’s request.Sources & context

    Contributors: Greshake and collaborators

    What changed: The indirect prompt-injection work demonstrates how externally controlled material influences answers and subsequent API calls. Retrieval relevance therefore cannot establish trust or authority.

  6. 2025The Lethal TrifectaPrivate data, attacker-controlled content, and external communication compose into a theft path.Sources & context

    Contributors: Simon Willison

    What changed: HTTP requests and image loads can supply the outbound channel. The framework directs attention to breaking a required path through capability separation or enforced communication restrictions; it is not a complete threat taxonomy.

Follow the shift from protected access to model behavior, untrusted context, and external effects. Independent enforcement remains essential throughout. Spacing is not to scale.

When content behaves like instruction

Prompt injection occurs when attacker-controlled content causes an instruction-following model to cross the application's intended task or instruction boundary. A direct injection arrives from the requester. An indirect injection is embedded in a webpage, document, message, image, memory record, or tool result that the application later places in context. A jailbreak principally targets the model's behavioral restrictions; prompt injection targets the application's separation between instructions and data. An input may do both.

One mechanism, two entry points

Suppose an application must summarize a support attachment. A direct attacker can ask the assistant to ignore its summarization task. An indirect attacker can place the same instruction inside the attachment. The second route is more easily overlooked because the user request remains legitimate; the hostile instruction arrives through material the system intended to treat as evidence.

Why marking is not mediation

Delimiters, role labels, Base64 encoding, instruction-hierarchy training, and classifiers can reduce some failures, but they do not create a hard authorization boundary. Parameterized SQL works differently: an independently enforced parser treats bound values as values rather than executable query structure. Natural-language marking asks the model to honor a distinction; it does not prevent marked content from influencing the model. Classifiers add another fallible decision and must be tested against both attacks and legitimate text containing attack-associated words.

III. From influence to effects

The exfiltration path

Exfiltration is an unauthorized transfer of information to an attacker-controlled or otherwise impermissible destination. Simon Willison's 2025 lethal trifecta names three capabilities that create a particularly direct path: access to private data, exposure to attacker-controlled content, and an external communication channel. Each capability can have a legitimate product purpose; their composition is the danger.

A browser assistant illustrates the chain. A webpage contains a hidden instruction. The assistant can read authenticated context. Generated Markdown or a navigation action constructs a request to an attacker-controlled destination with private information in the URL. The renderer or fetcher—not the language model response by itself—turns generated text into external communication. Historical disclosures involving browser and coding assistants demonstrate this two-boundary pattern: untrusted source material influenced generation, and a downstream consumer created the network effect.

Three capabilities compose into exfiltration

Example

The attack succeeds through a complete data path; mediating any required edge can stop this path while leaving other risks.

Attacker contentHidden instruction in a page, message, or document.
Assistant contextProcesses untrusted content while working for the user.
Private dataAuthenticated records or conversation context.
Renderer or fetcherInterprets generated URL or image markup as a network action.
Attacker destinationReceives the unauthorized request payload.
A renderer or fetcher converts generated content into an external request. Breaking any required edge stops this illustrated path; it does not establish protection against every attack.
Read the diagram as text
  • Attacker content. Hidden instruction in a page, message, or document.
  • Assistant context. Processes untrusted content while working for the user.
  • Private data. Authenticated records or conversation context.
  • Renderer or fetcher. Interprets generated URL or image markup as a network action.
  • Attacker destination. Receives the unauthorized request payload.
  • Attacker content → Assistant context: untrusted instruction.
  • Private data → Assistant context: private context.
  • Assistant context → Renderer or fetcher: generated outbound payload.
  • Renderer or fetcher → Attacker destination: external request.

The trifecta is precondition analysis, not a theorem or complete threat taxonomy. Denying private-data access, preventing untrusted content from controlling privileged decisions, or restricting outbound destinations can break this path. Its absence does not rule out corrupted answers, unauthorized writes, resource exhaustion, or attacks through another sink.

A valid request is not an authorized action

A tool interface gives the model a machine-readable operation and argument contract; Structured Outputs and Tool Calling owns the full mechanism. The crucial boundary here is that the model proposes a call and application code executes it. A schema-constrained object can establish that employee_id is a string and fields is an allowed list. It cannot establish that the requester may read that employee's salary.

Separate claims at the action boundary

  • Parsing — The bytes represent a value.
  • Schema validation — The value has the permitted fields, types, and enumerated values.
  • Semantic validation — The target exists, prerequisites hold, and values make sense in current state.
  • Authentication — The system knows which actor or workload made the request.
  • Authorization — That actor may perform this operation on this resource.
  • Consequence policy — The effect is allowed now or requires a transaction-specific approval.
  • Verification — Authoritative state establishes what actually occurred.

A tool proposal crosses separate gates

Example

Each successful check answers one question and supplies no automatic evidence for the next.

Action proposalModel-selected operation and arguments.
Parse and schemaChecks representation, fields, types, and allowed values.
Semantic checksChecks target existence, prerequisites, and state invariants.
Authenticate actorVerify the represented user and tenant independently of model output.
Resource authorizationChecks the represented actor, operation, tenant, and target.
Consequence policyAllows, denies, or requires approval for the exact versioned action.
DeniedNo protected operation occurs.
Human approvalBind approval to the exact target, arguments and version; reject changed details.
Protected executionExecute the authorized operation with the approved target, arguments and version.
Verify actual stateCheck the protected system’s result rather than the model’s account.
Authentication identifies the actor; authorization checks the requested resource. Approval binds to the exact action and version. A protected operation is followed by verification of its actual result.
Read the diagram as text
  • Action proposal. Model-selected operation and arguments.
  • Parse and schema. Checks representation, fields, types, and allowed values.
  • Semantic checks. Checks target existence, prerequisites, and state invariants.
  • Authenticate actor. Verify the represented user and tenant independently of model output.
  • Resource authorization. Checks the represented actor, operation, tenant, and target.
  • Consequence policy. Allows, denies, or requires approval for the exact versioned action.
  • Denied. No protected operation occurs.
  • Human approval. Bind approval to the exact target, arguments and version; reject changed details.
  • Protected execution. Execute the authorized operation with the approved target, arguments and version.
  • Verify actual state. Check the protected system’s result rather than the model’s account.
  • Action proposal → Parse and schema: structured value.
  • Parse and schema → Semantic checks: shape valid.
  • Parse and schema → Denied: invalid shape.
  • Semantic checks → Authenticate actor: state valid.
  • Semantic checks → Denied: invalid state.
  • Resource authorization → Consequence policy: authorized resource.
  • Resource authorization → Denied: unauthorized.
  • Consequence policy → Protected execution: low-risk allow.
  • Consequence policy → Human approval: approval required.
  • Consequence policy → Denied: policy deny.
  • Human approval → Protected execution: exact action approved.
  • Human approval → Denied: rejected or changed.
  • Authenticate actor → Resource authorization: verified identity.
  • Authenticate actor → Denied: identity invalid.
  • Protected execution → Verify actual state: authoritative result.

Approval must bind to the material action, target, arguments, and version the user inspected. If those details change before execution, the earlier approval no longer applies. This is a time-of-check/time-of-use problem: a correct check over one proposal does not authorize a substituted resource used later.

Delegated authority and accumulated harm

Tool abuse occurs when an available operation is used outside the represented user's authority, intended task, or acceptable overall effect. The classic confused deputy is a component that possesses stronger authority than its caller and mistakenly applies that authority to a caller-selected target. An agent running with a broad service credential can reproduce this failure even when its tool call is syntactically perfect.

Scope has several dimensions: operation, resource, tenant, destination, duration, and cumulative effect. Hiding a dangerous tool name does not help if another tool or credential provides equivalent reach. Read-only access to an entire service can still expose unrelated private records. A narrow capability for incident-related messages is materially different from general read access to every channel.

Authority patterns

Separate judgment from authority, and bound the whole attempt rather than each call in isolation. Agent Engineering develops accumulated effects.
PatternModel may doIndependent control
Read-only retrievalPropose queries and summarize authorized resultsPer-record and per-tenant authorization on every read
Propose and approvePrepare a versioned mutationHuman approves exact target and arguments; server rechecks at execution
Bounded autonomous writeSelect and execute within a narrow taskScoped capability, quota, idempotency, audit, and postcondition checks

IV. Bound reach and protect assets

Isolation defines the blast radius

Isolation constrains what an untrusted workload can observe or affect under stated escape assumptions. Sandboxes and Execution Isolation develops the mechanisms. For security review, inspect filesystem mappings, process and kernel boundaries, outbound networking, secret placement, CPU and memory limits, maximum lifetime, teardown, and tenant separation independently.

Linux namespaces isolate resource views and cgroups constrain consumption, but ordinary containers continue to interact with a shared host kernel. Seccomp can reduce the exposed system-call surface, sometimes at a compatibility cost. Userspace kernels and virtual machines move or narrow parts of that boundary. None of these names is a complete security claim: mapped files, allowed services, host configuration, network policy, and control-plane permissions still determine reach.

Keep secrets outside the untrusted workload where possible. A mediator can perform a narrowly authorized service operation without revealing the credential. Denied-by-default egress limits where readable data can be transmitted, while explicit service bindings preserve required capabilities. Resource caps bound both malicious denial of service and ordinary generated mistakes such as infinite loops. A sandbox still does not decide whether an allowed business action is authorized.

A configured isolation boundary limits the workload’s reach. Filesystem access, resource limits and network policy are separate controls; business authorization and credentials remain external. Escape assumptions and teardown still depend on the runtime.

Sensitive data leaves more than answers

Sensitive information can appear in source datasets, retrieval indexes, embeddings, prompts, provider requests, caches, memory, traces, evaluation corpora, feedback stores, generated outputs, and human-review tools. A harmless visible answer does not prove that upstream copies were minimized or that no provider, cache, or log received the value. Inventory these artifacts separately because each has its own readers, retention, and deletion path.

One datum creates several governed artifacts

Example

Restricting the final answer does not remove upstream copies, derived representations, provider transmissions, or operational records.

1 / 3 · Source and index

The source record remains distinct from its retrieval representation; authorization must govern eligibility before retrieval.

New this stepSource recordAuthoritative sensitive value with resource-level permissions.
New this stepRetrieval representationIndexed text, metadata, or embedding derived from the record.
Possible artifacts from one sensitive record, not a mandatory pipeline. Each copy or representation has its own readers, retention and deletion rules; transmission alone does not establish provider retention.
Read the diagram as text
  • Source record. Authoritative sensitive value with resource-level permissions.
  • Retrieval representation. Indexed text, metadata, or embedding derived from the record.
  • Assembled request. Selected source material placed into model context.
  • Provider transmission. Request crosses into the approved model-service path.
  • Operational trace. Recorded execution evidence, governed by its logging policy.
  • Generated answer. May reproduce, combine, infer, or omit sensitive information.
  • Human review record. A further audience and retention boundary.
  • Cache entry. Reusable result under its own eligibility and retention rules.
  • Source record → Retrieval representation: indexed as.
  • Retrieval representation → Assembled request: retrieved into.
  • Assembled request → Provider transmission: transmitted to.
  • Provider transmission → Generated answer: generates.
  • Assembled request → Operational trace: recorded by.
  • Generated answer → Human review record: reviewed through.
  • Generated answer → Cache entry: may cache.
  1. Source and index. The source record remains distinct from its retrieval representation; authorization must govern eligibility before retrieval. Active: Source record, Retrieval representation. New: Source record, Retrieval representation.
  2. Request crosses a provider boundary. Selected content becomes model input and may also create an operational trace. Active: Source record, Retrieval representation, Assembled request, Provider transmission, Operational trace. New: Assembled request, Provider transmission, Operational trace.
  3. Output and review add audiences. Answers may create a cache entry or a review record, each with separate access and retention rules. Active: Source record, Retrieval representation, Assembled request, Provider transmission, Operational trace, Generated answer, Human review record, Cache entry. New: Generated answer, Human review record, Cache entry.

Name the threatened representation

Different privacy attacks establish different things. Membership inference asks whether an already supplied record participated in training. Model inversion infers sensitive features or constructs an input associated with an output, often using auxiliary information. Training-data extraction seeks to recover content itself. Application-context exfiltration is different again: it transfers data supplied at runtime rather than recovering training information. Conflating these mechanisms leads to mismatched controls.

Retrieval must apply current tenant and record-level authorization before information enters model context. Similarity is not permission, and a tenant boundary does not imply that every user within the tenant may read every record. Cached answers and retained conversations require the same scope discipline. Disclosure also depends on audience, destination, purpose, and reuse, as explained in Privacy and Data Governance.

Models and artifacts are supply-chain assets

An AI release is a chain of distinct artifacts: datasets, dependencies, training code, checkpoints, adapters, tokenizers, configurations, system instructions, evaluation sets, deployment bundles, and serving credentials. Attackers can poison inputs, substitute an artifact, exploit a loader, or query a serving API to approximate valuable behavior.

Loading crosses a boundary

Treat downloaded models as potentially executable software. Flexible serialization formats and framework conveniences can execute code during deserialization. PyTorch's security guidance consequently recommends provenance checks, separating weights from Python code, and isolated loading. A pre-load scanner can detect some unsafe constructs, but a clean scan is not permission to load arbitrary untrusted serialized code.

Artifact provenance can bind a digest to a signed claim about source, builder, build type, and parameters. Verification detects substitution only under the stated trust assumptions. It does not prove that an authentic model is behaviorally safe. BadNets demonstrated why ordinary validation accuracy and backdoor absence are separate claims. Supply-chain assurance therefore combines provenance, restricted loading, scanning, promotion controls, access control, and behavior-specific tests.

V. Place and test defenses

Mediate every outbound sink

Model output becomes another component's input. A chat renderer may interpret Markdown, a backend may build a query or file path, and an agent runtime may convert structured output into a tool request. Each consumer creates a distinct sink. Protecting only the displayed response leaves URL fetches, webhooks, emails, tool arguments, logs, telemetry, and network requests unmediated.

Controls must match the channel. A destination allowlist is separate from URL syntax validation. Redirects and DNS resolution can move an apparently permitted request toward an internal or unexpected address. A credential broker can attach a secret after an untrusted workload has selected an allowed operation, but the broker must still authorize the requested resource and effect. Field-level disclosure policy asks whether this recipient may receive these values for this purpose.

Output filters and data-loss-prevention classifiers can catch known patterns, but transformed, split, or encoded information can evade content inspection. Rate and volume limits reduce cumulative exposure without deciding whether any one transfer is legitimate. Blanket egress denial offers a smaller communication surface but may eliminate the product's required work. The useful design is the narrowest independently enforced channel that still performs the task.

Controls belong on attack paths

Defense in depth is not a stack of labels. Each control should remove, narrow, detect, or repair a stated attack precondition. Source curation and provenance reduce hostile ingestion. Context minimization reduces exposed data. Instruction-hierarchy training and classifiers reduce some model-level failures. Resource authorization mediates protected operations. Isolation bounds execution, egress policy restricts communication, monitoring exposes suspicious sequences, and recovery limits duration and consequence.

Independent enforcement

Place the decisive control where the effect reaches the protected resource. A model instruction can express intended policy, but it cannot serve as the reference monitor when the model can ignore or rewrite it. Similarly, a model-based monitor can help identify sensitive operations without replacing deterministic authorization or human review for high-impact actions.

Controls can share failure modes. A gateway, sandbox, and audit system may all trust the same incorrect tenant claim. A classifier and model judge may both interpret the same injected candidate. Human review can degrade into approval fatigue when prompts are frequent or omit material transaction details. Record the owner, enforcement point, failure mode, false-positive cost, and residual path for every material control.

Adversarial tests need observable effects

An adversarial evaluation selects cases to exercise attacker goals, capabilities, entry points, and complete paths. A model-only probe can establish how a model responds to a supplied prompt. It cannot establish whether retrieval permissions held, whether a tool executed, or whether data reached an external destination. End-to-end tests must include relevant identities, mutable state, tools, sinks, and side effects.

Check effects, not rhetoric

Define the oracle around an observable property. A controlled exfiltration test can place a canary value in a synthetic private store and instrument the only permitted external destination. A state-changing test can inspect the environment after execution. AgentDojo separates legitimate task success, success without adversarial side effects, and attacker-objective success. Executable vulnerability witnesses likewise distinguish a real program effect from an agent's assertion. Evals and Benchmarks develops requirement-specific oracles.

Security test record

Preserve enough context to interpret the result and reproduce the assessed configuration.
FieldWhy it matters
System and policy versionsBinds the result to the tested configuration
Attacker capability and preconditionsDefines what access the test assumes
Attack population and attempt budgetBounds what zero or nonzero success means
Oracle and controlled effectDistinguishes persuasive text from achieved harm
Blocked stage and residual gapShows which boundary worked and what remained untested

VI. Detect, respond, and decide

Evidence from proposal to effect

A security record should distinguish intent, prevention, transport, and effect. Preserve attributable actor and tenant identity, input provenance, context-source versions, model and policy versions, proposed operation, authorization result, execution identity, destination, stable operation identifier, provider acknowledgement, and authoritative postcondition. Shared API-key logs may prove credential use while leaving both the user and acting workload unknown.

A model explanation says what the model believes happened. A tool response says what the integration returned. An accepted remote request says the provider received it. None alone proves the resulting business state. When the outcome is unknown, preserve the operation identity and reconcile against authoritative provider state before retrying; otherwise an investigation can create a second effect.

Useful detections include cross-tenant actor/resource mismatches, repeated denied actions, unusual tool sequences, new outbound destinations, secret-like payloads, and rate anomalies. Alerts are starting points, not verdicts: behavior drift, instrumentation changes, and legitimate administrative work can produce similar signals. Logs are sensitive assets themselves and need minimized payloads, access control, integrity protection, and defined retention.

Contain, reconcile, and restore

Incident response first limits new harm. Identify affected assets and tenants; revoke credentials and capabilities; disable compromised models, tools, destinations, or retrieval sources; and preserve relevant evidence. Containment can overlap investigation. It should be narrow enough to preserve unaffected service paths where the architecture permits that distinction.

Investigation reconstructs the exact model, prompt assembly, retrieved content, tool definitions, policy version, user and service identities, generated artifacts, trajectory, and external receipts. Preserve provenance and restrict evidence access. An investigation can itself damage evidence: altered timestamps may be immaterial when the question is whether data exists, but fatal when the question is who knew what when.

Recovery separates several responsibilities. Rollback changes future software or model behavior. Revocation prevents future credential use. Reconciliation establishes whether an uncertain external operation committed. Deletion removes qualifying retained state. Compensation creates a new effect that counters completed work. A sent disclosure or message is not erased by restoring an earlier deployment. Software Engineering Fundamentals explains unknown external outcomes, while its recovery section distinguishes rollback from compensation.

Recovery does not erase completed effects

Example timings

Containment, investigation, reconciliation, and restoration overlap, but only reconciliation or compensation addresses effects already committed externally.

Incident response0–10 relative timeDuration 10 relative time
External effect0.5–1.2 relative timeDuration 0.7 relative timeWithin Incident response
Detection and triage1–3 relative timeDuration 2 relative timeWithin Incident response
Containment2–5 relative timeDuration 3 relative timeWithin Incident response
Evidence preservation2–6 relative timeDuration 4 relative timeWithin Incident response
Investigation3–7.5 relative timeDuration 4.5 relative timeWithin Incident response
Reconcile or compensate5–9 relative timeDuration 4 relative timeWithin Incident response
Verified restoration7–10 relative timeDuration 3 relative timeWithin Incident response
Illustrative relative phases share one time axis. The external effect occurs before containment. Response activities overlap, so their durations must not be summed as elapsed time. Rollback changes future execution; it does not undo the earlier effect.
Read the diagram as text
  • Incident response. Overall coordinated response period. 0 to 10 relative time; duration 10 relative time.
  • External effect. Unauthorized disclosure, message, deletion, or transaction occurs. 0.5 to 1.2 relative time; duration 0.7 relative time. Parent: Incident response.
  • Detection and triage. Validate the report and identify likely scope and urgency. 1 to 3 relative time; duration 2 relative time. Parent: Incident response.
  • Containment. Stop new effects and revoke relevant authority. 2 to 5 relative time; duration 3 relative time. Parent: Incident response.
  • Evidence preservation. Protect relevant records, artifacts, and provenance. 2 to 6 relative time; duration 4 relative time. Parent: Incident response.
  • Investigation. Reconstruct identities, versions, trajectory, and external receipts. 3 to 7.5 relative time; duration 4.5 relative time. Parent: Incident response.
  • Reconcile or compensate. Establish unknown outcomes and address completed effects. 5 to 9 relative time; duration 4 relative time. Parent: Incident response.
  • Verified restoration. Deploy verified inputs, confirm service, and monitor heightened risk. 7 to 10 relative time; duration 3 relative time. Parent: Incident response.

Replay attacks only in controlled environments: cloned or synthetic state, inert tools, synthetic credentials, blocked or instrumented egress, and explicit reset procedures. Restoration should use verified artifacts, rotate compromised authority, add regression cases, obtain service-owner confirmation, and continue heightened monitoring until stated exit criteria hold.

Make a bounded security decision

Security assurance is bounded evidence, not a permanent certificate. Return to each consequential attack path and record the attacker assumptions, preventive and detective controls, independent enforcement points, test conditions, plausible blast radius, recovery mechanism, owner, and residual uncertainty. A control's existence is different from evidence that it operated under the relevant conditions.

Decision dimensions

Compare designs without collapsing unlike dimensions into a universal security score.
DesignReach and effectsSecurity obligation
Read-only retrievalAuthorized records; no intended mutationCurrent per-record authorization, data minimization, and disclosure control
Propose and approveVersioned consequential proposalMeaningful transaction-bound review and final server-side authorization
Autonomous bounded writeNarrow resources and cumulative effectsScoped capability, complete mediation, containment, quotas, evidence, and recovery

Stronger isolation, review, and monitoring can reduce risk while increasing latency, friction, false positives, cost, and operating burden. The acceptable balance depends on asset value, consequence, reversibility, and responsible ownership. Reassess when models, prompts, tools, permissions, providers, data paths, policy engines, or credible attacker capabilities change.

Open questions

  1. How can systems preserve useful reasoning over untrusted content while enforcing information-flow and capability policies that remain practical for dynamic tools? Progress would mean policies with clearly defined source and sink semantics, low bypass rates under adaptive attack, and tolerable false denials on real workflows.

  2. How should intent-based access distinguish a legitimate newly discovered task requirement from prompt injection or an over-eager agent? Progress would require independently evaluated decision procedures, explicit escalation behavior, and evidence about both unauthorized grants and unnecessary denials.

  3. How can red-team results support stronger assurance across multi-turn sessions, persistent memory, live tools, and changing application state? Progress would include reproducible stateful environments, effect-based oracles, adaptive attack budgets, and explicit uncertainty over the tested attack population.

  4. How can security telemetry remain useful for investigation without becoming another sensitive-data repository? Progress would combine minimal but attributable event schemas, protected payload access, integrity evidence, artifact-specific retention, and validated detections that do not require indiscriminate prompt capture.

Follow the curated reading path through the speakers and demonstrations behind this entry.

25 min

AI Engineer World's Fair 2024 · 2024

Building security around ML

Dr. Andrew Davis

Cited in this entry

A system-level tour of poisoning, model extraction, prompt injection, and unsafe model loading that connects model-specific risks to ordinary security boundaries.

Watch talk

Explore more talks

The rest of the library, beyond the curated path. Cited talks support this entry; reviewed transcripts were processed in full. Metadata candidates have not been reviewed as sources or verified as topic members.

194 matching talks

Every catalogued talk on this subject: Safety & governance →

TalkSpeakerEventYear
Nagkumar Arkalgud, Keiji KanazawaAI Engineer World's Fair 20252025
Fouad MatinAI Engineer World's Fair 20252025
Bobby Tiernay, Kam SweenAI Engineer World's Fair 20252025
Security Firewall for Agents

Transcript reviewed

Ryan DahlAI Engineer World's Fair 20262026
AI Engineering 101

Transcript reviewed

Noah HeinAI Engineer Summit 20232023
RAG for VPs of AI

Transcript reviewed

Jerry LiuAI Engineer World's Fair 20242024
Roy DerksAI Engineer Summit 20252025
How to Build Trustworthy AI

Cited in this entry

Allie HoweAI Engineer World's Fair 20252025
Don Bosco DuraiAI Engineer Summit 20252025
Eugene YanAI Engineer World's Fair 20262026
Diego CarpinteroAI Engineer Europe 20262026
Lovina DmelloAI Engineer World's Fair 20262026
Erik MeijerAI Engineer World's Fair 20262026
Kenton VardaAI Engineer World's Fair 20262026
Simon WillisonAI Engineer World's Fair 20242024
Idan GazitAI Engineer World's Fair 20262026
Paola Estefanía de CamposAI Engineer World's Fair 20262026
Yohei NakajimaAI Engineer World's Fair 20262026
Michael AlbadaAI Engineer World's Fair 20252025
Nick HeinerAI Engineer World's Fair 20262026
Rene BrandelAI Engineer World's Fair 20252025
Sander SchulhoffAI Engineer World's Fair 20252025
Daniel ChalefAI Engineer World's Fair 20262026
Tun Shwe, Jeremy FrenayAI Engineer Europe 20262026
Samuel ColvinAI Engineer World's Fair 20252025
Sunil PaiAI Engineer Europe 20262026
Vinoth GovindarajanAI Engineer World's Fair 20262026
Tushar JainAI Engineer World's Fair 20262026
Harshil AgrawalAI Engineer Europe 20262026
Cedric VidalAI Engineer World's Fair 20252025
AI’s Jurassic Park Period

Cited in this entry

Aaron StanleyAI Engineer World's Fair 20262026
Anna Marie BenzonAI Engineer World's Fair 20262026
Simon WillisonAI Engineer Summit 20232023
Sam MorrowAI Engineer Europe 20262026
Nico AlbaneseAI Engineer Summit 20252025
Sarthak AggarwalAI Engineer World's Fair 20262026
2025 in LLMs so far

Transcript reviewed

Simon WillisonAI Engineer World's Fair 20252025
Trust, but Verify

Transcript reviewed

Shreya RajpalAI Engineer Summit 20232023
Akele Reed, Dave Revere, Doug KellerAI Engineer World's Fair 20262026
Ian WebsterAI Engineer World's Fair 20242024
Christopher Lovejoy, Saul HowardAI Engineer World's Fair 20262026
Sonny Merla, Mauro Luchetti, Mattia RedaelliAI Engineer Europe 20262026
Joel HronAI Engineer World's Fair 20252025
Manoj Nair, Ezra, RandallAI Engineer World's Fair 20262026
Build Systems, Not Code

Cited in this entry

Angie JonesAI Engineer World's Fair 20262026
Philipp SchmidAI Engineer World's Fair 20252025
Gabe De MesaAI Engineer World's Fair 20262026
Uri Rolls, Thom WolfAI Engineer World's Fair 20262026
Ameya BhatawdekarAI Engineer World's Fair 20262026
Abhishek BhardwajAI Engineer World's Fair 20262026
Tobin SouthAI Engineer World's Fair 20252025
Building Reactive AI Apps

Transcript reviewed

Matt WelshAI Engineer Summit 20232023
Ritvik PandyaAI Engineer World's Fair 20262026
Daniel WhitenackAI Engineer World's Fair 20242024
Ben Hylak, Sid BendreAI Engineer World's Fair 20252025
Ezra Tanzer, Dan ArpinoAI Engineer World's Fair 20262026
Steve YeggeAI Engineer World's Fair 20262026
Abhishek BhardwajAI Engineer World's Fair 20252025
Mani KhanujaAI Engineer World's Fair 20252025
Sarah KhalifeAI Engineer World's Fair 20242024
Vasek MlejnskyAI Engineer World's Fair 20242024
Jared HansonAI Engineer World's Fair 20252025
Steven MoonAI Engineer Summit 20252025
Remy GuercioAI Engineer Europe 20262026
Liam McGarrigleAI Engineer Europe 20262026
Nick TaylorAI Engineer Europe 20262026
Gunjan PatelAI Engineer World's Fair 20242024
John DickersonAI Engineer World's Fair 20252025
Damien MurphyAI Engineer World's Fair 20252025
Sharmila Chokalingam, ShubhiAI Engineer World's Fair 20242024
Jacob LauritzenAI Engineer Europe 20262026
Ian Butler, Nick GregoryAI Engineer World's Fair 20252025
AI SDK v6

Metadata candidate

Nico AlbaneseAI Engineer Europe 20262026
Justin SmithAI Engineer World's Fair 20262026
Stephen Chin, Jonathan LoweAI Engineer Summit 20252025
Gagan Bhat, Isabella Kai HeAI Engineer World's Fair 20262026
Henry MaoAI Engineer World's Fair 20252025
Robert BrennanAI Engineer Code 20252025
Ivan BurazinAI Engineer World's Fair 20252025
Nimrod HauserAI Engineer Europe 20262026
Sunny MadraAI Engineer World's Fair 20242024
Julián Duque, Anush DSouzaAI Engineer World's Fair 20252025
Mahesh MuragAI Engineer Summit 20252025
Jerry LiuAI Engineer World's Fair 20252025
Ekaterina DeynekaAI Engineer World's Fair 20262026
Lou BichardAI Engineer World's Fair 20252025
Building Cursor Composer

Metadata candidate

Lee RobinsonAI Engineer Code 20252025
Simrat HanspalAI Engineer Summit 20232023
Marlene Mhangami, Liam HamptonAI Engineer Europe 20262026
Michael FesterAI Engineer World's Fair 20252025
Abed MatiniAI Engineer World's Fair 20262026
Prasenjit SarkarAI Engineer Europe 20262026
Michael GrinichAI Engineer World's Fair 20252025
Thariq ShihiparAI Engineer Code 20252025
Pedro RodriguesAI Engineer Europe 20262026
Mahesh SathiamoorthyAI Engineer World's Fair 20262026
Develop at Idea Velocity

Metadata candidate

Jeffrey Lee-ChanAI Engineer World's Fair 20262026
Tomas ReimersAI Engineer World's Fair 20252025
Joseph Wang, SidAI Engineer World's Fair 20262026
Ofer MendelevitchAI Engineer Code 20252025
Sam BhagwatAI Engineer World's Fair 20262026
Alex Shaw, Ryan MartenAI Engineer World's Fair 20262026
Katelyn LesseAI Engineer Code 20252025
Gaurav MishraAI Engineer World's Fair 20262026
Gateways are All You Need

Metadata candidate

Karan SampathAI Engineer Europe 20262026
Ruben CasasAI Engineer Europe 20262026
Mark MyshatynAI Engineer World's Fair 20252025
Brian JohnAI Engineer Code 20252025
Tanmai GopalAI Engineer World's Fair 20242024
Dr Bryan Bischof, Dr Bryan BischofAI Engineer World's Fair 20242024
Donald HruskaAI Engineer World's Fair 20252025
Vaibhav Page, Infant VasanthAI Engineer World's Fair 20252025
Eno ReyesAI Engineer World's Fair 20262026
Jaspreet SinghAI Engineer World's Fair 20252025
Zhou YuAI Engineer Summit 20252025
Kyle Jaejun LeeAI Engineer World's Fair 20262026
Identity for AI Agents

Metadata candidate

AI Engineer Code 20252025
Philip Kiely, Yineng ZhangAI Engineer World's Fair 20252025
Andreas Kolleger, Zach Blumenthal, Michael Hunger, TomaszAI Engineer World's Fair 20242024
Raymond FengAI Engineer World's Fair 20262026
Ben HolmesAI Engineer World's Fair 20262026
Will BrownAI Engineer World's Fair 20262026
Move Fast Break Nothing

Metadata candidate

Dedy KredoAI Engineer Summit 20232023
Atita Arora, Deanna EmeryAI Engineer World's Fair 20242024
Maggie AppletonAI Engineer Europe 20262026
Jeronim MorinaAI Engineer World's Fair 20242024
Mario ZechnerAI Engineer Europe 20262026
Pragmatic AI With TypeChat

Metadata candidate

Daniel RosenwasserAI Engineer Summit 20232023
Steve KorshakovAI Engineer World's Fair 20262026
Kuba RogutAI Engineer Europe 20262026
Rewiring the State

Metadata candidate

Eoin MulgrewAI Engineer Europe 20262026
Joshua SnyderAI Engineer Europe 20262026
Robert BrennanAI Engineer World's Fair 20252025
Daniel HanAI Engineer World's Fair 20262026
Peter Steinberger, swyxAI Engineer Europe 20262026
Brandon WaselnukAI Engineer Europe 20262026
Sohail Shaikh, Ankush RastogiAI Engineer World's Fair 20262026
Barr YaronAI Engineer World's Fair 20262026
Patrick DeboisAI Engineer World's Fair 20252025
Christopher Harrison, John PeckAI Engineer World's Fair 20252025
Jack CableAI Engineer World's Fair 20262026
Armin Ronacher, Cristina Poncela CubeiroAI Engineer Europe 20262026
The Future of Work

Metadata candidate

Toran Bruce Richards, Silen Naihin, PootsAI Engineer Summit 20232023
The Log Is The Agent

Metadata candidate

Ishaan SehgalAI Engineer World's Fair 20262026
Itamar FriedmanAI Engineer Code 20252025
Jonathan MortensenAI Engineer World's Fair 20252025
Forrest Brazeal, Matt BallAI Engineer World's Fair 20252025
Chris LattnerAI Engineer World's Fair 20242024
Erik HanchettAI Engineer World's Fair 20262026
Veo 3 for developers

Metadata candidate

Paige BaileyAI Engineer World's Fair 20252025
Fryderyk WiatrowskiAI Engineer Europe 20262026
Lucas PalmaAI Engineer World's Fair 20262026
Sai Krishna RallabandiAI Engineer World's Fair 20262026
Benjamin CowenAI Engineer Europe 20262026
Andy TriedmanAI Engineer Summit 20252025
Frédéric BartheletAI Engineer Europe 20262026
Diane LinAI Engineer World's Fair 20262026
Prukalpa SankarAI Engineer World's Fair 20262026
Ravi MadabhushiAI Engineer World's Fair 20262026
Dan FarrellyAI Engineer World's Fair 20262026
Veronica HylakAI Engineer World's Fair 20262026
Rustin BanksAI Engineer World's Fair 20252025
Sachin KumarAI Engineer World's Fair 20262026
Patrick SmythEvent unknownUnknown
Tanmai GopalEvent unknownUnknown
Kamil PotrecEvent unknownUnknown
Marina PetzelEvent unknownUnknown
Sachin MalhotraAI Engineer World's Fair 20262026
Averi Kitsch, Prerna KakkarEvent unknownUnknown
Javier GarzaAI Engineer World's Fair 20262026
Killian Carlsen-PhelanAI Engineer World's Fair 20262026
Gabriel Spencer-HarperAI Engineer Code 2025Unknown
Mikyo KingEvent unknownUnknown
Joshua MoAI Engineer World's Fair 20262026
Oliver KennonEvent unknownUnknown
Aidan SochowskiEvent unknownUnknown
Shu FangEvent unknownUnknown
Al ChenEvent unknownUnknown
CJ Avilla, Harrison StallEvent unknownUnknown
Tariq ShaukatEvent unknownUnknown
Isabelle WilliamsEvent unknownUnknown
Vinoo GaneshEvent unknownUnknown
Branislav BrzakAI Engineer NYC 2026 Online Talks2026
Tim TylerEvent unknownUnknown
Mark Lummus, Navinkumar PatilAI Engineer World's Fair 20262026
Andrey RybkaEvent unknownUnknown
Allie Howe, Ian Livingstone, Michael Davis, Dick HardtEvent unknownUnknown
Jiachen JiangAI Engineer NYC 2026 Online Talks2026
Sarah SandersEvent unknownUnknown
Jediah KatzEvent unknownUnknown
Eli CohenAI Engineer World's Fair 20262026
Taylor Luttrell-WilliamsEvent unknownUnknown
Stephanie JarmakEvent unknownUnknown
Micah SilvermanAI Engineer World's Fair 20262026

References

Coverage and source review
Processed transcripts
60 processed in full · 5 in the curated path
Automated source review
Passed
Metadata candidates
139 unreviewed; not verified topic membership
Corpus version
1bd8e407b26a07b33815594e1b2db5f41827119a2b3cb6fbf240f9fc571fc767

Automated review checks source support; it is not publication approval.

A synthesis of selected conference talks and technical references. Citations link to the source material; they do not imply that every talk on this subject is included.

  1. NIST FIPS 199: Security objectives and losses

    https://nvlpubs.nist.gov/nistpubs/FIPS/NIST.FIPS.199.pdf

  2. Stealing Machine Learning Models via Prediction APIs

    https://www.usenix.org/system/files/conference/usenixsecurity16/sec16_paper_tramer.pdf

  3. NIST SP 800-30 Rev. 1: Guide for Conducting Risk Assessments

    https://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-30r1.pdf

  4. NIST AI 100-2e2025: Adversarial Machine Learning

    https://doi.org/10.6028/NIST.AI.100-2e2025

  5. Balancing Innovation with Security & Safety

    https://www.youtube.com/watch?v=G7aSH6N7qY4&t=559

  6. OWASP LLM10:2025 Unbounded Consumption

    https://genai.owasp.org/llmrisk/llm102025-unbounded-consumption/

  7. OWASP Threat Modeling Cheat Sheet

    https://cheatsheetseries.owasp.org/cheatsheets/Threat_Modeling_Cheat_Sheet.html

  8. AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases

    https://arxiv.org/html/2407.12784v1

  9. Invariant Labs: MCP Security Notification—Tool Poisoning Attacks

    https://invariantlabs.ai/blog/mcp-security-notification-tool-poisoning-attacks

  10. Saltzer and Schroeder: Basic Principles of Information Protection

    https://web.mit.edu/saltzer/www/publications/protection/Basic.html

  11. NIST CSRC Glossary: Reference Monitor

    https://csrc.nist.gov/glossary/term/reference_monitor

  12. Explaining and Harnessing Adversarial Examples

    https://arxiv.org/html/1412.6572v3

  13. BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain

    https://arxiv.org/html/1708.06733v2

  14. Simon Willison: Prompt injection attacks against GPT-3

    https://simonwillison.net/2022/Sep/12/prompt-injection/

  15. Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection

    https://arxiv.org/abs/2302.12173

  16. The Protection of Information in Computer Systems

    https://www.cs.virginia.edu/~evans/cs551/saltzer/

  17. Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection

    https://arxiv.org/pdf/2302.12173

  18. How to Build Trustworthy AI

    https://www.youtube.com/watch?v=n6wHJDqlS1I&t=733

  19. Why, and how you need to sandbox AI-Generated Code? — Harshil Agrawal, Cloudflare

    https://www.youtube.com/watch?v=AHtGAgQ0Q_Q&t=198

  20. OWASP SQL Injection Prevention: prepared statements and bound values

    https://cheatsheetseries.owasp.org/cheatsheets/SQL_Injection_Prevention_Cheat_Sheet.html

  21. The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions

    https://arxiv.org/abs/2404.13208

  22. Building security around ML

    https://www.youtube.com/watch?v=GVbPiq3Pet0&t=925

  23. InjecGuard: Benchmarking and Mitigating Over-defense in Prompt Injection Guardrail Models

    https://arxiv.org/html/2410.22770v1

  24. Simon Willison: The lethal trifecta for AI agents

    https://simonwillison.net/2025/Jun/16/the-lethal-trifecta/

  25. GitHub Copilot Chat: From Prompt Injection to Data Exfiltration

    https://embracethered.com/blog/posts/2024/github-copilot-chat-prompt-injection-data-exfiltration/

  26. From Arc to Dia: Lessons learned in building AI Browser

    https://www.youtube.com/watch?v=o4scJaQgnFA&t=794

  27. OWASP Server-Side Request Forgery Prevention Cheat Sheet

    https://cheatsheetseries.owasp.org/cheatsheets/Server_Side_Request_Forgery_Prevention_Cheat_Sheet.html

  28. AI Engineering with the Google Gemini 2.5 Model Family

    https://www.youtube.com/watch?v=zK9lYrLbjSg&t=3465

  29. OWASP Input Validation Cheat Sheet

    https://cheatsheetseries.owasp.org/cheatsheets/Input_Validation_Cheat_Sheet.html

  30. OWASP Access Control

    https://owasp.org/www-community/Access_Control

  31. OWASP Transaction Authorization Cheat Sheet

    https://cheatsheetseries.owasp.org/cheatsheets/Transaction_Authorization_Cheat_Sheet.html

  32. OWASP: Transaction Authorization

    https://cheatsheetseries.owasp.org/cheatsheets/Transaction_Authorization_Cheat_Sheet.html

  33. MITRE CWE-367: Time-of-check Time-of-use Race Condition

    https://cwe.mitre.org/data/definitions/367

  34. Norm Hardy: The Confused Deputy

    https://pdos.csail.mit.edu/6.828/2009/readings/hardy-confused-deputy.html

  35. Securing Agents with Open Standards

    https://www.youtube.com/watch?v=FZoMSupg37E&t=133

  36. OWASP LLM06:2025 Excessive Agency

    https://genai.owasp.org/llmrisk/llm062025-excessive-agency/

  37. Unlock Agent Autonomy: The Runtime for AI-Native Systems

    https://www.youtube.com/watch?v=zaGyGgLW3SM&t=421

  38. Your Insecure MCP Server Won't Survive Production — Tun Shwe, Lenses

    https://www.youtube.com/watch?v=BurJvbqFr4c&t=328

  39. OWASP Transaction Authorization Cheat Sheet

    https://cheatsheetseries.owasp.org/cheatsheets/Transaction_Authorization_Cheat_Sheet.html

  40. Stripe API Reference: Idempotent requests

    https://docs.stripe.com/api/idempotent_requests

  41. Beyond permission prompts: making Claude Code more secure and autonomous

    https://www.anthropic.com/engineering/claude-code-sandboxing

  42. gVisor Security Model

    https://gvisor.dev/docs/architecture_guide/security/

  43. From fork() to Fleet: Designing an Agent Sandbox Cloud — Abhishek Bhardwaj, OpenAI

    https://www.youtube.com/watch?v=OqM67QG_Ikk&t=744

  44. gVisor security model: containment and host dependencies

    https://gvisor.dev/docs/architecture_guide/security/

  45. Realtime multiplayer, automation, and you!

    https://www.youtube.com/watch?v=iQ5xldZ9StU&t=720

  46. Cloudflare Dynamic Workers: Egress control

    https://developers.cloudflare.com/dynamic-workers/usage/egress-control/

  47. Why, and how you need to sandbox AI-Generated Code? — Harshil Agrawal, Cloudflare

    https://www.youtube.com/watch?v=AHtGAgQ0Q_Q&t=751

  48. NIST Privacy Framework 1.0: lifecycle and minimized audit evidence

    https://nvlpubs.nist.gov/nistpubs/CSWP/NIST.CSWP.01162020.pdf

  49. RedisVL Extensions: Semantic Cache and Message History

    https://redis.io/docs/latest/develop/ai/redisvl/concepts/extensions/

  50. Through the AI Fog: The architectural decision the next 24 months of agentic security depends on.

    https://www.youtube.com/watch?v=1EZdpEhwmNc&t=587

  51. Membership Inference Attacks against Machine Learning Models

    https://arxiv.org/pdf/1610.05820

  52. Model Inversion Attacks that Exploit Confidence Information and Basic Countermeasures

    https://www.cs.cmu.edu/~mfredrik/papers/fjr2015ccs.pdf

  53. Extracting Training Data from Large Language Models

    https://arxiv.org/html/2012.07805v2

  54. Microsoft: Design a secure multitenant RAG inferencing solution

    https://learn.microsoft.com/en-us/azure/architecture/ai-ml/guide/secure-multitenant-rag

  55. PyTorch Security Policy: using models securely

    https://github.com/pytorch/pytorch/security/policy

  56. How to Build Trustworthy AI

    https://www.youtube.com/watch?v=n6wHJDqlS1I&t=430

  57. Building security around ML

    https://www.youtube.com/watch?v=GVbPiq3Pet0&t=1285

  58. SLSA v1.2: Verifying Artifacts

    https://slsa.dev/spec/v1.2/verifying-artifacts

  59. Building security around ML

    https://www.youtube.com/watch?v=GVbPiq3Pet0&t=223

  60. OWASP LLM05:2025 Improper Output Handling

    https://genai.owasp.org/llmrisk/llm052025-improper-output-handling/

  61. MCP Security Best Practices: token passthrough and confused deputies

    https://modelcontextprotocol.io/docs/2025-11-25/tutorials/security/security_best_practices

  62. What does Enterprise Ready MCP mean?

    https://www.youtube.com/watch?v=0MqYA52iWQU&t=555

  63. Your LLM Stack Is a 2008 Database With Better Marketing

    https://www.youtube.com/watch?v=XjI-AR4pt7Y&t=335

  64. Safety and security for code-executing agents

    https://www.youtube.com/watch?v=w7IMuYsBNr8&t=640

  65. OWASP: prompt injection and evaluator trust boundaries

    https://genai.owasp.org/llmrisk/llm01-prompt-injection/

  66. NIST AI Risk Management Framework Core

    https://airc.nist.gov/airmf-resources/airmf/5-sec-core/

  67. Microsoft Foundry: AI Red Teaming Agent concepts

    https://learn.microsoft.com/en-us/azure/foundry/concepts/ai-red-teaming-agent

  68. AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents

    https://arxiv.org/html/2406.13352v3

  69. Teaching AI to Find Real Vulnerabilities — Prof. David Brumley, Bugcrowd

    https://www.youtube.com/watch?v=ZFxh7sqbUZo&t=451

  70. Mitigating the risk of prompt injections in browser use

    https://www.anthropic.com/research/prompt-injection-defenses

  71. OWASP Logging Cheat Sheet

    https://cheatsheetseries.owasp.org/cheatsheets/Logging_Cheat_Sheet.html

  72. It's 10pm. Do You Know Where Your Agents Are?

    https://www.youtube.com/watch?v=I3znWC3MEXM&t=469

  73. OpenTelemetry: Context propagation

    https://opentelemetry.io/docs/concepts/context-propagation/

  74. NIST SP 800-61r3: incident response and verified recovery

    https://doi.org/10.6028/NIST.SP.800-61r3

  75. Balancing Innovation with Security & Safety

    https://www.youtube.com/watch?v=G7aSH6N7qY4&t=885

  76. NCSC: Secure operation and maintenance of AI systems

    https://www.ncsc.gov.uk/collection/guidelines-secure-ai-system-development/guidelines/secure-operation-maintenance

  77. RFC 7009: OAuth 2.0 Token Revocation

    https://www.rfc-editor.org/rfc/rfc7009.html

  78. AI’s Jurassic Park Period

    https://www.youtube.com/watch?v=1lgFGaHoGq8&t=142

  79. Google Machine Learning Glossary: training, inference, tokens, context, and RAG

    https://developers.google.com/machine-learning/glossary

  80. Dia Security Bulletins: Security Story of fetch_web_content

    https://www.diabrowser.com/security/bulletins

  81. Model Context Protocol: Cancellation, 2025-11-25 specification

    https://modelcontextprotocol.io/specification/2025-11-25/basic/utilities/cancellation

  82. Build Systems, Not Code

    https://www.youtube.com/watch?v=ZD9-4fW2HhM&t=838

  83. Build Systems, Not Code

    https://www.youtube.com/watch?v=ZD9-4fW2HhM&t=930

  84. Defeating Prompt Injections by Design

    https://arxiv.org/html/2503.18813v2

  85. Unlock Agent Autonomy: The Runtime for AI-Native Systems

    https://www.youtube.com/watch?v=zaGyGgLW3SM&t=502

  86. From Arc to Dia: Lessons learned in building AI Browser

    https://www.youtube.com/watch?v=o4scJaQgnFA&t=931

  87. It's 10pm. Do You Know Where Your Agents Are?

    https://www.youtube.com/watch?v=I3znWC3MEXM&t=456

  88. We Gave an Agent Production Code Access and Then Tried to Sleep at Night

    https://www.youtube.com/watch?v=LqLoYksJ6do&t=400

  89. Teaching AI to Find Real Vulnerabilities — Prof. David Brumley, Bugcrowd

    https://www.youtube.com/watch?v=ZFxh7sqbUZo&t=207