AI Engineer World's Fair 2025

How to Build Trustworthy AI

Allie Howe· Founder, Growth Cyber24:22

Loading slides…

Read the talk

How to Build Trustworthy AI

Trustworthy AI needs protection before a model loads, adversarial testing before and after release, and runtime controls that keep the application within its intended purpose.

From a talk by Allie Howe

Before you start: Basic familiarity with model inference, APIs, and retrieval-augmented generation will help; no security specialization is required.

A chatbot offers a car for a dollar

A dealership chatbot offers a customer a Chevy Tahoe for $1. The problem is not simply that the answer is wrong: the application has produced an offer its business never intended to make. Allie Howe, a Growth Cyber vCISO working across AI security and compliance, opens with this reported 2023 incident because it makes trustworthy AI a concrete product requirement. A conversational interface must remain within the boundaries of the service it represents.

Opening slide with a Slack prompt-injection headline, a dealership chatbot exchange and a Fortnite Vader clip.Open full source frame
The opening examples put prompt injection, a dealership chatbot and Fortnite’s Vader character on the same slide.

The consequences become more serious when the boundary protects private information. Howe describes a 2024 Slack AI prompt-injection case involving private-channel data leakage; this is an account of an attack, not evidence here of a confirmed customer breach. She then turns to Epic Games’ conversational Darth Vader NPC in Fortnite. According to Howe, players elicited racist and homophobic responses before its behavior improved substantially. She compares the episode to Microsoft’s Tay chatbot, whose public failure occurred in 2016. Across these examples, the common problem is an AI application behaving outside its intended limits.

0:010:29
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:01 · section reference included

Who owns the boundary?

Depending on a model provider does not remove the application builder’s responsibility. Howe illustrates this with the radio-host defamation lawsuit against OpenAI, referring to a May 20 development and emphasizing warnings that ChatGPT can produce incorrect information. The summary-judgment order was filed on May 19, 2025; defense counsel’s account describes a holding about defamatory meaning. The case does not establish that a disclaimer automatically transfers liability to users. Howe’s operational point remains useful: whatever the legal allocation in a particular case, customers encounter your application, and your brand bears the consequences of its behavior.

That makes trust a shared responsibility rather than a security-team add-on.

TeamPrimary concerns
ProductRelevant topics, correct behavior, helpful answers
SecurityInappropriate content, prompt injection, jailbreaks
EngineeringImplementation, cost, latency

These concerns overlap. An off-topic answer can simultaneously be a product failure, a policy violation, and an engineering problem that needs a control in the request path.

Howe separates two directions of harm: AI security asks how the outside world can harm the application; AI safety asks how the application can harm the world. Prompt injection belongs naturally to the first category. Harmful advice or discriminatory output belongs naturally to the second. A trustworthy application needs both kinds of protection, and a single runtime control may address both.

2:032:23
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:03 · section reference included

Extend security from the pipeline to runtime

Traditional DevSecOps puts security checks into CI/CD: static application security testing examines code, while dependency and supply-chain checks inspect what the application incorporates. AI development adds surfaces that those checks may not cover, including Databricks workspaces and Jupyter notebooks. Security needs to reach those environments, rather than assuming that everything important passes through the existing application pipeline.

The deployed system also needs attention. Prompt-injection techniques evolve, teams replace models, and the same application can produce different responses across interactions. Shifting right means inspecting behavior at runtime without abandoning checks earlier in development. Howe organizes the work into three stages:

StageControlsPurpose
BuildModel scanning, provenance, AI/ML bills of materialsUnderstand and inspect artifacts
TestAI red teamingFind security and safety failures
RuntimeInput and output validationIntercept failures during use

The stages support one another: build-time checks address what enters the system, testing exposes failure patterns, and runtime validation acts on actual traffic.

4:024:15
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:02 · section reference included

A model file can execute code when loaded

MLSecOps, or machine learning security operations, extends security coverage to the places where models are developed and handled. Howe’s practical starting points include looking for exposed secrets in notebooks and Databricks, then establishing model provenance: who created the model, where it came from, what data trained it, and what rights or safeguarding obligations accompany that data. Even the creator’s relationship to a nation-state can become a relevant risk question. Her call to move beyond traditional DevSecOps is about broader coverage, not an incompatibility between ML work and CI/CD.

The most immediate artifact risk is unsafe deserialization. A malicious payload can be embedded when a model is serialized and execute when someone later deserializes it. Python’s pickle documentation explicitly warns that malicious pickle data can execute arbitrary code during unpickling. The dangerous operation is loading the artifact; an attacker does not need to wait for a user to submit an inference request.

For a model downloaded from a repository or model zoo, inspection must therefore happen before loading. Unsafe operators can lead to credential theft, data loss, or model poisoning. Howe points to Protect AI’s open-source ModelScan as one inspection tool and to its MLSecOps community as a place to learn the broader practices. ModelScan’s documented workflow includes both ML and CI/CD integration.

The basic scan is a separate operation from loading the model:

bash

pip install modelscan
modelscan -p ./models

In the repository example Howe walks through, saving the model embeds an unsafe payload that outputs AWS secrets. Loading the model triggers that payload and exposes the credentials. Scanning the artifact first instead reveals a critical finding for an unsafe operator, giving the developer a chance to stop before deserialization. The sequence matters: a finding after loading cannot undo code that has already executed.

Repository warnings provide another opportunity to stop. Howe describes a Protect AI scanning integration with Hugging Face and advises inspecting unsafe-file notices when selecting models. Such a warning belongs in the artifact-review process, alongside your own decision about whether the model and its source are trustworthy.

5:526:09
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:52 · section reference included

Use adversarial testing to shape controls

AI red teaming tests both directions of harm. Security tests attempt prompt injection and jailbreaks. Safety tests probe whether the application supplies bomb-making or chemical-weapons instructions, exhibits bias, or produces racist and homophobic responses. Howe calls for continuous testing rather than testing only when application code changes. Interaction can change context and application state; it does not necessarily update the underlying model’s weights. The practical requirement is to test the system as it actually evolves.

A discovered failure should inform a control. If a particular prompt pattern repeatedly induces an unsafe response, that finding can guide runtime restrictions on prompts or topics. Red teaming then becomes a feedback process: it identifies behavior to prevent, rather than merely producing a report of failures.

The same test questions can also help with model selection. Ask several LLMs the same question and investigate a model whose response differs sharply from the others. Howe suggests that unusual behavior might raise suspicion of a backdoor or deliberately deceptive behavior. A response difference is a reason to investigate, not proof of either. It may nevertheless reveal that a model behaves poorly for the application’s requirements and should be replaced.

9:259:40
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:25 · section reference included

Intercept the request and the response

For an organization starting an AI security practice, Howe would prioritize runtime security. Red teaming can consume considerable time and money, and retraining in response to its findings adds overhead. A runtime product can often be integrated through an API or a Python module, making it an accessible place to begin protecting deployed behavior.

The inspection surface is broader than the user’s message:

  • Indirect prompt injection: Instructions arrive inside a scraped website or a document retrieved by a RAG system.
  • Direct prompt injection: A user sends instructions intended to redirect a chatbot or agent.
  • Jailbreaks: Inputs attempt to bypass behavioral restrictions. Howe’s Lakera AI slide illustrates a semantically strange, chaotic-looking text example.
  • Unsafe or off-topic requests: Inputs ask for harmful instructions or content the application should not provide.

Each route can bring unwanted behavior into the same model call, even though the content enters through different parts of the application.

Runtime validation offers two intervention points. On the input side, reject or deflect a request before the model answers it. On the output side, inspect the generated response and withhold content that violates the configured policy before it reaches the user. These controls let the application act on the traffic it actually receives, including attack patterns that were absent from its earlier tests.

11:2411:39
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:24 · section reference included

Where would a voice agent validate content?

Returning to Vader, Howe sketches a hypothetical architecture, not a verified description of Fortnite’s implementation. In her proposed flow, player proximity audio and possible context such as skins or configuration reach Fortnite servers. ElevenLabs converts speech to text, Gemini generates Vader’s response, and ElevenLabs converts that text back into speech. Fortnite servers then return the audio to the player. The speech-to-text assignment and server flow are assumptions in this architectural exercise.

Her first placement choice is close to the model. Validate the incoming text before Gemini processes it, then validate Gemini’s answer before the response continues downstream. An inappropriate user request can be stopped on the way in; an inappropriate generated answer can be stopped on the way out. The same locations can check safety concerns and security attacks such as prompt injections or jailbreaks.

Estimated Vader runtime architecture connecting Gemini, ElevenLabs and Fortnite servers, with runtime-security checks.Open full source frame
The speaker’s estimated Vader architecture places runtime security checks around the model and voice-processing path.

Additional checks could sit earlier or later in the voice pipeline. Filtering before an ElevenLabs call could avoid paying for downstream processing of a request that should be rejected. Inspecting generated audio after ElevenLabs could check the final material the player will hear. But every additional check must justify its latency and cost.

PlacementMain opportunity
Before downstream voice processingReject early and avoid further calls
Around the language modelValidate model inputs and generated text
After speech generationInspect the audio returned to the player

The design target is acceptable cost, low latency, and high detection accuracy—not simply the greatest number of checks.

Howe also sees an operational advantage in runtime vendors with established research teams: they can maintain detection as new prompt injections and jailbreaks emerge. That reduces the burden on a product team trying to ship its own application while keeping pace with adversarial techniques.

13:2813:39
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:28 · section reference included

Match patients to trials without changing their records

A generic harmful-content filter does not fully describe what an application is allowed to do. Howe demonstrates that distinction using Pillar, one runtime-security product she uses, with a multi-agent system for ALS clinical-trial matching. The system finds studies on the web and recommends suitable patients from a database. Its intended output is a list of patients for each trial; it is not supposed to answer questions about individual patients or modify their records.

In the demonstration, a request to change a patient’s FVC percentage to 50 is blocked by Pillar’s configured topic and keyword restrictions. The request asks for a database mutation, while the application’s purpose is matching. This is a demonstrated prompt block, not a separate database authorization guarantee. The important configuration choice is to express the application’s specific boundary, rather than relying only on a broad definition of unsafe content.

Runtime policies can serve several distinct purposes:

  • Privacy: Check for personally identifiable information on inputs, outputs, or both, according to the application’s needs.
  • Safety: Detect toxic or otherwise inappropriate responses.
  • Application scope: Restrict actions such as updating patient data.
  • Business policy: Restrict unwanted recommendations—for example, Howe imagines Tesla choosing not to recommend Ford.

The last example is not the same kind of risk as a patient-record modification, but both require a policy tailored to the product rather than a universal content filter.

16:0916:22
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

16:09 · section reference included

Make the control visible to customers

Once a runtime control exists, its implementation can become evidence in a governance, risk, and compliance process. Howe starts with a risk-register entry: the company could suffer reputational damage because its AI application produces harmful or off-topic output. She places this in the context of recurring risk assessment, which she describes as an at-least-annual SOC 2 requirement.

Her Vanta example connects that risk to a custom control for validating AI outputs. The practical sequence is:

  1. Define the risk the control addresses.
  2. Create the output-validation control.
  3. Submit evidence supporting the control.
  4. Expose the relevant input/output validation controls in a customer-facing trust center.

This lets buyers see AI-specific protections alongside the controls they already expect in a SOC 2-oriented review.

The commercial value is less uncertainty during procurement. Howe suggests that visible evidence could reduce the need for a lengthy security questionnaire, using a 200-question questionnaire as an illustration. The control is doing security work inside the application; the trust center makes that work inspectable by prospective customers.

18:3218:44
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

18:32 · section reference included

Trust determines what can be deployed

For an AI product, correct, safe, relevant behavior is part of what the customer buys. Howe argues that this brings cybersecurity risk and business risk unusually close together: an application that consistently produces irrelevant or inappropriate output cannot deliver its intended value. Deferring security can therefore mean deferring a working product, not merely accepting a hidden technical liability.

AI also magnifies gaps in ordinary security practice. Missing training-data inventories, unclear model origins, and weak supply-chain controls become harder to ignore when those inputs shape application behavior. Howe points to ISO 42001 and the EU AI Act as further reasons to formalize AI governance. Her accompanying example of an approximately 20 million facial-database fine needs a legal distinction: the closest verified match is CNIL’s €20 million Clearview AI fine in October 2022 for GDPR violations, not an EU AI Act fine. The unnamed example in the talk cannot be conclusively identified as that case.

Sector-specific obligations matter too. Disclosing patient data may create HIPAA exposure, depending on the organization and circumstances. Howe also mentions FDA AI/ML guidance issued earlier in the year of the talk. The engineering task is to identify which obligations apply to the actual product and industry, then build the relevant safeguards before an incident forces the issue.

The strongest reason to do this work is the technology it can make usable. Howe imagines protein discovery eventually helping enable new transplant organs and refers to an unnamed model trained on 1.3 million cells involving highly confidential data. These are an aspiration and an attributed example, respectively, rather than established results demonstrated here. Their role in the argument is practical: powerful healthcare systems still need compliance, safety, and security before people can responsibly rely on them.

Responsibility begins before a lawsuit or a reputational failure. Carry it through the system’s lifecycle: inspect model artifacts and development environments, red-team application behavior, and incorporate runtime validation at deployment. Security protects the application from the world; safety protects the world from the application. Building both into the product is what makes its capabilities usable with trust.

20:1720:27
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

20:17 · section reference included

Resources

From the talk

  • Open-source model artifact scanner with installation instructions, unsafe-operation reporting and serialization-attack examples.

  • Python's serialization documentation explains why unpickling untrusted data can execute arbitrary code.

  • Howe discusses agent capabilities, security risks, runtime monitoring and community work on agent security.

  • The French regulator's report documents its €20 million Clearview AI fine and the associated GDPR violations on printed page 81.

Transcript download
Full transcript

Loading transcript…