AI Engineer World's Fair 2024
LLM Safeguards: Security, Privacy, Compliance, Anti-Hallucination
Loading slides…
Read the talk
LLM Safeguards: Security, Privacy, Compliance, Anti-Hallucination
Enterprise LLM safety spans factual consistency, model supply chains, private data, serving infrastructure and the permissions that turn generated text into action.
From a talk by Daniel Whitenack
Before you start: Familiarity with LLM prompts, APIs and retrieval-augmented generation will help; the article explains the security mechanisms as they appear.
What happens when the copilot reaches the enterprise?
Everyone gets a copilot, every employee gains an assistant, and work becomes easier. That is the promise. Enterprise adoption can look more like the opening slide: a startled llama in an office surrounded by debris and flames. The practical question is how to turn a model that generates plausible text into a system a company can safely use.
Daniel Whitenack, founder and CEO of Prediction Guard, approaches that question through experience deploying secure, accurate systems for enterprise customers. His scope is open-access large language models, which may form only part of an organization's AI strategy. The internals of closed systems are outside that scope. Starting with the familiar path—user input, LLM, output—he builds a checklist of what can go wrong around it.
The first problem appears in the output itself. Training text can contain accurate information, outdated information and confident nonsense. A request for the health benefits of eating glass illustrates the danger: a model may accommodate a false premise and produce an authoritative answer instead of rejecting it.
Whitenack makes the stakes concrete with a customer application that assists field medics in disaster relief and military settings. He asks the audience to imagine a medic dealing with 16 casualties. That is an illustrative workload, not a reported clinical result, but the consequence is clear: inaccurate advice in this setting can cause harm. Other enterprise applications may have lower immediate stakes while still creating liability when confident answers are wrong.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The model arrives with code and runs on a server
An open model is more than a set of weights. Running it can involve a package such as Transformers, model-specific code and further third-party imports. A malicious change to those assets or dependencies introduces a familiar software supply-chain risk through a new acquisition path. The server may be in company infrastructure or a cloud VM; neither location removes that dependency chain.
The serving endpoint is another attack surface. Whether inference runs on GPUs, Groq hardware or Gaudi processors, callers typically reach it through an API. A compromised server can expose sensitive prompts through logs, request caches or memory. Running that service resiliently also requires distributed-systems expertise; knowing how to develop a model does not automatically prepare a team to operate its serving infrastructure at scale.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Useful context creates a privacy boundary
A request to summarize an email is not useful until the email is included. Retrieval systems generalize that step: take a user's question, fetch a relevant document chunk and insert it into the prompt. This makes organizational information available to the model without fine-tuning, but it also moves potentially sensitive data across a processing boundary.
Consider a support ticket that records an employee's response, email address and home information. If that ticket becomes context for a customer-facing answer, the model may disclose those personal details in its completion. The organization has created a data leak through ordinary use of its knowledge base. Regulatory and compliance rules can also restrict where that information may be sent or how it may be used.
Prompt injection adds deliberate manipulation to this accidental-disclosure risk. An attacker might insert instructions to disregard previous directions and reveal a server IP. The instruction is dangerous to the extent that the application supplies sensitive context or connects the model to knowledge bases, databases and other internal systems. Agentic connections expand the possible consequences beyond an unwanted answer. The checklist now covers hallucination, supply-chain compromise, vulnerable or unreliable servers, data breaches and malicious instructions.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Check whether the answer follows its source
Retrieval-augmented generation is a natural response to hallucination. Supply the email, and the model has something specific to summarize. But this intervention also increases exposure of company data, and it leaves a separate question unanswered: how does the application distinguish a successful summary from one that fails badly? Providing evidence and checking the answer are separate operations.
Whitenack turns to models that evaluate factual consistency between two pieces of text. He names UniEval and BARTScore as relevant research. These are not interchangeable binary error detectors: UniEval supports consistency evaluation against source text, while BARTScore is a broader generation-based evaluation metric. The useful common idea is to evaluate generated text in relation to supplied evidence.
Prediction Guard's approach, as Whitenack describes it, uses an ensemble in an inference framework. The application supplies both the generated answer and the source material, then receives a score for their factual consistency:
- Put retrieved source material into the generation prompt.
- Generate the answer.
- Pass the answer and source material to the consistency evaluator.
- Return the answer together with its score.
The score concerns agreement with the supplied source; it does not independently establish that the source is true. An LLM used as a judge is another model-based evaluation option.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Curate and verify the models you execute
Supply-chain controls need not be novel to be useful. Start with a trusted model registry: approved models from trusted publishers, with licenses appropriate to the intended use. Verify downloaded artifacts through attestation or a hash checked against a trusted expected value. Approved models can also be mirrored into an organization's own Hugging Face repository or an AWS-hosted registry.
A small Python utility illustrates the artifact-integrity check. It reads the expected SHA-256 digest from an approved manifest and refuses a mismatch:
python
import hashlib
import hmac
import sys
from pathlib import Path
def verify_sha256(path: Path, expected: str) -> None:
digest = hashlib.sha256()
with path.open("rb") as artifact:
for chunk in iter(lambda: artifact.read(1024 * 1024), b""):
digest.update(chunk)
if not hmac.compare_digest(digest.hexdigest(), expected.lower()):
raise ValueError(f"Integrity check failed: {path}")
if __name__ == "__main__":
model_file = Path(sys.argv[1])
expected_digest = Path(sys.argv[2]).read_text().split()[0]
verify_sha256(model_file, expected_digest)
This checks that an artifact matches the approved bytes. The expected digest must itself come from a trusted source; a matching hash does not establish that arbitrary code is safe.
Whitenack compares indiscriminate model downloads to automatically searching GitHub for some code and immediately executing whatever turns up. Use curated models and established libraries, and avoid enabling model-supplied code casually. The Transformers option he paraphrases is trust_remote_code; current security guidance confirms that identifier. Keeping it False avoids that remote-code execution path, but it is only one part of supply-chain protection.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Operate inference as an enterprise service
The model server needs the controls expected of any enterprise API: endpoint monitoring, file-integrity monitoring, penetration testing and internal red teaming. Whitenack suggests using SOC 2 readiness as a practical question: what evidence would you need to produce about this server and its operation? That is a way to identify missing controls, not a claim that a particular deployment is certified.
The same questions apply when a vendor operates a private tenant for you. Ask where the models run, what security controls protect those servers and how endpoint monitoring works. Delegating inference does not remove the need to understand the service's security boundary.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Filter private inputs and protect the processing environment
Traditional NLP provides useful components for detecting personally identifiable information, or PII. More specialized detectors can identify protected health information, or PHI. Put these checks before inference when that information should not reach the model, then choose a policy appropriate to the application.
| Policy | What reaches the model |
|---|---|
| Block | No prompt when private information is detected |
| Strip | Prompt with detected private information removed |
| Replace | Prompt with detected private information replaced by fake values |
These are configurable responses to detection. Blocking prevents the request from proceeding; stripping and replacement preserve more of the surrounding task while changing its content.
Filtering does not make the original request disappear from the rest of the system. It may already be present in logs or unencrypted memory, where someone with sufficient access could recover it. Whitenack therefore adds confidential computing and environment attestation to the server-side discussion. Intel SGX protects application enclaves; TDX protects confidential virtual machines. These are scoped protections, rather than encryption of every component of a server. Attestation lets a caller verify a protected environment before sending a request; it does not establish the correctness of the answer that environment will generate.
Prompt-injection detection adds another filter before the LLM. Whitenack describes a custom layer combining a growing collection of injection examples with ensembled classification models. It estimates whether incoming content is likely to be an injection and applies a configurable filtering policy.
Together, these controls turn the simple input–model–output picture into a larger system. The practical goal is to incorporate established security measures and more specialized safeguards from the beginning, while users are forming their trust in the application. Whitenack says this architecture reflects roughly a year of work at Prediction Guard; he closes the presentation by inviting implementation questions through the company's Discord community.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Spend latency where a large model is needed
The first audience question asks what happens to latency as the diagram accumulates components. Every additional operation costs time, so the choice of evaluator matters. Whitenack favors a smaller, specialized factual-consistency model when it can perform the required check without a second chained LLM call.
Whitenack illustrates the trade-off with a roughly four-second LLM call and a roughly 200-millisecond check by a smaller NLP model running on CPU. These are illustrative estimates, with no specified model, hardware configuration, input length or concurrency, rather than a measured benchmark. The architectural point is that a specialized check can have a different cost profile from another full generation.
Vector search offers another useful operation. For prompt injection, maintain examples of both attacks and non-attacks, compare an incoming input semantically with those examples, and use a maximum or average similarity score as one signal. Whitenack tentatively points to Rebuff as a similar approach. Example matching is not his system's sole detector, and Rebuff also includes model-based detection. The fast database comparison should not be confused with making the entire detection pipeline model-free. Where possible, use smaller NLP models and vector operations around the LLM, reserving expensive generation for the work that needs it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make detection independently inspectable
A question about pre-production testing leads to an interface design choice: expose the injection detector independently as well as inside a chat call. A team investigating a blocked input can send it directly to the detector instead of reproducing the entire application interaction.
That separation makes the safeguard easier to understand and tune. Whitenack contrasts it with opaque moderation, where an application receives a block without enough information to understand why. In the system he describes, developers can inspect scores and adjust thresholds. His answer provides a way to examine and configure the component; it does not lay out a complete pre-production red-team protocol.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Observe security events without collecting every prompt
Security operations teams need to know which new events an LLM application should send to their security information and event management system, or SIEM. Whitenack begins with a retention boundary: he says Prediction Guard's production model servers save very little information and do not log prompts or completions, because the company does not want access to that content.
Useful monitoring can still focus on artifacts and behavior. A model cache introduces security-relevant files that are data rather than conventional application code. File-integrity monitoring should account for those artifacts, since changes can affect both model behavior and security.
Request count alone is also an incomplete view of resource abuse. Large token inputs or unusually high requested output-token counts can tie up a model server. Anomalies in those input parameters are therefore relevant denial-of-service signals, alongside ordinary API traffic monitoring.
The audience then asks whether injection-filter flags should be forwarded to the SIEM. Whitenack agrees: security teams should have visibility into those detections. Repeated PII or PHI entering prompts is another behavioral signal worth surfacing. The resulting monitoring focuses on what happened—an integrity change, unusual resource demand or a detector flag—without requiring blanket retention of inference content.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Let agents propose changes before they execute them
The final question extends the architecture to agents and human involvement. The earlier diagram primarily covers single-turn requests and chat threads passed into a model. Chained agents introduce an additional concern: granting too much permission too early. Whitenack brings up the OWASP LLM Top 10 and its excessive agency category.
His example is a desktop assistant asked to switch three applications to dark mode. If that request leads to broad administrative permissions, a hallucination or mistaken action can change far more than those settings. Two controls address different parts of the risk: restrict the agent's permissions, and have it produce a dry run of its intended changes for a person to approve or modify. A proposed change remains a proposal until that approval step.
The value of this workflow is clearer in network configuration. Generating the initial set of changes can be tedious; a skilled network engineer may find it much easier to inspect and edit a draft than to create everything from scratch. Whitenack reports that this draft-and-review approach has been useful in the customer cases he has seen. The agent supplies the labor-intensive starting point, while a knowledgeable person decides what should actually change.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
The original paper formulates text-quality evaluation as Boolean question answering across dimensions such as consistency and fluency.
Code and examples for evaluating summaries, dialogue responses, and factual consistency against source text.
A generation-based metric for evaluating text from perspectives including informativeness, fluency, and factuality.
A prototype combining heuristic, LLM, vector-database, and canary-token defenses against prompt injection. Archived in May 2025.
PostgreSQL extension for exact and approximate vector similarity search, with installation and query examples.
Further reading
Describes access policies for unstructured storage and RAG data, including an Amazon S3 integration and access auditing.
Updates since the talk
Current guidance on remote model code, revision pinning, and safer model serialization.
Full transcript
Loading transcript…