AI Engineer Summit 2023
Trust, but Verify
Loading slides…
Read the talk
Trust, but Verify: Put a Validation Boundary Around LLM Outputs
A prototype can look reliable until someone else uses it. Application-specific checks and corrective retries give model outputs a clearer path into dependable software.
From a talk by Shreya Rajpal
Before you start: Familiarity with model API calls and retrieval-augmented generation will help; the Python example uses callable functions and a bounded retry loop.
The prototype works—until someone else tries it
Why does an AI application work for its builder, then fail when someone else tests it? That is the practical problem behind Shreya Rajpal’s trust-but-verify approach. Introducing herself as co-founder and CEO of Guardrails AI, Rajpal brings experience leading machine learning infrastructure at Predibase, working across self-driving systems, and researching classical AI and deep learning.
The excitement is easy to understand: AutoGPT expanded expectations for autonomous applications, while new tools appeared in mental health, sales, and software engineering. Rajpal’s search-interest graph shows a peak around ChatGPT’s release. But rapid adoption does not necessarily produce lasting use. Drawing on Sequoia’s Generative AI’s Act Two, she contrasts fast adoption with weaker month-one mobile-app retention among AI-first applications than traditional incumbents.
The engineering symptom is a gap between a convincing prototype and reliable behavior outside the developer’s own tests. Rajpal attributes that gap to the nondeterministic behavior of machine learning systems: a successful response during development does not establish a dependable contract for future responses.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Variable outputs become compounding errors
Consider a database query asking how much a particular user spent last month. With the same underlying data and a correctly specified query, the answer represents the stored records. This is a question of correctness relative to the data, separate from uptime or availability. That predictable relationship lets one software component consume another component’s output.
A model API offers a different relationship. Repeating the same question can produce different responses, and downstream components may depend on whichever response arrives. Once those outputs feed further decisions, errors can compound. The failures include hallucinated content, incorrect structure, and vulnerability to prompt injection. In the prompt-centered development approach Rajpal describes, developers try to control these behaviors through English instructions, without an independent enforcement mechanism.
Human-facing interfaces can absorb some of this uncertainty. A developer can ignore a bad GitHub Copilot suggestion; a ChatGPT user can explain what is wrong and ask again. Applications where correctness is critical cannot always depend on that human repair loop. The goal is to retain a model’s flexibility while adding explicit checks before its output becomes an application decision.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Insert verification before consumption
Rajpal introduces a proposal she attributes to Alex Graveley: check a generation against constraints, then, on a violation, return both the generated content and the broken rule to the model and ask it to regenerate. Guardrails provides an open-source framework around this pattern. Instead of sending a model’s raw response straight back to the application, route it through a verification suite.
The suite reflects the application’s risks. A healthcare chatbot and a code generator need different checks, even if both call the same model.
- Sensitive information: detect personally identifiable information or protected health information.
- Language and commercial policy: check profanity and competitor references. Rajpal’s example is a McDonald’s chatbot that should not recommend Burger King as the best burger in town.
- Executable code: check whether generated code works within the target environment.
- Source grounding: check whether a summary or free-form response is supported by the material the application treats as authoritative.
These are independent constraints, assembled into a suite appropriate to the product.
Release an output only after the required checks pass. If a check fails, feedback can help the model correct itself: identify what went wrong, request another response, and validate again. That correction is a possibility, not an assurance. Each retry also consumes latency, tokens, and money, so the application needs a budget for the loop.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Separate the check from the failure policy
A guard packages the requirements around a model call. In the talk’s API, guards can be constructed from XML-style RAIL specifications, Pydantic models with structure and validation criteria, or string-based definitions. Prompt and model information can also be supplied during initialization. At runtime, the guard surrounds the LLM callable and validates material entering or leaving it. These are the historical interfaces discussed in the recording; current package examples use a different API surface.
When an output is invalid, the next step depends on the failed constraint and its configured policy.
| Policy | Response to failure |
|---|---|
| Re-ask | Request a corrected generation |
| Filter | Remove offending content |
| Fix | Apply a programmatic correction |
| Fallback | Use another system |
| Refrain | Decline to answer |
| No-op | Record the failure without corrective action |
Validation and enforcement are separate decisions. A no-op can preserve the check result and failure reason while allowing execution to continue. It therefore does not implement the strict release gate described above. An application that requires rejection must configure failure handling accordingly.
The framework handles more than calling individual validators. It supports custom checks, supplies a catalog of common validators, and orchestrates verification to manage its latency cost. It also compiles validation requirements into the prompt, communicating the desired constraints before generation while retaining the checks afterward.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Context is not a compliance check
Why not use a better prompt or a fine-tuned model? Prompt engineering can communicate requirements, and retrieval-augmented generation, or RAG, can inject relevant context. Neither operation checks that the resulting response actually follows the instructions or uses the supplied material. Providing evidence and verifying adherence are different operations.
Rajpal reports that five runs of the same LLM-evaluator experiment, with no parameter changes and temperature zero, produced different benchmark numbers. She does not identify the model, benchmark, or magnitude of variation. The observation reinforces the need to examine actual outputs rather than treating a fixed prompt and sampling configuration as a guarantee of compliance.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Training and model updates add operational work
Controlling behavior through model training introduces another set of costs. Rajpal connects this to her earlier work simplifying machine learning infrastructure: assembling a dataset, experimenting with hyperparameters, training, and serving remain substantial work even with better tooling.
A commercial model API shifts those responsibilities to a provider, but it can also limit the application developer’s control over model-version changes. Rajpal describes prompts that had worked previously losing effectiveness after an underlying model changed. Requirements encoded only in prompts are consequently exposed to changes in how a model interprets them.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Choose a checker that fits the constraint
There is no single implementation that makes every guardrail reliable. When possible, ground the check in an external system. For generated code, that can mean a runtime containing application-specific data. In text-to-SQL, a sandbox with the database and schema can supply concrete feedback for a corrective retry. Rajpal reports that coupling this SQL sandbox with re-asking substantially improved query correctness, without supplying a numerical result. A runtime check provides evidence about the query’s behavior; successful execution alone does not establish that it answers the intended business question.
Other constraints call for smaller mechanisms:
- Rules-based heuristics: for interest-rate extraction from a document whose rates use percentage signs, check for that expected notation. The rule is useful because it encodes a property of the source format.
- Traditional machine learning: use a targeted model for a narrow classification problem.
- High-precision deep learning classifiers: check for categories such as toxicity, harmful advice, or misleading content. Rajpal favors smaller classifiers where suitable instead of invoking an LLM for every constraint.
- LLM self-reflection: ask a model to assess an output when the task calls for that form of judgment.
The choice follows the evidence needed to evaluate the constraint, rather than defaulting to the largest available model.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Define correctness for a help-center chatbot
The concrete application is a chatbot answering questions over an organization’s help-center articles. Its requirements combine factual support with product policy: responses should not hallucinate, swear at customers, or mention competitors. A single general judgment of whether an answer is good would obscure these separate failure modes.
Provenance asks where a generated claim came from. After retrieving help-center context, the application should be able to trace the response back to supporting material within that context. This checks support against the supplied sources; it does not establish universal truth or repair an incorrect source article.
Rajpal names embedding similarity, natural-language-inference classifiers, and LLM self-reflection as techniques used for provenance checks. Embedding similarity looks for related source material, while inference and model-based judgments can assess the relationship between that material and the generated response. The guard configuration selects provenance, profanity, and competitor-reference validators, then wraps the LLM call with the resulting guard.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Repair an unsupported password-change answer
A customer asks how to change their password. The RAG application constructs a prompt using retrieved material, but the model invents the location of the password setting. The answer may sound plausible while still failing to describe the actual application.
The correction proceeds through a specific sequence:
- The provenance check identifies the unsupported portion of the answer.
- Guardrails constructs a re-ask prompt containing the failed response, the relevant context, and an explanation of the unsupported content.
- The model generates a revised response.
- The verification suite checks that response again before the application releases it.
In Rajpal’s toy example, the revised answer passes verification. The key transition is from an unchecked candidate to a checked response, with corrective generation between them—not an assumption that a second answer must be right.
The following Python function expresses that control flow for the same password question. The caller supplies the retrieved help-center context, a model callable, and a verifier that returns concrete violation messages. It keeps a failed candidate out of the return path and stops when its retry budget is exhausted.
python
from collections.abc import Callable
def answer_password_question(
context: str,
generate: Callable[[str], str],
verify: Callable[[str, str], list[str]],
max_reasks: int = 1,
) -> str:
if max_reasks < 0:
raise ValueError("max_reasks must be nonnegative")
question = "How do I change my password on your application?"
base_prompt = (
f"Help-center context:\n{context}\n\n"
f"Question: {question}\n"
"Answer using the context. Do not use profanity "
"or mention competitors."
)
prompt = base_prompt
for attempt in range(max_reasks + 1):
candidate = generate(prompt)
violations = verify(candidate, context)
if not violations:
return candidate
if attempt < max_reasks:
feedback = "\n".join(f"- {item}" for item in violations)
prompt = (
f"{base_prompt}\n\n"
f"Previous response:\n{candidate}\n\n"
f"Failed checks:\n{feedback}\n\n"
"Correct the response to satisfy these checks."
)
raise ValueError("No response passed verification")
The verifier remains the consequential component: the loop can enforce its decisions, but it cannot make an inadequate provenance check more accurate.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make application policy executable
The same pattern extends beyond help-center grounding. Validators can enforce restrictions on financial or healthcare advice, check generated code, detect requests for private information, and address competitor mentions, profanity, or prompt injection. Each requirement needs an appropriate check and a deliberate response to failure.
Guardrails packages four responsibilities around those requirements: custom validation, orchestration of verification and re-prompting, a catalog of reusable validators, and compilation of requirements into prompts. That division lets the application communicate what it wants before generation and decide what it will accept afterward.
For implementation, Rajpal points to the repository under its historical name, ShreyaR/guardrails, which now redirects to guardrails-ai/guardrails, and to the project website and documentation. The practical work begins with specifying the conditions under which an output is acceptable—and what the application should do when those conditions fail.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Source code and installation examples for composing validators around LLM inputs and outputs.
Sequoia’s September 2023 account of AI adoption, mobile-app retention, and the transition from demos to useful products.
Further reading
An earlier interview and transcript exploring Guardrails’ origins, XML-style specifications, SQL validation, and re-asking.
Updates since the talk
A November 2023 walkthrough of embedding and LLM-based provenance checks, with historical Python examples.
Documentation for wrapping model calls, validating existing outputs, inspecting results, and limiting re-asks.
Examples of configuring what happens after a validator fails, including logging without blocking.
Full transcript
Loading transcript…