AI Engineer Summit 2023
Building Blocks for LLM Systems & Products
Loading slides…
Read the talk
Building Blocks for LLM Systems & Products
Task-specific evaluations, carefully ranked context, factuality checks, and useful feedback turn model capabilities into products that can improve reliably.
From a talk by Eugene Yan
Before you start: Familiarity with LLM prompts and basic machine-learning evaluation will help; retrieval augmentation and factuality checks are explained as they appear.
Did that change actually improve the product?
You changed a prompt, added retrieval, or fine-tuned a model. How do you know whether the product improved? That question connects the four building blocks in Eugene Yan’s talk: evaluations, retrieval-augmented generation, guardrails, and collecting feedback. Evaluations are the foundation because they make the effects of the other changes visible.
In eval-driven development, task-specific tests guide what to build next. The same tests then become checks to run before deployment, helping catch regressions when a prompt or another part of the system changes. Yan points to OpenAI managers spending time writing and reviewing evaluations as an indication that this work deserves direct attention, not treatment as an afterthought.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
What does a benchmark score measure?
Conventional machine learning offers relatively settled metrics: root mean square error for regression, precision and recall for classification, and nDCG for ranking. LLM evaluation introduces another variable between the task and the score: how the task is presented. MMLU, for example, tests knowledge and reasoning through multiple-choice questions spanning subjects such as computer science, mathematics, and US history. But a shared question set does not ensure a consistent evaluation procedure.
In Evaluating LLMs is a minefield, Arvind Narayanan and Sayash Kapoor ask whether an evaluation measures the model or the prompt’s ability to elicit the desired response. Anthropic’s Challenges in evaluating AI systems supplies a particularly small example: changing parentheses in multiple-choice formatting can change accuracy. Without a consistent procedure, apparently comparable benchmark scores can reflect differences in prompting as well as differences between models.
Reference quality creates a separate problem. In the summarization results Yan presents, generated summaries receive higher human ratings than the dataset’s reference summaries. His XSum example includes InstructGPT, rather than GPT-4. This comparison concerns the quality of existing references; it does not establish that models outperform human summarizers generally. A benchmark can become less informative when its reference answers no longer represent the quality the application needs.
Finally, even a carefully run benchmark may measure the wrong task. MMLU might be relevant to a college-level chatbot, but its availability does not make it a useful measure for every application. Evaluate the application on the work it is supposed to do. That is a different question from asking which model has the highest academic benchmark score.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Start small and make the task measurable
There is no settled recipe for every LLM evaluation, but that need not prevent a useful first version. Yan cites Teknium starting a domain-expert evaluation with 40 questions. The example makes starting small concrete; it does not establish a sufficient sample size for every application. A narrow set of relevant cases can begin guiding development while the evaluation grows.
The next move is to simplify the task until success can be checked directly.
| Task | What to check |
|---|---|
| Content moderation | Precision and recall for the target category |
| SQL generation | Whether execution returns the expected result |
| JSON extraction | Whether keys and values match the expected output |
Moderation can be framed around specific targets such as toxicity or bias. SQL evaluation can inspect the returned result rather than merely whether the query looks plausible. JSON extraction can compare the actual structure and values with an expected answer. These tasks remain tractable because the evaluator has something concrete to check.
Open-ended dialogue is harder to reduce to an expected answer. A strong LLM may be needed to judge the output, but the evaluator then has its own cost. Yan cites Jerry’s report that running 60 evaluations with GPT-4 was expensive, without supplying a dollar amount. The choice of judge therefore affects how frequently the team can afford to evaluate changes.
Automated checks also leave room for direct inspection. Jonathan at Mosaic used a prompt that generated games for children of different ages, then inspected the outputs across training epochs. That made qualities visible that the available evaluations did not capture. Manual inspection complements the test suite by revealing what the team has not yet learned to measure.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
An answer in the context can still be missed
Retrieval-augmented generation, or RAG, supplies knowledge as input context instead of relying entirely on what the model learned during training. Yan describes this as a cheaper and more precise way to supply new knowledge than continuously fine-tuning. Retrieving the right documents is difficult, but another problem remains after retrieval succeeds: the generator still has to use those documents correctly.
The question-answering experiment discussed in Lost in the Middle makes that distinction concrete. It uses historical Google queries and hand-annotated answers from Wikipedia. The presented setup supplies 20 documents of at most 100 tokens each, with one answer-bearing document and 19 distractors. That is at most 2,000 tokens of document content, not a limit on the complete prompt. The experiment changes the position of the answer-bearing document and measures how that affects answering.
In the result Yan presents, accuracy is highest when the answer-bearing document comes first and remains reasonably good when it comes last. In the middle, performance can fall below the baseline without retrieval augmentation. A larger context window therefore does not remove the need for good ranking: fitting the answer into the prompt is different from putting it where the model will use it effectively.
Yan reports roughly 75% accuracy with the answer-bearing document first in this 20-document question-answering experiment. That is a result for the presented setup, not a universal ceiling for RAG. It nevertheless exposes an important boundary: even ideal placement of the retrieved answer does not guarantee correct generation. Retrieval quality and the model’s use of evidence need to be evaluated separately.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
When the context does not support a recommendation
A second RAG failure appears when the retrieved context does not support the requested answer. Yan supplies a list of 20 science-fiction movies as examples of movies he likes, then asks whether he would like Twilight. Twilight is a romantic fantasy involving a girl, a vampire, and a werewolf, so the preference evidence is a questionable fit. The prompt includes an explicit escape route: respond with not applicable when the supplied movie preferences do not support the recommendation.
The response first recognizes that Twilight belongs to a different genre and is not really science fiction. Then it draws a connection to E.T. through interspecies relationships. Recognizing a mismatch did not produce abstention; the model found a way to answer anyway. Yan connects this behavior to the pressure to be helpful: with fuzzy notions such as taste and relevance, a capable model can manufacture a plausible connection instead of acknowledging that the evidence is insufficient.
Search and recommendation systems already contain useful defenses. Rank the most relevant items highly, and apply a relevance threshold before passing context to the generator. For the Twilight example, Yan proposes measuring item distance and stopping before generation if the candidate is too far from the liked movies. This is a proposed filter, not a demonstrated threshold or a validated fix. Its purpose is to keep unsuitable evidence out of the prompt rather than depend entirely on the model to reject it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Check factual consistency one claim at a time
Production guardrails include harmful-content checks, such as the hate, harassment, and self-harm categories Yan points to in OpenAI’s moderation API. They also include factual consistency: whether an output is supported by its source. Unsupported claims can destroy user trust even when the output passes a content-safety check. A factuality guardrail can therefore also be understood as an evaluation for hallucination.
Summarization research offers a useful approach through natural language inference, or NLI. Given a premise and a hypothesis, the task distinguishes three relationships. For the premise “John likes all fruits,” Yan’s examples are:
| Hypothesis | Relationship | Reason |
|---|---|---|
| John likes apples. | Entailment | Supported by the premise |
| John eats apples daily. | Neutral | Not enough information |
| John dislikes apples. | Contradiction | Conflicts with the premise |
The neutral category matters: a statement can lack support without being contradicted by the source.
For summarization, use the source document as the premise and the summary’s claims as hypotheses. The granularity of this comparison matters. In Yan’s example, the final sentence of a summary is incorrect, but a single judgment over the whole document and summary misses the error. Checking at sentence level exposes the incorrect final sentence. The presented ablation reports better factual-inconsistency detection as document granularity becomes finer, moving from document to paragraph to sentence. The point is to prevent an otherwise plausible summary from hiding one unsupported claim.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Agreement and strong-model judging
Another approach uses repeated generation as a consistency signal:
- Generate several summaries from the same input document.
- Compare them with measures such as n-gram overlap or BERTScore.
- Treat substantial disagreement as a possible hallucination signal.
The intuition is that summaries grounded in the same document should share information, while unstable details may indicate unsupported generation. This remains a heuristic: agreement alone does not establish that the shared claims are supported by the source.
A strong LLM can also receive the source document and summary directly and return a score. That is conceptually simple, but running a capable model as a judge adds expense. Yan reports seeing simpler factual-consistency methods outperform LLM-based approaches at substantially lower cost. This is a reason to test simpler methods against the application’s needs, not a universal ordering of factuality evaluators.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A copied answer is not a correct answer
Collecting feedback closes the loop between product use and evaluation. It reveals what customers like and dislike, and it can supply examples for both evaluation and fine-tuning. Those datasets are transferable assets: models may change, while the accumulated record of useful and unsuccessful behavior remains valuable.
The challenge depends on how the feedback is collected. Explicit ratings are sparse. Yan asks how many audience members use ChatGPT, then how many click its thumbs-up or thumbs-down buttons. The much smaller response to the second question illustrates why adding rating buttons does not guarantee a useful volume of feedback. It is an audience demonstration, not a measured response rate.
Implicit feedback comes from ordinary product use and can be more plentiful, but its meaning is noisy. Clicking a copy-code button shows that someone wanted to take the snippet elsewhere; it does not show that the code works. Yan’s example supplies nrows to pandas.read_parquet. In the versioned pandas API relevant to this example, nrows is not a pandas-level row-limit parameter. Treating every copied snippet as positive feedback would therefore turn this erroneous answer into a positive training example.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make preference visible through normal use
GitHub Copilot offers an example of feedback embedded in the task itself. A developer writes a function signature or comments, receives a code suggestion, and can accept it, reject it, or move to another suggestion. These choices recur throughout the working day. They expose user decisions without requiring a separate trip to a feedback form, although acceptance still should not be mistaken for proof of correctness.
In the Midjourney interface shown in the talk, a prompt produces four candidate images. The user can rerun generation, request a variation using a V control, or upscale an image using a U control. Yan reads these actions as different strengths of preference:
| Action | Yan’s reward interpretation | Inferred preference |
|---|---|---|
| Rerun | Negative | None of these candidates works |
| Vary | Small positive | This candidate has potential |
| Upscale | Larger positive | This is an image to use |
These are interpretations of observable choices, not verified Midjourney training labels or reward weights. The design lesson is to make progress through the product also reveal what users prefer, while remembering that any implicit signal needs interpretation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Automate the checks and reuse what works
The operational starting point is modest: annotate 30 or 100 examples, then work out how to automate their evaluation. Run the checks as prompts, retrieval, and fine-tuning change. Manual inspection can remain a final check, but it cannot support every iteration at scale. Yan reports running tens of experiments each day and credits automated evaluations with making that pace possible. The value is both faster experimentation and a safer path to deployment.
The retrieval system does not need to start from scratch either. BM25, metadata fetching and matching, two-stage retrieval and ranking, and filtering already address the problem of selecting useful information. These established techniques aim to put the most relevant items first—the same property that matters when retrieved documents become model context.
The final building decision is how people encounter the model. Copilot puts assistance inside an IDE; ChatGPT makes it available through a chat window. Users can work in a familiar context without calling an API themselves. That interface also determines which choices become visible as feedback. UX therefore shapes both the immediate usefulness of the model and the evidence available to improve the product next.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Seven production patterns spanning evaluations, retrieval, fine-tuning, caching, guardrails, UX, and feedback.
Annotated slides on prompt sensitivity, construct validity, and benchmark contamination.
Anthropic's account of practical difficulties in multiple-choice evaluations and broader model assessments.
Versioned API reference for reading Parquet files into pandas.
Further reading
Yan's slides and talking points, with links to supporting research.
- Lost in the MiddlePaper
Controlled experiments testing how answer position affects question answering over retrieved documents.
- SummaCPaper
Sentence-level natural language inference and aggregation for detecting inconsistent summaries.
- SelfCheckGPTPaper
A method that uses agreement across sampled generations as a hallucination signal.
Human evaluation of LLM summaries, benchmark references, and summaries commissioned from freelance writers.
Full transcript
Loading transcript…