AI Engineer Summit 2023

Building Blocks for LLM Systems & Products

Eugene Yan· Senior Applied Scientist, Amazon17:24

Loading slides…

Read the talk

Building Blocks for LLM Systems & Products

Task-specific evaluations, carefully ranked context, factuality checks, and useful feedback turn model capabilities into products that can improve reliably.

From a talk by Eugene Yan

Before you start: Familiarity with LLM prompts and basic machine-learning evaluation will help; retrieval augmentation and factuality checks are explained as they appear.

Did that change actually improve the product?

You changed a prompt, added retrieval, or fine-tuned a model. How do you know whether the product improved? That question connects the four building blocks in Eugene Yan’s talk: evaluations, retrieval-augmented generation, guardrails, and collecting feedback. Evaluations are the foundation because they make the effects of the other changes visible.

In eval-driven development, task-specific tests guide what to build next. The same tests then become checks to run before deployment, helping catch regressions when a prompt or another part of the system changes. Yan points to OpenAI managers spending time writing and reviewing evaluations as an indication that this work deserves direct attention, not treatment as an afterthought.

Slide titled “Why do we need Evals?” with three bullets and a Jason Wei tweet highlighting managers at OpenAI writing evaluations.Open full source frame
Why evaluations matter: checking changes, testing before deployment, and OpenAI’s attention to evals.
0:391:05
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:39 · section reference included

What does a benchmark score measure?

Conventional machine learning offers relatively settled metrics: root mean square error for regression, precision and recall for classification, and nDCG for ranking. LLM evaluation introduces another variable between the task and the score: how the task is presented. MMLU, for example, tests knowledge and reasoning through multiple-choice questions spanning subjects such as computer science, mathematics, and US history. But a shared question set does not ensure a consistent evaluation procedure.

In Evaluating LLMs is a minefield, Arvind Narayanan and Sayash Kapoor ask whether an evaluation measures the model or the prompt’s ability to elicit the desired response. Anthropic’s Challenges in evaluating AI systems supplies a particularly small example: changing parentheses in multiple-choice formatting can change accuracy. Without a consistent procedure, apparently comparable benchmark scores can reflect differences in prompting as well as differences between models.

Reference quality creates a separate problem. In the summarization results Yan presents, generated summaries receive higher human ratings than the dataset’s reference summaries. His XSum example includes InstructGPT, rather than GPT-4. This comparison concerns the quality of existing references; it does not establish that models outperform human summarizers generally. A benchmark can become less informative when its reference answers no longer represent the quality the application needs.

Finally, even a carefully run benchmark may measure the wrong task. MMLU might be relevant to a college-level chatbot, but its availability does not make it a useful measure for every application. Evaluate the application on the work it is supposed to do. That is a different question from asking which model has the highest academic benchmark score.

1:381:54
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:38 · section reference included

Start small and make the task measurable

There is no settled recipe for every LLM evaluation, but that need not prevent a useful first version. Yan cites Teknium starting a domain-expert evaluation with 40 questions. The example makes starting small concrete; it does not establish a sufficient sample size for every application. A narrow set of relevant cases can begin guiding development while the evaluation grows.

The next move is to simplify the task until success can be checked directly.

TaskWhat to check
Content moderationPrecision and recall for the target category
SQL generationWhether execution returns the expected result
JSON extractionWhether keys and values match the expected output

Moderation can be framed around specific targets such as toxicity or bias. SQL evaluation can inspect the returned result rather than merely whether the query looks plausible. JSON extraction can compare the actual structure and values with an expected answer. These tasks remain tractable because the evaluator has something concrete to check.

Open-ended dialogue is harder to reduce to an expected answer. A strong LLM may be needed to judge the output, but the evaluator then has its own cost. Yan cites Jerry’s report that running 60 evaluations with GPT-4 was expensive, without supplying a dollar amount. The choice of judge therefore affects how frequently the team can afford to evaluate changes.

Automated checks also leave room for direct inspection. Jonathan at Mosaic used a prompt that generated games for children of different ages, then inspected the outputs across training epochs. That made qualities visible that the available evaluations did not capture. Manual inspection complements the test suite by revealing what the team has not yet learned to measure.

3:544:02
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:54 · section reference included

An answer in the context can still be missed

Retrieval-augmented generation, or RAG, supplies knowledge as input context instead of relying entirely on what the model learned during training. Yan describes this as a cheaper and more precise way to supply new knowledge than continuously fine-tuning. Retrieving the right documents is difficult, but another problem remains after retrieval succeeds: the generator still has to use those documents correctly.

The question-answering experiment discussed in Lost in the Middle makes that distinction concrete. It uses historical Google queries and hand-annotated answers from Wikipedia. The presented setup supplies 20 documents of at most 100 tokens each, with one answer-bearing document and 19 distractors. That is at most 2,000 tokens of document content, not a limit on the complete prompt. The experiment changes the position of the answer-bearing document and measures how that affects answering.

In the result Yan presents, accuracy is highest when the answer-bearing document comes first and remains reasonably good when it comes last. In the middle, performance can fall below the baseline without retrieval augmentation. A larger context window therefore does not remove the need for good ranking: fitting the answer into the prompt is different from putting it where the model will use it effectively.

Yan reports roughly 75% accuracy with the answer-bearing document first in this 20-document question-answering experiment. That is a result for the presented setup, not a universal ceiling for RAG. It nevertheless exposes an important boundary: even ideal placement of the retrieved answer does not guarantee correct generation. Retrieval quality and the model’s use of evidence need to be evaluated separately.

5:445:59
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:44 · section reference included

When the context does not support a recommendation

A second RAG failure appears when the retrieved context does not support the requested answer. Yan supplies a list of 20 science-fiction movies as examples of movies he likes, then asks whether he would like Twilight. Twilight is a romantic fantasy involving a girl, a vampire, and a werewolf, so the preference evidence is a questionable fit. The prompt includes an explicit escape route: respond with not applicable when the supplied movie preferences do not support the recommendation.

The response first recognizes that Twilight belongs to a different genre and is not really science fiction. Then it draws a connection to E.T. through interspecies relationships. Recognizing a mismatch did not produce abstention; the model found a way to answer anyway. Yan connects this behavior to the pressure to be helpful: with fuzzy notions such as taste and relevance, a capable model can manufacture a plausible connection instead of acknowledging that the evidence is insufficient.

Search and recommendation systems already contain useful defenses. Rank the most relevant items highly, and apply a relevance threshold before passing context to the generator. For the Twilight example, Yan proposes measuring item distance and stopping before generation if the candidate is too far from the liked movies. This is a proposed filter, not a demonstrated threshold or a validated fix. Its purpose is to keep unsuitable evidence out of the prompt rather than depend entirely on the model to reject it.

7:498:03
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

7:49 · section reference included

Check factual consistency one claim at a time

Production guardrails include harmful-content checks, such as the hate, harassment, and self-harm categories Yan points to in OpenAI’s moderation API. They also include factual consistency: whether an output is supported by its source. Unsupported claims can destroy user trust even when the output passes a content-safety check. A factuality guardrail can therefore also be understood as an evaluation for hallucination.

Summarization research offers a useful approach through natural language inference, or NLI. Given a premise and a hypothesis, the task distinguishes three relationships. For the premise “John likes all fruits,” Yan’s examples are:

HypothesisRelationshipReason
John likes apples.EntailmentSupported by the premise
John eats apples daily.NeutralNot enough information
John dislikes apples.ContradictionConflicts with the premise

The neutral category matters: a statement can lack support without being contradicted by the source.

For summarization, use the source document as the premise and the summary’s claims as hypotheses. The granularity of this comparison matters. In Yan’s example, the final sentence of a summary is incorrect, but a single judgment over the whole document and summary misses the error. Checking at sentence level exposes the incorrect final sentence. The presented ablation reports better factual-inconsistency detection as document granularity becomes finer, moving from document to paragraph to sentence. The point is to prevent an otherwise plausible summary from hiding one unsupported claim.

Slide shows a document-to-summary inference diagram and a balanced accuracy table: document 57.4, paragraph 65.2, and sentence 70.3, with sentence-level summary granularity.Open full source frame
Sentence-level NLI checks and results across document granularities.
9:449:58
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:44 · section reference included

Agreement and strong-model judging

Another approach uses repeated generation as a consistency signal:

  1. Generate several summaries from the same input document.
  2. Compare them with measures such as n-gram overlap or BERTScore.
  3. Treat substantial disagreement as a possible hallucination signal.

The intuition is that summaries grounded in the same document should share information, while unstable details may indicate unsupported generation. This remains a heuristic: agreement alone does not establish that the shared claims are supported by the source.

A strong LLM can also receive the source document and summary directly and return a score. That is conceptually simple, but running a capable model as a judge adds expense. Yan reports seeing simpler factual-consistency methods outperform LLM-based approaches at substantially lower cost. This is a reason to test simpler methods against the application’s needs, not a universal ordering of factuality evaluators.

11:4111:59
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:41 · section reference included

A copied answer is not a correct answer

Collecting feedback closes the loop between product use and evaluation. It reveals what customers like and dislike, and it can supply examples for both evaluation and fine-tuning. Those datasets are transferable assets: models may change, while the accumulated record of useful and unsuccessful behavior remains valuable.

The challenge depends on how the feedback is collected. Explicit ratings are sparse. Yan asks how many audience members use ChatGPT, then how many click its thumbs-up or thumbs-down buttons. The much smaller response to the second question illustrates why adding rating buttons does not guarantee a useful volume of feedback. It is an audience demonstration, not a measured response rate.

Implicit feedback comes from ordinary product use and can be more plentiful, but its meaning is noisy. Clicking a copy-code button shows that someone wanted to take the snippet elsewhere; it does not show that the code works. Yan’s example supplies nrows to pandas.read_parquet. In the versioned pandas API relevant to this example, nrows is not a pandas-level row-limit parameter. Treating every copied snippet as positive feedback would therefore turn this erroneous answer into a positive training example.

12:3212:45
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

12:32 · section reference included

Make preference visible through normal use

GitHub Copilot offers an example of feedback embedded in the task itself. A developer writes a function signature or comments, receives a code suggestion, and can accept it, reject it, or move to another suggestion. These choices recur throughout the working day. They expose user decisions without requiring a separate trip to a feedback form, although acceptance still should not be mistaken for proof of correctness.

In the Midjourney interface shown in the talk, a prompt produces four candidate images. The user can rerun generation, request a variation using a V control, or upscale an image using a U control. Yan reads these actions as different strengths of preference:

ActionYan’s reward interpretationInferred preference
RerunNegativeNone of these candidates works
VarySmall positiveThis candidate has potential
UpscaleLarger positiveThis is an image to use

These are interpretations of observable choices, not verified Midjourney training labels or reward weights. The design lesson is to make progress through the product also reveal what users prefer, while remembering that any implicit signal needs interpretation.

Four generated images of yellow blocks above Midjourney controls, with arrows labeling upscale as positive reward, variations as small positive reward, and rerunning as negative reward.Open full source frame
Yan’s slide interprets Midjourney choices as positive, small positive, and negative rewards.
14:1414:23
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

14:14 · section reference included

Automate the checks and reuse what works

The operational starting point is modest: annotate 30 or 100 examples, then work out how to automate their evaluation. Run the checks as prompts, retrieval, and fine-tuning change. Manual inspection can remain a final check, but it cannot support every iteration at scale. Yan reports running tens of experiments each day and credits automated evaluations with making that pace possible. The value is both faster experimentation and a safer path to deployment.

The retrieval system does not need to start from scratch either. BM25, metadata fetching and matching, two-stage retrieval and ranking, and filtering already address the problem of selecting useful information. These established techniques aim to put the most relevant items first—the same property that matters when retrieved documents become model context.

The final building decision is how people encounter the model. Copilot puts assistance inside an IDE; ChatGPT makes it available through a chat window. Users can work in a familiar context without calling an API themselves. That interface also determines which choices become visible as feedback. UX therefore shapes both the immediate usefulness of the model and the evidence available to improve the product next.

15:4315:55
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

15:43 · section reference included

Resources

From the talk

Transcript download
Full transcript

Loading transcript…