AI Engineer World's Fair 2024

What We Learned From A Year of Building With LLMs

Eugene Yan· Senior Applied Scientist, AmazonHamel Husain· Founder, Parlance LabsJason Liu· Consultant (Instructor)Bryan Bischof· Head of AI, HexCharles Frye· AI Engineer, ModalShreya Shankar· UC Berkeley35:21

Loading slides…

Read the talk

What We Learned From A Year of Building With LLMs

Useful LLM products depend on domain expertise, deliberate hiring, and a feedback loop that turns real user interactions into evaluations, better decisions, and production guardrails.

From a talk by Eugene Yan, Hamel Husain, Jason Liu, Bryan Bischof, Charles Frye and Shreya Shankar

Before you start: Familiarity with LLM APIs and retrieval-augmented generation will help; no model-training background is required.

Six practitioners encounter the same problems

What happens when six practitioners discover that they are all about to write the same lessons-learned article? After a year of posting, blogging, and comparing frustrations in a group chat, Eugene Yan, Hamel Husain, Jason Liu, Bryan Bischof, Charles Frye, and Shreya Shankar combined their experiences into What We’ve Learned From a Year of Building with LLMs, published through O’Reilly. Their recurring problems crossed three levels: choosing what to build, organizing the work, and making the system reliable.

Article page titled “What We’ve Learned From a Year of Building with LLMs,” overlaid with “…became a white paper…”Open full source frame
A group chat became “What We’ve Learned From a Year of Building with LLMs.”

The collaboration became a 30-page article and then an unusual six-person keynote, with room for sharper opinions that had missed the written version. The Avengers comparison fit the format, though the authors pushed back on being mistaken for opportunists arriving from crypto: their experience extended back to writing neural-network gradients by hand. Charles and Bryan begin with strategy, Jason and Hamel move into operations, and Shreya and Eugene finish with production techniques.

0:240:42
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:24 · section reference included

Build a product advantage that survives better models

How do you build an LLM application without getting outmaneuvered or wasting everyone’s money? Training a custom model can look like the obvious route: pretraining, fine-tuning, perhaps some reinforcement learning from human feedback. Bryan rejects that as the default. For almost everyone in the audience, the model is not the moat. Product expertise, an existing customer relationship, or a well-understood niche offers a more useful starting point.

Build where model providers are unlikely to concentrate. Providers must serve requirements shared across many customers; reproducing those common capabilities creates exposure to the next platform release—the steamrolling risk Bryan invokes. Treat the underlying model like another SaaS dependency and replace it when a competitor becomes clearly better. His contemporary comparison is GPT-4o versus Claude 3.5 Sonnet, illustrating the willingness to switch rather than prescribing a permanent winner.

Benchmark performance does not define a customer outcome. Bryan uses an illustrative 87% SpiderSQL score to explain why benchmark accuracy does not imply automating 87% of a company’s data requests. High MMLU performance and a HumanEval score are similarly difficult to sell as products in themselves. A product still needs good design, a job to be done, and a way to make its user more capable.

That suggests three durable places to invest:

  • Adaptability: capabilities that remain useful as models become smarter and faster.
  • Quality control: mechanisms that maintain the product’s quality bar despite uncertain outputs.
  • Continuous improvement: ways to learn from use and make the next version better.
3:233:31
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:23 · section reference included

Evaluation belongs inside an improvement loop

The reason to collect data and create evaluations is to move an improvement loop forward. Charles starts with Hamel’s diagram of that loop, then connects it to Shreya’s work on operationalizing machine learning: collect data, experiment, evaluate and deploy, monitor the result, and feed what you learn into the next iteration. Evals occupy the center because they help determine whether a change worked. Collecting them is not the endpoint.

Four connected boxes show Data Collection, Experimentation, Evaluation and Deployment, and Monitoring and Response, with a return arrow above.Open full source frame
The ML engineering workflow loops monitoring back into data collection.

The pattern predates LLMs. MLOps inherited iterative improvement and production monitoring from DevOps. DevOps, in turn, drew on the Lean Startup movement’s use of measurement to improve an entire business. Charles traces the lineage further back to the Toyota Production System and Kaizen, or continuous improvement. Each version addresses uncertainty by observing the real system and using that observation to decide what to change.

Toyota’s Genchi Genbutsu principle brings this down to a concrete practice: go to the real place and inspect the real thing. For executives, that meant the factory floor. For LLM engineers, it means reading actual application data—the requests, responses, and interactions that show whether users receive value.

Tools can make the loop easier to operate, but they can also become a distraction. Charles invokes a line he attributes to Shigeo Shingo: “Value is only created when metal gets bent.” Building an evaluation framework or calculating concept drift matters when it leads to a better experience for users. The machinery of improvement must eventually produce an improvement.

6:066:14
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:06 · section reference included

Get real interactions into the loop

Evals express objectives: what should the system do? Starting with a convenient bespoke metric reverses that relationship. Bryan recommends beginning with any adequate system for capturing the behavior that matters, rather than waiting for the perfect evaluation framework.

A beta can start that process. Bryan points to Apple Intelligence, Photoshop, and Hex as examples of products using that route to put capabilities in front of people. Once users interact with the system, inspect the responses with human eyes. AI review can help, but it does not remove the need for direct human inspection.

Start with binary human feedback, then add richer explanations where they help. Those interactions expose more than quality problems: users ask chatbots to do things the product does not yet support. Repeated unmet requests can reveal where product-market fit might lie. The same stream of data can therefore guide both reliability work and the product roadmap.

10:0110:11
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:01 · section reference included

Prototype for economics that may become possible

Immediate user feedback does not eliminate the need to plan for future capabilities. Alan Kay and colleagues at PARC projected Moore’s Law forward, then built expensive, slow, buggy systems to experience the computing environments that would eventually become affordable. A prototype could be economically impractical and still teach its builders what the future product should feel like.

Charles applies that approach to inference costs, while acknowledging that the scaling evidence is less established than the industrial evidence behind Moore’s Law. Charles reports roughly tenfold cost reductions every 12–18 months at three capability levels: GPT-3 davinci, text-davinci-002, and GPT-4/Sonnet. His shorthand is about fifteen months per tenfold reduction, faster than Moore’s Law. The companion analysis uses five-shot MMLU as a capability proxy; matching a benchmark score does not establish equivalent behavior in a particular product.

The planning question is which currently uneconomical applications might become viable by the next fundraising round. Charles estimates a chatbot-NPC game at about $625 per hour in 2023, compared with roughly $6 per hour for inflation-adjusted 1980 Pac-Man play. If the assumed cost trend continued, two tenfold reductions over about thirty months from mid-2023 would bring the chatbot game to roughly $6 per hour. Those are approximate economics and a conditional forecast, including the expectation that people would pay. But they motivate useful work before commercialization: experience the game, design its interactions, and learn what makes it compelling.

11:5712:17
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:57 · section reference included

Buying another shovel does not tell you where to dig

Jason turns the operational discussion into deliberately bad advice. Drawing on consulting work with Hamel across roughly twenty companies, from pre-seed startups to public businesses, he offers a recipe for ruining a business: keep buying shovels. Where should you dig? Buy another shovel. When should you stop? Surely the shovel will tell you. Should you dig one deep hole or many shallow ones? More shovels.

The software equivalents are familiar: a failing RAG application prompts a vector-database swap; an ineffective method prompts implementation of another paper; weak product-market fit prompts an embedding-model upgrade. Any of those changes might solve an identified technical problem. None supplies the missing decision process by itself.

The satire targets substitution of tools for expertise and process. Teams still need to decide where to experiment, when to stop, and when to exploit a mechanism that already works. A better tool cannot make those choices on their behalf.

14:5215:10
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

14:52 · section reference included

Premature model ownership creates the wrong work

Jason’s next satirical recommendation is to replace a $2,000 monthly OpenAI bill with a $250,000 hire plus 1% equity. The new engineer can fight CUDA build errors and server cold starts. Anxiety about being a wrapper then becomes another reason to fine-tune, even when the harder business problem is building something worth charging for. The comparison exposes the costs that disappear from view when API spend is considered in isolation.

Hiring a machine-learning specialist before a data-generating product also creates a mismatch between the hire and the work. The urgent task may be fixing a Vercel TypeScript build, not training a model. Jason extends the joke to claims that Python is dead, that its ecosystem can be rebuilt over a few weekends, and that tooling, product sense, and data literacy transfer effortlessly between communities. The sarcasm depends on the opposite being true: these are substantial bodies of expertise.

A catch-all title does not resolve that mismatch. Ninja, wizard, data scientist, prompt engineer, or AI engineer can all conceal an unspecified set of outcomes. Jason hands the problem to Hamel with a familiar prompting joke: “take a deep breath, think out loud, step by step.” The operational task is to become more explicit about the skills and results the business actually needs.

16:2316:41
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

16:23 · section reference included

Evaluation is a core engineering skill

Hamel values the AI engineer label popularized by Swyx because it brings a community together. The problem is inflated expectations about what the title guarantees. Talent is a major lever for founders and engineering leaders, so misunderstanding the required skills directly affects what the team can deliver.

A familiar skills diagram places AI engineering on the application side of an API dividing line, emphasizing chains, agents, tooling, and infrastructure. Evals and data are conspicuously absent. Read too literally, that diagram suggests that application builders can ignore evaluation. Tools and vibe checks can get a team from zero to one quickly, but without measurement the team cannot reliably identify improvements. The initial speed gives way to stagnation.

Slide titled “Without Evals, We Cannot Make Progress,” with a Stagnation banner across a loop between prompting tools and Vibe Checks.Open full source frame
Without evals, the prompt-and-vibe-check loop leads to stagnation.

Hamel reports that four to six weeks of deliberate practice can make engineers effective at writing evals and working with data. This is his and Jason’s consulting experience, not a guaranteed training schedule. The practical implication is that evaluation and data literacy belong in the core of AI engineering and can be deliberately developed within the team.

18:2418:40
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

18:24 · section reference included

Hire for the next stage of the product

An ambiguous AI role can become the destination for every AI-related failure, regardless of whether the person has the skills to fix it. Data science provides a precedent: the original unicorn combined software engineering, statistics, mathematics, and domain expertise. Over time, organizations separated that work into roles such as decision scientist, machine-learning engineer, and data engineer. AI engineering may undergo a similar differentiation.

Specify both the skills and the product context. Hamel uses GitHub Copilot job postings to illustrate that specificity, then describes a hiring sequence tied to maturity:

  1. Build the application. Application development, software engineering, and AI engineering get the product from zero to one.
  2. Capture its data. Platform and data engineering make interactions available for analysis and improvement.
  3. Add machine-learning specialization. Hire ML engineers once there is data for them to work with.

The warning is straightforward: do not hire a machine-learning engineer into a product that has no data. Developing eval and data skills in the existing team can extend its effectiveness before that specialization is needed. The recurring failure mode is an AI engineer role that is over-scoped but under-specified.

21:0021:19
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

21:00 · section reference included

Turn broad quality goals into testable criteria

Eugene begins the tactical discussion with a Hacker News observation: the importance a team gives evaluation separates poor outputs from real products. The useful question is what to evaluate for this product. He points to Apple’s foundation-model evaluation, which uses summaries representative of actual features. Apple’s published evaluation sampled 750 responses for each summarization use case: email, messages, and notifications. The unit of evaluation is the feature people use, rather than a generic model score.

Start by decomposing the task. Product-attribute extraction becomes checks for title, price, and rating. Summarization becomes dimensions such as factual inconsistency, relevance, and informational density. Natural-language-to-SQL generation can include checks for expected tables and columns. Narrow criteria make failures easier to identify and give simple assertions useful work to do.

For an illustrative product record, the extraction checks can be ordinary Python assertions. This fixture represents a description of a Trail Mug priced at $18.50 with a 4.6 rating; each field has a separate failure message.

python

from decimal import Decimal


def check_product(actual: dict, expected: dict) -> None:
    assert actual["title"] == expected["title"], "incorrect title"
    assert Decimal(str(actual["price"])) == expected["price"], "incorrect price"
    assert actual["rating"] == expected["rating"], "incorrect rating"


expected = {
    "title": "Trail Mug",
    "price": Decimal("18.50"),
    "rating": 4.6,
}
extracted = {"title": "Trail Mug", "price": "18.50", "rating": 4.6}
check_product(extracted, expected)

These checks isolate field correctness; they do not require a model to judge the entire response as one undifferentiated object.

Assertions eventually reach their limits. For criteria requiring semantic judgment, Eugene suggests a factual-inconsistency classifier or a relevance reward model. Formulating the criterion as classification or regression gives an evaluator a more precise job than deciding whether an output is broadly good.

23:5424:05
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

23:54 · section reference included

Choose an evaluator you can align and maintain

Shreya makes the case for LLM-as-a-judge as an easy starting point. Write a prompt describing the criterion, then provide a few examples of good and bad outputs to express your preferences. A trained evaluator demands more upfront work: collecting data and building a training pipeline before it can help.

Eugene’s concern is alignment with business-specific criteria. A prompted judge may be easy to create but difficult to make precise. Eugene suggests that a few hundred to a few thousand examples may suffice to fine-tune a simpler, more precise evaluator. He also estimates five to eight seconds per request for a judge using chain-of-thought, versus roughly ten milliseconds for a classifier or reward model. These are estimates without a specified model, hardware, or load; their practical significance is the possible effect on throughput and response latency.

Both approaches create maintenance obligations, whether the validator runs asynchronously or in the request’s critical path. A prompted judge needs examples and instructions that continue to match the current definition of good and bad. A trained validator needs new production data and continued fine-tuning to resist drift. An evaluator can become stale even when the code calling it has not changed.

DecisionPrompted LLM judgeTrained evaluator
Starting investmentCriteria prompt and examplesLabeled data and training pipeline
Alignment workRevise prompts and examplesTrain on task-specific labels
Ongoing maintenanceRefresh the definition of goodUpdate with production data
Best initial fitLow-volume, quick prototypesGreater resources and product confidence

Shreya frames the choice around resources and application maturity. For a prototype needing quick evals with little development effort, start with a judge and invest in keeping it aligned. As confidence in the product and available resources grow, a trained evaluator becomes more attractive.

25:4126:00
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

25:41 · section reference included

Inspect recognizable failures with their execution context

Where do the evaluation criteria come from? Regular inspection of production data. Shreya describes a startup that creates a dedicated Slack channel whenever it ships a new agent and streams the agent’s outputs there in real time. After a couple of weeks, it transitions to daily batch inspection. The cadence changes, but observation continues beyond the initial deployment.

Look for characterizable slices: outputs from a particular source, containing a keyword or phrase, or concerning a topic. A collection labeled bad is hard to act on if the failures have no recognizable common property. A slice gives the team something it can locate again, investigate, and use to assess a pipeline change.

Preserve the execution context alongside those outputs. Record the GitHub commit, model version, and prompt version so an inspected trace can be connected to the system that produced it. Shreya points to MLflow as an example of the traceability that traditional MLOps tools made useful, with replay as a possible next step.

Finally, pin model versions where the provider supports it. Shreya’s historical examples are GPT-4 1106 and GPT-4o; in an implementation, a model-family name alone does not establish that a fixed snapshot is selected. The purpose is to limit behavior changes that are difficult to quantify and to preserve a useful explanation of what ran. Version records aid diagnosis, but do not by themselves guarantee exact reproduction of a hosted model’s output.

“How to ‘look at your data’” checklist, with earlier inspection steps dimmed and “minimize data skew,” version tracing, and model pinning highlighted.Open full source frame
Inspect data slices, trace logs to prompt or codebase versions, and pin model versions.
28:3428:44
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

28:34 · section reference included

Turn reference-free evaluations into guardrails

Human inspection cannot examine every output continuously. Automated guardrails extend the checks across production traffic. Eugene adapts Brandolini’s law to emphasize the asymmetry: generating an output is easy; detecting and fixing its defects takes much more work. Basic checks include toxicity, personally identifiable information, copyright, and the expected language.

Even apparently straightforward requirements can fail because the application does not control all of its context. An advertisement submitted in another language to an English website can cause attribute extraction or summarization to return that language. A movie summary generated from a trailer description can introduce spoilers or other facts absent from the supplied text. The model’s attempt to be helpful can violate the actual product requirement.

The useful distinction is whether evaluation requires an ideal answer for each input.

Evaluation styleComparison targetProduction implication
Reference-basedA gold outputRequires an ideal sample for comparison
Reference-freeSource document and criteriaCan check newly generated outputs

Reference-free does not mean evidence-free or label-free. Labels can still train and evaluate the checking model. What disappears is the need to write a gold response for every new input.

For summaries, inspect whether the source supports the generated claims and whether they contradict it. That turns factual consistency into a reusable hallucination check: the incoming source document supplies the evidence for judging each new summary. An evaluation that works without a gold response can also become a production guardrail. The investment in defining and training the check can therefore serve both offline evaluation and new outputs.

Guardrails slide contrasts comparison to a gold reference with comparison to a source document, then recommends reusing reference-free evals for new outputs.Open full source frame
Reference-free evaluations can also serve as guardrails.
30:4330:50
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

30:43 · section reference included

The surrounding system still needs maintenance

The closing production-system diagram puts the model inside a much larger body of engineering work. Data verification, feature engineering, and infrastructure monitoring all require maintenance. Replacing a traditional model with an LLM does not remove those surrounding responsibilities or the technical debt they can create.

The responsibilities often return in a different form:

Traditional ML responsibilityLLM application counterpart
Feature engineeringRetrieve and shape relevant context through RAG
Data validation and verificationEvals and deployed guardrails
Infrastructure monitoringMonitor the production system around the model

RAG requires decisions about which context to retrieve and how to present it without distracting the model. Those decisions involve experiments and ongoing maintenance, especially as underlying models change. Evals and guardrails likewise have to remain useful as the application evolves. A model wrapped in software is only one component of the investment needed to operate the product.

32:3232:52
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

32:32 · section reference included

A compelling demo can precede a product by decades

Eugene ends with an observation attributed to Andrej Karpathy: a large class of problems is easy to imagine and demonstrate, yet extremely difficult to turn into products. Autonomous driving makes the gap tangible. The ALVINN paper from 1988 demonstrated neural-network road following under particular conditions. It was an early concrete result, not a finished unrestricted driving service.

The keynote then jumps twenty-five years to a demo ride Eugene says Karpathy took in 2013, labeled Waymo on the slide, and another decade to San Francisco driverless passenger service. The precise 2023 milestone was CPUC approval expanding paid driverless service to all times of day; earlier non-fared authorization already existed. The timeline shows how much development can separate an impressive demonstration from a service operating under real-world requirements.

Three-part timeline shows a 1988 neural-network car paper, a 2013 Waymo demo drive, and a 2023 driverless permit, connected by 25-year and 10-year arrows.Open full source frame
From a neural-network-driven car in 1988 to a Waymo driverless permit in 2023.

Eugene’s suggestion that the service might reach all of California a few years later remains a forecast within the talk. His closing point is more durable: getting from demo to production takes time. The work around the model—learning from users, defining quality, maintaining data and context, and catching failures—is how a team spends that time productively.

34:1734:28
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

34:17 · section reference included

Resources

From the talk

Updates since the talk

Transcript download
Full transcript

Loading transcript…