AI Engineer World's Fair 2024

Productionizing GenAI Models – Lessons from the world's best AI teams

Loading slides…

Read the talk

Productionizing GenAI Models – Lessons from the world's best AI teams

Lukas Biewald’s local voice assistant shows why production AI depends on exact outputs, fast responses, recorded experiments, and evaluations that guide the next change.

From a talk by Lukas Biewald

Before you start: Familiarity with LLM prompts, API calls, and basic Python is helpful; no model-training experience is required.

In production, but is it working?

How many people already have LLM applications in production at their companies? When Lukas Biewald asks the room, he estimates that more than 70% raise their hands. Why attend a talk on productionizing AI, then? An attendee answers: “It's not going great.” Deployment and satisfactory performance are different milestones.

Even the deployment category needs unpacking. Biewald estimates roughly 30% participation when he asks about custom solutions. Purchased solutions initially draw fewer hands, until someone points out that GitHub Copilot belongs in that category. The apparent adoption picture changes with what people count as an AI application.

Looking back ten and fifteen years—with traditional ML explicitly included—Biewald recalls an earlier expectation that AI would become accessible through AutoML or graphical interfaces. Instead, chat and conversational interfaces made that access commonplace. He reports seeing AI use in every company he speaks with and suspects that probably every Fortune 500 company is investing in custom AI, often combining GenAI with traditional ML. These are observations from his customer conversations, but the practical shift is clear: production use now extends beyond the companies people instinctively consider technologically adventurous. The question is no longer only when GenAI will reach production; it is how to make the applications already there work well.

Slide stating “Democratization of AI is here” beside a chart comparing AI maturity across industries in 2021 and 2024.Open full source frame
Democratization of AI, illustrated by industry maturity in 2021 and 2024.
0:180:28
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:18 · section reference included

A compelling demo hides the distance to production

AI is easy to demo and hard to productionize. An astonishing example can persuade a CEO to stop listening to customers and release something those customers find terrible. Biewald sees this in demonstrations from his own team and from startups pitching him. Software already has a gap between a promising prototype and a dependable product; AI makes that gap unusually easy to underestimate.

Dark slide displaying “AI: Easy to Demo, Hard to Productionize,” with the speaker inset at lower left.Open full source frame
AI: Easy to Demo, Hard to Productionize.

Biewald founded Weights & Biases to address a tooling gap in AI operations. Its users now span three groups, with different starting points:

  • Foundation model builders develop the underlying models and push the platform’s scale.
  • AI engineers build applications combining traditional ML and GenAI.
  • Software developers can now attempt applications that previously required building a custom model, but must learn how to move beyond a demo.

The last group is much larger than the first two. Its arrival expands both the number of possible AI applications and the need for tools that support the development process.

Biewald says W&B works with nearly all foundation model companies, naming OpenAI GPT, Mistral, and Llama as model families built using the platform. Yet most of its customers are application builders, not foundation model developers. Their work reaches healthcare, agricultural technology, and manufacturing—a broader set of tangible applications than the search ranking, finance, and ad targeting opportunities he remembers from the beginning of his career. That application experience is the basis for the production practices that follow.

3:243:33
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:24 · section reference included

The reusable asset is what the experiments taught you

Why do AI applications need different development tools? Adding an AI label to observability, CI/CD, or code versioning does not by itself answer that question. Biewald’s distinction concerns the work those tools must support: software development is generally additive, while LLM development is experimental. A new prompt or workflow is a hypothesis to test, not a feature whose addition guarantees progress. Outputs can vary, and a change can improve one case while breaking another. Broad behavioral evaluation therefore cannot be reduced to expecting every case to pass, even though narrow must-pass tests still have a role later in the workflow.

The learning is the intellectual property. A final model or prompt is an outcome of the development process. The unsuccessful prompts, rejected workflows, and reasons an approach failed tell the next engineer where to go—and where not to spend another week. If those discoveries live only in one person’s memory, the ability to continue iterating leaves with that person. Saving the final artifact alone does not preserve the work that produced it.

Reproducibility makes that learning usable, but recording enough information is painful, especially when experiments call nondeterministic endpoints. Biewald recommends passive capture by processes outside the engineer’s manual note-taking routine. People forget details even when they intend to document everything. Background tracking gives a colleague a way to reconstruct the experiment and pick up the investigation. Its practical return is shorter iteration time: preserved learning enables collaboration, and collaboration gets the application from demo to useful product faster.

6:216:35
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:21 · section reference included

Turning a weather question into a function call

Biewald encountered the production gap in a project he started during time off after his second child was born. His older daughter, Matilda, asked Alexa to play her favorite song and was surprised that it did not know which song she meant. She repeatedly requested Baby Shark; the preference seemed obvious to her father. He speculates that privacy considerations might explain the missing memory, but the immediate product request came from Matilda: could he build an Alexa?

He looked at Mycroft, which he found dated and not using LLMs at the time. Then a neighbor showed him a seven-billion-parameter model running on a ROCK Pi costing roughly $200; Biewald tentatively identifies the model as Llama 2. That suggested a feasible local assistant. He built a library of skills for music, weather, math, and news. The API wrappers were straightforward: the weather skill was the sort of short function Copilot could generate. The difficult part was recognizing speech and translating the resulting words into the correct API call.

For the input “What's the weather in Boston?”, the on-device LLM must produce a weather function call with Boston as the location. The essential Python contract can be expressed as:

python

expected_call = 'weather(location="Boston")'

def matches_weather_call(generated_call: str) -> bool:
    return generated_call.strip() == expected_call

This illustrates a strict output contract, not the project’s unspecified scoring implementation. A sentence explaining that the weather function should be called is not the same thing as producing the call. The downstream weather skill needs the function name and argument in the format it accepts before it can query its API.

The local stack combines Whisper for speech recognition with Llama for interpreting the request. Biewald chose llama.cpp to run the language model on inexpensive hardware; he also mentions vLLM and hosted providers as alternatives. Running the system himself was part of the appeal, and he reports that large LLMs could run in real time on a MacBook.

“How does it work” slide showing Whisper, a Boston weather question, an LLM block, a weather function call, and a Weather Skill block with an audio icon.Open full source frame
A weather question passes through Whisper, an LLM, and a weather skill.
9:229:34
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:22 · section reference included

Fast enough to answer, exact enough to run

Biewald skips the live demonstrations because he is unsure the audio will work, then turns to a constraint that becomes obvious when using the assistant: latency is part of correctness from the user’s perspective. Speech must first be transcribed, and the transcription then passes through an LLM to generate the function call. He describes delays beyond a couple hundred milliseconds as painful in this interaction; this is a responsiveness judgment, not a measured latency result. It motivates choosing a small model and running it quickly. At the time, that led him to Llama 2.

Biewald reports 0% function-call accuracy for Llama 2 with the default prompt. The accuracy figures throughout this project are his reported results; the recording does not specify evaluation-set size, train/test separation, scoring rules, or whether speech-recognition errors were included. The first intervention was ordinary prompt engineering: describe the available functions more clearly. That made the answer look closer to the desired call, but it still included a leading instruction to call the weather function rather than cleanly producing runnable output. Semantic proximity was not enough for the interface.

Switching to Llama 2-Chat, which was trained for conversation, raised reported function-call accuracy to 11%. Inspecting its errors and incorporating feedback raised reported accuracy to 75%. This was enough for a pleasing demo when Matilda phrased her request just right, but it remained an annoying assistant to use. The distinction matters: a carefully chosen successful interaction can hide the frequency of failures across ordinary requests.

11:5012:04
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:50 · section reference included

A new model, then better training examples

Mistral arrived during the project. Replacing Llama 2 with Mistral raised Biewald’s reported function-call accuracy from 75% to 79% without fine-tuning. In his setup, the replacement used the same API, so he could take advantage of the new model without rebuilding the application. This is a useful reason to keep model replacement easy: a new release can improve the existing task at little implementation cost. But the remaining failures were still too frequent to satisfy his daughter.

The next step was fine-tuning. LoRA and QLoRA had made this more tractable, with hosted services also available for teams that did not want to run training themselves. Biewald ran QLoRA in a few hours on a GPU he remembers, uncertainly, as a 4080. The remaining prerequisite was a dataset.

His data-building procedure was small enough to do personally:

  1. Write a few examples of requests paired with the desired skill calls.
  2. Ask ChatGPT to generate additional examples.
  3. Review the generated examples by hand and remove unsuitable ones.
  4. Specify an output schema and supply more examples to elicit a wider variety of cases.

Biewald estimates that about 95% of the generated examples were good and that he obtained a couple thousand fine-tuning examples in roughly 15 minutes. The larger model supplied training material for the smaller local model; manual review remained part of that process.

Fine-tuning Mistral raised Biewald’s reported function-call accuracy to 98%. At that point, the assistant began to feel potentially shippable to him. The change was not merely a more impressive isolated answer: it was a substantial improvement in how often the system produced the required function call.

Multilingual behavior was a welcome byproduct. Biewald tested Japanese and tentatively identifies the displayed example as French. Whisper could transcribe other languages, and the LLM could interpret those requests. For anyone following the repository setup, multilingual use requires a multilingual Whisper checkpoint: Otto’s README defaults to the English-only base.en checkpoint. The broader attraction of the project was that useful, interesting AI systems were again within reach of an individual hobbyist, even after the growth of model sizes had made that seem doubtful.

13:3513:46
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:35 · section reference included

Failed experiments and the problem of version two

The project is available as Otto. Its progression shows why prompting and fine-tuning are not opposing strategies: prompt engineering helped from a poor starting point, and fine-tuning supplied later gains. Biewald extends that observation to production applications more generally, where retrieval-augmented generation, prompting, and fine-tuning may all contribute. He does not describe adding RAG to this assistant; the point is that the appropriate techniques can be complementary.

The clean sequence of improvements leaves out much of the work. The project took a couple of weeks, and most experiments failed. A colleague who inherited only the final model and this success story could easily repeat those failures before catching up. Biewald says he recorded the experiments—including unsuccessful ones—in W&B and made them available for inspection. That history is the concrete version of preserving learning rather than just preserving the winning prompt.

The first broader production practice is to build an evaluation framework and take it seriously. Biewald recalls asking an unnamed CEO on Gradient Dissent how his company tested GenAI applications. The CEO said he tested by vibes and declined Biewald’s offer to remove the remark. There is some value in that approach: personally trying a product can reveal glaring failures that should have prevented a release.

The limit appears when it is time to release a second version. A model or prompt change can make some requests better and others worse. A subjective impression will not reliably distinguish the relatively small improvement Otto gained from its model swap, so it cannot confidently justify the replacement. Evaluation makes those tradeoffs visible. When Biewald checked back with the CEO a year later, the company was doing substantial evaluation work after confronting this problem. A framework for comparing versions is what turns experimentation into a release decision.

15:5616:07
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

15:56 · section reference included

Build the whole loop, then measure what matters

Evaluation belongs alongside three other practices: start with a lightweight prototype, incorporate end-user feedback, and iterate. Biewald sees an enterprise failure mode in which teams try to perfect the first stage before building the second, indefinitely postponing the moment anyone can use the application. A small end-to-end prototype should reach its intended audience, whether internal employees or external customers. Their feedback then gives the next experiment a purpose.

When Biewald asks how many attendees feel confident in their evaluations, participation is much lower than in the deployment poll. Teams that do mature their evaluations tend to use several sets and techniques rather than one all-purpose score:

Evaluation layerIntended role
Restricted must-never-fail setTarget 100% on simple, critical cases
Small, quick suiteReturn feedback in 20–30 seconds
Larger nightly suiteCover more cases outside the immediate iteration loop

The restricted set explains where an all-pass requirement can make sense. It protects a narrow contract; it does not imply that the application’s full behavioral evaluation should be perfect. The quick suite supports interactive experimentation, while the nightly suite supplies broader feedback.

Metrics must correspond to user experience or customer value. That relationship is application-dependent and difficult to establish. W&B initially expected teams to track perhaps 10–20 metrics per model or application. Biewald reports thousands or tens of thousands of metrics in production self-driving and mission-critical banking applications. Some customers even request regex search because they cannot find the metric they need among so many. The point is not to maximize the number of metrics; it is that a consequential application can fail in many distinct ways, and a single aggregate score may hide the failures users care about.

Actually running evaluations is what makes the other decisions tractable. Should this application use RAG? Would fine-tuning help? Is another technique worth adding? With a useful evaluation system, the team can try the change and answer for its own workload instead of treating the choice as a general debate. The evaluation loop connects a proposed intervention to evidence about whether it improves the product.

“Evaluation Best Practices” slide listing multiple evaluation sets or techniques, metrics correlated with user experience, and “Actually do it.”Open full source frame
Evaluation best practices: use multiple evaluation sets, align metrics with user experience, and actually evaluate.

Biewald closes by inviting attendees to see Weave, with the qualification that it can help with some production problems, not all of them. The current Weave evaluation guide is a follow-up resource rather than an API demonstrated in this recording. Tools can support the loop, but the application’s users and failure modes still determine what it needs to measure.

19:0519:19
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

19:05 · section reference included

Resources

From the talk

Updates since the talk

Transcript download
Full transcript

Loading transcript…