AI Engineer World's Fair 2024
LLM Quality Optimization Bootcamp
Loading slides…
Read the talk
LLM Quality Optimization Bootcamp
Follow a PII-redaction workflow from training data to LoRA deployment and evaluation, with the model choices, serving economics and live-demo failure that shape the result.
From a talk by Thierry Moreau and Pedro Torruella
Before you start: Familiarity with Python, chat-completion messages and JSON tool definitions will help you follow the notebook workflow.
When the prototype costs too much and still falls short
What do you do when a promising GenAI prototype still misses the production quality bar—or its inference bill has already become painful? Thierry Moreau opens with those two problems. He describes customers spending tens or hundreds of thousands of dollars per month while sometimes still struggling to deliver acceptable output.
The exercise follows a full deployment cycle: collect data, fine-tune a model, deploy it and evaluate its output. Its worked task is personally identifiable information (PII) redaction, a narrow application where a compact model can learn a specific behavior. GPU scarcity makes resource efficiency consequential: the goal is to move from an expensive prototype below the quality threshold to a system whose quality and operating costs support production margins.
OpenPipe handles managed fine-tuning, while OctoAI supplies the serving infrastructure. Moreau previews 47% better accuracy on the demonstrated task and a 99.5% inference-cost reduction, described as a 200× difference between GPT-4 Turbo and Llama 3. These are the opening claims; the later live evaluation provides the concrete scores and the generated-token prices behind the comparison.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Choose the intervention that matches the failure
Start by improving what you ask the existing model to do. Moreau's crawl–walk–run progression begins with prompt specificity, few-shot examples and chain-of-thought prompting. These techniques leave the model weights unchanged and try to elicit better behavior from the model already available.
Next, determine whether the model lacks information. Its parametric memory—information encoded in its weights—reflects its training data and training cutoff. Retrieval-augmented generation supplies additional context through vector similarity search, API calls, search engines or database queries. That external information lets the model answer questions about material absent from its training.
Fine-tuning becomes attractive when the remaining problem is a focused behavior that high-quality examples can teach. Moreau presents the progression as a simplification, not a requirement to build retrieval for every task. Classification, extraction, formatting and function calling are good candidates when prompting and relevant context improvements have plateaued. The prerequisites are a sufficiently narrow task and good training data; high inference bills add an economic reason to try a smaller specialized model.
The resulting workflow is a loop:
- Build a representative dataset.
- Fine-tune the model.
- Deploy it at the required volume.
- Evaluate quality and feed new failures back into the dataset.
Production inputs change. Monitoring, dataset updates and retraining keep the model aligned with those changes, so completing one pass through the notebook is only the first iteration.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Build data that resembles the work
Training examples should resemble the inputs and desired outputs the system will encounter in production. Moreau describes three practical sources:
| Prompts | Responses | Practical tradeoff |
|---|---|---|
| Real human interactions | Human responses | Close to the task, but potentially difficult to acquire |
| Real user inputs | Strong-model responses | Realistic inputs with cheaper response generation |
| Synthetic inputs | Synthetic labels or responses | Accessible starting point without existing logs |
Customer-service conversations illustrate the first option. For the middle option, GPT-4 or Claude can generate responses to real prompts, and OpenPipe can log the input-output pairs for training. Fully synthetic datasets from Hugging Face or Kaggle provide the starting point used in this exercise.
Keep the roles of the data partitions separate. Training data updates model parameters. Validation data supports development decisions such as hyperparameter selection. A third test set evaluates the resulting model outside those decisions. Evaluating examples the fine-tune has already seen cannot establish how well it handles new inputs.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Choose the base model and the artifact you will serve
The first model decision concerns control over the trained result. Moreau favors open models because exported weights let the operator choose where and how to serve them. A proprietary fine-tuning service may keep the result inside its own hosting ecosystem. Access to the resulting weights therefore affects both deployment flexibility and dependence on a vendor.
The next decision is size. Llama 3 8B is the small-model example; Mixtral 8x22B is the much larger mixture-of-experts alternative. Moreau associates larger models with stronger reasoning and broader context options, but also higher training and serving costs, potentially requiring hardware such as H100s. His recommendation is to start with Llama 3 8B or Mistral 7B, measure the specialized task, and increase model size only if necessary. This exercise chooses Llama 3 8B.
Training methodology determines the artifact you deploy. LoRA, a parameter-efficient fine-tuning method, adds a compact learned adaptation to a base model. Full-parameter fine-tuning instead updates the model more broadly and produces a full checkpoint.
| Choice | Artifact | Serving implication |
|---|---|---|
| Parameter-efficient tuning | LoRA adapter | Multiple adapters can share a resident base model |
| Full-parameter tuning | Full checkpoint | Each customized model carries its own full weights |
Moreau contrasts an approximately 50 MB adapter with a 15 GB checkpoint. The adapter size is additional model state, not total inference memory: the base model must also be available. In the demonstrated serving design, several users' adapters share a GPU and endpoint, reducing the resources allocated to each customization. Moreau contrasts that with separately provisioned checkpoints. Full-parameter tuning provides more adjustable parameters for demanding tasks such as reasoning; he considers LoRA sufficient for the specialized classification, labeling and function-calling task here.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Decide which infrastructure to operate
You can operate the training stack yourself or use a managed service. Axolotl is the named DIY option: it gives you training tooling, while GPU acquisition, library setup and hyperparameter choices remain your responsibility. OpenPipe packages data collection and fine-tuning into a managed workflow, making the transition from a general model to a specialized one more approachable. Starting with managed training does not prevent a later move to operating the stack directly.
Serving presents a similar choice. vLLM, MLC LLM, TensorRT-LLM and Hugging Face TGI can run models on infrastructure you control. They do not remove the need to provision GPUs, scale with demand and maintain reliability under load. OctoAI is the workshop's managed alternative, exposing familiar hosted-model endpoints while handling those operational concerns.
Moreau reports approximately 150 tokens per second for Llama 3 8B on OctoAI, attributing the speed to in-house serving optimizations. He quotes $0.15 per million tokens for that model versus $30 for GPT-4, with no additional serving charge for a customized adapter. He also reports platform volume above 20 billion tokens per day. These are historical vendor figures; the speed claim does not specify workload or concurrency conditions. For organizations requiring their own environment, he introduces OctoStack for on-premises or VPC deployment, including healthcare, banking and government use cases.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Define what a good answer means
Evaluation depends on the output contract. Some properties can be checked directly: whether generated code or SQL executes, whether extracted labels match expected values, or whether an output satisfies a JSON format. Other properties, such as politeness and professionalism, require a judgment about which answer is better.
- Direct checks: use executable tests, labels and schema validation for properties with a precise expected result.
- Preference judgments: use human comparisons or an LLM judge for qualities that lack a simple pass/fail rule.
An LLM judge introduces its own behavior into the measurement. Moreau warns that GPT-4 judging a comparison involving GPT-4 may prefer its own answer. The evaluation method therefore needs scrutiny alongside the model being evaluated.
The evaluation also determines when to repeat the deployment cycle. There is no universal retraining interval: Moreau suggests that the right cadence could be weekly or yearly, depending on the application and its constraints.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Extract sensitive values before deciding how to redact them
The practical exercise moves into a Google Colab notebook, shared with attendees through a QR code, with questions directed to the event's bootcamp Slack channel. Its source is AI4Privacy's PII Masking 200K, a synthetic dataset introduced as covering 54 PII classes across 229 discussion subjects. Examples span customer support, banking and conversations between individuals.
The slide's email-like example contains a credit-card number, an IP address and a job title. The desired redacted version preserves the surrounding text while replacing the sensitive values with category labels such as CREDITCARDNUMBER, IPV4 and JOBTITLE. This retains information about what was removed without retaining the original value.
The LLM does not directly write that redacted text. It receives a system prompt describing the redaction task, the supported PII categories, a tool specification and the user's message. Its output is a call to redact containing the sensitive strings and their categories. A large tool specification makes this a useful test of function-calling behavior: the model must select both the right spans and the right labels.
Separate detection from replacement policy. Once the model has produced structured arguments, application code can replace each value with its category, obscure it completely, or consult a database for a stable fictional substitute. The last option preserves consistency across records: repeated references to one person can receive the same fake name or address, keeping the text useful for downstream processing. A returned tool call supplies proposed arguments; the application still performs the replacement.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn labeled examples into tool-call conversations
The recorded setup requires OpenPipe and OctoAI accounts, plus an OpenAI API key for the GPT-4 comparison. The Python dependencies are openai and Hugging Face datasets; the notebook accepts the OctoAI token and OpenAI key. Moreau proceeds past a displayed Colab dependency error without diagnosing it. Treat the OctoAI setup and deployment steps as historical: its former homepage now redirects to NVIDIA, so the recording is not a current account-creation or endpoint guide.
The dataset provides source text, a redacted target and a privacy mask pairing sensitive values with their classes. The system prompt explains the categories in natural language, and the tool specification constrains each extracted value to a PII type. Moreau contrasts that interface with hand-written regular expressions. Here he describes 56 categories in the prompt and enum, whereas the dataset introduction specified 54; the recording does not explain the difference.
After loading the source dataset, the notebook converts labeled examples into records that look like logged LLM conversations. Each record contains the system prompt, user text, tool specification and expected assistant tool call. The following compact Python example shows that record structure with an email address as the teaching input; a full dataset uses the complete category enum and all labeled values for each message.
python
import json
from pathlib import Path
text = "Contact Alex at alex@example.com about the invoice."
fields = [{"string": "alex@example.com", "pii_type": "EMAIL"}]
tool = {
"type": "function",
"function": {
"name": "redact",
"description": "Identify sensitive values for redaction.",
"parameters": {
"type": "object",
"properties": {
"fields_to_redact": {
"type": "array",
"items": {
"type": "object",
"properties": {
"string": {"type": "string"},
"pii_type": {
"type": "string",
"enum": ["EMAIL"]
}
},
"required": ["string", "pii_type"]
}
}
},
"required": ["fields_to_redact"]
}
}
}
record = {
"messages": [
{
"role": "system",
"content": "Identify email addresses and call redact."
},
{"role": "user", "content": text},
{
"role": "assistant",
"tool_calls": [{
"id": "call_redact_1",
"type": "function",
"function": {
"name": "redact",
"arguments": json.dumps({"fields_to_redact": fields})
}
}]
}
],
"tools": [tool],
"tool_choice": {"type": "function", "function": {"name": "redact"}}
}
Path("openpipe_dataset.jsonl").write_text(
json.dumps(record, ensure_ascii=False) + "\n",
encoding="utf-8"
)
The nested arguments value is a JSON string inside the outer JSON record. The training target is the assistant's structured extraction, not a rewritten email and not the result of executing redact.
The demonstrated export contains 10,000 records in openpipe_dataset.jsonl. The upload procedure is straightforward:
- Create a dataset inside an OpenPipe project.
- Name it under Settings; the demonstration uses the June 26 Lunch and Learn.
- Upload the downloaded JSONL file under General.
- Wait for processing to format inspectable examples and create the training/held-out split.
At this point Moreau calls the held-out partition validation data.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Configure training, then inspect a prepared run
Once processing finishes, the dataset view exposes individual examples and a roughly 90/10 split, now described as training and test. The selected model is a Llama 3 8B variant with a 32K context window, associated with Nous Research's Theta model; the corresponding public model family is OpenPipe Hermes-2-Theta-Llama-3-8B-32k. The displayed training estimate is $40 for the demonstrated dataset. Moreau explains that cost scales with sample count and suggests that additional examples can improve task performance. He also mentions a possible $100 introductory OpenPipe credit, qualifying that offer as his understanding at the time.
The upload is slow, so the demonstration moves to prepared artifacts rather than waiting through a fresh training job. Moreau uses a cooking-show analogy: one model goes into the oven while a previously prepared model comes out. Training completion would trigger an email, and the result of OpenPipe's parameter-efficient training is the LoRA intended for deployment.
With the current upload still unfinished, he opens a dataset uploaded days earlier. The input column contains the long system prompt, user message, tool choice and PII tool specification. The output column contains the assistant's redact call, with a list of sensitive strings and PII types. This inspection checks the actual learning contract: what the model will receive and exactly what response it should learn to produce.
The Fine-tune dialog exposes the model selector and training controls, but Moreau deliberately does not click Start Training. He instead opens an existing trained model showing 9,000 training examples and 1,000 examples labeled test. OpenPipe manages settings such as learning rate and epoch count, reducing the manual tuning needed to get started. The UI's test label should not obscure the earlier distinction between validation used during development and an independent final test.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Export the adapter and register a serving asset
OpenPipe exposes both LoRA exports and merged checkpoints. A merged checkpoint incorporates the adaptation into full model weights; exporting one does not mean the original training updated every parameter. The demonstration chooses an FP16 LoRA, available as a small ZIP or a download link. Copying the link provides the handoff to OctoAI without manually moving the archive between interfaces.
The intended deployment sequence is:
- Install the OctoAI CLI in the notebook.
- Authenticate and confirm the token is valid.
- Replace the LoRA URL placeholder with the OpenPipe export link.
- Choose an asset name.
- Register the LoRA as a
safetensorsasset associated with Llama 3 8B.
OctoAI's asset library stores custom adapters and checkpoints; Moreau notes that it also supports image-generation assets such as those for Stable Diffusion. Training, export preparation and availability of the download link can introduce separate waits before registration begins.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Recover from the upload failure and call the model
The fresh LoRA upload fails, and retrying does not resolve it. Moreau asks Pedro Torruella to check the capability with the product team, saying that it had worked earlier that day. To continue, he retrieves previously uploaded assets, suggesting octoai asset list during the lookup, then selects an existing adapter and assigns its name in the notebook. The successful continuation therefore uses an earlier registered asset; the failed upload is not repaired during the recording.
Inference uses OpenAI-style chat completions against the OctoAI endpoint. The request supplies the system prompt and example email, selects the OpenPipe Llama 3 8B-32K model, and includes the parameter-efficient fine-tune setting identifying the chosen LoRA asset. The returned response contains a tool call to the redaction function. That demonstrates structured inference with the existing adapter, leaving application code to execute the chosen redaction policy.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Measure extraction quality against the privacy labels
The dataset's privacy masks provide ground truth, so this evaluation does not require an LLM judge. The custom score penalizes missed sensitive information—false negatives—and incorrectly added PII—false positives. A similarity-distance comparison matches predictions to labels and allows partial credit.
In the scoring illustration, a perfect match receives 1.0. Labeling Billy as a first name rather than a middle name still receives a high score because the prediction is close to the expected extraction. An illustrated GPT-4 response misses two of three sensitive items and receives roughly one-third. These examples show why the reported score is not exact-match accuracy or a direct measurement of PII recall: detecting the right sensitive value and assigning precisely the right category contribute differently to the result.
For the live comparison, Moreau reduces the evaluation to 100 examples and calls the OctoAI fine-tune and OpenAI's GPT-4 back-to-back. He encourages larger runs, but notes the expense of the baseline: approximately $1 for 100 examples or $10 for 1,000 examples in this exercise.
The displayed results include both individual examples and the aggregate:
| Custom redaction score | GPT-4 | Fine-tuned Llama 3 8B |
|---|---|---|
| Shown example A | 0.49 | 0.85 |
| Shown example B | 0.30 | 1.00 |
| Mean over the 100-example run | 0.68 | 0.97 |
The fine-tune scores substantially higher on this comparison. The displayed means imply about a 43% relative increase in the custom score, so they do not directly reproduce the opening 47% claim. The exact scoring formula and the evaluation sample's independence from training and model selection are not established in the recording. Moreau connects the result to a production quality threshold, but a mean partial-credit score of 0.97 does not establish that 97% of messages have every sensitive value safely removed.
The closing price comparison is $30 per million generated tokens for GPT-4 versus $0.15 for Llama 3 8B on OctoAI, a quoted 200× difference. Those prices imply a 99.5% reduction in generated-token charges. They describe the workshop's historical output-token comparison, not a measured reduction in total application cost including input tokens, training and operations.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keep the quality loop running
The durable workflow begins with prompting and relevant retrieval, then uses high-quality task data to teach the behavior that remains missing. Once deployed, the fine-tune needs continued quality monitoring and new training examples as inputs change. Managed tools reduce the effort of repeating that cycle, but the dataset remains the foundation of the result.
The serving economics also depend on work below the model API. Moreau points attendees toward Pedro's related discussion of compiler, systems and infrastructure optimization, and says OctoAI's inference prices support positive margins rather than subsidized serving. He directs SaaS and enterprise-deployment questions to the team's conference booth.
The event's LLM Quality Optimization Bootcamp Slack channel remains the offered route for questions and follow-up, alongside a historical survey promotion for additional credits. Moreau ends by inviting attendees to try the notebook, explicitly contingent on fixing the LoRA upload issue. The recording closes with that deployment problem unresolved: structured inference and evaluation worked with a prepared adapter, while the fresh export-to-serving handoff still needed repair.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Synthetic multilingual PII examples with source text, masked targets and labeled sensitive spans.
OpenPipe's 32K Theta model repository, including model lineage and inference examples.
Open-source framework for configuring datasets and running full or parameter-efficient model fine-tuning.
Further reading
Original explanation of adapting frozen model weights with trainable low-rank matrices.
Local PII processing with Python and CLI interfaces for tagging, redaction and consistent fake-data replacement.
Full transcript
Loading transcript…