AI Engineer World's Fair 2024
How to construct domain-specific LLM evaluation systems.
Loading slides…
Read the talk
Building a Domain-Specific LLM Evaluation System
Rechat’s real estate assistant shows how assertions, trace review and curated examples turn an impressive prototype into a system whose improvements can be measured.
From a talk by Hamel Husain and Emil Sedgh
Before you start: Familiarity with unit tests, LLM prompts and tool-calling assistants will help you follow the evaluation workflow.
An impressive demo with unknown reliability
A real estate application already has contacts, marketing tools, listing data and internal APIs. What happens when an AI assistant can operate them on an agent’s behalf? Rechat began with exactly that opportunity: its existing software supplied both the data and the actions an assistant would need.
The first prototype used GPT-3.5. It was slow and frequently wrong, but a successful interaction made the product feel compelling. A real estate agent could ask it to create a contact, send an email, find listings or build a website. Those successes established the product’s potential; they did not establish whether it was ready for production.
The trouble appeared when the team tried to improve it. They would make a change, invoke the assistant a few times and form an impression that it worked better. They still could not tell how often it would succeed. Worse, a prompt change that helped one use case could break another. The missing capability was a way to distinguish improvement from regression.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn iteration into a measurement loop
Prompt engineering, retrieval-augmented generation and agents can get an MVP surprisingly far when paired with informal checks. Hamel Husain does not dismiss that approach: it can work well for discovering a product. But its usefulness runs out when developers can no longer tell whether their changes are making progress. At that point, more experimentation without measurement leads to stagnation.
The next layer is an evaluation workflow that repeatedly connects application behavior to evidence and then to changes. The diagram brings together model invocations, assertions, trace logging, evaluation and curation, prompt engineering, and fine-tuning. No single evaluator supplies the whole system; the value comes from closing the loop between observing behavior and improving it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Start with failures that ordinary tests can catch
The foundation is familiar software engineering: unit tests and assertions. Before adding an LLM judge or a collection of generic scores, inspect application outputs and write down the failures that can be checked directly. Rechat’s examples included email actions that did not happen, invalid placeholders and details repeated when they should not be. These tests come from observed behavior, not from a universal definition of a good answer.
For an email workflow, separate the assistant’s claim from the recorded action. A Python check can encode that distinction in a small evaluation record:
python
import re
def check_email_case(case: dict) -> dict[str, bool]:
successful_send = any(
event["tool"] == "send_email"
and event["status"] == "succeeded"
and event["recipient"] == case["recipient"]
for event in case["events"]
)
return {
"requested_email_sent": successful_send,
"no_template_placeholders": re.search(
r"\{\{[^{}]+\}\}", case["email_body"]
) is None,
}
case = {
"recipient": "recipient_hamel",
"email_body": "Hi {{first_name}}, here are the listings.",
"events": [
{
"tool": "send_email",
"status": "failed",
"recipient": "recipient_hamel",
}
],
}
checks = check_email_case(case)
assert checks == {
"requested_email_sent": False,
"no_template_placeholders": False,
}
Here the fixture represents a request to send an email, not merely draft one, and the placeholder rule checks the {{...}} syntax. The general mechanism is to express a concrete failure as a named check whose result can be inspected independently. Once an output and its action records exist, such checks provide immediate feedback at very little additional cost.
Run the first version using infrastructure you already have. CI is a reasonable starting point, even if the evaluation workload eventually outgrows it. Persist check results in a database so they can be tracked over time. Rechat used Metabase to visualize those results: CI runs the checks, storage retains their outcomes, and dashboards show whether recurring failures are becoming less common. There is no need to purchase a new evaluation stack before this basic process works.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make traces easy to inspect and label
Assertions catch known failures. Trace logging and human review make it possible to discover the next ones. Tracing is an exception to Husain’s caution about adopting tools early: dedicated tooling can be useful immediately, and Rechat chose LangSmith. But recording traces accomplishes little if nobody looks at them.
Rechat found that general-purpose interfaces imposed too much friction on review. Its traces contained domain-specific information, and reviewers needed application-specific filters and metadata close to the conversation. The team built a small interface that let people inspect and label examples without hunting through separate sources. Gradio, Streamlit and Shiny for Python are possible tools for this kind of application; Husain used Shiny. The displayed interface brings tool and scenario filters, conversation and function records, editable output, and accept/reject actions into one place.
Removing review friction is part of evaluation engineering. If understanding a trace requires too much effort, people stop reviewing. That removes the source of new failure categories, better tests and useful labels. The custom interface is valuable because it keeps that activity practical, not because every team needs to build another dashboard.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Bootstrap coverage before users arrive
A new application may have no users and therefore little real traffic to evaluate. That does not prevent it from having test cases. Rechat used an LLM to act as a real estate agent and generate requests for Lucy, its assistant. The generator supplied inputs across the application’s features, scenarios and tools.
The useful distinction is between generating a request and deciding whether the response is correct. Role-play supplies questions and tasks to exercise the application; it does not automatically supply trustworthy answer labels. Organizing generation around features, scenarios and tools makes it a coverage exercise rather than an unrestricted request for random examples. Those inputs can then pass through the same assertions and review process as user traffic.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Test the evaluation workflow itself
With assertions, traces and human review in place, the next job is to exercise the system. Prompt engineering provides an accessible first intervention: change the application’s instructions, then use the evaluation workflow to understand what happened. Repeat this process enough times to discover whether the workflow actually supports improvement.
Each iteration tests two things at once: the assistant and the machinery used to evaluate it.
- Make a prompt change intended to improve a known behavior.
- Run the test cases and inspect whether coverage captures the relevant effects.
- Check that traces were recorded correctly.
- Review the outputs and remove any remaining obstacles to understanding them.
A missing trace, an uncovered use case or a cumbersome review screen is a defect in the improvement process. Fixing it makes the next application change easier to assess.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Use evaluation to curate fine-tuning data
The same workflow creates a path into fine-tuning. Husain identifies data curation as most of that work. Synthetic generation supplies examples, evaluations help filter promising cases, and human review determines which examples should enter the training data. Evaluation is therefore useful both for measuring a model and for preparing the material used to improve it.
Failed cases also have a place in this process. The review workflow gives the team a way to work through them and continuously update the fine-tuning dataset. As automated checks become more comprehensive, Husain reports that human review becomes less costly because more of the checking is automated and confidence in the data grows. The result is a shared workflow for identifying failures, curating examples and checking whether subsequent changes help.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Calibrate model judges against domain experts
Some qualities cannot be expressed adequately as assertions or unit tests. Once the basic workflow exists, an LLM judge can address those gaps. But the judge introduces another reliability question: why should its assessment be trusted? Its output needs to be aligned with human judgment, rather than accepted simply because it looks like an evaluation.
Husain’s starting point is deliberately simple: a spreadsheet, a domain expert who labels and critiques examples, and repeated iterations on the judge using that feedback. The goal is confidence that the judge is applying the intended standard. The talk supplies neither a numerical acceptance threshold nor a complete calibration protocol; the essential requirement is a human reference against which the judge can be examined and improved.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keep the process specific to the application
The most common breakdowns begin before sophisticated evaluation is needed. Teams do not inspect their data, or they select tools before learning what their review process requires. Working manually with existing tools first teaches the team what a new tool must make easier. Without that experience, it is difficult to evaluate the evaluation product itself.
Generic scores create a similar risk. Conciseness and toxicity checks can be useful, but they cannot substitute for checks tailored to the application’s actual failures. A concise response can still fail to perform the requested action. Husain’s objection is to using readily available metrics as a crutch, not to their existence.
Premature reliance on model judges can also hide simpler opportunities. Close inspection often reveals many more failures that ordinary assertions can catch. Use those checks where they fit, reserve judgment-based evaluation for what remains, and retain human calibration when introducing a model judge.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The behaviors prompting alone did not deliver
After implementing the loop, Emil Sedgh reports that Rechat rapidly increased the application’s success rate, without giving a numerical rate. Measurement made the work tractable, but it did not make every desired behavior achievable through prompting. In Rechat’s experience, few-shot prompting remained insufficient even with newer, more capable agents. The team would have preferred a simple wrapper if that had delivered the required experience.
Two requirements explain why Rechat still needed fine-tuning:
- Mixed output formats. Lucy needed to combine natural language with embedded interface elements. Producing the right experience required structured and unstructured output together, which the team could not make reliable without fine-tuning.
- Requests for missing information. Some user commands could not be executed immediately. The assistant had to ask for more input, and that interaction also had to work with the embedded interface elements.
These were Rechat’s product-specific constraints, not a claim that every assistant requires fine-tuning.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From listing search to a coordinated marketing workflow
Complex command execution was the third reason Rechat fine-tuned its assistant. The closing demonstration begins with a request that Sedgh describes as requiring roughly five or six tools. Lucy must decompose the request into function calls while preserving a dependency: search for listings first, then create a website and an Instagram marketing asset only for the most expensive of the three listings. The selection constraint matters as much as the ability to invoke each tool.
The demonstrated sequence proceeds from search results to generated assets and then communication:
- Find three listings matching the requested criteria.
- Create a website for the most expensive listing.
- Create and render an Instagram post video, also described as a story.
- Prepare an email to Hamel containing information about the listings and links to the website and Instagram asset.
- Include a dinner invitation and create a follow-up task.
The displayed result is an email draft, with a pending follow-up task below it. Preparing that communication should not be confused with evidence that the email was sent or that the follow-up was completed.
Sedgh estimates that this workflow could take a less technically experienced real estate agent a couple of hours, versus about a minute with the assistant; this is an estimate, not a controlled benchmark. The substantive result is the coordinated workflow: the assistant carries a selection constraint through search, asset creation, a prepared email and follow-up. Sedgh credits the comprehensive evaluation framework with making that experience feasible.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Current documentation for recording and inspecting application traces, with runs, threads and contextual metadata.
Build interactive Python applications and dashboards with examples that can support custom data-review interfaces.
Create dashboards from queries and charts, then add filters to inspect results across different conditions.
Further reading
- Your AI Product Needs EvalsArticle
Hamel Husain's detailed Rechat case study connects assertions, trace review, human evaluation and fine-tuning data curation.
Rechat's announcement and pitch transcript document Lucy's early real estate assistant concept.
Full transcript
Loading transcript…