AI Engineer World's Fair 2024

How to construct domain-specific LLM evaluation systems.

Loading slides…

Read the talk

Building a Domain-Specific LLM Evaluation System

Rechat’s real estate assistant shows how assertions, trace review and curated examples turn an impressive prototype into a system whose improvements can be measured.

From a talk by Hamel Husain and Emil Sedgh

Before you start: Familiarity with unit tests, LLM prompts and tool-calling assistants will help you follow the evaluation workflow.

An impressive demo with unknown reliability

A real estate application already has contacts, marketing tools, listing data and internal APIs. What happens when an AI assistant can operate them on an agent’s behalf? Rechat began with exactly that opportunity: its existing software supplied both the data and the actions an assistant would need.

The first prototype used GPT-3.5. It was slow and frequently wrong, but a successful interaction made the product feel compelling. A real estate agent could ask it to create a contact, send an email, find listings or build a website. Those successes established the product’s potential; they did not establish whether it was ready for production.

The trouble appeared when the team tried to improve it. They would make a change, invoke the assistant a few times and form an impression that it worked better. They still could not tell how often it would succeed. Worse, a prompt change that helped one use case could break another. The missing capability was a way to distinguish improvement from regression.

Rewind Back a Year slide lists a slow GPT-3.5 prototype with a high chance of failure, followed by an improvement phase unable to establish progress or detect breakage.Open full source frame
The prototype worked, but improvement relied on “vibes” and “feelings.”
0:360:51
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:36 · section reference included

Turn iteration into a measurement loop

Prompt engineering, retrieval-augmented generation and agents can get an MVP surprisingly far when paired with informal checks. Hamel Husain does not dismiss that approach: it can work well for discovering a product. But its usefulness runs out when developers can no longer tell whether their changes are making progress. At that point, more experimentation without measurement leads to stagnation.

The next layer is an evaluation workflow that repeatedly connects application behavior to evidence and then to changes. The diagram brings together model invocations, assertions, trace logging, evaluation and curation, prompt engineering, and fine-tuning. No single evaluator supplies the whole system; the value comes from closing the loop between observing behavior and improving it.

Virtuous Cycle diagram links LLM invocations, unit tests, logging traces, evaluation and curation, fine-tuning, prompt engineering, and model improvement.Open full source frame
The evaluation recipe connects tests and traces to a repeated improvement cycle.
3:013:18
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:01 · section reference included

Start with failures that ordinary tests can catch

The foundation is familiar software engineering: unit tests and assertions. Before adding an LLM judge or a collection of generic scores, inspect application outputs and write down the failures that can be checked directly. Rechat’s examples included email actions that did not happen, invalid placeholders and details repeated when they should not be. These tests come from observed behavior, not from a universal definition of a good answer.

For an email workflow, separate the assistant’s claim from the recorded action. A Python check can encode that distinction in a small evaluation record:

python

import re


def check_email_case(case: dict) -> dict[str, bool]:
    successful_send = any(
        event["tool"] == "send_email"
        and event["status"] == "succeeded"
        and event["recipient"] == case["recipient"]
        for event in case["events"]
    )
    return {
        "requested_email_sent": successful_send,
        "no_template_placeholders": re.search(
            r"\{\{[^{}]+\}\}", case["email_body"]
        ) is None,
    }


case = {
    "recipient": "recipient_hamel",
    "email_body": "Hi {{first_name}}, here are the listings.",
    "events": [
        {
            "tool": "send_email",
            "status": "failed",
            "recipient": "recipient_hamel",
        }
    ],
}

checks = check_email_case(case)
assert checks == {
    "requested_email_sent": False,
    "no_template_placeholders": False,
}

Here the fixture represents a request to send an email, not merely draft one, and the placeholder rule checks the {{...}} syntax. The general mechanism is to express a concrete failure as a named check whose result can be inspected independently. Once an output and its action records exist, such checks provide immediate feedback at very little additional cost.

Run the first version using infrastructure you already have. CI is a reasonable starting point, even if the evaluation workload eventually outgrows it. Persist check results in a database so they can be tracked over time. Rechat used Metabase to visualize those results: CI runs the checks, storage retains their outcomes, and dashboards show whether recurring failures are becoming less common. There is no need to purchase a new evaluation stack before this basic process works.

4:144:18
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:14 · section reference included

Make traces easy to inspect and label

Assertions catch known failures. Trace logging and human review make it possible to discover the next ones. Tracing is an exception to Husain’s caution about adopting tools early: dedicated tooling can be useful immediately, and Rechat chose LangSmith. But recording traces accomplishes little if nobody looks at them.

Rechat found that general-purpose interfaces imposed too much friction on review. Its traces contained domain-specific information, and reviewers needed application-specific filters and metadata close to the conversation. The team built a small interface that let people inspect and label examples without hunting through separate sources. Gradio, Streamlit and Shiny for Python are possible tools for this kind of application; Husain used Shiny. The displayed interface brings tool and scenario filters, conversation and function records, editable output, and accept/reject actions into one place.

Slide beside an LLM Data Review interface showing tool and scenario filters, pending status, conversation and function records, editable output, and Accept and Reject buttons.Open full source frame
A custom review interface puts trace inspection, filtering, and annotation together.

Removing review friction is part of evaluation engineering. If understanding a trace requires too much effort, people stop reviewing. That removes the source of new failure categories, better tests and useful labels. The custom interface is valuable because it keeps that activity practical, not because every team needs to build another dashboard.

6:336:41
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:33 · section reference included

Bootstrap coverage before users arrive

A new application may have no users and therefore little real traffic to evaluate. That does not prevent it from having test cases. Rechat used an LLM to act as a real estate agent and generate requests for Lucy, its assistant. The generator supplied inputs across the application’s features, scenarios and tools.

The useful distinction is between generating a request and deciding whether the response is correct. Role-play supplies questions and tasks to exercise the application; it does not automatically supply trustworthy answer labels. Organizing generation around features, scenarios and tools makes it a coverage exercise rather than an unrestricted request for random examples. Those inputs can then pass through the same assertions and review process as user traffic.

8:478:50
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:47 · section reference included

Test the evaluation workflow itself

With assertions, traces and human review in place, the next job is to exercise the system. Prompt engineering provides an accessible first intervention: change the application’s instructions, then use the evaluation workflow to understand what happened. Repeat this process enough times to discover whether the workflow actually supports improvement.

Each iteration tests two things at once: the assistant and the machinery used to evaluate it.

  1. Make a prompt change intended to improve a known behavior.
  2. Run the test cases and inspect whether coverage captures the relevant effects.
  3. Check that traces were recorded correctly.
  4. Review the outputs and remove any remaining obstacles to understanding them.

A missing trace, an uncovered use case or a cumbersome review screen is a defect in the improvement process. Fixing it makes the next application change easier to assess.

9:389:44
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:38 · section reference included

Use evaluation to curate fine-tuning data

The same workflow creates a path into fine-tuning. Husain identifies data curation as most of that work. Synthetic generation supplies examples, evaluations help filter promising cases, and human review determines which examples should enter the training data. Evaluation is therefore useful both for measuring a model and for preparing the material used to improve it.

Failed cases also have a place in this process. The review workflow gives the team a way to work through them and continuously update the fine-tuning dataset. As automated checks become more comprehensive, Husain reports that human review becomes less costly because more of the checking is automated and confidence in the data grows. The result is a shared workflow for identifying failures, curating examples and checking whether subsequent changes help.

10:3710:50
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:37 · section reference included

Calibrate model judges against domain experts

Some qualities cannot be expressed adequately as assertions or unit tests. Once the basic workflow exists, an LLM judge can address those gaps. But the judge introduces another reliability question: why should its assessment be trusted? Its output needs to be aligned with human judgment, rather than accepted simply because it looks like an evaluation.

Husain’s starting point is deliberately simple: a spreadsheet, a domain expert who labels and critiques examples, and repeated iterations on the judge using that feedback. The goal is confidence that the judge is applying the intended standard. The talk supplies neither a numerical acceptance threshold nor a complete calibration protocol; the essential requirement is a human reference against which the judge can be examined and improved.

11:5512:03
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:55 · section reference included

Keep the process specific to the application

The most common breakdowns begin before sophisticated evaluation is needed. Teams do not inspect their data, or they select tools before learning what their review process requires. Working manually with existing tools first teaches the team what a new tool must make easier. Without that experience, it is difficult to evaluate the evaluation product itself.

Generic scores create a similar risk. Conciseness and toxicity checks can be useful, but they cannot substitute for checks tailored to the application’s actual failures. A concise response can still fail to perform the requested action. Husain’s objection is to using readily available metrics as a crutch, not to their existence.

Premature reliance on model judges can also hide simpler opportunities. Close inspection often reveals many more failures that ordinary assertions can catch. Use those checks where they fit, reserve judgment-based evaluation for what remains, and retain human calibration when introducing a model judge.

13:0013:09
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:00 · section reference included

The behaviors prompting alone did not deliver

After implementing the loop, Emil Sedgh reports that Rechat rapidly increased the application’s success rate, without giving a numerical rate. Measurement made the work tractable, but it did not make every desired behavior achievable through prompting. In Rechat’s experience, few-shot prompting remained insufficient even with newer, more capable agents. The team would have preferred a simple wrapper if that had delivered the required experience.

Two requirements explain why Rechat still needed fine-tuning:

  • Mixed output formats. Lucy needed to combine natural language with embedded interface elements. Producing the right experience required structured and unstructured output together, which the team could not make reliable without fine-tuning.
  • Requests for missing information. Some user commands could not be executed immediately. The assistant had to ask for more input, and that interaction also had to work with the embedded interface elements.

These were Rechat’s product-specific constraints, not a claim that every assistant requires fine-tuning.

15:1515:27
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

15:15 · section reference included

From listing search to a coordinated marketing workflow

Complex command execution was the third reason Rechat fine-tuned its assistant. The closing demonstration begins with a request that Sedgh describes as requiring roughly five or six tools. Lucy must decompose the request into function calls while preserving a dependency: search for listings first, then create a website and an Instagram marketing asset only for the most expensive of the three listings. The selection constraint matters as much as the ability to invoke each tool.

The demonstrated sequence proceeds from search results to generated assets and then communication:

  1. Find three listings matching the requested criteria.
  2. Create a website for the most expensive listing.
  3. Create and render an Instagram post video, also described as a story.
  4. Prepare an email to Hamel containing information about the listings and links to the website and Instagram asset.
  5. Include a dinner invitation and create a follow-up task.

The displayed result is an email draft, with a pending follow-up task below it. Preparing that communication should not be confused with evidence that the email was sent or that the follow-up was completed.

Email draft lists three properties, links to a website and Instagram story, asks about dinner availability, and appears above a pending follow-up task; the highlighted original request remains alongside.Open full source frame
The demo shows a prepared email with listing details, marketing links, and a follow-up task.

Sedgh estimates that this workflow could take a less technically experienced real estate agent a couple of hours, versus about a minute with the assistant; this is an estimate, not a controlled benchmark. The substantive result is the coordinated workflow: the assistant carries a selection constraint through search, asset creation, a prepared email and follow-up. Sedgh credits the comprehensive evaluation framework with making that experience feasible.

16:5217:13
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

16:52 · section reference included

Resources

From the talk

  • Hamel Husain's detailed Rechat case study connects assertions, trace review, human evaluation and fine-tuning data curation.

  • Rechat's announcement and pitch transcript document Lucy's early real estate assistant concept.

Transcript download
Full transcript

Loading transcript…