AI Engineer Summit 2023

Best Practices for Evaluating Large Language Model Applications with llmeval: Niklas Nielsen

Niklas Nielsen· CTO and co-founder, Log109:33

Loading slides…

Read the talk

Evaluating LLM Applications with llmeval

A correct answer can still break an application. Niklas Nielsen demonstrates how to test output contracts, diagnose prompt regressions, and extend evaluation from Python checks to human-informed model judges.

From a talk by Niklas Nielsen

Before you start: Basic familiarity with Python, command-line tools, and LLM prompts will help you follow the examples.

What counts as a good response?

How do you safely change an LLM application when you cannot yet say what a good response looks like? After shipping features built on GPT, teams face decisions about prompts, configurations, model providers, self-hosting, and fine-tuning. Each change can alter the generated output, and without an evaluation criterion, it is difficult to tell whether the application improved or regressed.

Slide with three numbered points about evaluating GPT applications, risky changes, and introducing llmeval, beside a photograph of a van.Open full source frame
We are stuck without evaluation.

Niklas Nielsen, CTO and co-founder of Log10, introduces llmeval as a local command-line tool for making those decisions more concrete. Its starting point is to give prompts, test cases, and acceptance criteria a repeatable structure.

0:170:37
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:17 · section reference included

Prompts, Python metrics, and repeated trials

Nielsen presents the initial setup as four lines of code. Initialization creates a folder structure for prompts and tests, while Meta’s Hydra supplies the configuration system. The evaluation logic lives in Python, so a metric can perform a simple comparison, run more involved validation, or call another LLM. Reports then summarize the application and its individual tests.

A test needs both a response criterion and a rule for aggregating attempts. Because model responses vary, Nielsen proposes flexible acceptance criteria: for example, require three of five attempts to pass. That is an example policy for accepting a test, rather than a measured reliability rate. Repeated trials let the report describe behavior that a single response cannot capture.

1:131:33
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:13 · section reference included

Define an exact-output contract

The terminal demonstration begins with a small setup sequence:

  1. Create a working directory and enter it.
  2. Create a Python virtual environment.
  3. Install llmeval and initialize its prompt and test folders.

These are the historical CLI workflow and examples from the recording; the package is now archived.

The first prompt asks what A plus B is, then adds an output constraint: “Only return the answer without any explanation.” That second sentence matters because the test compares the actual response with the expected response as a string. A mathematically correct explanation can therefore fail the application’s contract.

The comparison allows one narrow normalization: it strips leading whitespace. Nielsen explains that Claude sometimes prepends spaces in this example. The test then exposes a metric for the report and a separate pass/fail result. Its concrete inputs are 4 and 5, with the expected output "9". The comparison can be expressed directly in Python:

python

def exact_match(actual: str, expected: str) -> dict[str, bool]:
    matches = actual.lstrip() == expected
    return {"metric": matches, "result": matches}

expected = "9"
assert exact_match(" 9", expected)["result"]
assert not exact_match("4 + 5 = 9", expected)["result"]

This preserves the distinction between tolerating leading whitespace and tolerating extra explanation. The latter still changes the required output.

VS Code displays math.yaml with an exact_match metric, a whitespace-stripping comparison, inputs a: 4 and b: 5, and expected output "9".Open full source frame
An exact-match test checks inputs 4 and 5 against the expected answer 9.
2:282:35
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:28 · section reference included

Remove one instruction and inspect the failure

To expose what the output constraint does, Nielsen removes the answer-only instruction and runs the arithmetic test. The CLI can execute the entire suite or select the math prompt through a configuration override:

bash

# Run the complete suite.
llmeval run

# Run only the math example.
llmeval run prompts=math

Nielsen says the default is five samples per test; for this demonstration, he overrides it to one sample. The defaults live in the llmeval YAML configuration. This run samples Claude, GPT-4, and GPT-3.5 once each.

Generating a report makes the failure visible. GPT-3.5 supplies an explanatory sentence, while Claude writes out the equation. Both convey the correct sum, but neither meets the exact-output requirement. The displayed report distinguishes the responses as follows:

Model in the reportResponse formatExact match
GPT-4Bare 9Pass
GPT-3.5-turboAnswer with explanatory textFail
Claude-2Written-out equationFail

The useful finding is a formatting dependency: removing a prompt clause changes whether the output can satisfy the consumer’s contract.

llmeval report table shows three model outputs for adding 4 and 5, with a green pass for the bare answer 9 and failures for answers containing extra text.Open full source frame
The arithmetic report shows GPT-4 passing while GPT-3.5-turbo and Claude-2 fail exact matching.

Nielsen restores the instruction and reruns the test. The latest run passes, although generating the report can still announce failures because earlier runs remain in its scope. Per-run results and the overall report answer different questions: the latest result tells you whether the change repaired this attempt; the overall summary tells you whether any included run failed. Reading the aggregate without checking the run can make a successful fix look unsuccessful.

4:014:12
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:01 · section reference included

Check whether generated Python parses

The next example, framed as tool use, asks the model to generate Python. Here too, the prompt contains several clauses suppressing surrounding explanations. The acceptance criterion changes, however: instead of requiring one exact string, the test asks whether the response parses as a Python program. A Python metric can implement that boundary with ast.parse:

python

import ast


def parses_as_python(output: str) -> bool:
    try:
        ast.parse(output)
    except SyntaxError:
        return False
    return True

This accepts different valid programs without requiring identical source text. It checks syntax; it does not execute the program or establish that the program computes the requested result.

The displayed report shows the generated-Python tests passing. Its rows contain examples for adding two numbers and finding an inverse square root across the three models, with True values and green pass indicators. Those results complete the demonstrated parsing check.

VS Code report displays six generated-code rows for adding two numbers and finding an inverse square root across three models, each with True and a green pass indicator.Open full source frame
Generated Python examples receive passing results in the evaluation report.
6:046:19
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:04 · section reference included

Use a model when the criterion needs judgment

Exact matching and parsing give precise answers to narrow questions. For a more interpretive task, Nielsen introduces a model-based evaluation of Mermaid diagrams: define a set of criteria, ask for a score from one to five, and request a reason. Writing tests and collecting cases still take work, but a judge model provides a way to evaluate qualities that are harder to capture with a direct comparison.

In this arrangement, typically a larger model evaluates another LLM’s output. The judgment can take several forms:

  • Pass/fail: Decide whether the output meets the criteria.
  • A grade: Assign a score on a small scale, such as one to five.
  • A preference: Choose between alternatives and explain the choice.

The reasoning accompanying a judgment can make the result more useful for diagnosis than a bare score.

The judge also introduces its own failure modes. Nielsen cautions that, when comparing models, judges can favor output from their own model. He also warns about the quality of point scores on zero-to-one or zero-to-100 scales. These cautions affect how much confidence to place in a ranking or numerical difference: a more nuanced-looking result still depends on the evaluator’s behavior. The next step in the demonstration is to bring human feedback back into that process.

6:487:02
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:48 · section reference included

Generate suggestions from a reviewer’s history

Nielsen’s final demonstration bridges model evaluation and human review. Suppose John has already reviewed a collection of completions. The proposal is to model that accumulated feedback as an “Auto John,” which generates suggested reviews for incoming completions. Human judgments become an input to the evaluator instead of being removed from the feedback process.

The interface shows two existing human feedback entries: one with a score of five and another with a more nuanced review. A further item is pending feedback. Clicking it reveals an AI-suggested answer. The visible transition is from a pending review to an available suggestion; it does not establish that a human accepted the suggestion. This leaves a practical role for automation: extend a reviewer’s prior judgments to new work while keeping the distinction between human feedback and generated advice visible.

Nielsen closes by directing viewers to the llmeval documentation and inviting viewers to contact him by email. He also gives Niklas Crafoord as the name to look for on X.

8:178:38
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:17 · section reference included

Resources

From the talk

  • llmeval 0.4.1Documentation1:02

    Historical Log10 evaluation package released in October 2023. The project is now archived and no longer receives updates.

  • Introduction to configuration composition, command-line overrides and parameter sweeps. The page currently displays development-version documentation.

Updates since the talk

Transcript download
Full transcript

Loading transcript…