# Judge the Judge: Building LLM Evaluators That Actually Work with GEPA — Mahmoud Mabrouk, Agenta AI

Mahmoud Mabrouk

AI Engineer Europe 2026

[Talk page](https://ai.engineer/talks/X4dEHRzBLmc-judge-judge-building-llm-evaluators-that-actually) · [Video](https://www.youtube.com/watch?v=X4dEHRzBLmc)

## Judge the Judge: Building LLM Evaluators That Actually Work with GEPA

A customer-support evaluator learns airline policy from annotated failures, revealing how seed prompts, reflection, candidate diversity and data quality shape reliable evaluation.

Familiarity with LLM prompts, agent tool calls, training and validation splits, and basic classification metrics will help with the notebook walkthrough.

### The monitor says everything is fine

[0:00](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=0s)

An agent is in production, and the team adds a hallucination judge to monitor reliability. The dashboard looks healthy. Customers complain anyway, and the traces confirm that the agent is failing. Under the hood, the evaluator’s prompt amounts to: given an LLM output, decide whether it is a hallucination, and make no mistakes. The instruction supplies neither the missing facts nor the criteria needed to recognize a failure.

![Slide titled “Let's make an LLM-as-a-judge” showing a prompt asking whether an LLM output is a hallucination, followed by “MAKE NO MISTAKES!” and an output placeholder.](https://ai.engineer/api/talk-essay-media/d8b07990034501744f815551cc4ba0336b6cbdcc0683300c7599b35921df2336)

A minimal hallucination judge prompt: “MAKE NO MISTAKES!”

**An evaluator needs a standard of correctness that matches the application.** Mahmoud Mabrouk’s workshop builds toward that standard by calibrating an LLM judge against annotations, using [GEPA](https://github.com/gepa-ai/gepa) to optimize its prompt. Here, calibration means making the judge’s decisions agree with the reference judgments; it is not a claim about calibrated probability estimates.

### Evaluation determines how fast the application can improve

[1:18](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=78s)

Offline development follows a familiar loop: change the prompt or harness, run evaluations, retain useful changes, and repeat. Human review can provide a strong signal, but reviewing the entire test set at every iteration is slow. An automated judge speeds up the loop only if its decisions track the judgments that matter. Otherwise, the team moves faster without knowing whether the application improved.

Online evaluation extends the same requirement to production. A judge aligned with business goals can help reveal regressions, improvements and changes in how users interact with an agent. Those observations can feed a **data flywheel**: inspect traces, turn edge cases into new evaluations, optimize the harness, and observe again. If both the harness and its evaluators can improve from new annotations, more of that cycle becomes amenable to automation.

![Slide listing three evaluation uses beside a trophy and circular arrows connecting “Optimize Harness,” “Add New Evals,” and a partially obscured observation box.](https://ai.engineer/api/talk-essay-media/363246a7077b3462665d514b4f540c1c848035ae5d4a6e2baf18bafc8d97e064)

Offline evals, online evals, and the data flywheel.

Mabrouk introduces himself as Agenta’s co-founder and CEO, with a machine-learning background that includes computational biology and protein structure prediction. Agenta brings together observability, prompt management and evaluation; his current work includes sampling and automatic optimization. The practical exercise is an airline customer-support judge, taken through metric design, data curation, annotation, optimization and validation. The [workshop repository](https://github.com/Agenta-AI/judge-the-judge-talk-2026) accompanies the notebook walkthrough.

### A cancellation can succeed and still violate policy

[5:47](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=347s)

The exercise uses airline conversations from the [τ-bench](https://github.com/sierra-research/tau-bench) benchmark family. The agent can manage reservations and retrieve flight and user information, but those tools operate under detailed rules about what it may change and when. A successful tool call is therefore insufficient evidence of a successful interaction: the agent may cancel a reservation without first establishing that it qualifies for cancellation.

Mabrouk describes an initial collection of 599 traces, approximately 62% compliant and 38% non-compliant, generated across multiple models and trials. Assertion-based results are converted into trace-level verdicts with explanations. In the cancellation example, the useful annotation identifies both the outcome—non-compliant—and the missing prerequisite: checking the airline’s cancellation rules before approving the request. The policy is complex, and Mabrouk describes the collection as imperfect demonstration data.

### Derive the metrics from observed errors

[8:10](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=490s)

The workflow has four steps: design the metrics, annotate the data, optimize the judge, and validate the result. The first step requires application knowledge. A generic hallucination score does not capture every way an airline agent can fail; a subject-matter expert needs to examine actual conversations and identify what went wrong. Mabrouk points to Hamel Husain’s [error-analysis workflow](https://hamel.dev/blog/posts/field-guide/): begin with open-ended comments on trajectories, then group recurring failures into categories.

The Agenta demonstration produces four categories:

| Evaluation axis | What the reviewer checks |
|---|---|
| Policy adherence | Whether actions obey the airline’s rules |
| Response style | Whether the response follows the expected style |
| Information delivery | Whether the customer receives necessary information |
| Tool calls | Whether tools are called correctly |

Information delivery is distinct from performing the operation: an agent might make a change and fail to tell the customer. Each category becomes a separate judge, keeping the decision narrower than an all-purpose success score.

![Error Analysis slide with an open annotation dropdown listing Policy Adherence, Response Style, Information Delivery, and Miscalled tools.](https://ai.engineer/api/talk-essay-media/df65bc957caadc3ec3cbb8615a81b95f50a11fcb325401a6bffb2e6226e01721)

Error analysis groups failures into specific categories.

**Use a binary verdict with a reason before attempting a graded score.** Even agreement on compliant versus non-compliant is difficult to learn. A one-to-five scale adds distinctions that human annotators may struggle to apply consistently, making the target harder to define as well as harder to optimize.

### The explanation supplies the missing policy knowledge

[11:46](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=706s)

In an annotation queue, the reviewer receives a named criterion such as policy adherence and supplies a verdict plus supporting reasoning. The reason is essential training information. A non-compliant label alone tells the optimizer that its answer was wrong; explaining that the agent approved cancellation before checking eligibility tells it which rule is missing. Some narrow tool failures may be evident from the trace itself, but reconstructing a complex business policy from unexplained binary labels is much harder.

Metric design and annotation receive less screen time than optimization, but Mabrouk identifies them as the hardest parts. The data must cover the relevant cases, and its annotations must contain enough information to learn a useful decision rule. This exercise has a small, unevenly distributed collection and a complex target policy. Its explanations are AI-generated from assertions and the original data, rather than collected from human reviewers. The companion [extraction code](https://github.com/Agenta-AI/judge-the-judge-talk-2026/blob/main/workshop/core/extract.py) identifies τ²-bench simulation inputs and policy-related assertion tags: the resulting task is derived policy-adherence classification, distinct from the original benchmark’s agent-success score. The recommended human-calibration workflow and the demonstration’s generated reference labels should therefore remain separate.

### Generate candidates, test them, preserve complementary strengths

[14:32](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=872s)

GEPA resembles a genetic search over prompts. Start with a seed, generate revised candidates, evaluate them, filter the candidate pool, and repeat. Understanding these operations matters because practical optimization requires tuning their behavior. The seed in this workshop asks whether a customer-service agent violated policy and begins by assuming compliance.

New candidates come from two operations:

- **Mutation:** Run a candidate judge on a trajectory, examine its input, output and evaluation feedback, and use an LLM’s reflection to propose a better prompt.
- **Merge:** Combine useful instructions from existing candidates—for example, guidelines learned by one prompt with complementary guidelines learned by another.

The first iteration has only the seed to mutate. As the pool grows, merging becomes possible. Proposed candidates are tested on mini-batches rather than the entire evaluation collection, and candidates that improve on their starting point can enter the pool.

Selection then asks more than which prompt has the highest average score. Mabrouk explains **Pareto-aware selection** through per-trajectory performance: preserve candidates with complementary strengths so that a prompt that solves a difficult case is not discarded merely because another prompt has a better overall average. The pool can collectively cover more cases than any single member. Merging attempts to consolidate those strengths into one rubric; it does not guarantee that consolidation will succeed. Mutation, evaluation and merging continue from the filtered pool until the compute budget is exhausted.

### The evaluator is the optimizer’s feedback channel

[19:44](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=1184s)

Mabrouk names DSPy as a prominent implementation of prompt optimization, then uses the GEPA authors’ library through `optimize_anything`. Its candidate is the configuration being optimized: a prompt string, a dictionary containing a prompt and temperature, or potentially a chain of prompts. The interface can therefore search over more than the wording of a single instruction.

The evaluator executes the system configured by that candidate. For this task, it runs the judge, compares its verdict with the reference annotation, and supplies diagnostic information for reflection. A scalar score tells GEPA which candidate did better; the trajectory, verdict, reasoning and annotation help explain how to improve it. Configuration controls the search budget and other behavior, while an objective supplies context for refinement.

The feedback boundary can be expressed in Python without tying it to a particular GEPA release. In this example, `run_judge` receives the candidate rubric and trajectory; the reference label and explanation are added afterward for the optimizer’s reflection. The function returns both an agreement score and the evidence needed to revise the rubric.

```python
from collections.abc import Callable
from typing import Literal, TypedDict

Verdict = Literal["compliant", "non-compliant"]

class Example(TypedDict):
    trajectory: str
    verdict: Verdict
    annotation: str

class Judgment(TypedDict):
    verdict: Verdict
    reasoning: str


def evaluate_candidate(
    candidate: str,
    example: Example,
    run_judge: Callable[[str, str], Judgment],
) -> tuple[float, dict[str, object]]:
    judgment = run_judge(candidate, example["trajectory"])
    score = float(judgment["verdict"] == example["verdict"])
    feedback = {
        "trajectory": example["trajectory"],
        "judge_verdict": judgment["verdict"],
        "judge_reasoning": judgment["reasoning"],
        "reference_verdict": example["verdict"],
        "reference_reasoning": example["annotation"],
    }
    return score, feedback
```

The recording uses its workshop wrapper around `optimize_anything`; current GEPA documentation confirms the same candidate/evaluator/diagnostic-feedback pattern, but is not a pinned version of that historical interface.

### Keep related tasks out of the validation split

[22:26](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=1346s)

The notebook uses dotenv, LiteLLM and GEPA, with some helper functions extracted into separate files. Optimization has already run before the recording, so the walkthrough inspects its outputs rather than waiting for another run. Mabrouk’s runtime estimates range from roughly an hour to a couple of hours across the walkthrough.

After loading the processed annotations, the notebook separates optimization data from final validation by benchmark task. The notebook shows 480 training traces and 112 validation traces. The [saved optimization summary](https://github.com/Agenta-AI/judge-the-judge-talk-2026/blob/main/workshop/results/best/optimization_summary.json) confirms those split sizes; together they total 592, leaving the earlier stated collection size of 599 unexplained. Repeated runs of the same task and model create redundancy within the data, but Mabrouk says that redundancy does not cross the training/validation boundary. This separation matters because another run of a familiar task would be a weaker test of generalization than an unseen task.

![Notebook section “Load and inspect the data” showing dataset-loading code and output reporting 480 training traces and 112 validation traces, with compliant and non-compliant counts.](https://ai.engineer/api/talk-essay-media/0b939c901d01418b23462bf60f552228362a508bb8416f573b06db53cf5e2200)

Loaded training and validation datasets with trace counts.

The annotation examples make the learning target concrete. A compliant trajectory correctly identifies a basic-economy reservation; a non-compliant one fails to recognize that a customer has regular membership. Those details carry the policy knowledge. Without an explanation of why each case is correct or incorrect, the optimizer would have to infer both the hidden business rules and how to apply them from verdicts alone.

### Start with a judge that does not invent violations

[26:06](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=1566s)

The initial rubric is deliberately simple: evaluate whether the agent violated policy, assume compliance, and change the verdict only when a specific reason supports non-compliance. This is an engineered starting point. Earlier prompts that simply asked the model to judge policy violations encouraged it to substitute its own assumptions for rules it had never been given. Mabrouk found those nearly random starting judgments difficult to improve.

Mabrouk reports 61% validation accuracy for the seed judge, with very low recall for non-compliant cases. Its strong tendency toward compliance is intentional at this stage: the seed has little policy knowledge, so it should not manufacture reasons to fail a trace. Inspecting false-compliant cases then reveals the rules that the rubric needs to acquire. Seed sensitivity was consequential for this complex policy task, rather than proof that every judge should use the same default.

### Teach reflection to extract rules from failures

[29:09](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=1749s)

The next intervention changes the prompt that proposes better judge prompts. GEPA’s default reflection template did not produce the improvements Mabrouk expected, so he supplied a custom template with airline-domain context and an explicit instruction to learn policy criteria. Reflection sees the judge’s verdict alongside the reference annotation, then uses that discrepancy to decide what the rubric lacks.

The allowed edits are concrete: add rules, restructure existing rules, and reword instructions for clarity. For the cancellation failure, the useful change is a criterion about verifying eligibility before approval, rather than a general instruction to be more careful. The template directs the refiner toward reusable policy rules. Mabrouk reports that this domain prior helped, despite only limited iteration on the reflection template itself.

![Notebook section “Optimize the judge with GEPA” listing six optimization steps and explaining that annotations identify missing policy rules, above the beginning of a reflection template.](https://ai.engineer/api/talk-essay-media/4e26c463da3955d7b41b7bf03ba4b162857be9e0727420c565a8c623f449e4ff)

GEPA's optimization loop uses annotated failures to improve the rubric.

The experiment’s optimization wrapper constructs configuration from keyword arguments and calls `optimize_anything`. Its evaluator runs the candidate judge and supplies the trajectory and annotation as feedback. The resulting rubric acquires criteria for flight cancellations and refunds, flight modifications, and communication. The prompt has moved from a default assumption about compliance toward explicit instructions grounded in the annotated cases.

### Better accuracy does not mean the merge problem is solved

[32:46](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=1966s)

For the recorded experiment, Mabrouk reports validation accuracy rising from 61% to 74%. The rounded endpoints imply a gain of 13 percentage points. Non-compliant precision and recall also improve from near zero, indicating that the optimized rubric has learned to flag violations rather than simply passing almost everything.

Mabrouk also reports a compliance-related figure falling from 98% to 64%, but describes it inconsistently as bias, prediction frequency and recall. It should not be treated as a precisely defined metric. The companion repository currently reports different accuracy results, 65.2% to 69.6%; those checked-in artifacts have not been tied to the exact recorded run, so they do not replace the recording’s figures.

Training accuracy reportedly improves by nine percentage points. More strikingly, Mabrouk reports 100% training-task coverage across the Pareto candidate set: for every task, at least one generated candidate gets it right. That is a property of the collection, not the accuracy of the selected judge.

| Result | What it establishes |
|---|---|
| Better single-rubric accuracy | One judge makes more correct decisions |
| Complete candidate-set coverage | Some candidate succeeds on each training task |
| Successful consolidation | One rubric preserves those complementary strengths |

The experiment achieves the second property without fully achieving the third. GEPA struggles to merge all the useful information into one prompt, leaving the final judge substantially short of the strong agreement Mabrouk wanted. Imperfect data and limited tuning constrain the result, and reaching it required several unsuccessful experiments.

### Inspect one iteration before paying for a large search

[34:51](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=2091s)

The retrospective begins with model capacity. Using **GPT-4o** for both judging and refinement failed in Mabrouk’s trials on this complex airline-policy logic. He also tried models described as Mini and Nano, along with **Gemini** and **DeepSeek**. His best reported combination used Gemini for reflection and **Grok** for judging; he also says a model he calls **GPT-4 Mini** worked fairly well in both roles. The companion [Grok/Gemini experiment](https://github.com/Agenta-AI/judge-the-judge-talk-2026/blob/main/experiments/exp_007_grok_gemini.py) specifies **Grok 4.1 Fast** and **Gemini 3 Flash Preview**, clarifying that saved configuration without resolving all the model names in the spoken comparison.

The useful debugging unit is a small iteration, not a full search budget:

1. Run a short optimization and inspect the reflection reasoning.
2. Read the generated candidates and check which ones actually improve.
3. Stop at a failed example and identify what the refinement prompt misunderstood.
4. Add the missing context or prior, then test whether the next candidate fixes the training case.

Mabrouk stopped after the first iteration and used Claude Code to edit the refinement prompt. His immediate goal was to make the process fit the training examples before scaling it. That separates an optimizer that cannot learn the available signal from one that can learn but may not generalize. The strong collective frontier coverage then directs attention toward candidate merging as an additional unresolved problem.

![Slide titled “How the sausage is made” recommending smart models, inspecting reasoning traces and new candidates, understanding failure modes, and first overfitting to training data.](https://ai.engineer/api/talk-essay-media/0113dfbee8f30554dffd1ac19924a2a9bfa6f3d6a65ce3bc73808caf253116a0)

Inspect failures and candidate improvements before scaling optimization.

### Supplying the policy and learning the policy are different starting points

[37:04](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=2224s)

Mabrouk tested two seed families. One omitted the agent’s original policy; the other copied that policy directly into the judge prompt. The policy-free seed performed better in his experiments. His hypothesis is that the copied policy constrained the search around a difficult local optimum, while a simpler seed left more room to explore alternative rubric structures.

The better-performing seed was not learning without policy information. Its annotations still explained why trajectories complied or failed, carrying much of the policy into the reflection loop. The distinction is where the knowledge enters: as a large starting instruction, or incrementally through evidence about specific decisions. That makes this a finding about initialization and feedback in these experiments, rather than a reason to withhold necessary business rules from evaluators generally.

### Spend the optimization budget where it can reduce recurring cost

[38:16](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=2296s)

Mabrouk estimates that his small experiments cost about $200–$300 in model tokens. Long trajectories make every evaluation input expensive, and capable judge and reflection models add to that cost. He stopped an experiment with **GPT-4** because of expense; even the model he calls GPT-4 mini incurred meaningful spending. Moving to cheaper Nano or other smaller models did not produce adequate results in those trials.

The economic motivation for a larger refiner and a smaller judge remains compelling when the judge will evaluate many production traces. Refinement is an upfront search expense; judging becomes a recurring inference expense. Spending more to discover a rubric that a less expensive model can apply reliably may pay off over a large volume of online evaluations. The capability requirement still has to be demonstrated on the application’s actual cases.

Before increasing sampling, instrument the optimization and inspect its traces, generated prompts and training behavior. Mabrouk used Agenta for that inspection and estimates roughly 200–300 iterations per experiment. Batch size and other search parameters also needed tuning. The practical stopping point for the first small run is evidence that reflection can turn a diagnosed failure into a useful candidate; only then does a larger budget have a clear job to do.

## Resources

- [Judge the Judge workshop notebook, data and experiments](https://github.com/Agenta-AI/judge-the-judge-talk-2026) — Companion notebook, airline-policy dataset, judge rubrics and experiment scripts. Saved results allow exploration without rerunning paid model calls.
- [GEPA](https://github.com/gepa-ai/gepa) — Reflective optimization library with prompt evolution, candidate merging, DSPy integration and optimize_anything examples.
- [GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning](https://arxiv.org/abs/2507.19457) — Original research on learning prompt improvements from execution traces and combining complementary candidates through Pareto-based search.
- [A Field Guide to Rapidly Improving AI Products](https://hamel.dev/blog/posts/field-guide/) — Hamel Husain's guide to deriving evaluation criteria from observed failures, involving domain experts and maintaining trust in evaluations.
- [Original τ-bench repository](https://github.com/sierra-research/tau-bench) — Sierra's original tool-agent-user benchmark with airline and retail policy tasks. Its README now marks those task versions as outdated.
- [Grok judge and Gemini reflection experiment](https://github.com/Agenta-AI/judge-the-judge-talk-2026/blob/main/experiments/exp_007_grok_gemini.py) — Concrete experiment configuration showing the compliance-first seed, custom reflection template, annotation feedback and merge settings.

## Slides

### Judge the Judge

Mahmoud Mabrouk introduces LLM-as-a-judge calibration with GEPA, starting with a production monitor that looks fine while customers report failures.

[0:07](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=7s) [Title slide introducing calibration of LLM-as-a-judge with GEPA optimization.](https://assets.ai.engineer/sites/aiecode2025/sha256/6d5b69b03acc297ccc7d35a2b1fbfaee0af96eba466a7be6efd7b9f2dd4729f7.webp)

Agenta

Judge the Judge
Calibrating LLM-as-a-judge with GEPA Optimization

Mahmoud Mabrouk

### Then set up an online eval

An online hallucination judge shows a nearly flat rate across October, while customers report that the agent is not working.

[0:24](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=24s) [A chart shows a nearly flat hallucination rate over time.](https://assets.ai.engineer/sites/aiecode2025/sha256/95b98084d0260775d0d32b618876d7188bbb69fd14a51cfdcfb148bc319384ae.webp)

Then set up an online eval

Prompt Hallucination Rate Over Time
Legend: Hallucination Rate
Y-axis: Hallucination Rate (%)
Y-axis ticks: 0.0, 0.1, 0.2, 0.3, 0.4, 0.5
X-axis: Time (Days)
Timeline: Oct 1, 2023; Oct 3, 2023; Oct 5, 2023; Oct 7, 2023; Oct 10, 2023; Oct 13, 2023; Oct 15, 2023; Oct 17, 2023; Oct 21, 2023; Oct 25, 2023; Oct 27, 2023; Oct 29, 2023; Oct 31, 2023

Page: 2

### Let's now set up an online eval

The plotted hallucination rate stays near 0.05 on a percent-labeled axis, even as customers report that the agent is not working and traces show failures.

[0:31](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=31s) [A hallucination-rate chart appears beside a photo of a frustrated person at a laptop.](https://assets.ai.engineer/sites/aiecode2025/sha256/e66f1e8ac26bba042dafc7c4b6b5af58737721dd2b13e273d70dcebdecd96cad.webp)

Let's now set up an online eval

Prompt Hallucination Rate Over Time
Legend: Hallucination Rate
Y-axis: Hallucination Rate (%)
Y-axis ticks: 0.0, 0.1, 0.2, 0.3, 0.4, 0.5
X-axis: Time (Days)
Timeline: Oct 1, 3, 5, 7, 10, 13, 15, 17, 21, 25, 27, 29, 31, 2023

Laptop screen: [illegible]

3

### Let's make an LLM-as-a-judge

The example judge prompt asks an LLM to flag hallucinations from the output alone. Mabrouk questions how the judge could know whether the output is a hallucination.

[0:36](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=36s) [An example prompt asks a judge to rate whether an LLM output is a hallucination.](https://assets.ai.engineer/sites/aiecode2025/sha256/f1dc55303504b9170f206ac12fc6d9f0973f2dbcbbb8f1dce67166a33c30bf18.webp)

Let's make an LLM-as-a-judge

You will be given an LLM output.
rate whether it is a hallucination.

MAKE NO MISTAEKES!

{output}

4

### This talk

The talk explores calibrating LLM judges against human annotations using prompt optimization with GEPA.

[1:01](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=61s) [The talk focuses on building calibrated LLM-as-a-judges.](https://assets.ai.engineer/sites/aiecode2025/sha256/1ad37c58deeef57a376181ad5485d40aeac676f61be9f540e25e13a3f0e57628.webp)

This talk
How to build calibrated LLM-as-a-judges that works

5

### Why do we want this?

Offline evals guide repeated prompt and harness improvements; online evals track changes in production. The unlabeled chart stays high before dropping sharply.

[1:17](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=77s) [Offline evaluations support a loop between improving prompts or harnesses and measuring results.](https://assets.ai.engineer/sites/aiecode2025/sha256/108389b0d6c9519b3b6b305abda5438a9fe293e52f49171e68d6cd3765fc9d9c.webp)

Why do we want this?

1. Offline evals

Improve
(harness, prompt..)

Measure (evals)

6

[2:43](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=163s) [Offline and online evaluations listed beside a chart that falls sharply; the chart is partly obscured by the presenter.](https://assets.ai.engineer/sites/aiecode2025/sha256/d5c4da6aa1599061a796e9650ec5a5510ba50df3b5b01b0a4e058db24cc55e56.webp)

Why do we want this?

1. Offline evals
2. Online evals

7

### whoami

Mahmoud Mabrouk is Agenta’s co-founder and CEO. His 15+ years in machine learning include work in computational biology; he now works on auto-optimization workflows.

[4:14](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=254s) [Speaker introduction with a portrait and experience in machine learning, Agenta and auto-optimization workflows.](https://assets.ai.engineer/sites/aiecode2025/sha256/284f2eb559185b026c75d41332076ea576ce27c996fc2c6da6c68d909deae052.webp)

whoami

• Co-founder and CEO Agenta (OSS LLMOps platform)
• 15+ years in ML
• PhD in ML (applied to CompBio)
• Working on auto-optimization workflows (autoresearch) etc..

9

### What you will do today

The talk builds an LLM judge for a customer support agent, calibrated with human annotations. It covers metric design and data curation, then focuses on GEPA optimization and validation.

[4:58](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=298s) [Agenda covering a customer support agent, metric design, data curation, GEPA judge optimization and validation.](https://assets.ai.engineer/sites/aiecode2025/sha256/5e43305c2079719fefed11f0f08be0591de0456bb6a3987285e0f8e7551b600c.webp)

What you will do today

Practical Use case:
• Customer support agent (access tools)

Learnings:
• The general process (metric design, annotation, etc..)
• Data curation (labeling, split)
• Optimizing the LLM as a judge using GEPA (optimize_everything)
• Validating the results

10

### Dataset: tau-bench

The example uses tau-bench’s airline customer support agent, which has tools for reservations, flight information and user information, and must follow a complex policy.

[5:49](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=349s) [Dataset slide showing the tau-bench benchmark website and leaderboard.](https://assets.ai.engineer/sites/aiecode2025/sha256/369adb2f66c81fa8f830567373cdfc6c96dcdfbade43a38938930bf2812e6258.webp)

Dataset

τ-bench

Benchmarking AI agents in collaborative real-world scenarios. τ-bench challenges agents to coordinate, guide, and assist users in achieving shared objectives across complex enterprise domains.

View on GitHub
Submit Results
Read Papers
Blog Posts

TEXT Overall
#  MODEL  PASS^1
Claude Opus 4.5 — Anthropic — 70.2%
GPT-5.2 — OpenAI — 69.9%
Qwen3.5-397B-A17B — Alibaba Cloud — 68.4%
...

VOICE Overall
#  MODEL  PASS^1
xai-realtime — xAI — 38.3%
gpt-realtime-1.5 — OpenAI — 35.3%
gemini-live-2.5-flash-na... — Google — 25.8%
...

12

### Metric Design

Mabrouk recommends deriving metrics from the business use case and asking subject matter experts to define them, rather than relying on general measures such as hallucination.

[8:42](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=522s) [Step 1: Metric Design.](https://assets.ai.engineer/sites/aiecode2025/sha256/f979d0fefc30604579c183dc7115c1a08071fb7cbea7ee57ded34ff864bb2ccd.webp)

Step 1
Metric Design

16

### Error Analysis

A subject matter expert reviews conversation traces and clusters failures into four error types. Mabrouk plans a separate LLM judge for each type to keep evaluation focused.

[9:29](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=569s) [An annotation menu lists policy adherence, response style, information delivery, and miscalled tools.](https://assets.ai.engineer/sites/aiecode2025/sha256/93842ffaa10386cf73e071ea590ca0de9f3f2a6b35db7b161328c21e6549c7bb.webp)

Error Analysis

Annotations
Incomplete
Error Type
[illegible]
type
Response Style
Policy Adherence
Response Style
Information Delivery
Miscalled tools

17

### Annotating the Results

Annotators record a policy-adherence verdict and reasoning. The example explains why an unverified cancellation was non-compliant; that reasoning helps the judge learn during optimization.

[11:45](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=705s) [A policy-adherence evaluator uses a boolean assessment and reasoning, alongside an example cancellation-policy explanation.](https://assets.ai.engineer/sites/aiecode2025/sha256/f7156fcdb046d70c5964c96e053776b5f711db3f6a329b473fc9b067e5dd6747.webp)

Annotating the results

Create new evaluator
Evaluator name: Policy Adherence
Evaluator slug: policy-adherence
Evaluator description (optional): Determine whether the Agent adheres to the policy and if not explain which policies it does not adhere to

Feedback name: adherence
Feedback type: Boolean (True/False)
Optional

Feedback name: reasoning
Feedback type: String (Free note)
Optional

The agent is non-compliant because it approved a cancellation without verifying that the reservation met the airline’s cancellation rules. The customer did not have a valid policy exception, so the agent should have refused or escalated instead of canceling.

19

### Optimization

With the data annotated, the next step is to use GEPA for optimization. Mahmoud will explain the algorithm before applying it in a Jupyter notebook.

[14:39](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=879s) [Step 3: Optimization.](https://assets.ai.engineer/sites/aiecode2025/sha256/d1de1a23a64d78ce53ba6c6e0a6ddc2e95ecae3593b099f0890f313873be9f89.webp)

Step 3
Optimization

20

### How does GEPA work?

GEPA starts with a seed prompt, then repeatedly samples candidates, evaluates them and filters the results for the next iteration. The example seed judges customer service policy compliance.

[15:05](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=905s) [A cycle evaluates samples, filters candidates, and samples new candidates from the filtered ones.](https://assets.ai.engineer/sites/aiecode2025/sha256/16cb1eac9ee0583fbb9ba496650bcc44e97155656cc7de7d1cd2b20c6c0f4eca.webp)

How does GEPA work?

Evaluate and keep good samples
Filter candidates
Samples new candidates (from filtered ones)

21

[15:48](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=948s) [A seed candidate leads to candidate sampling, alongside an example policy-evaluation prompt.](https://assets.ai.engineer/sites/aiecode2025/sha256/a65d64d3adc8ea7d63b3b56530cf05f628d1147b7b9c0585bf64d22cdd80c357.webp)

How does GEPA work?

Evaluate whether this customer service agent violated policy.

Rules:
- Start by assuming the agent is COMPLIANT

Samples new candidates (from filtered ones)
Seed Candidate

22

### Sampling GEPA candidates

GEPA starts with a seed prompt and samples new candidates through reflective prompt mutation or by merging candidates. With only one seed, the first iteration uses mutation.

[16:07](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=967s) [Candidate sampling is highlighted beside a diagram of prompt mutation and system-aware merging.](https://assets.ai.engineer/sites/aiecode2025/sha256/3652cd4ee393e7b18071b24110137ba0177ee5ee88415a954876c21a326f0eb3.webp)

How does GEPA work?

Samples new candidates
(from filtered ones)

Seed Candidate

Propose New Candidate

Reflective Prompt Mutation
Reflect and Propose New Prompt
Obtain text feedbacks using μf
Rollouts: Execute P₂ on M
Minibatch M
P₂
Sample 1 candidate for mutation

System Aware Merge
Strategically select prompt for each module, either from P₂ or P₃.
If a module has evolved in P₂ but not in P₃, select from P₂ and vice-versa.
P₃  P₂
Sam[illegible]

Choose Strategy to create [illegible]

23

### Evaluating GEPA candidates

GEPA tests new prompts on evaluation minibatches. If performance improves over the starting point, it evaluates them on all tasks and adds them to the candidate pool; otherwise it discards them.

[17:29](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=1049s) [The workflow highlights evaluation and keeping good samples, beside a performance decision diagram.](https://assets.ai.engineer/sites/aiecode2025/sha256/362abef5dcd0546ea901730ac4d6cac601466600cec93ae089ab040c32077f37.webp)

How does GEPA work?

Evaluate and keep good samples

Discard P_new
Performance improved?
No
Yes
Perform Minibatch Eval
P_new
Eval on all tasks
+
Add P_new to Pool
Propose New Candidate
Reflective Prompt
System Aware

Samples new candidates
(from filtered ones)

Seed Candidate

24

### GEPA optimization loop

GEPA starts with a seed candidate, evaluates and filters candidates, then samples new ones through mutation and merging. It repeats the loop until the compute budget runs out.

[19:16](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=1156s) [Arrows connect candidate sampling, evaluation, and filtering in a loop, with a seed candidate below.](https://assets.ai.engineer/sites/aiecode2025/sha256/76e9ef479ea68101036fc01c9907c23263b99cb1cb3a88c90f40a30c3ce104f5.webp)

How does GEPA work?

Evaluate and keep good samples
Filter candidates
Samples new candidates
(from filtered ones)
Seed Candidate

26

### GEPA optimize_anything

The example passes a seed candidate and objective to optimize_anything. Its evaluator runs the system, logs output and errors, and returns a score; the configuration allows 100 metric calls.

[19:43](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=1183s) [Python example defines an evaluator and calls optimize_anything with a seed candidate, objective, and configuration.](https://assets.ai.engineer/sites/aiecode2025/sha256/9ded19b368f222810210835679e07210cdb2115c51142f6e15ade214c437afb7.webp)

GEPA optimize_anything

def evaluate(candidate: str) -> float:
    """Score a candidate and log diagnostics as ASI."""
    result = run_my_system(candidate)
    oa.log(f"Output: {result.output}")
    oa.log(f"Error: {result.error}")
    return result.score

result = optimize_anything(
    seed_candidate="<your initial artifact>",
    evaluator=evaluate,
    objective="Describe what you want to optimize for.",
    config=GEPAConfig(engine=EngineConfig(max_metric_calls=100)),
)
print("Best candidate:", result.best_candidate)

27

Interface text:
Chrome  File  Edit  View  History  Bookmarks  Profiles  Tab  Window  Help
Sat 4. Apr 03:50
LLM as a judge – Figma Slide
Presenter · LLM as a judge
Audience · LLM as a judge
figma.com/deck/[illegible]/LLM-as-a-judge?node-id=1-42
Work  Finish update

[20:23](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=1223s) [Code example defines an evaluator and calls optimize_anything with a seed candidate, objective, and configuration.](https://assets.ai.engineer/sites/aiecode2025/sha256/73aea468d30d7166bd29d3095061ac2198ea3a5e16dc9d79a8fd4b704f4efb4c.webp)

GEPA optimize_anything

def evaluate(candidate: str) -> float:
    """Score a candidate and log diagnostics as ASI."""
    result = run_my_system(candidate)
    oa.log(f"Output: {result.output}")
    oa.log(f"Error: {result.error}")
    return result.score

result = optimize_anything(
    seed_candidate="<your initial artifact>",
    evaluator=evaluate,
    objective="Describe what you want to optimize for.",
    config=GEPAConfig(engine=EngineConfig(max_metric_calls=100)),
)
print("Best candidate:", result.best_candidate)

27

### Load and inspect the data

Mabrouk splits the tau-bench data by task: 480 training traces for GEPA and 112 validation traces to check the result. The notebook reports the compliant and non-compliant counts for each set.

[23:10](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=1390s) [Notebook loads training and validation datasets and displays their trace counts and compliance breakdowns.](https://assets.ai.engineer/sites/aiecode2025/sha256/7a5cfbdee50f6df9a0b7aa45fcc96aeaaffba8f6ae32e2918474441b6c754cac.webp)

[illegible]
• Ground truth assertions about whether the agent behaved correctly

Our goal: build an LLM judge that reads a trace and decides "compliant" or "non-compliant." Then use GEPA to optimize the judge's rubric so it agrees with the ground truth.

3. Load and inspect the data

# Load the datasets
train = json.load(open("../datasets/airline_policy_v0/train.json"))
val = json.load(open("../datasets/airline_policy_v0/val.json"))
train_ann = json.load(open("../datasets/airline_policy_v0/train_annotated.json"))
val_ann = json.load(open("../datasets/airline_policy_v0/val_annotated.json"))

# Basic stats
compliant_train = sum(1 for e in train if e["ground_truth"])
compliant_val = sum(1 for e in val if e["ground_truth"])

print(f"Training set: {len(train)} traces ({compliant_train} compliant, {len(train)-compliant_train} non-compliant[illegible]
print(f"Validation set: {len(val)} traces ({compliant_val} compliant, {len(val)-compliant_val} non-compliant)")

Training set: 480 traces (299 compliant, 181 non-compliant)
Validation set: 112 traces (69 compliant, 43 non-compliant)

Interface text:
talk-judge-the-judge-apr2026 [SSH: hetzner-dev]
Upgrade to Pro · Agents Window
01_build_judge_with_gepa.ipynb · optimize.py 9+
Judge the Judge: Building LLM Evaluators That Actually Work > 2. The problem: evaluating customer service agents at scale
Add Code · Add Markdown · Run All · Restart · Clear All Outputs · Jupyter Variables · Outline
.venv (Python 3.12.3)
[4] 0.0s
SSH: hetzner-dev · gepa · main · meta-pytorch/OpenEnv#496 needs reviewers · talk-judge-the-judge-apr2026

[23:39](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=1419s) [Notebook code loads training and validation data; output reports 480 training traces and 112 validation traces.](https://assets.ai.engineer/sites/aiecode2025/sha256/85d4089413c2fcc728fb535d4f8636e842c6dd62c0e05317916e706e32a95eec.webp)

# Load the datasets
train = json.load(open("../datasets/airline_policy_v0/train.json"))
val = json.load(open("../datasets/airline_policy_v0/val.json"))
train_ann = json.load(open("../datasets/airline_policy_v0/train_annotated.json"))
val_ann = json.load(open("../datasets/airline_policy_v0/val_annotated.json"))

# Basic stats
compliant_train = sum(1 for e in train if e["ground_truth"])
compliant_val = sum(1 for e in val if e["ground_truth"])

print(f"Training set: {len(train)} traces ({compliant_train} compliant, {len(train)-compliant_train} non-compliant)")
print(f"Validation set: {len(val)} traces ({compliant_val} compliant, {len(val)-compliant_val} non-compliant)")

Training set: 480 traces (299 compliant, 181 non-compliant)
Validation set: 112 traces (69 compliant, 43 non-compliant)

What does a trace look like?

# Show a compliant trace (abbreviated)
for e in val:
    if e["ground_truth"]:
        print("=== COMPLIANT TRACE ===")
        print(f"Task: {e['task_id']}, Model: {e['model'][:25]}")

Interface text:
talk-judge-the-judge-apr2026 [SSH: hetzner-dev]
Upgrade to Pro · Agents Window
01_build_judge_with_gepa.ipynb · optimize.py 9+
Judge the Judge: Building LLM Evaluators That Actually Work > 2. The problem: evaluating customer service agents at scale
Add Code · Add Markdown · Run All · Restart · Clear All Outputs · Jupyter Variables · Outline
.venv (Python 3.12.3)
[4] 0.0s · Python
[5]
SSH: hetzner-dev · gepa · main · meta-pytorch/OpenEnv#496 needs reviewers · talk-judge-the-judge-apr2026

## Full transcript

[0:00](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=0s) Hello everyone, and welcome to my talk/workshop, Judge the Judge. And today we're gonna talk about LLM-as-a-Judge. Quite sure you know this scenario. You have an agent in production and someone from the team says, "We need to monitor the reliability."

[0:16](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=16s) So you go to one of the libraries and maybe use the hallucination LLM-as-a-Judge. You put it in production within your observability platform and things look fine. But customers are actually saying that the agent is not working, and you look at the traces, it's not working.

[0:35](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=35s) You look now under the hood about this hallucination LLM-as-a-Judge, and you'll find a prompt not very far from this one. You'll be given an LLM output, rate whether it's an hallucination, make no mistakes.

[0:47](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=47s) Now obviously, how the hell would the agent know whether it's a hallucination? If it could, then your app would have worked from the day one.

[0:55](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=55s) So today we're gonna talk about how can we build calibrated LLM-as-a-Judge that works. Calibrated mean calibrated with human annotation, and the way we're gonna calibrate them is by using optimization or prompt optimization.

[1:10](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=70s) Specifically, we're gonna use GEPA, um, quite good algorithm for optimizing prompts.

[1:18](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=78s) Now why do we want this? Why do we want calibrated LLM-as-a-Judges or good LLM-as-a-Judges? First thing is for our fl-- our offline evaluations. As you know, like usually to create, uh, a good agent or a good prompt, the way you do is you try to experiment with a prompt, then run your evals, see if it improves things

[1:39](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=99s) or not. If it does, good. If it does not, you go back and you improve it a little bit, improve the harness, the prompt, and do it again and again.

[1:48](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=108s) And the speed in which you move to production or add features is actually the speed into which you can complete this loop. And the bottleneck in this loop is actually the evaluation.

[1:59](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=119s) How fast can you evaluate? Obviously, the slowest possible evaluation is having human annotator really look at your whole test set and annotate it manually. Um, the quality is quite good, but then each iteration will take a lot.

[2:15](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=135s) Um, you can have faster ones by using an LLM-as-a-Judge, but then if that LLM-as-a-Judge does not, um, uh, correlate with a human annotation, then you'll end up with useless signal.

[2:30](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=150s) And although this loop will move fast, it won't go anywhere. So having calibrated LLM-as-a-Judge with a similar quality, let's say, as human annotator will make a development much faster.

[2:43](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=163s) The thing-- second thing is basically the online eval, like our example from the beginning. If you have an online eval and you want to see basically in production if things are improving or not improving, same thing.

[2:55](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=175s) If you have LLM-as-a-Judges that are calibrated with your business goals, then you can quickly see whether changes that you've made are improving, not improvement, whether, um, there is some change in the distribution of the data, how people are interacting, uh, with, with your agent or model, and basically you react rapidly.

[3:15](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=195s) And finally, and I would call this is the holy grail of AI engineering, is really to build this data fl- uh, flywheel, where you optimize your harness, observe some traces, and then add new evals based on these traces, the edge cases, and do it again and again.

[3:32](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=212s) And again here, if, if you have a way to kind of add new evaluations quickly, obviously automatic evaluations, um, from the traces, from kind of the annotations, the data, you can go through this loop faster and faster to the moment or to the point that you can think of it as an automatic loop, right?

[3:51](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=231s) Because you can optimize the harnesses with optimization techniques like GEPA, which we're gonna use here. But the same thing, you can do it for the evals and basically over time, your application will improve just with, uh, new observations.

[4:05](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=245s) So today we're gonna build and optimize these LLM-as-Judges that are calibrated with human annotations. Uh, but before going there, a small intro about myself. My name is Mahmoud. I'm the co-founder and CEO of Agenta.

[4:21](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=261s) Uh, Agenta is an open source LLMOps platform, uh, basically providing you all the tools from observability, prompt management, evaluation, covering the whole life cycle of building reliable agents. Um, my experience is in machine learning.

[4:36](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=276s) I have more than fifteen years experience in that. Uh, in a previous life, I was in academia, uh, worked on machine learning applied to computational biology, protein structure prediction.

[4:46](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=286s) And right now we're working a lot on these sampling and auto optimization workflows. So, uh, if you're interested in that, please reach out. We'd love to have a conversation and show you also what we're building.

[4:59](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=299s) So what's the plan for this talk? Basically, we're gonna work on a practical use case. It's a customer support agent that we want to evaluate, and we are going to build an LLM-as-a-Judge that is calibrated with the annotations, basically human annotations for that customer support agent.

[5:17](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=317s) The plan would be really to go over the whole process of building this, starting with how we design the metrics, how to think about the data curation, the labeling.

[5:27](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=327s) But the main focus would be really about, um, the part about optimizing the LLM-as-a-Judge using GEPA and obviously then validating the results. All the code and the data used in this, uh, will be found in GitHub, and you can find them in the links, uh, in this video and the last slide.

[5:47](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=347s) So let's start with the dataset. We're gonna use tau-bench. Tau-bench is a benchmark and a large dataset built by Sierra, a customer support, um, scale-up, I think. And they have like multiple benchmark for real-world scenarios for customer support agents.

[6:08](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=368s) Uh, one of them is the airline agent, which we are going to use, um, in this example. And basically what they have is an airline customer support agent that has access to multiple tools to manage reservations, access flight information, access user information, and has a quite complex policy to be held to, like when to change a reservation,

[6:30](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=390s) uh, when to provide information, and so on and so forth. Just like a customer support agent, a human one. And the data that we have is, is the agent itself, but most inter-- uh, most importantly, um, five hundred and ninety-nine conversation traces that are generated with annotations.

[6:50](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=410s) Uh, now the format, the original format of the annotations is like in the format of assertion, but, uh, I pre-processed or I post-processed the data so that we have for each trace an annotation like a human annotation, uh, where it says, for example, like in this case, you have the conversation that you see here, and then you

[7:09](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=429s) have an annotation that the agent is not compliant because it improved-- approved the cancellation without verifying that the reservation met the airline's cancellation rules. So basically the evaluation failed, and the reason is because the agent, uh, um, canceled the reservation without verifying something.

[7:28](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=448s) So basically, uh, it did not hold to the policy. And the data is, is more or less, uh, kind of, um, not very skewed. It's sixty-t-two percent compliant, thirty-eight percent non-compliant, and is generated with multiple models and trials.

[7:46](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=466s) Overall, the data is, is kind of-- the problem is quite complex because the policy is quite complex. The data has caveats. Honestly, it's not very clean, um, due to the reasons how it's also generated.

[7:59](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=479s) But for our use case, I think it's very, um, interesting use case to test, uh, GEPA and to kind of demo how it would work, uh, in a real, uh, test case.

[8:10](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=490s) The workflow that we have is four steps. First thing is designing the metrics. That is deciding what will the LLM-as-a-Judge measure, uh, what are the different axes that it would look to.

[8:25](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=505s) The second thing is annotating the data, uh, and then we will optimize the judge and validate the results. Now, the most important thing to take here is that the metrics need to come from, uh, the use case itself.

[8:39](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=519s) It does not make sense to have general metrics like hallucination, uh, when you're evaluating your AI agent. It really depends on kind of the business use case. And the best person, the best people to, to, um, to determine these metrics are the subject matter experts.

[8:57](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=537s) For example, in the case of a customer support agent, you need, you need to have, um, a subject matter expert look at the conversation, provide the feedback. And I think the, the workflow to do that, the best one that described it is Hamel, uh, from hamel.dev.

[9:13](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=553s) Um, I'm gonna share his, uh, blog and, uh, and the YouTube video. And he really described this idea of error analysis very well, but, but I'm gonna go over it very quickly and also the annotation workflow very quickly.

[9:27](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=567s) So the idea is that you, you provide your subject matter expert with all these traces of, uh, the trajectories of the conversation, and they would annotate them first by commenting what did work or what did not work.

[9:44](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=584s) But then, uh, kind of slowly trying to cluster, create clusters of the error types, like when it's failing, why is it failing? And, uh, here I'm showing in Agenta how it's done.

[9:58](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=598s) And basically here I have like these-- I, I discovered kind of these four error types while going through these, uh, traces. So there is sometimes issues with policy adherence, uh, sometimes issue with the response style, uh, information delivery, basically the agent is not informing the customer that they made a change or something like that.

[10:20](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=620s) And finally, some tools has not been colle-- uh, called correctly. And the idea is that we're gonna take these four error types, and then we're gonna build four LLM-as-Judges for these.

[10:34](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=634s) So it does not make sense to have one LLM-as-a-Judge, which is success, um, uh, basically try to evaluate all of these. It will make it too complex, and it's very hard to learn.

[10:45](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=645s) And you will see a little bit later that even with a simple, like when we're gonna simplify it, it's still hard to learn a calibrated LLM-as-a-Judge or optimize a cali-- a calibrated LLM-as-a-Judge.

[10:57](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=657s) So it makes a lot of sense to make things, uh, very specific in these metrics, uh, that we want to evaluate. And the second thing is to move away from one to five scores or like percentages, and instead have like really binary solution, like whether it adhered to policy or not, um, with obviously some reasoning.

[11:21](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=681s) Um, and the reason it's, again, it's already quite hard to calibrate an LLM-as-a-Judge with a true/false like bini-binary classification. Uh, adding another layer and saying, "Okay, it should be a number between one and five," it's, it's hard.

[11:34](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=694s) It's even hard for human annotator, like to have two human annera-- uh, annotator agree on the same score. So the moment that we have defined these different metrics, uh, start the point of annotation.

[11:46](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=706s) Um, here again, I'm using, uh, Agenta. Basically, um, uh, you would take your traces, create an annotation queue, and, and kind of specify for your annotator, like, uh, the name of the, um, the feedback or the evaluator policy adherence here, and then providing like they should provide each one with whether it's adhered to the policy, whether it

[12:08](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=728s) does not, and, uh, provide the reasoning. And the reasoning here is very important, uh, because without the reasoning, we will see, uh, the optimization algorithm will need to, by discover itself, like why it failed.

[12:24](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=744s) And it's gonna be very hard, like in, u-unless It's a very specific, uh, kind of feedback, for example, tool, um, failure where it can infer things. It's very hard to see, for example, from a conversation why it did not adhere to policy without someone providing information in the beginning about it.

[12:45](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=765s) So having that reasoning is very important later for the LLM-as-a-Judge to learn. And again, this reasoning, uh, as I've shown previously with the annotation, uh, it, it kind of describe why, for example, the agent here is non-compliant.

[13:01](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=781s) Uh, because, uh, it approved the cancellation without-- before verifying, um, that the reservation met the cancellation rules. So now that we have, so now that we have the annotations, we can go to the optimization.

[13:17](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=797s) But before going there, uh, I want to add a small note. Although we went very quickly through the first step and the second step, these are actually the hardest part of the problem.

[13:28](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=808s) Like in reality, uh, as every data scientist know, getting your data, um, is the hardest thing. And you need to make sure to look at the data, look at the annotation, make sure that the distribution is, is good, uh, that, uh, the annotation and the information within the data is enough for the algorithm to

[13:53](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=833s) learn a representation of the LLM-as-a-Judge that is meaningful. Uh, in our case here, the data is not that good. Uh, the number of traces is small. The problem is quite complex.

[14:06](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=846s) Uh, they're not very well distributed because of the reason they have been generated. Uh, also the annotation is actually, uh, kind of AI generated based on the as-assertion and the original data.

[14:18](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=858s) You can see a little bit more how it's done in, um, i-in the repository. Um, so it makes things, uh, quite complex for this problem. Um, but it's still quite a good demonstration of how it would, uh, work.

[14:32](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=872s) So now that we have the annotated data, we can start with the optimization. And for the optimization, we are going to use the GEPA algorithm. So I'm gonna explain in the beginning how the GEPA algorithm works, and then we can jump to the Jupyter Notebook and start optimizing based on the annotated data.

[14:49](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=889s) It's very important to understand how the algorithm works because you will see that practically you need to play around a little bit with the parameters and to get it to work.

[14:59](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=899s) And it's very hard obviously to play around with the parameters if you don't understand what do they do. Uh, the algorithm is, um, similar to genetic, uh, algorithm. So basically, the idea is that you start with the seed prompt, and then at each iteration, you try to sample new prompts, see which one works, and then, uh, basically

[15:20](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=920s) select the new ones and, and kinda improve over time. Uh, that's kind of the general shape of, uh, the algorithm, and we're gonna go, uh, and look at each step.

[15:30](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=930s) So it's three steps. Basically, you sample new candidate each times, evaluate them, see which one go-- work well, and then do some filtering using this kind of Pareto frontier I'm gonna talk about, and then do it again and again.

[15:45](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=945s) So let's see how it works. So the way, uh, it works is first you start with a seed candidate. Here in our case, we're gonna use kind of a, a very simple LLM-as-a-Judge, like ev-evaluate whether this customer service agent violated policy, and start with assuming is the agent is compliant.

[16:04](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=964s) And now in each iteration, we are going to sample new candidates from the filtered candidates from the last iteration. Now here in the first one, we have only one seed candidate, but in, in the next iteration, we will have a larger bag of candidates.

[16:21](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=981s) So GEPA has two strategies to sample new candidates. Uh, one is prompt mutation, and the other one is merging multiple candidates. For prompt mutation, which is what we're gonna u-use in the beginning since we have only one candidate, the idea is that you would run the first LLM-as-a-Judge here, um, with the, the trajectory, and,

[16:46](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=1006s) uh, if it fails, uh, it-- like this LLM-as-a-Judge will reflect and propose a new prompt. Basically, there'll be some kind of reflection, which means basically we're using the intelligence of the LLM to try to improve the prompt because it looks at the input, looks at the outputs, looks at the results, and try to infer how to make

[17:08](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=1028s) it better. The other strategy is the merge strategy, which we're gonna use in the next iteration. And basically here it takes two prompts and then kind of put them together.

[17:18](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=1038s) And if you think about it with an LLM-as-a-Judge, usually you, you have like these guidelines and basically probably what it's gonna try to do is to take guidelines from prompt A, prompt B, and put them together.

[17:30](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=1050s) So now after we generated a lot of samples, uh, we need to select which one are good. And basically, the way it works is that it would evaluate these new prompts against mini batches of the eval, so not everything.

[17:45](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=1065s) Um, and if they improve the performance, uh, compared to the starting point, uh, then we select them.

[17:54](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=1074s) And, uh, they will be added to our bag of prompts, and then starts the next iteration, which is the other, uh, innovation of this algorithm, which is the idea of the Pareto fr-frontier.

[18:11](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=1091s) Basically, the way we select which prompts or which candidates we're gonna use as a seed for the new iteration is not that we select the ones that have the average best score, like that would be the trivial, um,

[18:25](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=1105s) uh, solution, right? Look at my, my prompt, see which one Work most by looking at the average over everything and then select these. Uh, instead, what they do is that they, um, try to add diversity by trying to look at what are the best candidate per task, right?

[18:45](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=1125s) You have like a set of kind of tasks in your evaluation, like in our case, set of trajectories. And you look for each of these trajectory, which is the best candidate, and that is kind of the Pareto frontier.

[18:59](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=1139s) Um, and then you try to select from these. And basically what you do is you try to select a set at the end of the day that covers your whole test case.

[19:10](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=1150s) So basically for, for each test case, there is at least one candidate that solves it. And obviously, you see there that, uh, the idea is that you get like a good Pareto frontier, and then you start merging things, and at the end of the day, you have this prompt that solves everything in your training.

[19:26](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=1166s) Now that you have like this filtered set of candidate, again, we sample new candidates from these, uh, these ones using, uh, kind of the mutation and the merge strategy, and we keep doing this, um, until basically the compute, uh, budget, uh, finishes.

[19:44](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=1184s) Now, there is a lot of libraries that implement this algorithm. I think the most known is DSPy. It popularized the idea of optimizing prompts or harnesses. Uh, but now there's, uh, I think a new library by the authors of GEPA, an open source one called GEPA, and they have implemented, uh, in the last month, a new interface

[20:09](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=1209s) called Optimize Anything, a new, uh, API, which is what we're gonna use, and which can be used not only to optimize prompts, but really to optimize, um, any-- almost any algorithm using the same idea.

[20:23](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=1223s) And it's quite powerful. Uh, let me show you how it works. So basically, uh, the API here is called optimize_anything. You see this function, and what it takes is a seed candidate.

[20:34](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=1234s) The candidate is the configuration that you want to optimize. In our case, that would be the LLM-as-a-Judge prompt, right? Uh, we can make it even a dict if we want.

[20:44](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=1244s) So it's kind of, for example, the LLM-as-a-Judge, uh, prompt plus temperature, let's say so. Or it could be chain of prompt and so on or so forth. So it's not limited.

[20:55](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=1255s) Then we have, uh, the evaluator, which is basically the thing that would be used by GEPA to optimize. And the expectation from the evaluator is that, um, it would obviously run the system.

[21:09](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=1269s) In our case, it would run the LLM-as-a-Judge, uh, parameterized by the candidate. Uh, and then it would, uh, log, uh, kind of a diagnostic, not only the output, uh, but also the error of the reasoning.

[21:27](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=1287s) And, and the idea, you can add as much as you want, and you see-- you'll see this is how we're gonna do it with, with our optimization for the LLM-as-a-Judge.

[21:36](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=1296s) Uh, but the idea is that, uh, if you remember here, we used some kind of, uh, reflection and, and, uh, um, reasoning to improve our prompt, and that reasoning is something that we ourself will build, uh, through this evaluator.

[21:54](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=1314s) And then other than that, there is ways in the configuration, for example, uh, to configure how many calls to do per iteration. Um, the objective, basically providing context for, uh, the refinement prompt on how to, uh, improve, uh, and so on and so forth.

[22:12](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=1332s) But, but the, um, the core, uh, flow is actually quite simple. So now let's jump to the Jupyter Notebook and really look into how to do it, uh, step by step.

[22:26](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=1346s) So you'll find this Jupyter Notebook in the GitHub repository that you'll find in the links and in the last slide. So we start by, uh, installing the library. So it's dotenv, LiteLLM and, uh, uh, GEPA.

[22:39](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=1359s) I'm not installing here GEPA because I did the optimization. It takes, uh, kind of quite a time, I think a couple of hours, uh, before. So I'm gonna jump this step.

[22:50](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=1370s) Um, but obviously, if you want to run it within the reposi-- within the Jupyter Notebook, you should also install it. So we install this, and we kind of do our imports.

[23:00](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=1380s) We have kind of a couple of, um, of functions that, that I extracted outside. We're gonna look at them later.

[23:08](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=1388s) And we start by loading the data. Uh, as I mentioned, so we're using the data from tau-bench. Uh, I just kind of, uh, pre-process- processed it in the beginning to change the type of the assertion so that they look like the annotation I showed in the presentation.

[23:25](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=1405s) So I've already kind of split, uh, the data into training and a validation, um, uh, datasets. Uh, so one which we'll use with the GEPA and the other one, the second one, to validate the result at the end.

[23:40](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=1420s) And the way that I did the split is, uh, based on different tasks that are created in, um, uh, in the tau-bench. And if we look here, basically we have a, a training set with four hundred and eighty traces, uh, with, with around two third, uh, that are compliant, and a validation set with one hundred and twelve

[24:01](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=1441s) traces that are compliant. As I mentioned, the data here is not, not very, very nice because, uh, there are some redundancies, so there are sometimes the same task, uh, that is, um, being run with, with the same model multiple times.

[24:16](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=1456s) So there is a little bit of redundancies, but there are no redundancies between the training and the validation set. So we look here how the annotation, uh, looks like.

[24:25](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=1465s) And again, uh, basically we have an example of a compliant annotation or a non-compliant annotation. Basically a trajectory that, uh, that kind of adheres to the policy. That's the LLM-as-a-Judge that we want to learn, another one that does not.

[24:40](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=1480s) And we can see here, basically it describes like, okay, it's compliant because, uh, it correctly identified, uh, the basic economy reservation, while here it did not identify, uh, the user membershi- membership as regular.

[24:57](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=1497s) And, and this annotation is actually quite important for us, for the LLM-as-a-Judge to learn, uh, the policy. Especially in the case here, um, this policy adherence, it's kind of a very complex system, uh, that the LLM-as-a-Judge need to, uh, to learn.

[25:13](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=1513s) And without kind of information about, uh, what is compliant, like why is something, uh, correct or not correct, it would be, uh, quite impossible for the, uh, GEPA algorithm to, to reach, uh, a good LLM-as-a-Judge, right?

[25:32](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=1532s) I mean, it would be the same for a human, right? If you gave me all these trajectories, told me this is, uh, kind of correct, this is non-conformed to policy, but you did not give me, uh, more information, it would be very hard for me to, um, to basically make a judgment, learn how to assess the policies.

[25:50](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=1550s) So having this information and the quality of the annotation, as I mentioned, the quality of the data is really paramount of being able to learn this. Uh, and again, obviously this is kind of bit of a complex, um, LLM-as-a-Judge to learn.

[26:06](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=1566s) So, uh, first thing we start is, is with a naive, uh, judge and, uh, basically this is the seed judge, uh, we start with. Um, and, and it's, it's something that, that actually I engineer, engineered.

[26:22](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=1582s) I'm gonna, uh, talk a little bit about it later with the learnings on like how exactly did we reach this. And basically you can see here is like it evaluates whether the customer service agent violated policy, and it tells that you should start by assuming that the agent is compliant and, uh, only change to non-compliant if there

[26:45](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=1605s) is a specific reason, right? I mean, here we, we are starting with an LLM-as-a-Judge, which means that, um, like the seed judge, the initial judge should actually, uh, in my opinion, start by, by saying everything's all right, right?

[27:00](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=1620s) I mean, if I don't have any rules, uh, it should say everything is all right. I started in, uh, like in another example, when I started as I created an LLM as a judge that says, "Okay, you should check whether the agent violated policy."

[27:13](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=1633s) And what you end up being is basically the LLM with its own biases trying to make that decision by itself. Says, "Okay, this is violates, this doesn't violate," without having any information.

[27:24](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=1644s) So without telling it in the beginning that you should start assuming that it's compliant. I mean, if you don't have any reason to, to believe it's non-compliant, it should be compliant.

[27:35](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=1655s) Um, then, uh, basically you start with, with some kind of random, uh, LLM-as-a-Judge that would be very hard to fix later, uh, unless you end up with one prompt that discovered this thing, right?

[27:50](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=1670s) That it start with a compliant. So the, uh, I discovered here that the initial seed is actually very important, uh, in this case. There might be simpler scenarios where you don't need this, but in this case it's kind of, uh, quite important.

[28:02](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=1682s) So if we take this and we run this on the validation set, like this, uh, initial, uh, LLM-as-a-Judge, uh, we basically find sixty-one percent accuracy, but, but we look like the bias, it's actually most of the time saying it's compliant, which is actually what we want, right?

[28:18](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=1698s) I mean, if that's, I would say saying it's compliant ninety-eight percent of the time is actually the unbiased thing to do, uh, the logical place where to start. Um,

[28:31](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=1711s) so, uh, we run the metrics in the beginning, right? The accuracy sixty-one percent, ninety-eight percent is saying like the recall of, of compliance with very low to recall for non-compliance.

[28:44](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=1724s) Um, and I think as I mentioned, this is quite all right. It's biased towards compliance, and I've had experiments in the beginning where it was kind of almost random, but then it doesn't learn at all.

[28:56](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=1736s) And we can look into why or where it goes wrong and, uh, basically by looking at the places where it does, does not work and we see like it says compliant, but it's not compliant because it doesn't know the policy, right?

[29:09](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=1749s) So we start here like the main code of kind of optimizing the judge with GEPA. And what you will see is that I have actually, um, wrote, um, a reflection template.

[29:23](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=1763s) So that prompt that GEPA uses to reflect and to improve, uh, or to sample new candidates, I did not use their default, but actually I wrote one. Uh, I tried in the beginning by using the default prompt in, in, um, in, uh, in GEPA, but the results were not, uh, as good as I expected.

[29:44](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=1784s) It was very hard for it to, uh, to learn. And what I tried to do is, is to provide, um, basically a bias and prior within the re-reflection, uh, template.

[29:58](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=1798s) So you see here, for example, I mentioned obviously that, uh, it's basically, uh, the judge is reading an airline customer service, um, and it needs to kind of decide which is the basics.

[30:10](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=1810s) But then you can look at, um, more information, for example, uh, that our annotation that the, this, uh, kind of reflection template sees includes also, uh, the judge verdict, the ground truth, like the annotation.

[30:24](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=1824s) Like this is an important information that the reflection template should look into and improve. Um, and uh, basically I explain to, to, to the LLM how to do this.

[30:37](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=1837s) You can add rules, restructure existing one, uh, reword things for clarity, um, and, and try to think about it that The, um, the reflection template should basically create a real policy rules, right?

[30:52](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=1852s) Uh, it should find the right policies and abilities. And I think adding that, uh, was very, um, important to kind of improve the quality. Uh, otherwise, the default reflection template did not understand, uh, that, that it should kind of try to, uh, learn the, the policy more or less, right?

[31:14](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=1874s) With the LLM-as-a-Judge, uh, to some degree. Um, so that's the second thing, uh, kind of I changed, and I mean, honestly, I iterated only a little bit on it.

[31:24](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=1884s) And then we run the optimization. Run optimization is basically a wrapper around, uh, optimize_anything, and it just, uh, uh, kind of, uh, parameterize it since I ran like multiple experiments, uh, to find the one, and it's actually, uh, it will be part of the GitHub repository, and it's actually something that you can play around with and start

[31:46](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=1906s) with when you're exploring, uh, the space of design, uh, in GEPA. And you can see here, it basically tries, um, to build, um, the configuration based on the parameters like the, the kwargs, and then calls optimize_anything, uh, with these.

[32:01](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=1921s) And for the, uh, kind of, um, uh, the make evaluator, it's basically, uh, it calls the LLM-as-a-Judge and all-- adds all the side information. So it doesn't only provide the trajectory, but also kind of the annotation, which is quite important.

[32:16](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=1936s) Um, and then we run this. Uh, as I mentioned, it takes around an hour to run. Um, and you can see here the, uh, the basically, uh, the results.

[32:27](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=1947s) This is the optimized, uh, rubric, and you can see compared to the default where we started, it's, it learned part of the policy criteria, like the flight cancellation and refunds, uh, flight modification, um, uh, how to communicate, and so on and so forth.

[32:46](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=1966s) Now, if we look at the results, like with the kind of evaluate, uh, rubric to evaluate it, we see that the accuracy increased from sixty-one percent to seventy-four percent.

[32:57](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=1977s) And, uh, we removed the bias, right? So what especially changed is the recall for the non-compliant and the precision for the non-compliant, which was basically zero in the beginning.

[33:08](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=1988s) Now, uh, the LLM-as-a-Judge is, uh, has less bias at sixty-four percent, so previously it was ninety-eight percent, and it's really learned parts of the policy. So looking at the results, as I mentioned, like for the validation set, we had, uh, quite a lot of improvement, like fourteen percent.

[33:27](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=2007s) And, uh, for the training accuracy, it improved by nine points. And we can see that the Pareto frontier, interestingly, accuracy is now a hundred percent, meaning that for each task, there is one candidate that we generated that solved it.

[33:42](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=2022s) Uh, the issue the algorithm faced is how to merge all these candidates and all the information to have one prompt that solves everything. Um, and it struggled to do this.

[33:52](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=2032s) So at the end of the day, we improved the LLM-as-a-Judge, we improved its accuracy, but it's very still quite far from kind of ninety-five percent accuracy or something that is really well aligned with the human judgment.

[34:06](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=2046s) Obviously, here I didn't invest extreme amount of time in it, and as I mentioned at the beginning, the quality of the data is also, um, would be better, I would guess, uh, in other cases.

[34:18](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=2058s) It's really a tricky, uh, example. Um, but, but nevertheless, it took actually quite a number of iteration, I think that's the biggest learning to, uh, to reach this LLM-as-a-Judge.

[34:31](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=2071s) It's not an algorithm that you just take and it works from, from day one, unless for kind of, uh, toy examples. And, uh, I wanted to show a little bit in the end, like what are the experiments I, I tried in the beginning, uh, that failed, and how did I think about fixing them.

[34:51](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=2091s) Um, the first thing was actually, uh, using a smaller or older model, like using, uh, GPT-4o, uh, for both the refiner and the LLM-as-a-Judge. And that was a complete failure.

[35:04](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=2104s) Like, uh, smaller models really are very bad, at least in this example, to be either an LLM-as-a-Judge or a refiner. Um, uh, for the LLM-as-a-Judge, providing all this policy especially, it has a lot of kind of complicated logic.

[35:21](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=2121s) It just failed and could not improve it. Uh, I tried other models. I tried Mini and, uh, Nano and Gemini and DeepSeek and, and you see that the best kind of results with this kind of, uh, using, uh, Gemini for reflection and, uh, Grok for a judge.

[35:37](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=2137s) But I would say also using, uh, GPT-4 Mini for both is actually quite good, and the results quite well. The other thing was actually trying how to try to debug it.

[35:49](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=2149s) And, and what I tried to do from the beginning is really to not start sampling, doing big experiments from the beginning, but trying first, uh, kind of a small iterations, looking at the reasoning LLM, looking at the candidate, how do they improve, how many improved, and understanding what's happening.

[36:10](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=2170s) And that actually what, uh, uh, what allowed me to kind of think about improving the refined prompt, um, and, and basically adding some, uh, prior there to, to kind of help solve it.

[36:24](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=2184s) Basically, what I did is I stopped at the first iteration, found some example, and then looked a little bit, kind of fine-tuned, um, in Claude Code, uh, that refinement prompt to, uh, to basically allow it to improve, um, uh, the candidate.

[36:41](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=2201s) And, and as always in machine learning, like what you always try to do is to overfit for the training data. Not trying to run the whole algorithm, but really trying to find a way so that it works.

[36:51](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=2211s) And I think what we saw with the Pareto frontier reaching one hundred percent is we almost overfit to the training data. But obviously for the merge, there are like things I think we can do to improve.

[37:04](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=2224s) And the final thing was kind of this iteration on the seed prompt. There I actually, um, iterated on multiple seed prompts, and there were two families. One which, as you have seen here is, uh, uh, kind of was not-- did not include any information about the agent prompt because we have access to that, right?

[37:25](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=2245s) And the agent prompt does have access to the policy, and one which had access to the policy. So basically, it was this prompt that I've shown, uh, but then the policy of the agent, like really copy-pasted.

[37:38](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=2258s) And interestingly, uh, the, uh, prompt that did not have access to the policy did, uh, better because, um, my hypothesis is that if you have access to the agent policy from the beginning, then it's very hard to fine-tune it.

[37:54](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=2274s) You're already stuck in the local minima that you can not improve on. But if you don't have access to the policy and, uh, yet obviously you have access to the annotations that describe, in this case, all the policy, uh, or a large part of the policy, then you, you are able a little bit, uh,

[38:16](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=2296s) to, um, to explore the space of the prompts much better. Um, and finally, the last point is beware of the cost. I mean, um, even these small experiments I've done, I think they cost like two, three hundred dollars, uh, in tokens, especially since, uh, the trajectories are long, so there is a lot of input tokens, uh, in

[38:38](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=2318s) this case. Uh, but, um, the models that are used, um, are actually quite expensive, right? Uh, GPT-4, I mean, I tried a little bit to play around with GPT-4, but that ate a lot of money, so I stopped the experiment.

[38:55](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=2335s) But even GPT-4 mini is quite expensive to some degree. And then if you go, uh, nano, at least from what I've seen, it doesn't work right also. You, you go to kind of smaller model, cheaper model, it does not work.

[39:09](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=2349s) Usually, what they say you should use, uh, kind of a bigger model for the refinement prompt and smaller model, uh, for the LLM-as-a-Judge. I think it makes sense, especially if you're running LLM-as-a-Judge, uh, against a lot of traces, like in the case of online evaluation.

[39:23](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=2363s) It's obvious that it's, it's worthwhile the investment of spending money on the optimization to lower the cost on the long term. And I think there is a lot of use cases where it worked.

[39:36](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=2376s) So again, first, uh, overfit to the training data, start with a small iteration, uh, visualize. So basically, instrument the traces. I instrumented them using Agenta in this case. Uh, try to look at them, try to look at the prompts that have been generated, and understand how the algorithm is working before, uh, increasing the sampling.

[39:57](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=2397s) And in this case, I think we had, uh, around two hundred, three hundred iterations per experiment. Um, in addition to that, there is actually a number of parameters like the batch size and so on that you need to, uh, fine-tune to get the algorithm to work.

[40:12](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=2412s) So that's it. Thanks a lot for watching. I hope that has been helpful and that you'll build good LLM-as-a-Judges that helps you to improve your applications. Uh, I'd love if you check out Agenta, our open source LLMOps, uh, platform.

[40:30](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=2430s) Um, and you can follow me both on LinkedIn and X. And finally, if you're thinking and working about auto optimization, uh, about how to, uh, optimize prompts, uh, feel free to reach out or to write in the comments on YouTube.

[40:48](https://www.youtube.com/watch?v=X4dEHRzBLmc&t=2448s) Have a great day. Thank you.
