# The Future of Evals - Ankur Goyal, Braintrust

Ankur Goyal, Braintrust

AI Engineer World's Fair 2025

[Talk page](https://ai.engineer/talks/MC55hdWLq4o-future-evals) · [Video](https://www.youtube.com/watch?v=MC55hdWLq4o)

## The Future of Evals

Braintrust’s Loop moves evaluation beyond inspecting scores, using frontier models to propose changes to prompts, datasets, and scorers while keeping those edits visible.

### The dashboard still leaves the hard decision to you

[0:42](https://www.youtube.com/watch?v=MC55hdWLq4o&t=42s)

After an eval finishes, what should you change in your code or prompt? Running evaluations can become routine long before answering that question does. After almost two years working with AI product teams at Braintrust, Ankur Goyal describes substantial evaluation activity: Goyal reports that the average organization signing up for Braintrust runs almost 13 evals a day, while some customers run more than 3,000 a day. Some advanced customers also spend more than two hours a day in the product working through evaluations. These are reported product-usage figures; the observation window and customer cohorts are unspecified.

![Ankur Goyal beside a slide titled “Eval adoption is growing quickly, with no end in sight,” with company logos and figures 12.8, 3,000, and 2hrs.](https://ai.engineer/api/talk-essay-media/4c85970128c01a487f4a37da004c6a08c0c5df1c714510214cda695016957f10)

Braintrust’s eval adoption slide highlights daily experiment volume and time spent in the product.

Even teams building highly automated agents still do much of this improvement work manually. An eval produces a dashboard; a person inspects it, leaves the dashboard, and decides which code or prompt changes might improve the next result. **The missing step is turning evaluation results into useful changes.** A better dashboard helps people inspect the evidence, but it still leaves that decision with them.

### Evaluating models that improve evaluations

[2:00](https://www.youtube.com/watch?v=MC55hdWLq4o&t=120s)

[Loop](https://www.braintrust.dev/docs/loop) is an agent built into Braintrust to help perform that improvement work. Its development depended on evaluations of the models doing the improving: every quarter for two years, Braintrust tested frontier models on their ability to improve prompts, datasets, and scorers. Goyal says those models performed poorly until very recently.

Goyal reports that **[Claude 4](https://www.anthropic.com/news/claude-4)** performed almost six times better than the previous leading model in Braintrust’s internal tests of evaluation-development tasks. The talk does not specify the scoring metric, test-set composition, inference settings, previous leader, or which Claude 4 variant was used. The comparison explains Braintrust’s decision to pursue this workflow, but does not provide enough detail to reproduce the result.

Loop’s scope extends from optimizing prompts to improving complex agents. It also helps build datasets and scorers, because **evaluation quality depends on all three working together**.

| Component | What Loop helps improve |
| --- | --- |
| Prompts and agents | The behavior being evaluated |
| Datasets | The cases used to evaluate that behavior |
| Scorers | The criteria used to judge the results |

This makes the evaluation setup itself part of the work: an agent can help improve how a system is tested as well as the system under test.

### Enabling Loop and choosing a model

[3:00](https://www.youtube.com/watch?v=MC55hdWLq4o&t=180s)

The launch-era setup presented in the recording is short:

1. Sign in to Braintrust or create an account.
2. Enable the feature flag named `Loop`.
3. Use the default Claude 4 model or select another model you have access to.

Goyal names OpenAI models, **Gemini**, and users’ own LLMs as alternatives. These are the instructions and model choices announced at the talk; today’s Loop documentation describes access within the product and model selection through providers configured for the organization.

### Keep proposed changes visible

[3:24](https://www.youtube.com/watch?v=MC55hdWLq4o&t=204s)

Loop runs directly inside Braintrust, preserving the ability to inspect data and prompts while working on them. That matters because an optimization suggestion may change different parts of the evaluation. The interface shows proposed changes side by side, including:

- **Data edits:** changes to the examples being evaluated.
- **Scoring ideas:** suggestions for how to assess outputs.
- **Prompt edits:** changes to the instructions driving the system.

The proposal remains connected to the material it would change, so users can examine the edit directly in the working interface.

For users who want more autonomy, Goyal describes an optional toggle as “Just go for it”: Loop proceeds with optimization rather than stopping at suggestions for inspection. He says that mode works well, without providing a separate measurement of its effectiveness. The product therefore offers both visible proposed edits and an option to let the agent optimize autonomously.

### Evaluation becomes work for the agent, too

[4:10](https://www.youtube.com/watch?v=MC55hdWLq4o&t=250s)

Evals have become a critical part of building AI products, while the work of interpreting and improving them remains labor intensive. Speaking at the June 2025 [AI Engineer World’s Fair](https://ai.engineer/worldsfair/2025), Goyal forecasts that frontier models will transform evaluation itself over the following year. Braintrust’s intended direction is to incorporate those advances into the tools used to develop and evaluate AI systems.

That direction is still being shaped. Goyal invites users to try Loop and provide feedback, acknowledging that substantial work remains. He also invites people working on UI, AI, and infrastructure to join the team, closing with an on-screen QR code as a route to get in touch.

![Closing slide lists Play, Eval, Log, Review, and Monitor beside a yellow panel containing a QR code, “Book a demo today,” and braintrust.dev.](https://ai.engineer/api/talk-essay-media/abe9483968273851bdbf15b6d83bd0880ee647fb0e27497391ca1cb023608c38)

Braintrust’s closing invitation displays a QR code and braintrust.dev.

## Resources

- [Introducing Claude 4](https://www.anthropic.com/news/claude-4) — Anthropic's original announcement of Claude Opus 4 and Sonnet 4, including capabilities and benchmark methodology.
- [Loop agent documentation](https://www.braintrust.dev/docs/loop) — Instructions for using Loop to analyze traces, optimize prompts, generate datasets and scorers, select models, and configure edit approval.
- [Turn production data into better AI with Loop](https://www.braintrust.dev/blog/loop) — A November 2025 account of Loop's production-data analysis workflows, team use cases, and product access.
- [Webinar recap: Eval best practices](https://www.braintrust.dev/blog/webinar-best-practices) — A recap of Goyal's technical Q&A covering small initial datasets, scoring functions, and iterative evaluation.

## Slides

### Eval adoption is growing

Braintrust organizations log 12.8 experiments per day on average; the most active run over 3,000 per day. Goyal says some advanced teams spend over two hours a day working through evals, a manual process.

[0:46](https://www.youtube.com/watch?v=MC55hdWLq4o&t=46s) [Customer logos accompany Braintrust statistics: 12.8 experiments per day on average, over 3,000 for the most active organizations, and over two hours per day for top AI teams.](https://assets.ai.engineer/sites/aiecode2025/sha256/d228f3fc28f123a4792445f53ac3feb5636208b1022ba1daa4600af355631717.webp)

Eval adoption is growing quickly, with no end in sight

Airtable
instacart
Notion
ramp
replit
stripe
Vercel
zapier

12.8
On average, organizations log 12.8 experiments per day with Braintrust

3,000
Organizations running the most evals conduct over 3,000 experiments per day.

2hrs
Top ai teams spend 2+ hours per day in Braintrust

Interface text:
AI Engineer World's Fair
Microsoft
ANKUR GOYAL
CEO
braintrust

### Loop prompt optimization preview

Loop reviews a factuality score of 0.6 and suggests a more specific fitness prompt, while the playground keeps the prompt and a bench press example visible.

[3:09](https://www.youtube.com/watch?v=MC55hdWLq4o&t=189s) [Braintrust displays a proposed prompt edit beside the playground and Loop's evaluation feedback.](https://assets.ai.engineer/sites/aiecode2025/sha256/9126a6f1cf1b5803c8bc43360fb3b774da4d9b6471df07821020d3c090e298a9.webp)

Fitness friend

Base task
Suggested change
JSON
Unchanged context:
{
  "prompt": {
    "type": "chat",
    "messages": [
      {

Deleted text:
        "content": "You are an expert personal trainer",
        "role": "system"
      },
      {
        "content": "Which muscle groups does this workout target {{input}}",

Added text:
        "content": "You are an expert personal trainer and exercise physiologist with deep knowledge of anatomy and biomechanics. Your responses should be accurate, evidence-based, and clearly [clipped]

Comparison task
Claude 4 Sonnet
System
You are an expert personal trainer
User
Which muscle groups does this workout target {{input}}

Loop
and current performance:
Summarized playground — Completed
I notice the factuality score is only 0.6, which suggests there's room for improvement. Let me get more examples to understand the dataset better:

Fetched eval results — Completed
Based on the current results, I can see that the prompt is producing detailed responses but the factuality score is only 0.6. The current response is actually quite comprehensive and anatomically accurate, so the issue might be in the specificity and structure of the prompt. Let me optimize it to be more precise and focused on accuracy.

Edited prompt
Let me check the correct structure by looking at the task definition more carefully:

Task change suggestions
JSON
{
  "prompt": {
    "type": "chat",
    "messages": [
      {
        "content": "You are an expert personal trainer",
        "role": "system" [clipped]

Input: Bench press
GPT-4.1 — Base
Output: The bench press primarily targets the following muscle [clipped]

Interface text:
braintrustdata.com > ai-search > Playgrounds
Diff
Private
+ Experiments
Run
Reject
Accept
Params
+ Message part
+ Message
Tools
Text output
Save prompt
All rows
Filter
Fields
Grid
Height
+ Row
+ Task 3
+ Scorer 2
Workouts 1
Factuality
Humor
6d ago
0.1s
160
$0.001
What would you like to optimize or generate?
Continuous
AI Engineer World's Fair
Engineering the future of AI

### Loop prompt optimization

Loop shows a suggested prompt edit beside the playground. Its evaluation feedback reports a factuality score of 0.6 and proposes a more specific, accuracy-focused prompt.

[3:23](https://www.youtube.com/watch?v=MC55hdWLq4o&t=203s) [Braintrust shows the proposed prompt edit and Loop's evaluation feedback beside the presenter.](https://assets.ai.engineer/sites/aiecode2025/sha256/0562a6b525b716507df614d14f64b61f4846c5099aa820ee42a6950c075c23c6.webp)

Fitness friend

Base task
Suggested change
Reject    Accept
JSON
{
  "prompt": {
    "type": "chat",
    "messages": [
      {
-       "content": "You are an expert personal trainer",
-       "role": "system"
-     },
-     {
-       "content": "Which muscle groups does this workout target {{input}}",
+       "content": "You are an expert personal trainer and exercise physiologist with deep knowledge of anatomy and biomechanics. Your responses should be accurate, evidence-based, and clearly

Comparison task
Claude 4 Sonnet
System
You are an expert personal trainer
User
Which muscle groups does this workout target {{input}}

Loop
and current performance:
Summarized playground    Completed
I notice the factuality score is only 0.6, which suggests there's room for improvement. Let me get more examples to understand the dataset better:
Fetched eval results    Completed
Based on the current results, I can see that the prompt is producing detailed responses but the factuality score is only 0.6. The current response is actually quite comprehensive and anatomically accurate, so the issue might be in the specificity and structure of the prompt. Let me optimize it to be more precise and focused on accuracy.
Edited prompt
Let me check the correct structure by looking at the task definition more carefully:
Task change suggestions
JSON
{
  "prompt": {
    "type": "chat",
    "messages": [
      {
-       "content": "You are an expert personal trainer",
-       "role": "system"

Input: Bench press
GPT-4.1
Output: The bench press primarily targets the following muscle

Interface text:
braintrustdata.com > ai-search > Playgrounds
Diff    Private    + Experiments    Run
Params    + Message part    + Message    Tools    Text output    Save prompt
All rows    Filter    Fields    Grid    Height    + Row    + Task 3    + Scorer 2    Workouts 1
Factuality    Humor    Base
6d ago    0.1s    160    $0.001
What would you like to optimize or generate?
Continuous
AI Engineer World's Fair
Microsoft
aws
smol ai

[3:28](https://www.youtube.com/watch?v=MC55hdWLq4o&t=208s) [Braintrust playground showing a suggested prompt change alongside Loop’s evaluation and editing steps.](https://assets.ai.engineer/sites/aiecode2025/sha256/52a82e91ad67fed18a0383dcc61cc5190393ec47e19f8e11f4441eae631801cc.webp)

Fitness friend

Base task
Suggested change
Reject    Accept
JSON
{
  "prompt": {
    "type": "chat",
    "messages": [
      {
-       "content": "You are an expert personal trainer",
-       "role": "system"
-     },
-     {
-       "content": "Which muscle groups does this workout target {{input}}",
+       "content": "You are an expert personal trainer and exercise physiologist with deep knowledge of anatomy and biomechanics. Your responses should be accurate, evidence-based, and clearly

Comparison task
Claude 4 Sonnet
System
You are an expert personal trainer
User
Which muscle groups does this workout target {{input}}

Loop
and current performance:
Summarized playground    Completed
I notice the factuality score is only 0.6, which suggests there's room for improvement. Let me get more examples to understand the dataset better:
Fetched eval results    Completed
Based on the current results, I can see that the prompt is producing detailed responses but the factuality score is only 0.6. The current response is actually quite comprehensive and anatomically accurate, so the issue might be in the specificity and structure of the prompt. Let me optimize it to be more precise and focused on accuracy.
Edited prompt
Let me check the correct structure by looking at the task definition more carefully:
Task change suggestions
JSON
{
  "prompt": {
    "type": "chat",
    "messages": [
      {
-       "content": "You are an expert personal trainer",
-       "role": "system"

Input: Bench press
GPT-4.1
Output: The bench press primarily targets the following muscle

Interface text:
braintrustdata.com > ai-search > Playgrounds
Diff    Private    + Experiments    Run
Params    + Message part    + Message    Tools    Text output    Save prompt
All rows    Filter    Fields    Grid    Height    + Row    + Task 3    + Scorer 2    Workouts 1
Factuality    Humor    Base
6d ago    0.1s    160    $0.001
What would you like to optimize or generate?
Continuous

### Play, Eval, Log, Review, Monitor

Goyal describes evaluation as manual and expects frontier models to transform evals over the next year. He invites attendees to try Loop and scan the QR code to get in touch.

[4:12](https://www.youtube.com/watch?v=MC55hdWLq4o&t=252s) [A list of five Braintrust activities beside a yellow demo-booking panel with a QR code.](https://assets.ai.engineer/sites/aiecode2025/sha256/546b7d7b684f804ae9c277961bb8e2ea250c3c6e17e39b493485acda148d74c2.webp)

Play
Eval
Log
Review
Monitor

Book a demo today
braintrust.dev

## Full transcript

[0:00](https://www.youtube.com/watch?v=MC55hdWLq4o&t=0s) [upbeat music] [audience applauding] Awesome.

[0:22](https://www.youtube.com/watch?v=MC55hdWLq4o&t=22s) Uh, so today we're gonna talk a little bit about evals to date and where we think evals are gonna be going in the future.

[0:32](https://www.youtube.com/watch?v=MC55hdWLq4o&t=32s) Also, for those of you who saw my brother earlier, um, I'm gonna do my best to live up to his energy and, uh, and charisma. [laughs]

[0:42](https://www.youtube.com/watch?v=MC55hdWLq4o&t=42s) But, um, yeah, you know, it's been an amazing almost two-year journey for us at Braintrust. We have had the opportunity to work with some of the most amazing companies building, um, I think, the best AI products in the world.

[0:55](https://www.youtube.com/watch?v=MC55hdWLq4o&t=55s) Uh, I'm blown away by how many evals people actually run on the product. The average org that signs up for Braintrust runs almost thirteen evals a day. Some of our customers run more than three thousand evals a day.

[1:11](https://www.youtube.com/watch?v=MC55hdWLq4o&t=71s) Uh, and some of the most advanced companies that are running evals are spending more than two hours in the product every day working through their evals. And I think one of the things that stands out to me is while we have customers building some of the coolest, most automated, um, AI-based products and agents in the world, evals

[1:33](https://www.youtube.com/watch?v=MC55hdWLq4o&t=93s) are such a manual process. Um, you know, to date, every time you run an eval, the best thing you can do is look at a dashboard, and I think we have a pretty cool dashboard in Braintrust.

[1:44](https://www.youtube.com/watch?v=MC55hdWLq4o&t=104s) But still, it's just a dashboard that you look at and you walk away and think, "Okay, what changes can I make to my code or to my prompts so that this eval does better?"

[1:55](https://www.youtube.com/watch?v=MC55hdWLq4o&t=115s) Um, and I actually think that is all going to change.

[2:00](https://www.youtube.com/watch?v=MC55hdWLq4o&t=120s) Uh, so today I'm excited to talk about something called Loop. Loop is an agent that we've been working on for some time now that's built into Braintrust, um, and it's actually only possible because of evals.

[2:12](https://www.youtube.com/watch?v=MC55hdWLq4o&t=132s) Every quarter for the last two years, we've run evals on the frontier models to see how good they are at actually improving prompts, improving datasets, and improving scorers. And until very, very recently, they actually weren't very good.

[2:27](https://www.youtube.com/watch?v=MC55hdWLq4o&t=147s) In fact, we think that Claude 4, in particular, was a real breakthrough moment, um, and it performs almost six times better than the l- the previous leading model before it.

[2:39](https://www.youtube.com/watch?v=MC55hdWLq4o&t=159s) So Loop runs inside of Braintrust, and it can automatically optimize, uh, your prompts all the way to very complex, uh, agents. Um, but just as importantly, it also helps you build better datasets and better scorers because it's really the combination of these three things that make for really great evals.

[3:00](https://www.youtube.com/watch?v=MC55hdWLq4o&t=180s) This is a little preview of, of the UI. Um, you can actually start using it today if you are an existing Braintrust user or you sign up for the product.

[3:08](https://www.youtube.com/watch?v=MC55hdWLq4o&t=188s) There's a feature flag that you can just flip on called Loop and start using it right away. Um, by default, it uses Claude 4, but you can actually pick any model that you have access to and start using it, whether it's an OpenAI model, a Gemini model, or maybe some of you are building your own LLMs.

[3:24](https://www.youtube.com/watch?v=MC55hdWLq4o&t=204s) You can use those as well. Um, and as you can see, it runs directly inside of Braintrust. One of the things that we, uh, learned from working with a lot of users is how important it is to actually look at data and look at prompts while you're working with them, and we didn't want that to go away,

[3:43](https://www.youtube.com/watch?v=MC55hdWLq4o&t=223s) uh, when we introduced Loop. So every time it suggests an edit to your data, or it suggests a new idea for scoring, or it suggests an edit to one of your prompts, you can actually see that side by side directly in the UI.

[3:56](https://www.youtube.com/watch?v=MC55hdWLq4o&t=236s) Um, of course, for the more adventurous among you, there's also a toggle that you can turn on that says, like, "Just go for it," and it will go and optimize away, um, which actually works really well.

[4:10](https://www.youtube.com/watch?v=MC55hdWLq4o&t=250s) So just to recap, uh, to date, evals have been a critical part of building some of the best AI products in the world, but the task of actually doing evaluation has been incredibly manual.

[4:23](https://www.youtube.com/watch?v=MC55hdWLq4o&t=263s) And I'm excited about how over the next year, uh, evals themselves are going to be completely revolutionized by the latest and greatest that's coming out, um, from, you know, the frontier models themselves.

[4:34](https://www.youtube.com/watch?v=MC55hdWLq4o&t=274s) And we're very excited to incorporate that into Braintrust. Please, if you're not already using the product, try it out. Uh, try out Loop. Give us your feedback. Uh, we have a lot of work to do, um, and we'd love to talk to you.

[4:46](https://www.youtube.com/watch?v=MC55hdWLq4o&t=286s) We're also hiring, uh, so if you're interested in working on this kind of problem, whether it's the UI part of it, the AI part of it, or the infrastructure, uh, side of it, we'd love to talk to you.

[4:56](https://www.youtube.com/watch?v=MC55hdWLq4o&t=296s) Um, you can scan this QR code. Uh, it should be over there. Yeah. You can scan the QR code and, and get in touch with us. Uh, we'd love to chat.

[5:04](https://www.youtube.com/watch?v=MC55hdWLq4o&t=304s) Thank you. [audience applauding] [upbeat music]
