AI Engineer World's Fair 2025

Prompt Engineering is Dead

Nir Gazit14:19

Loading slides…

Read the talk

Prompt Engineering Is Dead: Building an Evaluation Loop for a Documentation Chatbot

Nir Gazit turns manual prompt revision into an automated research-and-evaluation loop, then confronts what its improved score does—and does not—establish.

From a talk by Nir Gazit

Before you start: Basic familiarity with retrieval-augmented generation and Python will help you follow the pipeline and scoring example.

A documentation chatbot that needs to behave

How do you make a documentation chatbot consistently answer the questions you want it to answer? Nir Gazit opens with a provocative claim: prompt engineering never really amounted to engineering, and he improved his sample chatbot fivefold without manually engineering its prompts. That opening characterization is broader than the fact-based score demonstrated later. The practical problem is more specific: replacing repeated appeals to the model with examples of acceptable behavior and a way to measure progress.

The application is a simple retrieval-augmented generation, or RAG, chatbot on Traceloop’s website. It retrieves documentation and answers questions about it. The first deployment worked only reasonably well, leaving three requirements to improve: stay on Traceloop topics rather than answering unrelated questions about things such as the weather, give users useful answers, and make fewer mistakes.

Slide titled “I tried making the prompt work well” with three goals on the left and two Traceloop Demo chat examples on the right.Open full source frame
The chatbot goals: answer about Traceloop, be useful, and make fewer mistakes.

Manually rewriting instructions is the obvious next step. But the desired interface is simpler: show the system good and bad examples, then have it learn to follow the requirements reliably. The engineering problem becomes how to turn those examples into feedback that can drive prompt revisions.

0:160:32
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:16 · section reference included

Put an improvement loop around the RAG pipeline

Gazit imagines an agent that researches prompting techniques online and repeatedly applies them to the chatbot’s prompts. For that machine to know whether a revision helps, it needs two things first: a dataset of documentation questions and an evaluator that scores the pipeline’s answers. The resulting system has three parts—the RAG pipeline, the evaluator, and the agent that proposes improvements.

The baseline pipeline uses Chroma to find relevant documents and OpenAI to generate an answer from the retrieved context. Gazit runs the question, “How do I get started with Traceloop?” The trace exposes the OpenAI calls and database retrieval, followed by the final call that receives the accumulated context and produces the user’s answer. This also makes the optimization surface visible: the pipeline contains multiple prompts that could be revised.

1:582:15
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:58 · section reference included

Choose what counts as a good answer

An evaluator runs each dataset question through the RAG pipeline, collects the answer, and returns a score, ideally with reasons explaining a low or high result. Those outputs will become the optimization agent’s feedback. Gazit chooses LLM-as-a-judge because it is easy to build and deploy for this experiment.

Classic NLP metrics and embedding-based comparisons are alternatives, including for translation and summarization. Gazit distinguishes them primarily by the reference information they need: classic metrics usually compare generated output with a ground-truth answer, whereas a model judge can also assess an answer using the question and retrieved context.

Evaluator approachReference information
Classic reference-based metricsExpected answer and generated answer
LLM judge with ground truthExpected answer or facts, plus generated answer
LLM judge without a reference answerQuestion, context, and generated answer

For this demonstration, Gazit knows what the answers should contain, so he chooses a ground-truth-based judge. The ability to evaluate without a reference answer is available, but it is not the approach used here.

Types of evaluators slide with green panels listing classic NLP metrics as ground truth based and LLM-as-a-judge as able to work without ground truth.Open full source frame
Classic NLP metrics and LLM-as-a-judge differ in their need for ground truth.
4:004:22
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:00 · section reference included

Decide where to evaluate

Choosing a judge does not yet specify what it will inspect. A RAG pipeline retrieves data and then generates an answer from that data, so evaluation can operate at several levels:

  • Individual components: Evaluate retrieval or generation separately, much like unit tests. For retrieval, ask whether the database fetched the information needed to answer the question.
  • End-to-end behavior: Give the evaluator the user’s question and the pipeline’s final answer, then assess how well the answer addresses the question.
  • Behavior with internal context: Inspect the question, retrieved context, and answer together to assess the response in light of the information the generator actually received.

These views answer different questions. A final-answer score tells you whether the overall result was satisfactory; inspecting retrieval and context helps locate the source of a failure.

6:076:16
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:07 · section reference included

Turn expected facts into a numerical objective

Gazit creates twenty questions and writes three facts that each answer should contain, yielding sixty fact checks. The judge examines the generated answer for each expected fact and returns a Boolean pass or fail. For failures, it also supplies a reason explaining why the fact is missing. Aggregating those decisions produces the optimization objective: passed fact checks divided by total expected facts.

The aggregation itself can stay small. In Python, the following function takes completed judge results, computes the fact score, and retains the failure explanations for the optimizer:

python

from dataclasses import dataclass


@dataclass(frozen=True)
class FactCheck:
    question: str
    expected_fact: str
    passed: bool
    reason: str


def summarize_checks(checks: list[FactCheck]) -> dict:
    if not checks:
        raise ValueError("At least one fact check is required")

    return {
        "score": sum(check.passed for check in checks) / len(checks),
        "failure_reasons": [
            {
                "question": check.question,
                "expected_fact": check.expected_fact,
                "reason": check.reason,
            }
            for check in checks
            if not check.passed
        ],
    }

The judge supplies the semantic decision; the arithmetic simply aggregates it. This objective measures expected-fact coverage, not every requirement of the chatbot. It does not separately score usefulness, off-topic behavior, or all possible factual errors.

In the demonstration, Gazit shows the questions and expected facts, then starts the evaluator. It calls the RAG pipeline for each question and checks the returned answer against the associated facts. A running score updates as evaluation proceeds. The presentation does not show the full run to completion; Gazit says it takes a couple of minutes.

7:137:25
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

7:13 · section reference included

Research, revise, and evaluate again

With the pipeline and evaluator in place, the agent has a concrete job. Gazit describes a researcher that finds prompting guides online, then combines that guidance with evidence from failed evaluations. The loop proceeds as follows:

  1. Evaluate the initial prompt to obtain a baseline score and failure reasons.
  2. Use the failure reasons and prompting guidance to propose a revised prompt.
  3. Run the evaluator with the revised prompt.
  4. Feed the new score and failures into another revision.

The important feedback is more than a scalar. A score says how the prompt performed; a failure reason gives the revising agent something specific to address. Gazit compares this process to classic machine learning training, with model-generated text revisions taking the place of a conventional parameter update.

The demonstration uses CrewAI. After kickoff, the agent invokes the evaluator and receives its results. The recording skips several minutes before showing the researcher examining why the prompt failed, producing new instructions, and calling the evaluator again. What matters in this sequence is the closed loop: a proposed improvement does not become an observed improvement until the revised prompt has been evaluated.

9:039:19
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:03 · section reference included

What the higher score means

The resulting prompt is long and explicit, resembling the output of substantial manual prompt work. It assigns expertise in answering Traceloop questions and specifies desired and undesired behavior. Gazit emphasizes that building the agent required considerable code, but the target prompt revisions happened automatically.

Gazit reports that the score rose from 0.4 to 0.9 after two iterations on the twenty-question, sixty-fact evaluation. He stopped at 0.9 because ninety percent of the expected facts passed. Those scores imply a gain of 50 percentage points, or 2.25 times the baseline score; they do not establish the opening fivefold characterization on this metric.

Two green prompt panels connected by a right-pointing arrow, with Score: 0.4 beneath the left panel and Score: 0.9 beneath the right.Open full source frame
The results slide compares prompt excerpts with scores of 0.4 and 0.9.

The practical lesson is to build an evaluator and optimize against its feedback. Gazit calls this a form of gradient ascent, but the analogy concerns progressively improving the score: the process revises prompt text rather than calculating numerical gradients. An agent can perform those revisions, or a person can read prompting guides and make changes manually. Either way, the evaluator supplies the common test of whether a change helped.

11:0611:24
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:06 · section reference included

The optimizer still needs an independent test

A prompt can improve on the examples used for optimization while remaining poor on new questions. Gazit explicitly acknowledges overfitting: the optimizer had access to all twenty examples. The reported gain therefore does not establish performance on unseen questions. His proposed remedy is to collect more examples and divide them into training and test sets, or training, test, and evaluation sets. Optimize against the training examples, then assess performance separately on held-out data.

Giving the optimizer too much example-specific context risks teaching it how to answer only those questions. There is also a second limit to the headline: Gazit did considerable manual prompt engineering to build the agent that optimizes the chatbot’s prompts. The work moved up a level. Optimizing the evaluator’s prompts or the optimizer’s own prompts might automate more of that work, but he presents those possibilities as future experiments, not demonstrated capabilities.

The implementation is available in Traceloop’s auto-prompting demo, which Gazit says he tested the day before presenting. For readers following the repository today, its current code should not be treated as a frozen copy of the recorded run: the optimization flow completes above 0.8 and limits retries, while the optimizer crew configures a specific document as research knowledge rather than establishing unrestricted web crawling. The recording ends with the repository handoff and an invitation for questions outside the session, without demonstrating deployment of the optimized prompt.

Links slide listing Full Repository at https://github.com/traceloop/auto-prompting-demo and Talk to me at https://www.traceloop.com/consult.Open full source frame
Closing links to the auto-prompting demo repository and Traceloop consultation page.
12:2712:46
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

12:27 · section reference included

Resources

From the talk

Transcript download
Full transcript

Loading transcript…