AI Engineer World's Fair 2025
Prompt Engineering is Dead
Loading slides…
Read the talk
Prompt Engineering Is Dead: Building an Evaluation Loop for a Documentation Chatbot
Nir Gazit turns manual prompt revision into an automated research-and-evaluation loop, then confronts what its improved score does—and does not—establish.
From a talk by Nir Gazit
Before you start: Basic familiarity with retrieval-augmented generation and Python will help you follow the pipeline and scoring example.
A documentation chatbot that needs to behave
How do you make a documentation chatbot consistently answer the questions you want it to answer? Nir Gazit opens with a provocative claim: prompt engineering never really amounted to engineering, and he improved his sample chatbot fivefold without manually engineering its prompts. That opening characterization is broader than the fact-based score demonstrated later. The practical problem is more specific: replacing repeated appeals to the model with examples of acceptable behavior and a way to measure progress.
The application is a simple retrieval-augmented generation, or RAG, chatbot on Traceloop’s website. It retrieves documentation and answers questions about it. The first deployment worked only reasonably well, leaving three requirements to improve: stay on Traceloop topics rather than answering unrelated questions about things such as the weather, give users useful answers, and make fewer mistakes.
Manually rewriting instructions is the obvious next step. But the desired interface is simpler: show the system good and bad examples, then have it learn to follow the requirements reliably. The engineering problem becomes how to turn those examples into feedback that can drive prompt revisions.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Put an improvement loop around the RAG pipeline
Gazit imagines an agent that researches prompting techniques online and repeatedly applies them to the chatbot’s prompts. For that machine to know whether a revision helps, it needs two things first: a dataset of documentation questions and an evaluator that scores the pipeline’s answers. The resulting system has three parts—the RAG pipeline, the evaluator, and the agent that proposes improvements.
The baseline pipeline uses Chroma to find relevant documents and OpenAI to generate an answer from the retrieved context. Gazit runs the question, “How do I get started with Traceloop?” The trace exposes the OpenAI calls and database retrieval, followed by the final call that receives the accumulated context and produces the user’s answer. This also makes the optimization surface visible: the pipeline contains multiple prompts that could be revised.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Choose what counts as a good answer
An evaluator runs each dataset question through the RAG pipeline, collects the answer, and returns a score, ideally with reasons explaining a low or high result. Those outputs will become the optimization agent’s feedback. Gazit chooses LLM-as-a-judge because it is easy to build and deploy for this experiment.
Classic NLP metrics and embedding-based comparisons are alternatives, including for translation and summarization. Gazit distinguishes them primarily by the reference information they need: classic metrics usually compare generated output with a ground-truth answer, whereas a model judge can also assess an answer using the question and retrieved context.
| Evaluator approach | Reference information |
|---|---|
| Classic reference-based metrics | Expected answer and generated answer |
| LLM judge with ground truth | Expected answer or facts, plus generated answer |
| LLM judge without a reference answer | Question, context, and generated answer |
For this demonstration, Gazit knows what the answers should contain, so he chooses a ground-truth-based judge. The ability to evaluate without a reference answer is available, but it is not the approach used here.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Decide where to evaluate
Choosing a judge does not yet specify what it will inspect. A RAG pipeline retrieves data and then generates an answer from that data, so evaluation can operate at several levels:
- Individual components: Evaluate retrieval or generation separately, much like unit tests. For retrieval, ask whether the database fetched the information needed to answer the question.
- End-to-end behavior: Give the evaluator the user’s question and the pipeline’s final answer, then assess how well the answer addresses the question.
- Behavior with internal context: Inspect the question, retrieved context, and answer together to assess the response in light of the information the generator actually received.
These views answer different questions. A final-answer score tells you whether the overall result was satisfactory; inspecting retrieval and context helps locate the source of a failure.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn expected facts into a numerical objective
Gazit creates twenty questions and writes three facts that each answer should contain, yielding sixty fact checks. The judge examines the generated answer for each expected fact and returns a Boolean pass or fail. For failures, it also supplies a reason explaining why the fact is missing. Aggregating those decisions produces the optimization objective: passed fact checks divided by total expected facts.
The aggregation itself can stay small. In Python, the following function takes completed judge results, computes the fact score, and retains the failure explanations for the optimizer:
python
from dataclasses import dataclass
@dataclass(frozen=True)
class FactCheck:
question: str
expected_fact: str
passed: bool
reason: str
def summarize_checks(checks: list[FactCheck]) -> dict:
if not checks:
raise ValueError("At least one fact check is required")
return {
"score": sum(check.passed for check in checks) / len(checks),
"failure_reasons": [
{
"question": check.question,
"expected_fact": check.expected_fact,
"reason": check.reason,
}
for check in checks
if not check.passed
],
}
The judge supplies the semantic decision; the arithmetic simply aggregates it. This objective measures expected-fact coverage, not every requirement of the chatbot. It does not separately score usefulness, off-topic behavior, or all possible factual errors.
In the demonstration, Gazit shows the questions and expected facts, then starts the evaluator. It calls the RAG pipeline for each question and checks the returned answer against the associated facts. A running score updates as evaluation proceeds. The presentation does not show the full run to completion; Gazit says it takes a couple of minutes.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Research, revise, and evaluate again
With the pipeline and evaluator in place, the agent has a concrete job. Gazit describes a researcher that finds prompting guides online, then combines that guidance with evidence from failed evaluations. The loop proceeds as follows:
- Evaluate the initial prompt to obtain a baseline score and failure reasons.
- Use the failure reasons and prompting guidance to propose a revised prompt.
- Run the evaluator with the revised prompt.
- Feed the new score and failures into another revision.
The important feedback is more than a scalar. A score says how the prompt performed; a failure reason gives the revising agent something specific to address. Gazit compares this process to classic machine learning training, with model-generated text revisions taking the place of a conventional parameter update.
The demonstration uses CrewAI. After kickoff, the agent invokes the evaluator and receives its results. The recording skips several minutes before showing the researcher examining why the prompt failed, producing new instructions, and calling the evaluator again. What matters in this sequence is the closed loop: a proposed improvement does not become an observed improvement until the revised prompt has been evaluated.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
What the higher score means
The resulting prompt is long and explicit, resembling the output of substantial manual prompt work. It assigns expertise in answering Traceloop questions and specifies desired and undesired behavior. Gazit emphasizes that building the agent required considerable code, but the target prompt revisions happened automatically.
Gazit reports that the score rose from 0.4 to 0.9 after two iterations on the twenty-question, sixty-fact evaluation. He stopped at 0.9 because ninety percent of the expected facts passed. Those scores imply a gain of 50 percentage points, or 2.25 times the baseline score; they do not establish the opening fivefold characterization on this metric.
The practical lesson is to build an evaluator and optimize against its feedback. Gazit calls this a form of gradient ascent, but the analogy concerns progressively improving the score: the process revises prompt text rather than calculating numerical gradients. An agent can perform those revisions, or a person can read prompting guides and make changes manually. Either way, the evaluator supplies the common test of whether a change helped.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The optimizer still needs an independent test
A prompt can improve on the examples used for optimization while remaining poor on new questions. Gazit explicitly acknowledges overfitting: the optimizer had access to all twenty examples. The reported gain therefore does not establish performance on unseen questions. His proposed remedy is to collect more examples and divide them into training and test sets, or training, test, and evaluation sets. Optimize against the training examples, then assess performance separately on held-out data.
Giving the optimizer too much example-specific context risks teaching it how to answer only those questions. There is also a second limit to the headline: Gazit did considerable manual prompt engineering to build the agent that optimizes the chatbot’s prompts. The work moved up a level. Optimizing the evaluator’s prompts or the optimizer’s own prompts might automate more of that work, but he presents those possibilities as future experiments, not demonstrated capabilities.
The implementation is available in Traceloop’s auto-prompting demo, which Gazit says he tested the day before presenting. For readers following the repository today, its current code should not be treated as a frozen copy of the recorded run: the optimization flow completes above 0.8 and limits retries, while the optimizer crew configures a specific document as research knowledge rather than establishing unrestricted web crawling. The recording ends with the repository handoff and an invitation for questions outside the session, without demonstrating deployment of the optimized prompt.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Source code and setup instructions for the CrewAI demonstration, including data loading, RAG execution, and iterative prompt optimization.
Current documentation for retrieving documents through embedding similarity search in Chroma.
Current guidance for configuring agent roles, goals, tools, and collaboration in CrewAI.
Further reading
Gazit describes the documentation-chatbot experiment and the evaluation and optimizer work behind it, with a link to the presentation.
Full transcript
Loading transcript…