Contents
  1. Specifying behavior in a model call
    1. What prompt engineering changes
    2. Specify the task
    3. Define acceptable responses
    4. Learn from demonstrations
  2. How prompting developed
    1. From prediction interfaces to instructions
      1. Tasks expressed as predictions
      2. Training for direct requests
      3. Searching for effective inputs
  3. Choosing and testing demonstrations
    1. Choose informative examples
    2. Test order and formatting
    3. Protect independent assessment
  4. Improving reusable prompts
    1. Revise through testable hypotheses
    2. Search prompts automatically
    3. Inspect DSPy's prompt artifacts
    4. Use diagnostic feedback with GEPA
  5. Transfer and practical choices
    1. Reassess transfer and sufficiency
    2. Frequently asked questions
      1. Is in-context learning training?
      2. How many demonstrations should I include?
      3. Do roles, wording or example order reliably help?
      4. Does a written explanation verify the answer?
      5. When is automatic optimization worth its cost?
      6. Does GEPA require DSPy?
      7. When should I retest a saved prompt?
      8. When should I supply context or try fine-tuning?
  6. Check understanding
  7. Open questions
  8. Selected talks
  9. References
  10. Talk library
← All topics

Prompting & in-context learning

Prompt engineering is the practice of designing and testing instructions and examples that guide a model’s response without changing its learned parameters, or weights. In 2023, Karina Nguyen of Anthropic used a fashion recommender to show what it means to “clearly communicate the task objective.” She gave Claude a shopper’s request and a catalog item, then asked for a yes-or-no decision about whether to recommend that item. Nguyen’s task-prompting walkthrough

Specifying behavior in a model call

What prompt engineering changes

On Nguyen’s slide, the shopper asks for a long dress in the style of The Great Gatsby. The candidate is a Maxi Godet Denim Skirt In Light Blue Wash. The length might fit, but the shopper asked for a dress and the item is a skirt. Claude’s reply on the last line, <answer> NO </answer>, passes two separate checks. The tags contain one of the two allowed labels, and NO fits this particular garment mismatch. Testing the recommendation rule requires checking its decisions on other items as well.

A denim skirt is evaluated against a Gatsby-style dress query; the response is no.
Curation strategy 1: denim skirt example — Writing Principles for Task-Tuned Prompt Engineering · Karina Nguyen · Watch at 12:29

The prompt lists the allowed answers, but it gives Claude no example of when to choose either one. A demonstration pairs an earlier input with its desired answer. In their GPT-3 study, Brown and colleagues at OpenAI called a request with no demonstrations zero-shot, one with a single demonstration one-shot, and one with several few-shot. Nguyen’s request is zero-shot. The NO visible on the slide is the answer Claude generated. Language Models are Few-Shot Learners

An engineer can write the rule, show the desired behavior, or use software to select what the model sees.
ApproachWhat the author suppliesWhat reaches the next callHow to check the effect
Nguyen’s handwritten instructionA rule for judging relevance and an answer formatThe rule, the query and the candidate itemCheck format and relevance separately on new items.
GPT-3 in-context demonstrationsInputs paired with desired completionsThe pairs followed by a new, unanswered inputCompare predictions with an instruction-only baseline on the same cases.
DSPy prompt compilationTask fields, examples and a scoring ruleInstructions or demonstrations chosen using development and validation resultsFreeze the selected program and assess it on untouched cases.

These choices combine. A DSPy compiler can select examples for a few-shot prompt that also contains a handwritten rule. In each case the intervention changes the input the model uses to predict its response. Deciding which documents, conversation history and tools also belong in each call is the broader work of context engineering.

Specify the task

An instruction tells the model what operation to perform. In Nguyen’s example, the query expresses a requirement and the item description supplies evidence about one candidate. Calling the operation recommendation still leaves decisions to the author. If a shopper mentions summer, should a winter coat qualify because it matches the requested color? Nguyen makes season and weather relevant when the shopper specifies them. A reviewer can apply that criterion to other garments. Calling the model a fashion expert leaves the priority between color and season open.

Missing evidence requires a decision too. If fabric matters and the description omits it, one catalog might exclude the item and another might send it for review. For the second catalog, the instruction could allow UNKNOWN when a required attribute cannot be determined. That would extend Nguyen’s binary rule and change what the caller must handle. Try giving the proposed instruction to two reviewers. A disagreement about which answer is allowed identifies a requirement worth settling before experimenting with phrasing.

A role can be useful when voice is part of the requirement. For Claude.ai’s conversation titles, Nguyen asked Claude to act as an editor and supplied previous titles so it could see the style to follow. Where an interface supports a system message, the application can use that channel for instructions. Its message boundaries depend on the model interface. Placement within a message is a separate variable. On long-document questions for Claude and Claude Instant, Nguyen reported better results when the question followed the document. Moving the same question around the same document is a way to test whether that result applies elsewhere. Special tokens and chat templates

Define acceptable responses

The caller needs to know where to find the result, what values it can take and what each value means. For the skirt response, a parser can locate the answer tags. A second check can verify that the enclosed text, after trimming spaces, is an allowed label. Now imagine replacing NO with YES. Both checks would still pass. Judging the value requires comparing the offered garment with the requested one. The same distinction applies to JSON. Constrained generation can restrict the model to continuations permitted by a schema, but choosing the right permitted value requires task evidence. Structured Outputs and Tool Calling

Input delimiters serve a different purpose. A tag such as <item> identifies material the model should inspect. A command appearing inside a product description can still influence the response. If an application lets a model request actions, the application must independently decide whether the action and its arguments are authorized.

Response requirements also affect what can fit. A token is a model-specific unit used to represent text and may be shorter than a word. The context window limits the sequence available to a call. Count the rendered instructions, message markers, examples and task data together. Where input and output share capacity, leave room for generation and respect any separate input and output limits. Counting the complete request and reserving room for generation

Asking for intermediate steps changes the requested output. Chain-of-thought prompting can also show those steps in worked demonstrations, which adds to the input as well. Assess the final answer independently and check any stated steps on their merits. In experiments with answer hints and biased answer positions, models sometimes justified a changed answer without mentioning the cue that influenced it. What a reasoning trace establishes

Learn from demonstrations

In-context learning occurs when a model uses information in the current request to adapt how it performs a task while its weights stay fixed. Examples can tell it which task to perform, how to format the answer, or which label goes with which input. To test whether the model uses a particular label mapping, keep the task and the new input fixed and change only the mapping in the examples. A fresh call needs the mapping supplied again. An explicit instruction can state it directly. To repeat the demonstration condition, supply the examples again. Fitting model parameters can carry the association into later calls through the weights.

Consider two versions of a sentiment prompt using the unrelated label names Foo and Bar. These illustrative reviews and expected answers show the test. The last row is for scoring and is withheld from the model.
Same review textMapping AMapping B
Demonstration · I enjoyed every minute.FooBar
Demonstration · I hated it.BarFoo
New input · What a delight.UnansweredUnanswered
Expected label for the new inputFooBar

Recognizing delight as positive helps in both columns. Only the demonstrations say which word to emit. If predictions track the reversal on fresh reviews, that is evidence the model used the supplied association. Reversing the familiar words positive and negative makes the test harder to interpret because their prior meanings compete with the examples. Models differed on these tests. Unrelated and reversed label experiments

A complementary experiment randomized the labels attached to examples. On several classification and multiple-choice tasks, much of the gain over zero-shot prompting survived. The model still saw representative inputs, the available labels and how an answer should look. A correct prediction can therefore have several explanations. For an application, verify the labels and test the intended association directly. Demonstration ablations

How prompting developed

From prediction interfaces to instructions

An early version of the scientific question used numbers in place of words. In 2001, Hochreiter, Younger and Conwell trained a recurrent network on families of functions. Given a sequence of inputs and previous target values, its internal state could carry information about the particular function it was currently predicting. This separated two timescales of learning. Training built the capacity to adapt, and examples in a sequence let the network use that capacity. Learning to Learn Using Gradient Descent

Tasks expressed as predictions

For language models, a practical question was how to express useful work as a prediction the model already knew how to make. In 2019, GPT-2 used a summary cue after an article and an unfinished sentence pair for translation. LAMA probed factual associations with cloze statements, sentences with a missing answer. Completing a birthplace statement let researchers test a specific fact using the model’s existing vocabulary. GPT-2 report and LAMA study

Text could specify the task during training too. T5, described in 2019, prefixed inputs with task identifiers so that translation, classification and other tasks could share one text-to-text interface. Pattern-Exploiting Training (PET), proposed in 2020, used blanks for classification. A verbalizer mapped predicted words to classes. PET fine-tuned models on patterns and used their predictions to label additional data. Its few-shot setting counted scarce labeled training cases. The GPT-3 experiments that year varied the number of examples in the input to a fixed model. Both used text to express a task, but reproducing the former required trained state and reproducing the latter required the examples in the request.

Training for direct requests

Following a direct instruction became a training objective in its own right. In 2021, FLAN fine-tuned a model on tasks expressed as instructions and evaluated it on task types withheld from that training. BigScience’s T0 varied the natural-language prompts used to train on supervised tasks. More formulations per dataset improved median held-out performance and reduced variation across prompts in its experiments. An engineer using the resulting model could benefit from that training and still write a task-specific instruction for each call. Instruction tuning

Searching for effective inputs

Choosing the input itself posed a search problem. In 2020, AutoPrompt used gradients from a model to propose replacement tokens for a template and tested the resulting strings. Soft-prompt tuning in 2021 instead trained continuous numerical values to prepend to an input. In 2022, Automatic Prompt Engineer used a model to propose instructions from examples. In 2023, Omar Khattab and colleagues introduced DSPy compilation for programs containing several model calls. These methods change who proposes an input and what form it takes. The developer still has to settle questions like Nguyen’s garment rule before a search can reward the behavior the application needs.

Choosing and testing demonstrations

Choose informative examples

Choose a demonstration by asking what the model could learn from it that the other examples leave undecided. Suppose positive reviews in a prompt all end in exclamation marks and negative reviews all end in periods. Both sentiment and punctuation predict every label. A positive review ending in a period and a negative review ending in an exclamation mark show both exceptions to the punctuation rule. Check that the sentiment labels are unambiguous before spending space on the pair. A mislabeled boundary case teaches a different distinction. Data Quality and Curation

A fixed set accompanies every request. A query-dependent set is selected for the current input, for example by finding similar cases in a library. Figure 1 of the coverage-based selection study illustrates a request to schedule a meeting with someone’s manager. Inspect the operation each example contributes. A second appointment paraphrase repeats scheduling. A manager-lookup example supplies the missing operation that identifies the attendee. Selecting for coverage helped particularly on tasks that required combining operations into a structured query. Meeting example, Figure 1 · Search and Retrieval

Similarity itself can help. In the KATE study, GPT-3 classified IMDB movie reviews with three demonstrations drawn from the SST-2 sentiment dataset per request. At temperature zero, random selection averaged 87.95% accuracy over five runs. Selecting similar reviews with an unchanged RoBERTa-large text encoder achieved 91.99%. The selection procedure also determined the order. Three reviews in either condition could occupy different numbers of tokens, so the result answers a question about fixed example count rather than equal input cost. KATE study, Table 3

Increasing count is a separate intervention. With the February 2024 Gemini 1.5 Pro, news-summary overlap with reference answers on XSum improved through roughly 50 examples, then declined. With the same source of demonstrations and a different target collection, XLSum, adding examples generally helped. Some summaries fabricated dates or times. Choose a set using the quality measure the application needs, its rendered length and its coverage. Inspect repeated labels and answer positions for accidental cues, and exclude assessment answers from the example pool. Many-Shot In-Context Learning

Test order and formatting

Prompt sensitivity is a change in behavior associated with a change to the input. Moving an intact demonstration changes its position. Swapping answer options changes where a choice appears inside a case. Changing separators changes how the same examples are presented. To measure any one of these, keep the others fixed and score the variants on the same cases.

ComparisonIntervention and controlsWhat can be learned
Demonstration orderResearchers tried all 24 arrangements of four fixed, balanced SST-2 examples.Accuracy varied. Rankings of the orders agreed weakly across the GPT-2 and GPT-3 models tested.
Prompt formattingFormatSpread varied separators, capitalization and option numbering with demonstrations, order and evaluation cases fixed.Ten sampled formats produced a median spread of 7.5 accuracy points across tested settings. The principal measure ranked answer options by model probability.
Execution variationRun the identical request afresh several times with the same model and generation settings.Differences estimate within-prompt variation. Replaying a cached answer supplies no new execution.

If the product allows several equivalent phrasings, examine performance across those phrasings as well as the best score. A prompt that wins once and changes its answer on an innocuous reformat has a different deployment profile from one that works across the variants users actually send. Punctuation can change token boundaries. A difference between two formatted requests alone cannot isolate segmentation as its cause.

Protect independent assessment

A baseline is the prompt against which a change is judged. A metric specifies how to score the resulting behavior. For a relevance task, start with a clear instruction alone and compare it with the same instruction plus demonstrations. Give both versions the same new items and queries. Record whether the label can be extracted and whether it agrees with a reviewed decision. Pairing the results item by item reveals both newly correct decisions and correct decisions lost. Cases and trials · Paired comparisons

Separate the cases by the influence they are allowed to have. In Annabell Schäfer’s Langfuse classification experiment, a prompt assigned labels to research titles and abstracts. On fit cases, the optimizer could inspect mistakes and suggest an edit for a recurring confusion. On validation cases, the score determined whether to keep the edit. The test cases were withheld until search stopped. In the figure, follow the validation score to the frozen prompt. Seeing only a score still lets a developer or optimizer adapt to the cases behind it. Schäfer’s classification experiment

Which cases can influence the prompt?

A validation score has a route into the choice even when the optimizer cannot read the validation labels.

Fit groupsThe author can inspect expected and actual answers.
Candidate promptsInclude the original baseline.
Validation groups
Compare and selectUse the same validation tasks for the candidates.
Frozen promptSelected wording and ordered demonstrations.
Untouched test groupsWithheld throughout proposal and selection.
Final assessment
Fit errors guide candidate construction. Validation scores select a prompt, whose wording and examples are then frozen. Untouched task groups enter only at final assessment. Related instances of one task belong together before these paths separate.
Read the diagram as text
  • Fit groups. The author can inspect expected and actual answers.
  • Candidate prompts. Include the original baseline.
  • Validation groups.
  • Compare and select. Use the same validation tasks for the candidates.
  • Frozen prompt. Selected wording and ordered demonstrations.
  • Untouched test groups. Withheld throughout proposal and selection.
  • Final assessment.
  • Fit groups → Candidate prompts: Errors inform edits.
  • Candidate prompts → Compare and select: Prompts to score.
  • Validation groups → Compare and select: Cases and reference answers.
  • Compare and select → Frozen prompt: Scores determine choice.
  • Frozen prompt → Final assessment: Selected prompt.
  • Untouched test groups → Final assessment: Fresh tasks and references.

Split the data according to what will be new when the prompt is used. If one task was run several times, holding out a single run still lets the author inspect other attempts at the same task. To assess performance on new tasks, keep all runs of each task together before choosing demonstrations or drafting edits. If you suspect that repeated tasks inflated an earlier result, keep the validation cases fixed and compare prompts developed with and without related fit cases. Match the size and difficulty of the fit sets, the search budget and the scoring rule. Reserve whole unseen tasks for the final test. Assessment independence

Preserve the rendered prompt versions, the order of examples, model version, available facts, generation settings, answer extraction and per-case scores. Count labels consumed during development separately from demonstrations sent per call. Choosing a three-shot prompt can use hundreds of reviewed cases. If generation varies, repeat the matched runs. Repeats measure variability on the observed tasks, while independent task groups support conclusions about the next task. Sampling units and uncertainty

Improving reusable prompts

Revise through testable hypotheses

Read the actual request, reply and reference answer together before editing. A missing item description calls for an input repair. A mislabeled reference calls for review. If the facts and reference agree but the rule allows two interpretations, state which cases clearer wording should fix. In the Anthropic Prompt Doctor workshop, Jamie Neuwirth and Zack Witten collect cases in the evaluation interface as they revise a prompt. Keeping those inputs available makes it possible to see whether a revision changes the intended cases and whether it breaks others.

Suppose reviewers agree that an explicitly requested garment type is mandatory. To test how best to communicate that rule, compare these four requests on matched items. The table specifies an experiment, with no assumed winner.
InstructionWithout demonstrationsWith the same contrasting pairs
Judge relevance.BaselineCan the examples convey the priority?
Judge relevance. Require the specified garment type.Can the explicit rule settle the cases?Do examples still help once the rule is explicit?

Removing the examples and rerunning the task is an ablation. Read across the first row to see whether the examples help convey the garment rule. Read across the second to see whether they still help when the instruction states the rule explicitly. If two promising prompts differ in both wording and examples, test the two remaining combinations with the examples in the same positions. Score each combination on overall relevance and on any mandatory cases, since a gain on common items can hide failures on rare ones. A rule that resolves a shared ambiguity is easier to test than a growing list of exceptions that may conflict. Before opening the test set, freeze the chosen wording, ordered examples and settings, and save the results that justified the choice.

Search prompts automatically

Automatic prompt optimization searches for an instruction, an ordered set of demonstrations or a combination to reuse on future requests. Specify what the search may change and which model will execute the resulting prompts. In one search round, run the current instruction on fit cases and give the proposer its errors. Run the proposed replacement and the baseline on matched validation cases. The branch in the figure depends on those results. A plausible diagnosis earns a trial. A measured gain that meets the acceptance criteria earns a place in the next round. Generating several candidate answers for one user question instead spends computation on that instance.

The information available to the proposer determines what it can try. Automatic Prompt Engineer inferred instructions that could account for completed input–output pairs. OPRO showed the proposer earlier instructions alongside their measured scores. ProTeGi asked for a critique of errors on a small batch and generated edits addressing that critique, retaining several candidates for further search. Its textual gradient is feedback in words. AutoPrompt used numerical derivatives to suggest discrete token edits, while soft-prompt tuning fits numerical prompt values through training. For demonstration search, a simple baseline samples several sets from a reviewed pool and validates each set before selecting one.

One round of validation-guided search

The current instruction supplies both the failures that suggest an edit and the baseline against which to test it.

Current prompt
Inspect fit executionsCompare predictions with expected answers.
Propose an editAddress a recurring error.
Compare on validation casesUse the same task model and metric for both prompts.
Retain edited prompt
Retain current prompt
Run the current prompt on fit cases to obtain diagnostic feedback. Draft a candidate and compare it with the current prompt on matched validation cases. Carry the accepted prompt into another round only while budget remains. At the stopping condition, freeze the retained prompt for independent assessment.
Read the diagram as text
  • Current prompt.
  • Inspect fit executions. Compare predictions with expected answers.
  • Propose an edit. Address a recurring error.
  • Compare on validation cases. Use the same task model and metric for both prompts.
  • Retain edited prompt.
  • Retain current prompt.
  • Current prompt → Inspect fit executions: Instruction to execute.
  • Inspect fit executions → Propose an edit: Mistakes and diagnostics.
  • Propose an edit → Compare on validation cases: Edited instruction.
  • Current prompt → Compare on validation cases: Baseline instruction.
  • Compare on validation cases → Retain edited prompt: If validation improves.
  • Compare on validation cases → Retain current prompt: If validation ties or declines.

A rising score can conceal two different failures. In Automatic Prompt Engineer’s TruthfulQA experiment, optimizing truthfulness could favor uninformative refusals. Fresh questions scored for truthfulness alone can reward the same shortcut. The remedy is to assess informativeness too. A sound metric can still yield an overfit prompt when repeated selection favors wording that happens to suit a finite set of questions. That failure calls for independent assessment. Automatic Prompt Engineer

Bound the search by a target or a budget such as a maximum number of evaluation calls. Check candidate length against the serving allowance before accepting a winner. Proposal and evaluation calls are development expenses. Every additional token in the selected instruction recurs across the workload. A run that exhausts its budget can return the baseline if no deployable candidate qualifies.

Inspect DSPy's prompt artifacts

In DSPy, a signature declares a task’s input fields, output fields and intent. A module makes that task callable. An adapter renders the declaration and actual inputs into messages and parses the reply into fields. The caller can therefore keep the same interface while changing the instructions used to implement it. Consider the subject-to-haiku signature in the DSPy 3.4.0 first program.

Illustrative pseudocode

Python-like pseudocode
# With a DSPy model client configured
write_poem = dspy.Predict("subject -> haiku")
reply = write_poem(subject="rain")
print(reply.haiku)
dspy.inspect_history(n=1)

In the call history, look for the field descriptions, formatting instructions and subject value that the short declaration expands into. The caller reads the extracted text in reply.haiku. A four-line poem can occupy that field just as easily as a three-line one. If three lines are required, count them when scoring the unoptimized module. Field descriptions can communicate that requirement, and the metric determines whether a compiler gets credit for satisfying it. Kevin Madura’s call-history walkthrough

Two prompt-only compilers in DSPy 3.4.0 use labeled data in different ways. In both, train names data available to construct a prompt while the task model’s weights stay fixed.
OptimizerCandidate materialSelection
BootstrapFewShotSupplied labeled pairs and demonstrations generated by running the taskA metric checks generated predictions. Accepted examples populate the predictor up to configured limits.
MIPROv2Proposed instructions and bootstrapped demonstration setsBayesian optimization uses earlier scores to choose combinations to try. Trials can use small batches. Full-validation performance selects the returned program.

Inspect the selected program’s state-only JSON to see its instructions, demonstrations and signature metadata. To reproduce that behavior in a fresh process, instantiate the program, load the saved state and configure the model client separately. The restored program and its target model are the combination to assess on untouched subjects. DSPy saving and loading

Use diagnostic feedback with GEPA

GEPA, named for genetic Pareto, is a prompt optimizer that uses feedback about individual executions to propose edits. In DSPy’s published haiku example, the revised instruction specifies line and syllable counts, imagery and poem-only output. To see why detailed feedback matters, suppose a line-count check fails. The score alone cannot distinguish a four-line poem from a three-line poem preceded by a title. Seeing the title suggests asking for the poem alone. GEPA calls this use of an execution record (a trace) and feedback to propose a revision reflection. It tests the revision on a small group of cases (a minibatch) before spending more calls on a promising candidate. Published haiku edit · Lakshya Agrawal on reflective search

Illustrative scores for three prompt candidates. Each pass counts equally. Read down the last case before deciding which candidates are worth revising.
CandidateCase 1Case 2Case 3Mean
APassPassFail2/3
BFailFailPass1/3
CPassFailFail1/3

A dominates C because it matches or beats C on every case and improves on case 2. Only B handles case 3. GEPA’s Pareto selection keeps complementary case winners and samples parents according to their case-level strengths. B can contribute to another revision even though A leads on average. Merging candidates is optional and can introduce regressions. The original algorithm ultimately returns the highest mean on the selection set. If case 3 is mandatory, the deployment decision needs that additional check. Original GEPA algorithm, July 2025

The authors ran Qwen3-8B on held-out HotpotQA questions. Their exact-match scorer gives a case one point when the final answer matches a reference answer under DSPy’s matching rules and zero otherwise. The baseline passed 42.33% of cases, MIPROv2 passed 55.33%, and GEPA passed 62.33%. The two searches had matched caps on rollouts, meaning task executions. Actual usage differed, and GEPA received richer feedback and spent computation on reflection. Matching caps therefore leaves total cost open. DSPy 3.4.0 exposes a score-plus-text metric and an evaluation budget through its GEPA integration.

The quality of the diagnostic matters as much as its detail. In Mahmoud Mabrouk’s policy-judge experiment at Agenta, repeated runs of each task stayed on the same side of the development–validation split. That protected against repeated-task leakage. The reference judgments were still synthetic labels derived from assertions. Agreement with them measured agreement with that labeling procedure, leaving agreement with independent policy experts to be tested. Judge the Judge

Transfer and practical choices

Reassess transfer and sufficiency

Generalization concerns fresh cases. Transfer asks whether a result survives a change of condition, such as a different language, domain, input length or model version. A change in the kinds of cases encountered is a distribution shift. Replay the frozen prompt and its original baseline under the target condition before retuning. Their difference estimates the benefit that carried over. An improvement after editing on the new model answers a different question about how well the prompt can be adapted there. When deployment conditions shift

Choose a comparison that changes the suspected cause while preserving the rest of the task.
Observed difficultyComparisonWhat it tests
The request lacks a policy fact.Add the verified policy passage to the same question and instruction.Whether sufficient evidence helps. A gain directs attention to request assembly.
A valid rare category disappears.Inspect the saved instruction for exclusions. Repair the rule or demonstrated coverage and test fresh rare and common cases.Whether restoring the distinction repairs the omission without sacrificing existing behavior.
The selected prompt exceeds the serving allowance.Compare shorter sets of instructions or examples with room for the same required answer.Which distinctions survive at a deployable length.
The task still fails with sufficient facts and a clear convention.Check reviewer agreement. With reliable labeled data, compare a trained variant with the strongest deployable prompt on fresh cases.Whether parameter adaptation adds enough to justify maintaining the trained model.

Even within one domain, the better method can depend on the decision the application needs. In a March 2026 study, Smartling researchers compared expert-written zero-shot prompts, minimal DSPy signatures and GEPA-optimized signatures on linguistic work. Expert prompts retained advantages on some measures of detecting translation errors. Optimization helped on some measures of characterizing those errors. The methods had different access to expert effort and labeled data, and scoring was automatic. Choosing between them requires specifying which errors the workflow needs to find and what a missed error costs. Smartling comparison

Frequently asked questions

Is in-context learning training?

In ordinary in-context learning, the current input carries the examples and the model uses its existing weights. To repeat the demonstration condition in a fresh call, supply its examples again. An instruction can also state the same mapping. Fine-tuning requires loading the trained parameters. Soft-prompt tuning requires restoring the learned numerical prompt even when the base model is frozen.

How many demonstrations should I include?

Compare an instruction alone with a small set covering distinct decisions. Add a pair when it supplies a missing distinction, fits the rendered input budget and helps on fresh cases. Performance can level off or fall as the set grows, even on a task where the first examples helped.

Do roles, wording or example order reliably help?

A role can suggest a voice or perspective. A title such as expert supplies little guidance about a disputed decision. To test an accuracy claim, change the role, wording or ordering separately and compare on matched cases. An ordering selected on one model may rank poorly on another.

Does a written explanation verify the answer?

No. An explanation can be checked against evidence or an independent calculation. Its fluency alone establishes neither the conclusion nor what influenced the model. In controlled experiments, explanations sometimes omitted a cue that changed the prediction.

When is automatic optimization worth its cost?

For a recurring task, compare a bounded search with a handwritten baseline at the same required quality. Count expert and annotation time, proposal and evaluation calls, serving cost at expected volume, and the work of repeating the process after a model change. A higher validation score alone supplies no break-even estimate.

Does GEPA require DSPy?

No. GEPA can work with a standalone evaluator that executes a candidate and supplies diagnostic feedback. DSPy offers an integration for optimizing its programs and also supports other ways to select instructions or demonstrations.

When should I retest a saved prompt?

Retest after a change to the model, the rendered request or the workload. Run the saved prompt and its baseline on representative inputs under the new conditions. Preserve their results before making edits so that you can distinguish transfer from the benefit of retuning.

When should I supply context or try fine-tuning?

If a reviewer would need an absent fact, supply verified information and rerun the case. If the facts are available but the decision rule is unclear, clarify it or show contrasting examples. Fine-tuning is worth comparing when errors persist, reliable training pairs exist and independent cases can establish whether changing the model helps.

Open questions

  1. Which examples earn their space at a fixed token budget? Similar reviews can supply useful local context or repeat a distinction already covered. Comparing fixed, similarity-selected and coverage-selected sets on the same queries at matched rendered lengths would show which gains survive the actual serving constraint.

  2. When does automatic optimization repay its full cost relative to expert revision? Expert time and annotations enter at different stages, while a longer winning prompt incurs recurring expense. A comparison through one model or workload change, with both methods held to the same quality requirements, would give teams a basis for budgeting maintenance as well as launch.

  3. How can search improve a rare mandatory behavior when only a few independent examples exist? Repeatedly selecting on those examples weakens their value as evidence, yet an average dominated by common cases can reward losing the behavior altogether. Progress would mean satisfying both requirements on fresh task groups with a single deployable prompt.

Follow the curated reading path through the speakers and demonstrations behind this entry.

Explore more talks

The rest of the library, beyond the curated path. Cited talks support this entry; reviewed transcripts were processed in full. Metadata candidates have not been reviewed as sources or verified as topic members.

51 matching talks

TalkSpeakerEventYear
Jeronim MorinaAI Engineer World's Fair 20242024
Aparna DhinakaranAI Engineer Code 20252025
Christopher Harrison, John PeckAI Engineer World's Fair 20252025
Almog BakuAI Engineer Summit 20252025
Nick HeinerAI Engineer World's Fair 20262026
Dan ClearyAI Engineer Summit 20252025
Devendra Chaplot, Devendra Singh ChaplotAI Engineer World's Fair 20242024
Sander SchulhoffAI Engineer World's Fair 20252025
AI Engineering 101

Cited in this entry

Noah HeinAI Engineer Summit 20232023
Skills are the New SDKs

Transcript reviewed

Elvin AghammadzadaAI Engineer World's Fair 20262026
Ishan AnandAI Engineer World's Fair 20262026
Charles FryeAI Engineer Summit 20232023
Lukas BiewaldAI Engineer World's Fair 20242024
Prompt Engineering is Dead

Cited in this entry

Nir GazitAI Engineer World's Fair 20252025
Raahul Singh, Vanč LevstikAI Engineer World's Fair 20262026
Sharif ShameemAI Engineer World's Fair 20252025
Vayum Arora, Zhengyao Jiang, Dixing Xu, Dhruv SrikanthAI Engineer World's Fair 20262026
Nick Ung, Akshay SharmaAI Engineer World's Fair 20262026
Ayush BhardwajAI Engineer World's Fair 20262026
Maxime Rivest, Isaac MillerAI Engineer World's Fair 20262026
Lakshya AgrawalAI Engineer World's Fair 20262026
Pamela FoxAI Engineer World's Fair 20262026
Samuel ColvinAI Engineer Europe 20262026
Prompt Engineering Tactics

Metadata candidate

Dan ClearyAI Engineer Summit 20232023
Frank CoyleAI Engineer World's Fair 20262026
Shaan DesaiAI Engineer Summit 20252025
Vaibhav Page, Infant VasanthAI Engineer World's Fair 20252025
Shirsha ChaudhuriAI Engineer Summit 20252025
Mark HenningsAI Engineer Summit 20232023
Respect The Process

Metadata candidate

Andrew DumitAI Engineer World's Fair 20262026
Reid MayoAI Engineer Summit 20232023
Anita KirkovskaAI Engineer Summit 20252025
Sander DielemanAI Engineer Europe 20262026
Sandra KublikAI Engineer World's Fair 20242024
Louis-François Bouchard, Omar Solano, Samridhi VaidAI Engineer World's Fair 20262026
Abi AryanAI Engineer Summit 20232023
Rachel Lee Nabors (RL Nabors)AI Engineer World's Fair 20262026
Alex AtallahAI Engineer World's Fair 20252025
Preetika Bhateja, Daniel BumpAI Engineer World's Fair 20262026
Matt CareyAI Engineer Europe 20262026
Rémi LoufAI Engineer World's Fair 20242024
Aman KhanAI Engineer World's Fair 20252025
Daniel HanAI Engineer World's Fair 20262026
Manish SanwalAI Engineer Summit 20252025
Kobie CrawfordAI Engineer Europe 20262026
Ted JohnsonAI Engineer World's Fair 20262026
Rafal Wilinski, Vitor BaloccoAI Engineer World's Fair 20252025
Phil HetzelAI Engineer Europe 20262026
Nupur SharmaAI Engineer Europe 20262026
SallyAnn DeLuciaEvent unknownUnknown
Louis-François BouchardEvent unknownUnknown

References

Coverage and source review
Processed transcripts
28 processed in full · 5 in the curated path
Automated source review
Passed
Metadata candidates
28 unreviewed; not verified topic membership
Corpus version
c76f4395d1050f81b2bfd27df900224591ef94b026792d1b65896097daa9f77c

Automated review checks source support; it is not publication approval.

A synthesis of selected conference talks and technical references. Citations link to the source material; they do not imply that every talk on this subject is included.

  1. Writing Principles for Task-Tuned Prompt Engineering — Karina Nguyen

    https://www.youtube.com/watch?v=6d60zVdcCV4&t=310s

  2. Curation strategy 1: denim skirt example — Writing Principles for Task-Tuned Prompt Engineering

    https://ai.engineer/talks/6d60zVdcCV4-writing-principles-task-tuned-prompt-engineering?view=slides#transcript-6d60zVdcCV4-asset-19

  3. Language Models are Few-Shot Learners — Brown et al.

    https://arxiv.org/abs/2005.14165

  4. Language Models are Few-Shot Learners

    https://arxiv.org/html/2005.14165v4

  5. DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines

    https://arxiv.org/html/2310.03714v1

  6. MIPROv2 — DSPy 3.4.0

    https://dspy.ai/3.4.0/api/optimizers/MIPROv2/

  7. Optimization in DSPy

    https://raw.githubusercontent.com/stanfordnlp/dspy/main/docs/docs/learn/optimization/overview.md

  8. Language Models are Unsupervised Multitask Learners

    https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf

  9. DSPy: The End of Prompt Engineering

    https://www.youtube.com/watch?v=-cKUW6n8hBU&t=757

  10. Effective context engineering for AI agents

    https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents

  11. Writing Principles for Task-Tuned Prompt Engineering

    https://www.youtube.com/watch?v=6d60zVdcCV4&t=885

  12. Writing Principles for Task-Tuned Prompt Engineering

    https://www.youtube.com/watch?v=6d60zVdcCV4&t=2469

  13. Prompting best practices — Claude Platform Docs

    https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/claude-prompting-best-practices

  14. Writing Principles for Task-Tuned Prompt Engineering

    https://www.youtube.com/watch?v=6d60zVdcCV4&t=3069

  15. Transformers: model-specific chat serialization

    https://huggingface.co/docs/transformers/en/chat_templating

  16. Writing Principles for Task-Tuned Prompt Engineering

    https://www.youtube.com/watch?v=6d60zVdcCV4&t=1831

  17. Introducing Structured Outputs in the API

    https://openai.com/index/introducing-structured-outputs-in-the-api/

  18. Writing Principles for Task-Tuned Prompt Engineering

    https://www.youtube.com/watch?v=6d60zVdcCV4&t=722

  19. Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection

    https://arxiv.org/pdf/2302.12173

  20. Text-generation-launcher arguments

    https://huggingface.co/docs/text-generation-inference/basic_tutorials/launcher

  21. Conversation state: managing the context window — OpenAI

    https://developers.openai.com/api/docs/guides/conversation-state

  22. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models

    https://proceedings.neurips.cc/paper/2022/file/9d5609613524ecf4f15af0f7b31abca4-Paper-Conference.pdf

  23. Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting

    https://arxiv.org/pdf/2305.04388

  24. Optimizing LLMs for Speed and Memory

    https://huggingface.co/docs/transformers/main/en/llm_tutorial_optimization

  25. Larger language models do in-context learning differently

    https://arxiv.org/html/2303.03846v2

  26. Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?

    https://arxiv.org/pdf/2202.12837

  27. Prompt Engineering & AI Red Teaming

    https://www.youtube.com/watch?v=_BRhRh7mOX0&t=3133

  28. Learning to Learn Using Gradient Descent

    https://gwern.net/doc/reinforcement-learning/meta-learning/2001-hochreiter.pdf

  29. Language Models are Unsupervised Multitask Learners

    https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf

  30. Language Models as Knowledge Bases?

    https://aclanthology.org/D19-1250.pdf

  31. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

    https://www.jmlr.org/papers/volume21/20-074/20-074.pdf

  32. Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference

    https://aclanthology.org/2021.eacl-main.20.pdf

  33. Language Models are Few-Shot Learners

    https://arxiv.org/html/2005.14165v4

  34. Finetuned Language Models Are Zero-Shot Learners

    https://arxiv.org/pdf/2109.01652

  35. Multitask Prompted Training Enables Zero-Shot Task Generalization

    https://arxiv.org/pdf/2110.08207

  36. AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts

    https://aclanthology.org/2020.emnlp-main.346.pdf

  37. The Power of Scale for Parameter-Efficient Prompt Tuning

    https://arxiv.org/pdf/2104.08691

  38. Large Language Models are Human-Level Prompt Engineers

    https://arxiv.org/html/2211.01910

  39. Writing Principles for Task-Tuned Prompt Engineering

    https://www.youtube.com/watch?v=6d60zVdcCV4&t=2288

  40. Writing Principles for Task-Tuned Prompt Engineering

    https://www.youtube.com/watch?v=6d60zVdcCV4&t=2520

  41. Coverage-based Example Selection for In-Context Learning

    https://aclanthology.org/2023.findings-emnlp.930.pdf

  42. The LLM Triangle: Engineering Principles for Robust AI Applications

    https://www.youtube.com/watch?v=ro5HkZvzfiQ&t=1479

  43. What Makes Good In-Context Examples for GPT-3?

    https://aclanthology.org/2022.deelio-1.10.pdf

  44. Many-Shot In-Context Learning

    https://arxiv.org/html/2404.11018v2

  45. Calibrate Before Use: Improving Few-Shot Performance of Language Models

    https://proceedings.mlr.press/v139/zhao21c/zhao21c.pdf

  46. Optimizing LLMs in Insurance with DSPy: Beyond Manual Tuning

    https://www.youtube.com/watch?v=IAdZxqjZ45U&t=648

  47. Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity

    https://aclanthology.org/2022.acl-long.556.pdf

  48. Quantifying Language Models’ Sensitivity to Spurious Features in Prompt Design

    https://arxiv.org/html/2310.11324v2

  49. Persona Engineering: A Field Guide to AI Synthetic Personas — Ishan Anand, InsightSciences.ai

    https://www.youtube.com/watch?v=YnNF55QV0zs&t=463

  50. DSPy: The End of Prompt Engineering

    https://www.youtube.com/watch?v=-cKUW6n8hBU&t=2042

  51. Prompt Engineering & AI Red Teaming

    https://www.youtube.com/watch?v=_BRhRh7mOX0&t=4223

  52. Optimizing LLMs in Insurance with DSPy: Beyond Manual Tuning

    https://www.youtube.com/watch?v=IAdZxqjZ45U&t=1016

  53. Stop Burning Tokens: Why self-improvement needs domain expertise first - Annabell Schäfer, Langfuse

    https://www.youtube.com/watch?v=eAXxdtNlK04&t=296

  54. Prompt Engineering is Dead

    https://www.youtube.com/watch?v=jvKf6zXrNO4&t=766

  55. Judge the Judge: Building LLM Evaluators That Actually Work with GEPA — Mahmoud Mabrouk, Agenta AI

    https://www.youtube.com/watch?v=X4dEHRzBLmc&t=846

  56. NIST randomized blocks: applying controlled comparisons to RAG evidence

    https://www.itl.nist.gov/div898/handbook/pri/section3/pri332.htm

  57. True Few-Shot Learning with Language Models

    https://proceedings.neurips.cc/paper/2021/file/5c04925674920eb58467fb52ce4ef728-Paper.pdf

  58. OpenTelemetry GenAI semantic conventions: spans

    https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-spans.md

  59. Building with Anthropic's Claude - The Prompt Doctor is In — Jamie Neuwirth, Zack Witten

    https://www.youtube.com/watch?v=hkhDdcM5V94&t=512s

  60. Optimizing LLMs in Insurance with DSPy: Beyond Manual Tuning

    https://www.youtube.com/watch?v=IAdZxqjZ45U&t=494

  61. Writing Principles for Task-Tuned Prompt Engineering

    https://www.youtube.com/watch?v=6d60zVdcCV4&t=160

  62. Teach Better or Show Smarter? On Instructions and Exemplars in Automatic Prompt Optimization

    https://arxiv.org/pdf/2406.15708

  63. Playground in Prod - Optimising Agents in Production Environments

    https://www.youtube.com/watch?v=A48uhxfxbsM&t=3054

  64. Writing Principles for Task-Tuned Prompt Engineering

    https://www.youtube.com/watch?v=6d60zVdcCV4&t=2205

  65. Large Language Models as Optimizers

    https://arxiv.org/html/2309.03409v1

  66. Automatic Prompt Optimization with “Gradient Descent” and Beam Search

    https://aclanthology.org/2023.emnlp-main.494.pdf

  67. Stop Burning Tokens: Why self-improvement needs domain expertise first - Annabell Schäfer, Langfuse

    https://www.youtube.com/watch?v=eAXxdtNlK04&t=394

  68. DSPy: The End of Prompt Engineering

    https://www.youtube.com/watch?v=-cKUW6n8hBU&t=4287

  69. Playground in Prod - Optimising Agents in Production Environments

    https://www.youtube.com/watch?v=A48uhxfxbsM&t=2655

  70. DSPy: The End of Prompt Engineering

    https://www.youtube.com/watch?v=-cKUW6n8hBU&t=320

  71. Your first program — DSPy 3.4.0

    https://dspy.ai/3.4.0/getting-started/first-program/

  72. DSPy: The End of Prompt Engineering

    https://www.youtube.com/watch?v=-cKUW6n8hBU&t=2369

  73. DSPy: The End of Prompt Engineering

    https://www.youtube.com/watch?v=-cKUW6n8hBU&t=640

  74. BootstrapFewShot — DSPy 3.4.0

    https://dspy.ai/3.4.0/api/optimizers/BootstrapFewShot/

  75. Saving and loading — DSPy 3.4.0

    https://dspy.ai/3.4.0/getting-started/saving-and-loading/

  76. DSPy: The End of Prompt Engineering

    https://www.youtube.com/watch?v=-cKUW6n8hBU&t=3820

  77. Playground in Prod - Optimising Agents in Production Environments

    https://www.youtube.com/watch?v=A48uhxfxbsM&t=125

  78. GEPA optimization — DSPy 3.4.0

    https://dspy.ai/3.4.0/getting-started/gepa-optimization/

  79. Beating RL With Reflection: GEPA and Optimize Anything — Lakshya A. Agrawal, GEPA

    https://www.youtube.com/watch?v=OA-Mc60Rboo&t=192

  80. Judge the Judge: Building LLM Evaluators That Actually Work with GEPA — Mahmoud Mabrouk, Agenta AI

    https://www.youtube.com/watch?v=X4dEHRzBLmc&t=981

  81. GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning

    https://arxiv.org/html/2507.19457v1

  82. GEPA Overview — DSPy 3.4.0

    https://dspy.ai/3.4.0/api/optimizers/GEPA/overview/

  83. HotpotQA scoring function — GEPA paper reproduction artifact

    https://github.com/gepa-ai/gepa-artifact/blob/cbefbc1aa0f43dd39874ec4bf42211365dbda42e/gepa_artifact/benchmarks/hotpotQA/__init__.py

  84. DSPy: The End of Prompt Engineering

    https://www.youtube.com/watch?v=-cKUW6n8hBU&t=1529

  85. Prompt Engineering & AI Red Teaming

    https://www.youtube.com/watch?v=_BRhRh7mOX0&t=2660

  86. Trading Desks to Clinical Trials: Parallels in Applied Vertical AI

    https://www.youtube.com/watch?v=Yphdry8ttAQ&t=894

  87. To Write or to Automate Linguistic Prompts, That Is the Question

    https://arxiv.org/html/2603.25169v1

  88. Prompt Engineering & AI Red Teaming

    https://www.youtube.com/watch?v=_BRhRh7mOX0&t=1285

  89. Prompt Engineering is Dead

    https://www.youtube.com/watch?v=jvKf6zXrNO4&t=792

  90. Judge the Judge: Building LLM Evaluators That Actually Work with GEPA — Mahmoud Mabrouk, Agenta AI

    https://www.youtube.com/watch?v=X4dEHRzBLmc&t=1223

  91. DSPy: The End of Prompt Engineering

    https://www.youtube.com/watch?v=-cKUW6n8hBU&t=1197

  92. The Model Swap Workshop: GPT-5.5 vs Kimi vs DeepSeek vs Mistral — Pamela Fox, Microsoft

    https://www.youtube.com/watch?v=UGjk3Uk0MxM&t=1861

  93. Productionizing GenAI Models – Lessons from the world's best AI teams

    https://www.youtube.com/watch?v=uiq95JYpBGY&t=976

  94. AI Engineering 101

    https://www.youtube.com/watch?v=C0ZUdFg-iTo&t=7558

  95. Build Evals That Actually Matter - Nick Ung & Akshay Sharma, Lyft

    https://www.youtube.com/watch?v=3z2uT5aDx_Y&t=1767

  96. Judge the Judge: Building LLM Evaluators That Actually Work with GEPA — Mahmoud Mabrouk, Agenta AI

    https://www.youtube.com/watch?v=X4dEHRzBLmc&t=1633