AI Engineer Summit 2025

The Model Isn’t Wrong—You’re Just Bad at Prompting

Loading slides…

Read the talk

The Model Isn’t Wrong—You’re Just Bad at Prompting

Dan Cleary explains how instructions, examples and model-generated feedback improve prompts—and why early reasoning models change which techniques help.

From a talk by Dan Cleary

Before you start: Basic familiarity with LLM prompts is sufficient; no coding or model-training background is required.

What do you actually want the model to do?

Why does an LLM feature need prompt engineering if you can simply tell the model what to do? Because defining the desired behavior is already difficult. A request can sound clear to its author while leaving the model to guess what a successful answer looks like. That gap becomes consequential when the output is part of a product.

Prompting is a starting point within a larger system. Teams can access the same models, but their instructions, evaluation process and surrounding architecture differ. The opening diagram puts prompt engineering before testing and evaluation, with RAG and fine-tuning as further options. The useful question is whether better instructions can solve the problem before additional infrastructure becomes necessary.

Slide titled “An easy starting point” shows Prompt Engineering leading to Test and Evaluate, then to stacked RAG and Fine-tuning boxes.Open full source frame
Prompt engineering as a starting point: test and evaluate before considering RAG or fine-tuning.

Cleary connects this approach to Anthropic’s advice to seek the simplest solution when building agents. His cautionary example is spending an hour adjusting a prompt, declaring prompting insufficient, and moving immediately to a more complicated retrieval-augmented generation system. If prompting can meet the requirement, it leaves less machinery to manage; it deserves a serious iteration cycle before being ruled out.

Two techniques provide the starting toolkit: chain-of-thought prompting, which elicits intermediate reasoning, and few-shot prompting, which demonstrates the desired behavior through examples. These are selected methods rather than a complete taxonomy. Cleary points to additional reasoning templates in PromptHub for extending that toolkit.

0:450:58
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:45 · section reference included

Give the model room for intermediate work

Chain-of-thought prompting asks the model to work through a problem before producing its answer. Decomposing a task into smaller problems gives it intermediate work to build on, and the generated explanation can provide clues when troubleshooting an incorrect result. Cleary introduces this as a broadly applicable technique, then immediately qualifies it: reasoning models already incorporate reasoning behavior, so they do not necessarily need the same prompting intervention.

There are two straightforward ways to elicit that intermediate work:

ApproachWhat the prompt supplies
Zero-shot instructionA request to reason before answering
Few-shot demonstrationA worked example showing the desired solution steps

The zero-shot version adds a cue such as “Take a breath and think it through.” The demonstration version supplies another problem and its worked solution before the new question. For a math task, the example communicates the steps to follow as well as the final-answer format.

2:262:31
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:26 · section reference included

Use models to generate reasoning examples

Writing every worked example by hand is not the only option. Automatic Chain of Thought Prompting generates reasoning demonstrations with a model; Cleary introduces it as a more involved framework. He then presents AutoReason through a single prompt that accepts a task or question and generates a reasoning chain, using embedded reasoning examples. That offered prompt is the rationale-generation part of the workflow: the complete AutoReason method subsequently generates an answer from the question and rationale.

The same idea—eliciting intermediate reasoning in a structured format—also appears in model training. Cleary points to the DeepSeek training template, which separates thinking from the answer using think tags. The DeepSeek report labels that particular template DeepSeek-R1-Zero, rather than DeepSeek-R1. R1 uses a separate multistage training pipeline that includes cold-start supervised data and reinforcement learning; the template is a format for eliciting reasoning, not the whole training method.

For prompt experimentation, Cleary describes a shorter path: enter a task into one of PromptHub’s templates, generate a reasoning chain, and copy it for reuse. The collection he presents includes other reasoning and verification methods as well as chain-of-thought prompts, particularly for complex problems. These templates were offered as free resources in the talk.

3:303:39
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:30 · section reference included

Show the client’s style through brief–content pairs

Few-shot prompting turns examples into part of the specification. Cleary’s concrete task is generating content for a client. Instead of trying to describe the client’s tone exhaustively, the prompt supplies a brief and the finished content associated with it, repeats that pairing, then supplies a new brief whose content the model should produce. The examples demonstrate what a satisfactory transformation looks like.

The demonstrated structure can be used directly as a prompt template:

Write content for {client_name} using the examples below
as guidance for tone and style.

Brief: {example_brief_1}
Content: {approved_content_1}

Brief: {example_brief_2}
Content: {approved_content_2}

Brief: {new_brief}
Content:

Each completed pair connects an input to an acceptable output. The final empty Content field asks the model to apply that pattern to a new input. This communicates style through actual copy rather than adjectives such as friendly or professional.

Slide titled “Few-shot prompting example” shows client placeholders, two example brief–content pairs, and a final brief followed by Content. The presenter bubble sits toward the lower center.Open full source frame
A few-shot prompt supplies two brief–content examples before a new brief with an empty Content field.

Cleary recommends starting with one or two diverse examples, describing diminishing returns and possible degradation as more examples are added. That is a starting heuristic, not a measured optimum for this client-content task. Choose examples that cover different inputs the model is likely to encounter; duplicating nearly identical examples teaches less about the range of acceptable behavior.

4:364:48
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:36 · section reference included

Let an LLM help revise the prompt

Meta-prompting uses an LLM to create, refine or improve another prompt. It applies the same model assistance used elsewhere in development to the instructions themselves. The available approaches range from frameworks requiring code to user-facing prompt generators.

Cleary names Anthropic’s prompt-generation tool, OpenAI’s Playground and PromptHub’s enhancers. In the PromptHub workflow he describes, choosing a model provider selects a different underlying meta-prompt. The motivation is practical: instructions that work well with one provider’s models may need a different form for another provider’s models.

Prompt creation can then become a feedback loop:

  1. Run the current prompt.
  2. Inspect the output and give feedback about what should change.
  3. Use the feedback to revise the prompt and run it again.

Cleary describes PromptHub’s copilot as using ideas similar to TextGrad, where textual feedback guides iterative improvement. The relationship he claims is similarity of approach, not that the product necessarily uses the TextGrad package.

5:425:50
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:42 · section reference included

Reasoning models change what belongs in the prompt

The earlier techniques need another pass when the model has been trained to reason. Cleary turns to Microsoft’s From Medprompt to o1, first submitted in November 2024, and its experiments with o1-preview. Cleary reports that adding examples reduced performance in the Medprompt comparison he discusses. The result concerns that experimental setting, rather than establishing that demonstrations harm every reasoning model.

DeepSeek’s report likewise says few-shot prompting degraded performance for the evaluated R1 model and recommends a zero-shot problem description with an output format. Cleary also recalls OpenAI’s early o1-preview warning about excessive context. The historical guidance quoted in his companion discussion concerns keeping RAG context relevant: extra material can complicate the task rather than help resolve it.

Additional reasoning is a different intervention from additional examples. Cleary reports better results from an extended-reasoning prompt than from a quick-response prompt in the Medprompt work. The distinction is the amount of problem-solving effort encouraged at inference time, not the number of demonstrations placed in the input.

Cleary also describes longer reasoning traces alongside improved accuracy during DeepSeek training. This is a training observation, not a guarantee that making any answer longer improves it. No numerical effect sizes accompany these comparisons in the talk.

For the early reasoning models discussed here, start with a clear task and minimal prompting. If the output falls short, encouraging more reasoning may help, but that does not require scripting the model’s reasoning procedure. Cleary advises avoiding few-shot prompting by default in this setting; if examples are necessary, begin with one or at most two. He distinguishes asking for more reasoning from prescribing how to reason, which can interfere with behavior the model already learned.

The talk closes by pointing readers toward reusable material: the Prompt Engineering Substack, PromptHub’s blog, and community prompts contributed by the team and other users. Those resources extend the iteration process—from a clear task, to a tested prompt, to a targeted change when the output reveals what is missing.

6:476:58
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:47 · section reference included

Resources

From the talk

Transcript download
Full transcript

Loading transcript…