AI Engineer Summit 2023
Prompt Engineering Tactics
Loading slides…
Read the talk
Prompt Engineering Tactics
Small prompt changes can alter an AI product’s behavior. Task-specific personas, named sources, and emotional cues offer three ways to shape the response.
From a talk by Dan Cleary
Why not just ask for what you want?
Why engineer a prompt if you can simply tell a model what you want and usually get something useful back? That is the starting question for Dan Cleary, co-founder of PromptHub, a tool for teams to test, collaborate on, and deploy prompts. Getting a good answer once is useful; getting dependable behavior inside a product is a harder requirement.
Model outputs are nondeterministic, and small changes to a prompt can have outsized effects on the response. An answer that goes off the rails can damage trust in the product delivering it. Cleary illustrates the stakes with a slide pairing poor prompts with a headline about Replika chatbot behavior. Prompt quality becomes part of product quality when users encounter the output as a feature rather than an experiment.
Access to ChatGPT has also raised expectations: users want crisp answers that match their request, arrive quickly, and do not hallucinate. Cleary offers three tactics for improving responses, whether the prompt lives in an everyday chat or an AI product feature.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Let task-specific personas collaborate
Multi-persona prompting asks the model to approach a task through several relevant roles. Cleary attributes the method to University of Illinois research; the matching work, Solo Performance Prompting, uses a single language model to simulate collaborating personas. The agents here are perspectives generated within the model’s response, not necessarily separate running systems.
For a book-writing task, Cleary suggests an author, a publicist, and the intended audience. The model leads a brainstorming exchange among them, allowing their different priorities to influence the result. The author can develop the material, the publicist can consider its appeal, and the audience can respond to what would make it worth reading. A prompt can make that sequence explicit:
Help me develop a book idea.
Bring together three personas: an author, a publicist,
and a representative of the book's intended audience.
Lead a brainstorming discussion. Have the personas propose
ideas, respond to one another, and revise their suggestions.
Show the collaboration, then present a final recommendation.
The important operation is the exchange: personas respond to one another before the model produces its final answer. Merely listing three roles would leave that collaboration unspecified.
The visible discussion lets the reader inspect how the proposed answer developed. Cleary recommends the technique for complex tasks or tasks requiring additional logic, and personally favors it for generative work. In the book example, its value is in developing and revising ideas through several perspectives.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Name the source you want the answer to follow
The according-to method changes a smaller part of the prompt. Start with Cleary’s digestive-system question, then append a source:
In what part of the digestive tube do you expect starch
to be digested? According to Wikipedia.
The suffix directs the model toward a particular source’s account. In the matching original according-to study, the starch example concerns initial digestion, and both the baseline and grounded answers identify the mouth. The example demonstrates more source-overlapping detail, rather than correcting an initially false answer.
Naming a source does not fetch it. Although Cleary describes the model going to the source to retrieve information, the studied mechanism relies on recall from training data, without internet access. A source cue can influence the answer, but it does not establish that the model consulted or verified a live page.
Cleary reports that according-to prompting can reduce hallucinations by up to 20%. The original study does not establish that general hallucination-rate reduction: it measures quotation overlap with Wikipedia and, separately, question-answering performance. More overlap with a source is not itself proof that an answer is correct.
Cleary recommends this tactic when a fine-tuned or general model should consistently draw answers from a particular source. He attributes the research to Johns Hopkins University and introduces it as recent work. The useful design choice is to make the desired source explicit instead of leaving the model to infer it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Add an emotional cue
EmotionPrompt, which Cleary attributes to Microsoft and several universities, tests emotional stimuli appended to ordinary prompts. His analogy is a boss explaining that a project matters to your career or to a major client: the stakes may lead you to take the work more seriously. That human behavior motivates the prompting technique; the method tests whether similar language changes model responses.
The implementation is simply to preserve the task and add the cue at the end. The slide shows the addition “This is very important to my career.” For a writing request, that pattern could look like this:
Revise my cover letter for clarity and relevance to the job.
Preserve the facts in the draft.
This is very important to my career.
The task still specifies what to do. The emotional sentence supplies an additional signal about its importance.
Cleary says he has seen better outputs for tasks ranging from cover letters to change logs. Those are his practical observations. The September 2023 EmotionPrompt study reports relative benchmark improvements of 8% on Instruction Induction and 115% on BIG-Bench. Its expanded evaluation covers 24 Instruction Induction tasks and 21 BIG-Bench tasks across six models, with zero-shot BIG-Bench experiments. The study distinguishes original and APE prompt baselines, as well as average and best emotional-stimulus results; the headline percentages are not measured gains for the cover-letter or changelog examples. The slide pairs the prompt modification with an Original versus Ours results table across six models.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Try the prompts with your team
These methods fit both direct use in ChatGPT and prompts embedded in product features. Cleary closes by offering all three as PromptHub templates: copy a prompt, run it through the playground, and share it with teammates or access it through a link. The public multi-persona collaboration template provides a starting point for the persona-selection and iterative-discussion pattern. The practical next step is to try a tactic on the task you actually need the model to perform.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Research on a single language model identifying and simulating multiple personas to collaborate on a task.
A copyable PromptHub prompt with examples of persona selection, feedback and iterative revision.
The original study of source-directed prompting, Wikipedia quotation overlap and question-answering performance.
Emotional prompt suffixes evaluated across instruction-following benchmarks and a separate human-rated generation study.
A platform for managing, testing, sharing and deploying prompts with a team.
Further reading
An accessible explanation of how source cues encourage recall from training data without live internet access.
Full transcript
Loading transcript…