AI Engineer World's Fair 2024
Building with Anthropic's Claude - The Prompt Doctor is In
Loading slides…
Read the talk
Building with Anthropic's Claude - The Prompt Doctor is In
A live workshop moves from an overly talkative patient agent to JSON extraction, writing examples, persona control, image inputs, translation grading, and grounded summaries.
From a talk by Jamie Neuwirth and Zack Witten
Before you start: Basic familiarity with chat-model prompts, JSON, and API message roles will help with the structured-output examples.
What should a conversational agent say?
What should a medication-review agent say when someone asks who it is—and how do you turn that expectation into a prompt you can test? That is the first practical problem in Zack Witten’s live prompting workshop. The setting is the June 2024 AI Engineer World’s Fair, where Jamie Neuwirth opens by describing Anthropic’s support for startups: higher rate limits, access to prompting expertise, and early access to products.
The product backdrop is the newly introduced Claude 3.5 Sonnet, the Artifacts preview, and Claude Teams and Projects. Artifacts turns generated code and other content into something people can inspect and share: examples include games, diagrams, and illustrations of gene-sequencing discussions. Neuwirth also points builders toward Claude Cookbooks, the startup sales team, and Alex from developer relations.
Witten’s working material comes from the audience, through the AI Engineer Slack channel prompt-eng-live-workshop-anthropic. A useful submission contains a prompt template, variable slots, and concrete inputs that produce unwanted answers. The Console uses double-brace variables such as {{DOCUMENT}}. The submitter must also explain what counts as a good answer; otherwise, an apparent failure may simply reflect an unstated requirement. Most iteration happens in the Anthropic Console, with Claude for Sheets mentioned as another way to call Claude across spreadsheet cells.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Establish a baseline before cleaning up the prompt
Gordie’s medication-review agent provides the first case. Witten initially starts fixing capitalization and grouping information into XML tags. He describes the benefits of capitalization and grammar cleanup as largely anecdotal in his own work, while mentioning unspecified quantitative work on typos. His initial reason for choosing XML is Claude’s familiarity with it from training. Before these edits go further, an attendee asks for a before-and-after comparison. Witten undoes the changes and runs the original prompt.
The first input asks who the agent is. Its answer is much too long for the intended conversational interface; Gordie initially asks for no more than one sentence. Witten adds that case and two more to the Console’s Evaluate tab, then uses Run remaining to establish the baseline.
| Test input | Baseline problem |
|---|---|
| Who are you? | Too much explanation |
| Why do I need to do this appointment? | Another long answer |
| Can I schedule it tomorrow instead? | Disclaims scheduling information |
Gordie says the service has 24-hour availability. At this point, brevity and scheduling behavior look like two separate things to fix.
The first revision separates information from instructions, repairs numbering, and adds consistent line breaks. Asked whether Claude could perform the capitalization cleanup itself, Witten notes that a rewrite might also change words that should remain untouched. Asking Claude to generate capitalization code would make the transformation more explicit, but sentence boundaries still have edge cases such as ellipses. An IDE assistant is useful for a narrower job: completing nested XML closing tags while the author controls the content.
The purpose of the tags is semantic separation: this is background information, these are instructions, and this is the user’s message. XML is a convenient notation, not the essential ingredient. Witten subsequently separates the user input too and selects Rerun all. The answers remain long, and the scheduling disclaimer remains. Better boundaries make the prompt easier to interpret, but they have not supplied a concrete length target or the missing scheduling fact.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Replace vague requirements with observable behavior
Rule seven already asks the model to be concise and offer only relevant information. The problem is that concise has no fixed size: it might mean one word in one task and a page in a book review. Witten first tries a range of two to four sentences. Prompt length and tone can also provide cues, but a long prompt does not inherently require a long answer.
Two to four sentences still looks too long. He tightens the instruction to one or two sentences, never more than three. The revised patient-agent outputs contain two, two, and three sentences. That is shorter while retaining some flexibility for questions that need more explanation; it is not the strict one-sentence maximum initially requested.
The scheduling case then changes meaning. The submitter initially suggests that the agent should ask what time tomorrow would work, but explains that the 24-hour service information was deliberately omitted. With that fact absent, acknowledging uncertainty is the desired behavior. Define success against the information the model actually receives. A prompt cannot be expected both to avoid unsupported claims and to confidently use a business fact withheld from its context.
For component order, Witten recommends putting information above instructions. He reports that instructions nearer the bottom tend to be followed more closely, while explicitly recommending testing that ordering on the actual task. The same empirical attitude applies to emphasis: capitals, exclamation marks, and statements of importance can change behavior, but he does not know their relative strength compared with numbered rules. An audience question about punctuation in tokenizer code does not produce a tokenizer-level explanation; his evidence here is observed differences in responses.
For multilingual tasks, the next question is whether instructions should be in English or the target language. Witten prefers a well-written target-language prompt when the author knows that language. If the choice is between clear English and a poor prompt in an unfamiliar language, he prefers the language the author understands. His ideal workflow is to explain the use case to a native speaker and have that person write the prompt—even when the target language may be less familiar to Claude.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Reproduce the JSON failure—and notice truncation
The next prompt belongs to a unit-test reviewer in a multi-agent workflow. A writer agent creates tests; the reviewer is supposed to improve them and return JSON describing file updates and modifications. Despite repeated JSON-only instructions, the reviewer sometimes adds other text.
Reproducing this requires more than copying the reviewer’s instructions. In the original application, the tests are earlier in the conversation, and the project path identifies a local directory the application can access. Witten makes a standalone Console reproduction by explicitly supplying the earlier tests and converting template variables to double braces. Merely placing a directory path in this reproduction does not recreate the original application’s file access.
There is also a model difference: the submitter’s earlier tests mostly used Claude 3, while the live reproduction initially uses Claude 3.5. Witten normally chooses temperature zero for knowledge work. Asked about 0.01, he tentatively expects zero to produce marginally fewer hallucinations, but this session does not establish that comparison. The response appears to begin as JSON—then runs into the maximum output-token allowance before finishing. He switches to Claude 3 Haiku for faster iteration and raises the allowance. A response can begin in the right format and still fail because generation stops too early.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Control where generation starts and stops
Witten’s next tool is assistant prefill: supply the beginning of an assistant message, and the model continues as though it has already said those words. To expose the unwanted behavior, he removes the repeated JSON-only instructions. The model produces the familiar preamble before its data. Moving that preamble into the supplied assistant prefix lets the generated continuation get on with the JSON. His analogy is a child determined to do something: instead of repeatedly forbidding the preamble, let it have already happened.
Prefilling the opening JSON delimiter goes a step further: the response is already inside the data structure. The generated continuation will not repeat that supplied delimiter, so the application must restore it before calling json.loads. A second approach asks for JSON inside <json> tags and extracts only the enclosed text. Combining the approaches gives a supplied preamble, opening tag, and opening JSON delimiter; the continuation completes the JSON and closes the tag.
For an object-valued response, the corresponding Python extraction logic can retain the prefix explicitly. Here, continuation is the generated text after the supplied prefix:
python
import json
import re
PREFILL = 'Here is the updated JSON:\n<json>{'
def parse_review(continuation: str) -> dict:
full_text = PREFILL + continuation
match = re.search(r'<json>(.*?)</json>', full_text, re.DOTALL)
if match is None:
raise ValueError('Missing complete JSON block')
review = json.loads(match.group(1))
if not isinstance(review, dict):
raise ValueError('Expected a JSON object')
return review
The prefix belongs to the assembled text, while only the content inside the tags reaches the parser. Parsing rejects invalid JSON; it does not establish that the proposed test changes are correct. This example follows the workshop’s returned-closing-tag approach.
The combined live response still adds an explanation after its closing tag. Rather than make the prompt increasingly insistent about stopping, Witten proposes putting </json> in the API’s stop_sequences. That control was not exposed in the Console he was demonstrating, so he explains it rather than running it there. The purpose is to prevent generation of the unwanted epilogue, avoiding the associated output and work. Use code and API controls for boundaries that do not require judgment.
In the historical API pattern, the final entry in messages has role set to assistant and contains the partial response. This is model-specific: current API documentation says Claude 4.6 and later reject last-assistant-message prefilling. Current structured outputs offer schema-constrained JSON through output_config.format, are incompatible with message prefilling, and still require handling refusals and token-limit truncation. For either implementation, inspect the generation stop reason before deciding how to parse the response; the extraction code above specifically expects a returned closing tag.
Putting the same prefix at the end of the user’s instructions is not equivalent to supplying an assistant continuation. In the live experiment, that version introduces an extra opening delimiter. An attendee’s simpler JSON: suffix works in another attempt. Witten acknowledges multiple workable approaches, but continues to recommend prefill and tagged extraction for the models used in the workshop.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Check format in code; reduce oversized inputs
A successful example is only the beginning of evaluation. Testing at production scale requires a collection of cases. Asked whether an LLM should grade format compliance, Witten recommends ordinary code whenever the condition is mechanically checkable. JSON parsing is a direct test; conversational usefulness may instead need a model grader or a human. He briefly considers showing a test-generation feature, but no such demonstration follows.
For a summary surrounded by chatty introductory or closing text, tags can make the desired region extractable. Request <summary>...</summary>, extract its content, and use the closing delimiter as a stop sequence where supported. Detecting conversational clutter is a task an LLM could grade, but explicit boundaries may remove the need to judge that clutter in the first place.
Before the next case, Witten declines an audience request to test a potential attack vector; no jailbreak experiment follows. The workshop instead turns to a poorly formatted Excel/CSV file. Its submitter describes multiple clusters or datasets on one sheet, with extraction that invents or skips information. Witten’s proposed decomposition is straightforward:
- Give the model fewer sheets at a time.
- Include only the columns needed for the current task.
- Ask smaller questions that can be answered from those inputs.
The concrete CSV test stalls at getting the file’s text into the Console. Witten declines to open his work-computer downloads on the projector because they may expose private information, and the browser-only attempt ends without a tested extraction result.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Show the writing style you want
The social-media ghostwriter prompt asks for five to ten tweets from a long article, without hashtags, hyperbole, or cringe. Witten moves the article before the instructions, adds XML boundaries, and asks why the output needs to be a JSON array. The answer is automation: it needs to be parsable. He switches to tagged output and recommends examples to make the subjective style requirement concrete.
With an Anthropic blog post as input, the output avoids hashtags and does not seem especially hyperbolic. The phrase “New feature alert” becomes a specific candidate for what the audience might consider cringe; the absence of emojis is viewed favorably. The submitter calls the tweets adequate but not engaging. Witten adds an engagement instruction and clarifies the intended voice: an AI influencer rather than the company’s own account.
Asked about sentence complexity in prompts, he favors short sentences and familiar words, while allowing more academic language when seeking an academic tone. The revised tweets add questions and exclamation marks. Changing temperature from zero to one does not produce an obvious improvement to Witten’s eye in this informal comparison. The remaining obstacle is specification: engaging is still a subjective adjective. Examples of desired tweets—or complete source-document and tweet pairs—would tell the model much more.
Building those examples takes editorial work and input tokens, but Witten argues that the tradeoff can make sense when the outputs have economic value and are generated relatively infrequently. His proposed process is iterative:
- Start with a source document and generate candidate tweets.
- Select the strongest candidates, edit them, or write better ones yourself.
- Keep the literal document together with its approved tweets.
- Repeat with another document and add that pair to the prompt.
Generating 100 candidates and choosing seven is his illustrative workflow, not a reported experiment. The essential work is selecting and editing examples until they demonstrate the intended tone.
Witten considers this example curation one of the largest sources of prompting gains. The examples can live in one block or in alternating user and assistant messages. He usually chooses a single block because it is easier to author; his limited testing did not establish a confident performance preference. For subjective failures, add contrasting pairs: a poor tweet and an excellent tweet about the same document. Negative examples are useful, but they should accompany positive demonstrations of what to produce.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Put useful preparation before the answer
An explanation of why a model already wrote something is not necessarily a faithful account of how it produced the answer. Witten distrusts these retrospective explanations, comparing them with human rationalization. He distinguishes them from preparation generated before the answer: considering the relevant facts or alternatives first can help shape what comes next.
For the tweet task, that preparation becomes a concrete intermediate artifact: document → key points → tweets. Each example contains the document, a list of its key points, and then the approved tweets. The key points can be written by a person, generated and edited, or retained from a good model response. Witten also updates the requested output format to ask for key points before tweets, calling this lightweight chain-of-thought prompting. Although the audience question asks how to avoid receiving reasoning, this demonstration explicitly includes a preparatory output stage.
If long documents make examples expensive, he would start with one excellent complete example rather than several truncated ones. That choice needs testing: a single example can cause the model to fixate on the example document’s particular content and carry it into an unrelated document. Quality comes first, followed by checks that the learned pattern transfers without copying irrelevant details.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Separate persona routing from staying in character
The next prompt defines three roles and a small command protocol, including >p assist, for switching between them. Designers use the conversation for synthetic-user interviews. Witten has limited experience with multiple personas in one prompt, but suggests an application architecture: write a separate prompt for each persona and let code route the user’s command to the selected one. The submitter clarifies that the current setup runs entirely in chat; explicit routing would require building an API application. A follow-up proposal for another thread to aggregate the persona conversations receives no substantive answer.
Testing the existing chat prompt exposes a different problem. Witten selects Sam with >p Sam and asks how the persona finds good medication prices. Joe, another persona, is described as favoring convenience over cost savings. Sam’s response announces itself with “As Sam,” breaking the intended illusion of an ordinary person answering a question. Witten adds an instruction to prefix each response with the current persona name in brackets.
He then removes earlier assistant turns that contradict the new format and restarts the conversation. Those turns are examples too: leaving them in context would show the model behavior the revised prompt now discourages. The live protocol also needs the persona switch in a separate call before the question. The next answer drops the exact self-announcement but still foregrounds the persona awkwardly. A light instruction to say less about the persona and stay in character produces a visibly better response.
Witten generally prefers positive instructions, but this repair combines a brief negative constraint with the positive direction to stay in character. He warns against dwelling on forbidden content: repeatedly mentioning elephants while prohibiting them may make elephants more salient. His parenting comparison makes the same point with prunes—briefly state the restriction, then move on.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Role placement, counterexamples, and steering
Asked why he uses the system prompt so little, Witten describes his personal default: put the role there and most task instructions in the human message. He notes tool-use prompts as a possible exception because of explicit fine-tuning. Audience members report better behavior after moving instructions into the system prompt, and he acknowledges that his preference may be wrong for their cases. His proposed alternative is clearer separation between the actual user message and the surrounding task instructions, using labeled boundaries. This remains a disputed workshop heuristic, not a general rule about message priority.
Counterexamples are likewise optional. An attendee asks whether preference training makes positive and negative pairs especially important in prompts. Witten cautions that the training algorithm’s comparison need not mean the model sees both examples together in one context window. The useful question is whether a contrasting example clarifies this task—not whether every prompt should imitate a presumed training format.
Another attendee asks about control vectors and Scaling Monosemanticity, suggesting that a negative steering direction might replace a verbal prohibition. Witten distinguishes steering identified features from steering with differences in underlying activations. His limited experiments at the time favor prompting, but he does not make a settled prediction about which approach will ultimately work better. The attendee reports more success controlling style than deterministic task behavior on smaller models; Witten remains uncertain which techniques transfer between small and large models.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Improve the evidence in an image
A proposed dating-profile review has no screenshots attached, so the workshop moves to a supplied image of handwritten tube labels. Image pasting is unavailable in the demonstrated Console, prompting a switch to Claude.ai. The intended reading includes Mattie White and 8687; one label reads 867, which the submitter identifies as a human typo. The goal therefore includes reconciling repeated labels, not merely transcribing each one independently.
The first attempt misreads the handwriting. Witten’s strongest recommendation is to improve the input: crop to the relevant region, zoom in, and remove unnecessary details. The same principle applies to text and images—irrelevant material competes with the information needed for the task. Doing this programmatically can be difficult because the application must first decide which regions matter.
An attendee says the website describes downsampling to 1,000 × 1,000 pixels, but the discussion does not establish that historical limit. Current vision guidance describes model-dependent resizing that preserves aspect ratio, rather than a universal square size. The practical issue in this example is legibility. Witten suggests having the model describe what it sees before answering, while doubting that prompting alone can repair handwriting he finds difficult to read himself.
Describing the image first also has a failure mode. The attendee reports that a wrong interpretation of the first tube can propagate to later tubes. Witten agrees that the model’s drive for self-consistency can preserve an early mistake. Asked why use Claude rather than dedicated OCR, the attendee reports better results on some messy handwriting than with the Textract and OCR approaches they tried, especially after zooming in. No successful repair is demonstrated here; Witten closes the case with cropping and upsampling as the remaining suggestions.
Two brief questions follow. A feature called “fragments” turns out to mean Artifacts; Witten points to the bottom-left setting used to enable that workshop-era preview. Another attendee reports getting useful fixes by pasting entire tracebacks with no extra context. Witten attributes that improvement to model capability rather than a special prompting technique: sometimes the raw error already supplies enough information for a capable model.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Calibrate what a grade means
The translation-grading case starts with an English original and a poor Japanese translation. The prompt asks for a score from one to five and already includes analysis before the grade. The submitter says Claude can identify errors yet still score the translation badly. In the live translation test, Claude assigns 3 on a five-point scale where the submitter expects 1. The submitter describes roughly a dozen errors and suspects the grader is confusing acceptable underlying content with translation quality.
Witten describes a broader grading tendency: pleasant subject matter can receive favorable judgments on unrelated dimensions, while negative subject matter can attract criticism of style or logic. The prompt should explicitly identify the dimension being judged. It should also state that one is worst and five is best, even if the rubric descriptions already imply that direction.
A detour into Japanese tokenization does not resolve the grading problem. Witten says API inputs are tokenized but does not explain the pretraining setup. Audience comments mix questions about public tokenizer availability with conflicting claims about Japanese training and language quality. Those are distinct issues, and the exchange establishes no comparative language benchmark before he returns to editing the prompt.
The proposed repair separates ethical or risqué subject matter from translation quality, then anchors every score category with an example. For each category, show the original, the translation, an explanation of the relevant errors or strengths, and finally the grade. Explain how specific clues contribute to the overall score rather than merely asking the model to find clues. Claude can help draft examples, but they need expert editing; Witten cannot supply the Japanese judgments himself. No validated post-revision improvement is shown.
Numbers and verbal labels can both work as classes. Witten warns against assuming that a one-to-100 score is a calibrated measurement and prefers roughly five categories. More numerical resolution does not create more reliable distinctions when the grader has not learned what those distinctions mean.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Analysis before probabilities
Log probabilities could expose how probability is distributed across possible grades. Witten’s concern is what happens if obtaining those probabilities means asking for a grade immediately and dropping the analysis stage. He proposes a two-call design:
- Generate the analysis before the grade.
- Stop at the end of that analysis.
- Resend the analysis as an assistant prefill.
- Request probabilities for the grade that follows.
This requires a model interface supporting both prefill and log probabilities. Witten cannot identify one with both capabilities during the discussion, so the workflow remains a proposal.
His expectation is that analysis followed by a discrete grade would be more useful than a more nuanced probability distribution produced without analysis. The mechanism he points to is additional computation: explicitly generating the analysis gives the model more forward passes before it commits to the grade. This is his rationale, not a measured comparison in the workshop. The audience clarifies that he means analysis actually written before the answer, as in the prompt under discussion.
One last audience suggestion concerns rubric formatting: numbered criteria might influence score weighting, so dash bullets could be worth trying. The attendee posts an alternative prompt in Slack, but there is no live comparative test of that change.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Extract evidence before summarizing
The final prompt asks for a concise summary that preserves important information while avoiding introductory and closing chatter. Witten moves the instructions below the document and asks Matt for an input on which the prompt hallucinates. Prefill can address the unwanted introduction, but it does not by itself ground the summary in the source.
For grounding, Witten proposes extracting relevant quotations first, then writing the summary. This introduces an evidence-selection stage before synthesis. It also changes the response contract: a prefill that already starts the summary would skip the very stage the revised prompt now requests. The visible intermediate prompt has the quote-first instruction above an unchanged summary prefill—the mismatch he addresses next.
The intended sequence is now source document → relevant quotations → summary, with the prefix beginning the quotation stage. Matt is still obtaining the failing document when stage time expires. Witten offers to continue the case afterward, leaving a proposed repair rather than a tested before-and-after result. The final change makes the dependency explicit: gather the source evidence before generating the prose that depends on it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Anthropic's collection of notebooks and recipes for building applications with Claude.
Further reading
Exercises on prompt structure, examples, output formatting, and hallucinations, originally written around Claude 3 Haiku.
Historical announcement introducing Claude 3.5 Sonnet and the Artifacts preview.
Anthropic's companion explanation of its Scaling Monosemanticity research into interpretable features in Claude.
Updates since the talk
Current guidance for schema-constrained JSON responses, including truncation and refusal handling.
How to distinguish completed responses, token-budget truncation, and custom stop-sequence termination.
Current image-input guidance covering resizing, resolution limits, legibility, and image quality.
Full transcript
Loading transcript…