AI Engineer World's Fair 2024
What's new from Anthropic and what's next
Loading slides…
Read the talk
What's new from Anthropic and what's next
From electric factories to Artifacts, Alex Albert traces how better models and new interfaces can change the work a product makes possible.
From a talk by Alex Albert
Before you start: Basic familiarity with LLM prompts, API calls, and web application development will help with the examples.
A new motor in the same factory
Picture a factory in 1882: a large coal-fired steam engine at one end, overhead transmission lines running across the building, and hundreds of machines moving to the same rhythm. Alex Albert opens with the arrival of commercial electric power and a puzzle: why didn’t a superior power source immediately produce a more productive factory? His answer lies in the machinery surrounding the engine.
If one station needed power, the entire steam engine had to run. The transmission system also dictated where machines could sit, so the layout served power distribution rather than the manufacturing process or its workers. Replacing the steam engine with an electric motor removed the nearby coal furnace and made lighting easier, but left those constraints intact.
The larger change came when factories were redesigned around electricity. Smaller, specialized tools made the arrangement more flexible; workers could bring tools to an item instead of carrying the item back to a fixed workstation. The productivity opportunity was in redesigning the process around the new capability, not merely changing its power source.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The magic-star-icon phase
An LLM added behind a star-shaped button can preserve a product’s existing assumptions just as faithfully as an electric motor preserved a steam-powered factory’s layout. Albert draws a second comparison with early mobile development: shrinking a website onto a phone preceded applications designed around an always-available camera and GPS. Snapchat and Uber illustrate the latter approach—products whose behavior depended on what the new platform could do.
There are practical reasons AI products remain in this phase. LLMs are nondeterministic, reliability is difficult, and prompts can require repeated optimization. Developers are learning to build with a component that behaves differently from conventional software. In Albert’s account, the text box is still the main interaction pattern that has stuck; finding useful alternatives requires both stronger models and experiments with the product surface itself.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From benchmark scores to iterative coding
The model enabling the next set of examples is Claude 3.5 Sonnet, released the Thursday before this 2024 talk. It was the first release in the Claude 3.5 family, and Albert describes this middle-sized model as outperforming the previous flagship, Claude 3 Opus. He points to MMLU, HumanEval, GPQA, and tool-use evaluations, then turns from academic tests to application behavior.
Albert highlights a 200K-token context window for retrieval-augmented generation and claims near-perfect recall across it, without specifying the recall test conditions. A large context gives an application room to supply retrieved material, but that claim alone does not establish end-to-end RAG accuracy. For coding, his observation is more behavioral: Sonnet seems to understand debugging problems better and gets caught in repeated, unproductive loops less often than earlier models.
Pull requests provide a more concrete way to test multistep work. They have a defined objective, usually require several changes, and let a model iteratively write code and test its progress. Anthropic’s internal evaluation involved fixing bugs or adding functionality in open-source codebases with coding tools.
Albert reports scores of 64% for Claude 3.5 Sonnet and 38% for Claude 3 Opus on Anthropic’s internal pull-request evaluation. These figures describe that evaluation, rather than a general success rate for arbitrary pull requests; the talk does not provide its sample size, attempt budget, or full scoring protocol.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A table image becomes editable data
Vision extends the same model beyond text and code. Albert reports improvements over Claude 3 Opus on essentially every vision benchmark Anthropic tested. His concrete example is table transcription: supply an image of a table and ask the model to reproduce it in Markdown. He says the result was almost perfect and that he checked the numbers himself.
The Workbench frame shows the uploaded table and transcription request on the left, with the resulting earnings-history table on the right. This turns visual source material into a representation that can be copied, edited, or incorporated into another document—the kind of practical use that made vision particularly compelling to Albert.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Capabilities need affordable access
The launch offered three access paths: Anthropic’s API, AWS Bedrock, and Vertex AI. That choice let developers use Claude through the provider that fit their environment. The prices and availability here describe the launch-era offering, while the later model and steering announcements describe forecasts made during the talk.
Claude 3.5 Sonnet’s launch pricing was $3 per million input tokens and $15 per million output tokens. Albert describes it as five times cheaper than Claude 3 Opus. His application-level point is that intelligence, speed, and cost must work together: a model becomes easier to embed in a product when its useful capabilities are also economical to invoke.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Artifacts give the work its own surface
Artifacts separate generated content from the conversation used to create it. An essay, SVG, or React website can occupy its own work surface while the chat remains the place to request changes. This is the first product example that directly answers the factory analogy: the interface changes to accommodate what the model produces.
The experience combines Sonnet’s coding, reasoning, and vision abilities. A screenshot or Figma diagram supplies a visual target; Claude produces code and components that a developer can use. Albert demonstrates the combination by recreating the claude.ai chat layout in React from a single screenshot. The displayed result places the recreated interface beside the conversation that produced it.
Albert describes the idea as having hidden in plain sight for more than a year and a half. A tweet shown afterward frames LLM product development as early in its S-curve. The encouraging implication is that a useful interaction pattern can still emerge from capabilities developers have already been experimenting with.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Projects make context a shared resource
Projects, released the Tuesday before the talk, address another part of the workflow: grounding outputs in knowledge a team already has. Style guides, codebases, transcripts, and past work become context for collaboration with Claude. On the Claude Team plan, teammates can share projects and chats.
Albert reports that Anthropic engineers upload repositories and documentation, and that he has begun seeing people share chats and Artifacts in place of some Google Docs or site documentation. The Project view shows a chat input alongside a populated Project knowledge sidebar. The shared object is now more than a finished document: it includes the knowledge and conversation involved in creating the work.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Tools connect model output to application behavior
The developer-facing counterpart is the Tool Use API, which Albert says had launched about a month earlier. It lets an application supply custom client-side functions for Claude to select. Tool use also provides a way to request consistent structured JSON, so the model’s response can fit an application’s data contract rather than arrive only as prose.
For example, a tool definition for extracting a task from text can describe the expected fields explicitly:
json
{
"name": "record_task",
"description": "Extract a task and its owner from the supplied text.",
"input_schema": {
"type": "object",
"properties": {
"task": {
"type": "string",
"description": "The work to be completed."
},
"owner": {
"type": "string",
"description": "The named owner, or an empty string if none is given."
}
},
"required": ["task", "owner"]
}
}
The schema makes the desired structure explicit; a tool request still needs application code to handle it. Albert also reports seeing developers give Sonnet hundreds of tools at once, an observation about emerging usage rather than a measured reliability result.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A shorter path from task description to prompt testing
The developer console reduces the distance between describing a task and trying a prompt:
- Enter a task description into the prompt generator. Claude produces a prompt intended to suit that task.
- Open the generated prompt in the Workbench and edit it directly.
- Use variables to turn the prompt into a template, substituting inputs for cases such as RAG testing.
This gives developers a starting point and a place to iterate, rather than requiring every prompt to begin as a blank text field.
Albert then previews the console’s Evaluate feature, already visible with a beta tag. He promises further details and iteration; the demonstration at this point establishes the direction toward evaluation without walking through a complete testing or grading workflow.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Plan around a moving capability frontier
The roadmap preview names Claude 3.5 Haiku and Claude 3.5 Opus as models expected later that year. Albert describes the goals for each generation as greater intelligence, lower latency, and lower cost. His advice to developers follows from that pace: plan ambitiously enough that a new model arriving during development can expand what the product does. He expects meaningful improvement on a timescale of months rather than years.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Controlling features as well as prompts
A separate research direction asks whether developers can influence model behavior by manipulating internal features. Albert introduces Scaling Monosemanticity, covered in Anthropic’s research account. The research concerns Claude 3 Sonnet, distinct from the newly released 3.5 model. It identifies features that activate for different topics; clamping a feature’s value higher or lower can steer the output.
Golden Gate Claude made that mechanism visible to users by amplifying the Golden Gate Bridge feature. Instead of relying only on instructions in a prompt, the demonstration altered an internal feature’s influence. Albert then describes a few beta testers experimenting with a steering API that lets them find and clamp features for specific attributes. Broader developer access is something he hopes to offer, not a general release announced here.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Build a prototype and start the feedback loop
The immediate next step is a prototype: make something quickly enough to begin validating it and learning from feedback. To encourage that work, Albert announces a Build with Claude contest launched the previous day. Albert says the contest runs until July 10, with each of the top three projects receiving $10,000 in Anthropic API credits. He directs attendees to the on-screen destination and the top of the documentation page for the contest details.
For questions beyond the session, Albert offers to talk at the AWS booth down the hall over the next few hours. He also invites questions on X/Twitter and says he tries to read all his direct messages. The invitation leaves the next product experiment with the developers in the room.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Anthropic's historical announcement covering Sonnet's evaluations, launch pricing, availability and Artifacts preview.
How Projects introduced reusable knowledge, custom instructions and shared team conversations.
The historical tool-use announcement explains external-tool integration and structured data extraction.
Anthropic's account of the demonstration that amplified a Golden Gate Bridge feature.
Further reading
An accessible introduction to the feature-extraction and steering research behind Scaling Monosemanticity.
Updates since the talk
The July 2024 announcement explains test suites, generated inputs, output comparisons and human grading in the Console.
Full transcript
Loading transcript…