AI Engineer World's Fair 2024

What's new from Anthropic and what's next

Alex Albert· Head of Developer Relations, Anthropic13:38

Loading slides…

Read the talk

What's new from Anthropic and what's next

From electric factories to Artifacts, Alex Albert traces how better models and new interfaces can change the work a product makes possible.

From a talk by Alex Albert

Before you start: Basic familiarity with LLM prompts, API calls, and web application development will help with the examples.

A new motor in the same factory

Picture a factory in 1882: a large coal-fired steam engine at one end, overhead transmission lines running across the building, and hundreds of machines moving to the same rhythm. Alex Albert opens with the arrival of commercial electric power and a puzzle: why didn’t a superior power source immediately produce a more productive factory? His answer lies in the machinery surrounding the engine.

Black-and-white photograph of rows of factory machines beneath a dense network of overhead belts and shafts.Open full source frame
Factory machinery connected by overhead belts and shafts.

If one station needed power, the entire steam engine had to run. The transmission system also dictated where machines could sit, so the layout served power distribution rather than the manufacturing process or its workers. Replacing the steam engine with an electric motor removed the nearby coal furnace and made lighting easier, but left those constraints intact.

The larger change came when factories were redesigned around electricity. Smaller, specialized tools made the arrangement more flexible; workers could bring tools to an item instead of carrying the item back to a fixed workstation. The productivity opportunity was in redesigning the process around the new capability, not merely changing its power source.

0:170:31
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:17 · section reference included

The magic-star-icon phase

An LLM added behind a star-shaped button can preserve a product’s existing assumptions just as faithfully as an electric motor preserved a steam-powered factory’s layout. Albert draws a second comparison with early mobile development: shrinking a website onto a phone preceded applications designed around an always-available camera and GPS. Snapchat and Uber illustrate the latter approach—products whose behavior depended on what the new platform could do.

There are practical reasons AI products remain in this phase. LLMs are nondeterministic, reliability is difficult, and prompts can require repeated optimization. Developers are learning to build with a component that behaves differently from conventional software. In Albert’s account, the text box is still the main interaction pattern that has stuck; finding useful alternatives requires both stronger models and experiments with the product surface itself.

2:352:56
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:35 · section reference included

From benchmark scores to iterative coding

The model enabling the next set of examples is Claude 3.5 Sonnet, released the Thursday before this 2024 talk. It was the first release in the Claude 3.5 family, and Albert describes this middle-sized model as outperforming the previous flagship, Claude 3 Opus. He points to MMLU, HumanEval, GPQA, and tool-use evaluations, then turns from academic tests to application behavior.

Albert highlights a 200K-token context window for retrieval-augmented generation and claims near-perfect recall across it, without specifying the recall test conditions. A large context gives an application room to supply retrieved material, but that claim alone does not establish end-to-end RAG accuracy. For coding, his observation is more behavioral: Sonnet seems to understand debugging problems better and gets caught in repeated, unproductive loops less often than earlier models.

Pull requests provide a more concrete way to test multistep work. They have a defined objective, usually require several changes, and let a model iteratively write code and test its progress. Anthropic’s internal evaluation involved fixing bugs or adding functionality in open-source codebases with coding tools.

Albert reports scores of 64% for Claude 3.5 Sonnet and 38% for Claude 3 Opus on Anthropic’s internal pull-request evaluation. These figures describe that evaluation, rather than a general success rate for arbitrary pull requests; the talk does not provide its sample size, attempt budget, or full scoring protocol.

4:334:39
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:33 · section reference included

A table image becomes editable data

Vision extends the same model beyond text and code. Albert reports improvements over Claude 3 Opus on essentially every vision benchmark Anthropic tested. His concrete example is table transcription: supply an image of a table and ask the model to reproduce it in Markdown. He says the result was almost perfect and that he checked the numbers himself.

The Workbench frame shows the uploaded table and transcription request on the left, with the resulting earnings-history table on the right. This turns visual source material into a representation that can be copied, edited, or incorporated into another document—the kind of practical use that made vision particularly compelling to Albert.

Workbench interface with an uploaded table and the prompt “Transcribe this into a table” on the left, and an earnings-history table on the right.Open full source frame
Anthropic Workbench transcribes an uploaded table into a tabular response.
6:006:10
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:00 · section reference included

Capabilities need affordable access

The launch offered three access paths: Anthropic’s API, AWS Bedrock, and Vertex AI. That choice let developers use Claude through the provider that fit their environment. The prices and availability here describe the launch-era offering, while the later model and steering announcements describe forecasts made during the talk.

Claude 3.5 Sonnet’s launch pricing was $3 per million input tokens and $15 per million output tokens. Albert describes it as five times cheaper than Claude 3 Opus. His application-level point is that intelligence, speed, and cost must work together: a model becomes easier to embed in a product when its useful capabilities are also economical to invoke.

6:366:42
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:36 · section reference included

Artifacts give the work its own surface

Artifacts separate generated content from the conversation used to create it. An essay, SVG, or React website can occupy its own work surface while the chat remains the place to request changes. This is the first product example that directly answers the factory analogy: the interface changes to accommodate what the model produces.

The experience combines Sonnet’s coding, reasoning, and vision abilities. A screenshot or Figma diagram supplies a visual target; Claude produces code and components that a developer can use. Albert demonstrates the combination by recreating the claude.ai chat layout in React from a single screenshot. The displayed result places the recreated interface beside the conversation that produced it.

Albert describes the idea as having hidden in plain sight for more than a year and a half. A tweet shown afterward frames LLM product development as early in its S-curve. The encouraging implication is that a useful interaction pattern can still emerge from capabilities developers have already been experimenting with.

7:207:32
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

7:20 · section reference included

Projects make context a shared resource

Projects, released the Tuesday before the talk, address another part of the workflow: grounding outputs in knowledge a team already has. Style guides, codebases, transcripts, and past work become context for collaboration with Claude. On the Claude Team plan, teammates can share projects and chats.

Albert reports that Anthropic engineers upload repositories and documentation, and that he has begun seeing people share chats and Artifacts in place of some Google Docs or site documentation. The Project view shows a chat input alongside a populated Project knowledge sidebar. The shared object is now more than a finished document: it includes the knowledge and conversation involved in creating the work.

Claude Project page with a central chat input and a Project knowledge sidebar listing uploaded source files.Open full source frame
A Claude Project with source files listed in Project knowledge.
8:378:43
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:37 · section reference included

Tools connect model output to application behavior

The developer-facing counterpart is the Tool Use API, which Albert says had launched about a month earlier. It lets an application supply custom client-side functions for Claude to select. Tool use also provides a way to request consistent structured JSON, so the model’s response can fit an application’s data contract rather than arrive only as prose.

For example, a tool definition for extracting a task from text can describe the expected fields explicitly:

json

{
  "name": "record_task",
  "description": "Extract a task and its owner from the supplied text.",
  "input_schema": {
    "type": "object",
    "properties": {
      "task": {
        "type": "string",
        "description": "The work to be completed."
      },
      "owner": {
        "type": "string",
        "description": "The named owner, or an empty string if none is given."
      }
    },
    "required": ["task", "owner"]
  }
}

The schema makes the desired structure explicit; a tool request still needs application code to handle it. Albert also reports seeing developers give Sonnet hundreds of tools at once, an observation about emerging usage rather than a measured reliability result.

9:299:41
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:29 · section reference included

A shorter path from task description to prompt testing

The developer console reduces the distance between describing a task and trying a prompt:

  1. Enter a task description into the prompt generator. Claude produces a prompt intended to suit that task.
  2. Open the generated prompt in the Workbench and edit it directly.
  3. Use variables to turn the prompt into a template, substituting inputs for cases such as RAG testing.

This gives developers a starting point and a place to iterate, rather than requiring every prompt to begin as a blank text field.

Albert then previews the console’s Evaluate feature, already visible with a beta tag. He promises further details and iteration; the demonstration at this point establishes the direction toward evaluation without walking through a complete testing or grading workflow.

10:0810:24
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:08 · section reference included

Plan around a moving capability frontier

The roadmap preview names Claude 3.5 Haiku and Claude 3.5 Opus as models expected later that year. Albert describes the goals for each generation as greater intelligence, lower latency, and lower cost. His advice to developers follows from that pace: plan ambitiously enough that a new model arriving during development can expand what the product does. He expects meaningful improvement on a timescale of months rather than years.

10:5211:00
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:52 · section reference included

Controlling features as well as prompts

A separate research direction asks whether developers can influence model behavior by manipulating internal features. Albert introduces Scaling Monosemanticity, covered in Anthropic’s research account. The research concerns Claude 3 Sonnet, distinct from the newly released 3.5 model. It identifies features that activate for different topics; clamping a feature’s value higher or lower can steer the output.

Golden Gate Claude made that mechanism visible to users by amplifying the Golden Gate Bridge feature. Instead of relying only on instructions in a prompt, the demonstration altered an internal feature’s influence. Albert then describes a few beta testers experimenting with a steering API that lets them find and clamp features for specific attributes. Broader developer access is something he hopes to offer, not a general release announced here.

11:3111:35
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:31 · section reference included

Build a prototype and start the feedback loop

The immediate next step is a prototype: make something quickly enough to begin validating it and learning from feedback. To encourage that work, Albert announces a Build with Claude contest launched the previous day. Albert says the contest runs until July 10, with each of the top three projects receiving $10,000 in Anthropic API credits. He directs attendees to the on-screen destination and the top of the documentation page for the contest details.

For questions beyond the session, Albert offers to talk at the AWS booth down the hall over the next few hours. He also invites questions on X/Twitter and says he tries to read all his direct messages. The invitation leaves the next product experiment with the developers in the room.

12:2812:39
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

12:28 · section reference included

Resources

From the talk

Updates since the talk

Transcript download
Full transcript

Loading transcript…