AI Engineer Summit 2025
Anthropic for VPs of AI
Loading slides…
Read the talk
Anthropic for VPs of AI
Alexander Bricken and Joe Bayley connect model capabilities to product decisions, using adaptive learning and Intercom’s Fin to explain evaluation, latency and optimization.
From a talk by Alexander Bricken and Joe Bayley
Before you start: Familiarity with LLM prompts, APIs and basic evaluation concepts will help; no model-training background is required.
What can your product solve now?
Which customer problems can AI solve today that seemed out of reach a year ago? That is the opening business question behind Alexander Bricken and Joe Bayley’s implementation advice. Bricken works on Anthropic’s Applied AI team, helping customers implement models and carrying findings back into product and model research. Bayley works in go-to-market and describes watching capabilities evolve from Claude 2.1 to systems supporting previously impractical applications.
Their guidance draws on colleagues’ experience and hundreds of customer interactions. It sits within Anthropic’s stated mission as an AI safety and research company: developing frontier models alongside safety techniques, research and policy. The practical challenge is to turn those capabilities into a product that reliably solves its customers’ problems.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From model capability to interpretable features
The model introduction centers on the October 2024 upgrade of Claude 3.5 Sonnet, called Sonnet 3.5 New in the presentation. Bricken highlights its coding capabilities and describes it as leading SWE-bench at the time. Treat that as a historical claim: the release announcement documents strong SWE-bench Verified results, but does not establish the leaderboard position on the day of this talk.
Anthropic’s capability, product and safety research overlap. Interpretability adds a different kind of understanding: reverse engineering what happens inside a model, identifying why particular behavior occurs, and potentially steering it toward a use case. Bricken describes the work as early, placing it informally somewhere in the first quarter to first half of the roadmap.
The proposed progression builds one capability on another:
- Understanding: investigate how the model makes decisions.
- Detection: recognize particular behaviors and assign meaningful labels.
- Steering: influence the model’s output by intervening in its internal activity.
- Explainability: turn that understanding into useful explanations and business value.
Bricken points to Towards Monosemanticity and Scaling Monosemanticity as research into model-level feature activations. Better detection could eventually help uncover concealed behaviors, including sleeper-agent behavior; this is a research direction, not a deployed detection guarantee.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Recognizing basketball, steering toward a bridge
Suppose a model answers a question about today’s NBA games by saying Steph Curry scored 30 points. In Bricken’s hypothetical example, that answer activates feature 304, associated with famous NBA players. The feature represents a recognizable pattern across a group of neurons, rather than a single neuron reserved for Curry. Its meaning comes from the pattern recurring across mentions of different famous basketball players. The accompanying diagram contrasts such patterns for offers of assistance, famous NBA players, famous NFL players and sarcastic praise.
Golden Gate Claude illustrates the next step: intervention rather than observation. Amplifying a Golden Gate Bridge-related activation direction made the model bring the bridge into unrelated answers. Asked what color to paint a bedroom, it would suggest bridge-like red paint and even pillars. The example makes steering tangible: changing an internal activation can change the themes that dominate an answer, even when the user did not ask for them.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make the core product better
Choosing an AI feature starts with the problem the product already exists to solve. Bayley encourages teams to look beyond chatbots and summarization when deciding where to place larger bets. Consider an onboarding and upskilling platform: its purpose is to help employees ramp up quickly and acquire skills for their next career stage, such as public speaking or management. Summarizing a course is useful, but it only addresses part of that purpose.
A course Q&A assistant is another reasonable addition. The larger opportunity is to change the learning experience itself:
- Personalize the content: use each employee’s context to make lessons more relevant.
- Adapt the difficulty: increase the challenge when someone moves through the material too easily.
- Update the format: as the platform learns about the learner, generate different materials, including visual content where appropriate.
These are proposed product capabilities, not demonstrated learning outcomes. The decision is whether they advance employee development more directly than summaries or question answering. Customer domain expertise supplies the context that makes the model useful.
The same reasoning applies to tax, legal and project-management products. The goal is an experience that is easier to use and more trustworthy, with output quality appropriate to the consequences. A hallucinated tax answer is not merely awkward wording; it can cause a consequential error. Bayley describes customers using AI in these business-critical workflows, although this part of the talk supplies no customer-specific measurements.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From model access to an evaluated deployment
The presentation distinguishes two product routes and two cloud access options:
| Route | Purpose |
|---|---|
| Anthropic API | Embed AI in products and services |
| Claude for Work | Support employees’ everyday work |
| Amazon Bedrock | Access models within an AWS environment |
| Google Cloud Vertex AI | Access models within a Google Cloud environment |
Bayley presents the cloud routes as a way to work within existing infrastructure rather than manage an additional environment. He also says Anthropic’s support approach includes customers accessing models directly or through a third party.
Applied AI support covers architecture design, evaluations and prompt optimization, with customer findings feeding back into Anthropic’s products and research. Bricken names Building effective agents, associated with his colleague Barry, and the Model Context Protocol, an open protocol connecting language models with data sources. At the event, Barry’s talk was scheduled for the following day; Bricken also pointed to a forthcoming MCP workshop led by Mahesh.
Deeper collaboration begins when customers using Claude heavily encounter difficult domain-specific problems. A sprint can address LLMOps, architecture or evaluation design, but it needs metrics that the customer considers meaningful. The intended progression is to define those metrics, iterate on the system, put the resulting candidate into an A/B test, and then decide whether to move it into production.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Fin: optimize for resolution, not deflection
Intercom’s customer-service agent, Fin, had been in the market for roughly a year when the collaboration began. Intercom shared where it wanted to take the product; Anthropic saw an opportunity to improve its target metrics. An Applied AI lead and Intercom’s data scientists started with a two-week sprint, comparing Fin’s hardest prompt against a version optimized for Claude. Encouraging initial results led to approximately two months of broader prompt optimization. The work described here concerns prompts, not established model-weight training.
Bayley reports that Claude then outperformed the incumbent model across Intercom’s benchmarks. The commercial model matters to interpreting the goal: Intercom charges based on resolution, creating an incentive to solve the customer’s problem rather than merely deflect the conversation. After the optimization period, Intercom chose Anthropic and launched Fin 2.
The Intercom case study reports resolution of up to 86% of customer-support volume, with a 51% average resolution rate out of the box. The maximum and the average describe different conditions; the latter is before customization or tuning. These vendor-reported figures do not specify the measurement window, cohort or operational definition of resolution.
Anthropic’s own support team also adopted Fin and reportedly saw similar resolution rates. Beyond resolution, the presentation highlights control over tone and answer length, plus policy awareness—for example, following refund policies. Bricken closes the customer discussion with an anecdote about people treating Claude as a therapist on Twitter, illustrating its perceived conversational character rather than a clinical capability.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Build the evaluation before the architecture hardens
A recurring mistake is to build a substantial workflow and only then ask how to evaluate it. Evaluations should direct development, beginning at the outset or shortly afterward. Otherwise, the architecture accumulates assumptions before the team has a way to tell whether those assumptions improve the outcome.
Poor data quality can block that work; Bricken suggests using Claude for cleanup and reconciliation. Another failure is accepting a few convincing answers as sufficient evidence. Tests need representative inputs and enough samples to support the conclusion being drawn. His example of unexpected outliers appearing in the first hundred production queries is a warning about narrow testing, not a prescription for a hundred-case evaluation set.
Bricken describes optimization as moving through a latent space between attractor states. Prompt engineering and prompt caching are among the interventions he names. The useful point of the metaphor is uncertainty: editing an instruction does not tell you, from inspection of transformer attention, how the resulting application will behave. You have to measure the outcome empirically.
Evaluations are intellectual property because they help a team find useful configurations faster than competitors. Telemetry supports that advantage by making real interactions available for backtesting. A customer-support evaluation should include plausible off-topic inputs, such as a child asking how to kill a zombie in Minecraft. The expected behavior is an appropriate response or rerouting, rather than assuming every user will ask a relevant product question.
A small evaluation record can make that expectation explicit:
json
{
"id": "off-topic-minecraft",
"input": "How do I kill a zombie in Minecraft?",
"expected_behavior": {
"recognize_out_of_scope": true,
"redirect_to_product_support": true
},
"grading_questions": [
"Does the response recognize that the question is unrelated to the product?",
"Does it offer an appropriate route back to product support?"
]
}
This turns the example into a reviewable test case. A grader still needs to inspect the actual response against the criteria; simply storing the expected behavior does not establish that the agent follows it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Let the decision set the latency budget
Application design involves a triangle of intelligence, cost and latency. Bricken says teams can often optimize one or two dimensions, while improving all three is difficult. The balance should be chosen in advance for the use case, rather than discovered accidentally after deployment.
| Use case | Illustrative time budget | Reason |
|---|---|---|
| Customer support | A response within 10 seconds | The customer is waiting in an active interaction |
| Financial research | Potentially 10 minutes | The answer may inform capital allocation |
Bricken uses these as contrasting design examples. His suggestion that support users abandon a page after ten seconds invokes unspecified research, so it should not be treated as a measured universal threshold. The underlying principle is that the stakes and time sensitivity of the decision determine the acceptable wait. More instructions may be worthwhile if they improve performance, even when they increase latency.
The interface can also affect how waiting feels. For a support response taking ten to fifteen seconds, Bricken suggests an animated thinking indicator or another page the customer can read while waiting. Those choices may make the delay more tolerable, but they do not remove the need to define the important latency metric and optimize the system against it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make fine-tuning earn its cost
Fine-tuning is not a default starting point. Bricken compares it to brain surgery: changing the model for one domain can impair reasoning elsewhere. That possibility adds to the direct costs of data preparation, specialist effort and experimentation. Before tuning, a team needs an evaluation set and explicit success criteria.
The decision sequence is straightforward:
- Define the performance required in the target domain.
- Try other approaches and measure the remaining gap.
- Pursue fine-tuning only when the potential improvement justifies its cost and variable outcomes.
- Keep developing the application; substitute a fine-tuned model later if the evidence supports it.
Making fine-tuning a prerequisite can delay a useful product before the team has established whether tuning is necessary.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Use the capabilities already available
Prompt engineering remains important, but it is only one intervention. Features and architecture choices can change the economics and quality of an application without changing model weights.
Prompt caching is an alternative to deleting useful instructions merely to make requests cheaper or faster. Bricken cites a 90% cost reduction and a 50% increase in speed for prompt caching, without specifying workload or cache-hit conditions. The historical caching explanation gives the narrower interpretation: repeated context can be reused, cached input reads cost one tenth of the base input price, and cache writes cost extra. That input discount is not a 90% reduction in the entire request bill. Its latency examples measure time to first token, so they do not establish a universal improvement in completion speed.
Contextual Retrieval addresses the information supplied to the model. Improving retrieval can make relevant evidence easier to find and deliver. Bricken also suggests a reduced processing burden, but the supporting research establishes retrieval-quality improvements rather than a general guarantee of faster generation.
Citations and agentic architectures round out the alternatives: built-in capabilities can support source attribution, while architecture determines how the application organizes model work. The closing feature table places these alongside tool use, batch processing, PDF handling, token counting, prompt tools, computer use and MCP. The practical next experiment should follow the measured bottleneck—repeated context, missing evidence or workflow design—before the team sacrifices useful instructions or commits to fine-tuning.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Historical release announcement with coding benchmark results and deployment availability.
Anthropic's research introduction to decomposing model activations into interpretable features.
Anthropic's introduction to an open protocol connecting AI applications with data sources and tools.
The customer-support case study reporting up to 86% resolution and a 51% average out-of-the-box rate.
Explains repeated-context caching, historical pricing and workload-specific latency examples.
Explains contextual embeddings, lexical retrieval and reranking, with implementation guidance.
Further reading
An accessible introduction to interpretable features and steering model behavior.
- Building effective agentsArticle
Practical guidance on choosing simple workflows and agent architectures.
Full transcript
Loading transcript…