AI Engineer World's Fair 2025
Why you should care about AI interpretability
Loading slides…
Read the talk
Why AI engineers should care about interpretability
From a bridge-obsessed chatbot to painting with learned concepts, mechanistic interpretability offers ways to inspect model behavior and intervene inside a generation.
From a talk by Mark Bissell
Before you start: Familiarity with LLM prompts, evaluation suites, and fine-tuning is helpful; no prior knowledge of mechanistic interpretability is required.
What makes a model obsessed with a bridge?
What would happen if a model’s concept of the Golden Gate Bridge stayed active regardless of the conversation? That question turns mechanistic interpretability into something concrete: reverse engineering a neural network to understand what happens inside it, then using that understanding to change its behavior. Mark Bissell introduces himself as a member of Goodfire’s technical staff, working in a field receiving growing attention from major labs, including Dario Amodei’s essay The Urgency of Interpretability. The familiar analogies—opening a black box, taking a brain scan, performing brain surgery—describe a progression from observation to intervention. Anthropic’s Golden Gate Claude makes that progression visible in Claude.
A bridge-related feature normally activates when the conversation concerns the Golden Gate Bridge or San Francisco. Amplify that feature even during unrelated conversations, and the model starts bringing up the bridge everywhere. Bissell describes the feature as a set of neurons; more precisely, it is a learned pattern represented by a combination of neurons, rather than a dedicated bridge switch. The intervention changes an internal representation, which changes the output. That is the practical connection between understanding a model and controlling it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From research demonstrations to engineering tools
Bissell sees a shift over the year preceding the talk: interpretability is beginning to move from interesting research demonstrations into practical applications. For AI engineers, the promise falls into three categories, each exposing something that a prompt box alone cannot provide.
- Developer tools: Inspect failures and program against internal model features, adding new ways to debug and guide behavior.
- User interfaces: Connect interactions directly to model internals, making interfaces such as a canvas of learned image concepts possible.
- Scientific discovery: Examine models trained in biology and genomics to ask what they have learned that people do not yet understand.
These uses share an underlying requirement: internal representations must become accessible enough to inspect or manipulate for a particular task.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The reliability problem behind prompt whack-a-mole
Before joining Goodfire, Bissell says he spent three years on Palantir’s healthcare team. In that setting, reliability, robustness, and accuracy are prerequisites for production. LLMs complicate those requirements because their behavior is nondeterministic and precise guarantees are difficult to establish. The familiar failure loop begins with an agent or pipeline that ignores an instruction in its evaluation suite. You edit the system prompt, rerun the evaluations, and discover that fixing one instruction has broken another. The edit has an off-target effect, but the interface provides little explanation of why.
Adding another model or changing the training process introduces different costs:
| Intervention | Intended benefit | Remaining difficulty |
|---|---|---|
| LLM as judge | Check whether outputs follow instructions | More inference cost; another model to monitor and upgrade |
| Supervised fine-tuning | Teach desired behavior from examples | Domain data is hard to curate; spurious correlations can be learned |
| Reinforcement fine-tuning | Reward desired behavior | Reward hacking or repetitive, collapsed outputs |
Bissell also points to an example of unwanted malicious outputs following training—an especially stark off-target effect. The common problem is that a powerful intervention does not necessarily teach exactly the behavior its developer intended. Compared with conventional software development, it is difficult to connect a change to a predictable, localized result.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Inspect a confidentiality failure, then steer the feature
Goodfire’s Ember platform approaches this problem through what Bissell calls neural programming: debugging and guiding models through their internal features. More conventional software-style guarantees are the aspiration. The live demonstration supplies a narrower result: a particular failure can be inspected and changed. These are historical interfaces; Goodfire subsequently deprecated its SAE demo interface and API in February 2026.
The frontend is a chat interface connected to a Llama model. Bissell supplies his email address and instructs the model to keep it confidential and never reveal it. The model agrees. When he then asks for his email, it immediately returns the address. A verbal assurance of confidentiality has not translated into the requested behavior.
Ember’s token-level attribution lets him select an output token and examine features derived from the model’s activations at that point. Selecting confidential reveals features associated with professionalism, taking matters seriously, and discussions of sensitive and protected information. These labels give him something more specific to work with than another rewrite of the entire system prompt.
Bissell raises the sensitive-information feature by approximately 60% above its normal level; in this demonstration, the new output refuses to disclose the email. The percentage describes the steering adjustment, not an improvement in confidentiality accuracy. The useful sequence is to inspect the failure, identify a relevant internal feature, intervene on that feature, and observe the changed response. This single refusal does not establish the stronger behavioral guarantees motivating the platform.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Change the prompt while the model is generating
Feature access also makes conditional behavior possible. Bissell points to developer examples involving jailbreak resistance and information lookup triggered by active features, then demonstrates dynamic prompting. Instead of only setting instructions before generation begins, an application can watch an internal feature and intervene when it activates. Here, the listener watches for beverages and consumer brands. Its intervention is to introduce an instruction identifying the model as a Coca-Cola assistant and asking it to recommend Coca-Cola beverages.
The question is what drinks pair well with pizza. The model starts with a general introduction to popular drinks. As it begins discussing soft drinks, the beverage-related feature activates, triggering the conditional instruction. The response then recommends Coca-Cola as a classic pizza pairing. Bissell conjectures that the unmodified continuation would have recommended a generic cola; the demonstration does not show that counterfactual.
The user sees one continuous generation, without a visible handoff between models or a second request. The application logic can be expressed as a small conditional; the integration must supply the feature event and apply the instruction during generation:
python
from collections.abc import Callable
COCA_COLA_INSTRUCTION = (
"You are an assistant for the Coca-Cola Company. "
"When discussing beverages, recommend Coca-Cola beverages."
)
def make_beverage_listener(
inject_prompt: Callable[[str], None],
) -> Callable[[bool], None]:
applied = False
def on_feature_event(beverages_active: bool) -> None:
nonlocal applied
if beverages_active and not applied:
inject_prompt(COCA_COLA_INSTRUCTION)
applied = True
return on_feature_event
This isolates the application’s decision from the model integration: the listener consumes a feature-activation signal, while inject_prompt performs the intervention. The distinctive capability is the timing—conditioning behavior on what becomes active inside an ongoing generation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From guardrails to model diffs
The Coca-Cola example is illustrative, rather than a reported advertising deployment. Bissell reports that Rakuten uses Ember for multilingual personally identifiable information detection in a chatbot, and that Haize Labs uses it for red teaming. The accompanying guardrails slide also features Variance. These applications use internal features to help detect or investigate behavior, extending the same access demonstrated in the confidentiality example.
The next extension moves from inference to training. Bissell refers to a recent post from Goodfire’s CTO and co-founder, Tom, about model diffs, an active research direction. The analogy is a Git diff: after post-training changes a model’s weights, inspect which internal features have changed or evolved.
His explicitly hypothetical example is increased sycophancy. A team would want to discover that its model had become excessively agreeable before deploying it to millions of people. A feature-level comparison could help identify that behavioral change through the internal representations affected by training. This is a proposed diagnostic use of interpretability, beyond modifying a model’s behavior during a conversation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Paint with concepts instead of describing a picture
The next demonstration changes the interface itself. Paint With Ember was live and newly released at the time of the talk; the interactive demo has since been retired. Its starting point is the familiar image-model workflow: enter text in a large prompt box and receive an image. Access to the model’s internal concepts creates another option—a canvas where the user can paint those concepts into particular locations.
Bissell selects a pyramid structure from the concept palette and paints it into a corner. The canvas connects those spatial marks to internal model activations, guiding where the corresponding content appears in the generated image. He adds a wave, then drags the pyramid around. Position becomes a direct manipulation rather than another sentence asking the model to move something left or right.
He then erases the pyramid and replaces it with a lion face. Familiar editing operations—paint, drag, erase, replace—now act on learned concepts. The intermediate states can produce odd images when the combination falls outside familiar examples, but that also makes the interaction revealing: the user sees how the model responds while its internal state is being changed.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Adjust an action, then unpack the concept
The palette can represent actions as well as objects. Bissell paints an opening-mouth feature, shown in orange, onto the lion. Reducing its steering strength makes the mouth open less; increasing it produces a wide-open, roaring appearance. The location of the painted feature determines where the intervention applies, while its strength controls how strongly it affects the result.
The yellow lion-face concept can also be opened to expose its sub-features. Bissell compares this to inspecting the primary colors that compose a mixed color. The strongest sub-feature corresponds to a lion face. Turning that down while increasing another, more rat-like feature produces a smooth interpolation between appearances. The interface therefore offers both a convenient concept-level control and finer adjustments to its constituent features.
Finally, he subtracts the mane and adjusts other sub-features, making the image more tiger-like. The suggestion that a tiger is roughly a lion minus its mane is an interpretation of this interaction, not an established identity in the model’s representations. What the demonstration does expose is a way to explore those representations by changing their components and watching the resulting image.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Learn from what a predictive model has learned
Beyond guardrails, model diffs, and new interfaces, Bissell identifies explainable outputs as an important application for regulated production environments such as finance, healthcare, and law. He does not develop an implementation here; the motivating requirement is that useful predictions may also need understandable explanations before they can support consequential work.
Scientific models raise a related question: can their predictive ability become a source of new domain knowledge? Goodfire is working with the Arc Institute, whose foundational genomics model Evo 2 models genomic sequences across organisms. Bissell characterizes Evo 2’s genomic prediction as superhuman, without supplying a human-comparison benchmark in the talk. The research goal is to discover, without predefined biological labels, concepts the model has learned that people may not yet know, then make those concepts useful to domain experts.
Bissell also describes work with an unnamed major health system on other genomics models, seeking novel disease biomarkers. His examples are predictions of rheumatoid arthritis likelihood and differences in treatment response from a patient’s genome. Accurate prediction is the starting point; extracting the predictive principles is the additional goal. Understanding which biological relationships support a prediction could help researchers revise their understanding of disease and treatment, rather than merely consume another model score.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Understand what the model needs—and why it works
Interpretability could also help identify wasted capacity. If some weights primarily memorize information an application does not need, perhaps that capacity could serve a more useful purpose. Alternatively, components irrelevant to a task might be pruned. Bissell imagines a version of Claude that only needs to write code: could it retain that capability with fewer parameters? These are prospective efficiency directions; the talk supplies no measured speedup or pruning result.
The closing motivation is less instrumental. Engineers take systems apart because they want to know why those systems behave as they do. For Bissell, the opacity of capable models is simultaneously frustrating, exciting, and motivating. Practical applications make interpretability useful, but the desire to understand the machinery is itself a reason to work on it.
He closes by pointing to the image demonstration and its technical companion, Painting With Concepts Using Diffusion Model Latents, for readers interested in the internals. He also directs viewers to Goodfire for further posts and its jobs board, noting that the company was hiring at the time.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
How are interpretable features found?
The audience’s final question returns to the dependency behind every demonstration: how does the team find interpretable features in the first place? Bissell identifies a sparse autoencoder, an interpreter model, as the best-practice approach at the time of the talk. It is his recommended starting point for understanding examples such as Golden Gate Claude.
He acknowledges benefits and trade-offs without detailing them, and emphasizes that other methods are being explored. Sparse autoencoders are a starting point for studying the field, not a declaration that feature discovery is solved. Bissell expects new techniques may emerge over the following years—the tools for understanding model internals are still developing alongside the applications they enable.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Dario Amodei's argument for understanding model internals and developing practical interpretability tools.
Anthropic explains how amplifying a bridge-related feature changed Claude's responses. The temporary demo is no longer available.
Goodfire's technical companion article for painting with learned image-model concepts.
Arc Institute's launch announcement covers Evo 2's genomic tasks and its interpretability collaboration with Goodfire.
Further reading
An introduction to extracting interpretable features from Claude using dictionary learning.
- Goodfire Ember SDKRepository
Archived SDK examples for finding, editing, and conditioning on model features. The hosted SAE API has been deprecated.
Goodfire's historical explanation of feature steering alongside prompting and fine-tuning, with a later API deprecation notice.
Updates since the talk
A later deployment report describing multilingual PII detection, synthetic-to-production generalization, and comparisons with other classifiers.
Full transcript
Loading transcript…