▶ Watch ↗AI Engineer World's Fair 20242:21:54
Conference affiliation: Member of Technical Staff · Anthropic · 2026
Lance Martin is a member of technical staff at Anthropic, where his work includes Claude Managed Agents, infrastructure for agents running long assignments. Previously part of LangChain’s founding engineering team, he builds research agents, evaluation tools, and teaching materials. His development of open-deep-research illustrates a recurring concern across that work: a workflow that compensates for a model’s weaknesses can become an obstacle as the model improves.
Lance worked as a data scientist and engineer at Mascoma and a researcher at Lawrence Berkeley National Laboratory before completing his Stanford PhD in 2015. He joined Uber as a senior data scientist and later became its truck perception lead. In 2018, he joined Ike’s founding engineering team as perception lead; after Nuro acquired Ike, he became a tech lead manager on perception at Nuro. With Alvin Zhang, he coauthored an account of long-range object detection for automated trucking.
He joined LangChain in May 2023 and moved to Anthropic in January 2026. At LangChain, his work combined retrieval-augmented generation, agent development, evaluation, and developer education, including RAG From Scratch and the first LangChain Academy agent-building course.
Lance draws an explicit connection between improving vehicle-perception systems and improving language-model applications. Deployed systems expose difficult cases; developers collect and review those failures, turn them into evaluation examples, and test subsequent changes against them. In his language-model work, this means inspecting production traces and curating failures into reference questions and answers, rather than relying on an impressive demonstration.
His auto-evaluator supports that experimental approach to document question answering. Developers supply documents, generate or provide reference question-answer pairs, and compare settings such as chunk size, retrieval method, and model using model-based grading. The tool makes those choices testable against the material an application actually needs to handle.
In his 2024 work on controllable agents, Lance favored giving models freedom at selected points in an otherwise predefined process. His LangGraph implementations carried information between steps in shared state and used model decisions to choose particular branches. A retrieval system could check whether documents were relevant and supplement them with web search. A coding assistant could test generated imports and return errors to the model for correction.
The import check addressed a specific failure in ChatLangChain, LangChain’s documentation assistant: an answer could look useful until a developer copied its code and discovered that an import did not exist. Lance implemented a check that tested imports in isolation and fed failures back to the model. He preferred this inexpensive, executable check to generating tests automatically during the request, which would introduce another source of potentially incorrect output. The graph constrained where the model could change course while preserving the ability to recover from a known mistake.
His development of open-deep-research exposed the limits of that structure. The initial system, released in late 2024, divided a request into report sections, researched and wrote them in parallel, and combined the results. Parallelism made it fast, and avoiding tool calling removed a source of failure. But every request inherited the same research strategy, and independently written sections could produce a disjointed report.
As tool calling improved, the design also prevented him from taking advantage of the growing Model Context Protocol (MCP) ecosystem. Lance moved toward a multi-agent system with more freedom to choose tools and plan research. The reports remained fragmented because he had retained an older assumption: each research agent should write its own section.
Separating parallel information gathering from final writing addressed that problem. Research agents could gather material independently, while a final synthesis worked across their collected context. Lance credits Vadym Barda for initial evaluations, MCP support, and discussion, and Nick Huang for the multi-agent implementation and Deep Research Bench evaluations.
This experience sharpened his application of the Bitter Lesson: identify the assumptions built into an application and make them easy to revisit as models improve. He continued using LangGraph’s nodes, edges, and checkpointing. The prescribed research and writing strategy was what needed to change.
Several decisions recur across Lance’s projects and writing:
At Anthropic, Lance’s work extends these concerns to agents running asynchronously. In Claude Managed Agents, the team initially placed the harness directing the agent and its execution sandbox in one container. A container failure could lose the session.
The revised architecture separates the harness, execution environments, and an independent, append-only session log. A failed harness or sandbox can be replaced without erasing the history. Credentials stay in a separate vault, and the model can retrieve earlier information even after its working conversation has been compressed. The session log therefore serves both recovery and context management: information omitted from a summary remains available when a later step needs it.
Lance also advocates organization-level agent harnesses: shared systems with their own identities, credentials, and organizational context, accessible to multiple users. Colleagues can find prior experiments, avoid duplicating work, and begin with common tools and context instead of configuring everything individually. He expects asynchronous agents with this shared context to become more proactive, alerting people to relevant developments as well as responding to requests.
Lance’s work on getting more from frontier models applies the same scrutiny to inherited prompts. Instructions that helped earlier models—repeated demands to think step by step, double-check, or follow a prescribed reasoning ritual—can make stronger models spend unnecessary tokens. Contradictory rules create another problem: an earlier model might ignore one, while a model that follows instructions more closely can struggle to satisfy both.
He tested these effects with roughly 50 synthetic customer-support tickets, introducing prompting habits individually during a simulated upgrade from Opus 4.8 to Opus 5.5. Averaged across those variants, the newer model cost less and scored slightly higher; auditing the prompts reduced cost further and improved the score again. His recommendation is to test inherited instructions during model upgrades rather than carry every intervention forward.
He also distinguishes prompt changes from infrastructure savings. In multi-turn agents, much of the context repeats between calls. Prompt caching can avoid paying the full processing cost repeatedly, but changes to the shared prefix can invalidate the cache. Lance and colleagues found cost reductions of up to 72% on internal benchmarks by correcting cache use in their harnesses, with no meaningful performance change. That result concerned those configurations, rather than a general reduction available to every application.
For automated improvement, Lance starts with a measurable task and a narrowly defined thing to edit. He built an evaluation for the Claude API skill in Claude Code using questions drawn from Anthropic’s documentation; the skill initially scored 66%. He then gave Claude Code the skill text, documentation, and training-set failures so it could identify misleading instructions and revise them. Comparing training scores with scores on held-out questions provided a warning for overfitting: improvement on the training questions alone would not establish that the edits generalized.
Prompts and skills are convenient starting points because their text is easy to change and directly affects the behavior being tested. They are also easy to overfit. Lance warns that automated optimization can discover wording that raises an evaluation score without helping production work, or waste computation on ambiguous cases that no model can reliably satisfy. In a separate customer-support experiment, searching across model choice, reasoning effort, and prompt revisions reduced cost by 90% while improving performance. He treats stronger models at lower effort as configurations worth testing, rather than assuming that a cheaper model or maximum reasoning effort gives the best result.
Lance preserves a role for human inspection before optimization begins. Following feedback from Hamel Husain, he agreed that an evaluation-building workflow should show users the underlying examples before moving directly into analysis and test construction. A model can flag suspicious traces and run experiments; people still need to decide what matters and inspect whether the examples and grading criteria capture it.
▶ Watch ↗AI Engineer World's Fair 20242:21:54
▶ Watch ↗1:04:02
Loading more talks and highlights…