AI Engineer World's Fair 2024

Architecting and Testing Controllable Agents

Loading slides…

Read the talk

Architecting and Testing Controllable Agents

Build adaptive RAG workflows by constraining where models make decisions, then evaluate both the answers they produce and the paths they take.

From a talk by Lance Martin

Before you start: Familiarity with Python functions, dictionaries, and the basic idea of retrieving documents for an LLM will help with the examples.

Who decides the next step?

A RAG application starts with a question, retrieves documents from a vector store, passes them to an LLM, and returns an answer. The sequence is straightforward because the developer has already decided what happens next. This is a chain: its control flow is fixed, even though the generated answer is not.

Slide titled “A RAG chain” with arrows connecting Question, Vectorstore, LLM, and Answer.Open full source frame
A RAG chain passes a question through a vector store and an LLM to produce an answer.

An agent gives the model some responsibility for that sequence. After inspecting the output of one step, the LLM can decide whether to continue, go back, or choose another operation. The distinction is about who controls execution, rather than whether the application uses an LLM at all.

Tool calling supplies the interface for those decisions. A developer defines a function, exposes its description and argument schema, and binds that tool to the model. The model then emits a structured payload containing a function name and arguments. The application executes the function; the model proposes the call. For example, a payload naming step_two with an input of 3 still needs application code to dispatch it.

ReAct, short for reasoning and acting, turns this into a loop: the model selects an action, application code executes the tool, and the resulting observation returns to the model. The model can then request another tool or finish with an ordinary response. That loop can express many sequences: one tool alone, two tools in order, a reversed order, or repeated calls.

1:291:43
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:29 · section reference included

Make flexibility an explicit part of the graph

That flexibility creates several ways to fail. An agent can get stuck repeatedly calling the same tool. It can name step_three when the available tool is step_two, or supply 4 when the request specified 3. Selecting the operation and constructing its arguments are separate failure surfaces. Lance Martin reports that larger tool inventories and longer dialogues make these decisions harder.

Diagram connecting natural language and a structured tool to an LLM, with red annotations marking wrong payload and wrong tool.Open full source frame
Tool calling can fail through a wrong payload or a wrong tool choice.

Many applications do not need arbitrary sequences. A developer may know that step one must always run first and step two must always run last, while still needing a model to make a decision between them. LangGraph expresses that arrangement as a graph: state carries information through the run, nodes perform operations and update that state, and edges determine where execution goes next. An edge can follow a fixed route or use a model's decision.

This maps familiar agent capabilities onto explicit application structure. Short-term memory becomes graph state. Tools live inside nodes. Planning becomes decision-making at selected routing points. The useful constraint is that a model need not rediscover the entire workflow every time it acts.

5:165:25
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:16 · section reference included

Check retrieval before trusting it

Corrective RAG starts from a weakness in the opening chain: retrieved documents may not be relevant. Martin's workshop adaptation adds a document-grading step and a branch to web search when retrieval is inadequate. It is a simplified binary workflow inspired by the paper, whose full method also includes confidence-based actions and finer-grained filtering. Martin previously demonstrated the graph locally through Ollama with Mistral 7B.

The next extension checks the generated answer. Self-RAG motivates reflection on whether an answer is supported by retrieved evidence; the original work trains reflection-token behavior, while this graph uses explicit checking steps. Martin combines ideas from Corrective RAG, Self-RAG, and Adaptive RAG in a local Llama 3 workflow.

The combined flow separates decisions that an open-ended agent would otherwise make implicitly:

  1. Route the question to the vector index or web search.
  2. Retrieve documents and grade their relevance.
  3. If any retrieved document is irrelevant, supplement the retained evidence with web search.
  4. Generate an answer from the available documents.
  5. Check whether the answer is grounded in those documents and whether it answers the question.
  6. Return the answer when both checks pass.

Martin reports running this locally. His explanation for its reliability is architectural: the model makes decisions at discrete points, with the surrounding execution order already specified.

8:518:59
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:51 · section reference included

Use executable feedback to repair code

The same structure applies to generated code. AlphaCodium, discussed through Andrej Karpathy's flow-engineering framing, produces a candidate solution, checks it against tests, and feeds failures back to the model for another attempt. Tests can supply much sharper feedback than a general request to improve an answer.

Martin's LangGraph code-generation implementation targets a particularly frustrating RAG failure: an answer contains plausible Python code whose imports do not exist. He reports that adding import checks improved results in LangChain's internal RAG application, with production integration underway at the time. The workshop gives no numerical result isolating that check.

StructureWho chooses the sequence?Useful flexibility
ChainDeveloperFixed procedure
Constrained graphDeveloper, with selected model decisionsRouting and self-correction
Open-ended ReActModel, through repeated tool selectionArbitrary tool sequences

A constrained graph gives up some of ReAct's freedom. Martin acknowledges that a genuinely open-ended task may need that freedom; for many application workflows, however, routing and correction provide enough adaptability.

The Corrective RAG exercises offer three execution paths: a local Ollama notebook for suitable hardware, a Colab using a Google account and the required API keys, and a notebook using hosted models. The search component is replaceable, so the exercise does not depend on one particular web-search provider.

11:3711:48
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:37 · section reference included

Evaluate the answer and the route separately

How do you know the constrained agent is actually more reliable? That question requires a different feedback loop from the one repairing an individual answer. Martin separates in-app correction, pre-production testing, and production monitoring. LangSmith supplies optional evaluation, tracing, and monitoring around the application; the notebooks include it, but building the agent does not require using it.

An offline evaluation needs a dataset, an application to run against it, and evaluators that judge the resulting outputs. For RAG, a dataset can contain known question-answer pairs. An LLM judge compares the generated answer with the reference under an explicit grading prompt. A custom heuristic evaluator can instead calculate a property directly. The notebooks use both approaches.

A correct answer can conceal a bad execution path. An agent might eventually answer correctly after unnecessary calls or an unexpected detour. End-to-end answer grading will miss that behavior, so Martin also evaluates the sequence of tools used. His workshop experiment uses five RAG questions and three replicates per experiment, with aggregate answer and trajectory scores.

Martin reports 60% answer correctness for the local 8B LangGraph agent on the five-question evaluation, alongside strong trajectory consistency. Firefunction-v2, which he describes as a Fireworks fine-tune of Llama 70B, reaches 80% answer correctness and a 100% tool-trajectory score with LangGraph. These are small workshop experiments; they separate a model's answer quality from its adherence to the intended process.

With Firefunction-v2 in the ReAct arrangement, Martin observes worse answers and substantially worse trajectories. The GPT-4o ReAct examples retain strong answers but still take unexpected paths. He attributes the constrained graph's consistency to its fixed structure and the remaining answer-quality differences largely to model capacity. The trajectory score measures adherence to the evaluator's expected paths, rather than an unrestricted measure of reasoning ability.

15:2915:39
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

15:29 · section reference included

Build the system that measures improvement

Deployment enters the discussion as forthcoming work: Martin previews a Thursday announcement from Harrison about running LangGraph applications, and points to LangSmith for monitoring deployed runs. The broader investment is the system around a changing model—an evaluation framework, guardrails, and a data flywheel that make improvements measurable. Those principles do not depend on adopting either product.

The three feedback loops operate at different times. In-app checks repair the current attempt. Pre-production tests compare application versions before release. Online evaluators inspect production inputs and outputs, tag problematic runs, and make them available for later review. Together they connect application design to evidence from actual use.

Three-column diagram showing design, pre-production, and production phases with return arrows linking monitoring and testing to app design.Open full source frame
Feedback loops connect application design, pre-production testing, and production monitoring.
22:3022:42
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

22:30 · section reference included

Choose checks that fit the task

Exact trajectory matching becomes less useful as a task becomes more open-ended. For a short workflow—Martin suggests roughly five to ten steps—you may be able to enumerate several legitimate routes. For a long-running task with many valid solutions, use more selective properties: whether a required tool was called, whether calls repeat excessively, or whether the overall tool count is unreasonable. A custom evaluator can enforce those boundaries without prescribing every action.

Human intervention is another possible control point, but it is outside the supplied workshop notebooks. Martin points to separate LangGraph examples and previews additional support in the upcoming release.

Production introduces a different constraint: most incoming questions have no reference answer. Three useful RAG checks still work with the information already present in a run:

CheckCompareQuestion
Retrieval relevanceDocuments and questionIs the retrieved material relevant?
Answer relevanceAnswer and questionDoes the response address the request?
GroundingAnswer and documentsDoes the evidence support the response?

These checks assess different relationships. A grounded answer can still miss the question, and relevant documents can still yield an unsupported answer. Martin reports running these reference-free checks online in the internal application.

26:2126:41
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

26:21 · section reference included

Keep runtime checks cheap and bounded

Conventional tests remain useful alongside model-based evaluation. Martin describes LangSmith's pytest support, then distinguishes tests run outside the application from checks embedded in its correction loop. Import validity, code execution, and structured-output schema validation can all generate actionable error feedback. The same checks can also run independently in CI; further LangSmith CI integration was still forthcoming in the workshop.

An LLM judge on the request path adds latency. For a stable grading task, Martin suggests a small fine-tuned model, or even a conventional classifier. A binary yes/no decision is easier to specify than a loosely defined zero-to-five scale. Groq's inference hardware is another option he raises for fast model-based checks. In Corrective RAG, the important output of the grader is simply whether the document is relevant enough to retain.

Corrective RAG flowchart showing vector-store retrieval, document grading, a yes-or-no relevance branch, web search, and an answer.Open full source frame
Corrective RAG grades retrieved documents before routing to generation or web search.

LangGraph also supports multi-agent layouts and cycles. The custom Corrective RAG exercise uses a branch, while its ReAct counterpart loops. Set a recursion_limit to bound graph execution rather than allowing an agent to continue indefinitely. Martin's recollection of a default near twenty is tentative; it is not a version-independent setting, and graph steps should not be assumed to equal complete agent cycles.

32:3332:37
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

32:33 · section reference included

Inspect what each step costs and remembers

Model choice is only one part of latency control; first locate the expensive step. LangSmith's trace view exposes both whole-graph duration and individual operations. In the displayed trace, retrieval takes less than one second and document grading takes about four seconds. Martin initially identifies the grader as GPT-4o, then corrects himself after inspecting the trace: it uses ChatFireworks. Because documents are graded independently, possible improvements include bulk grading, different prompts, or faster inference, with Groq, a self-hosted fine-tune, and GPT-3.5 among the options discussed.

A GPT-4o ReAct trace makes the memory mechanism visible. The initial request contains a system prompt, a user question, and tool definitions. The assistant requests document retrieval; the tool runs and returns documents. On the next model call, the history contains that entire exchange, so the assistant can select document grading as its next operation. The enlarged tool panel marks grade_document_retrieval as the selected call.

Enlarged Functions & Tools panel listing retrieve_documents, grade_document_retrieval, web_search, and generate_answer, with CALLED beside grade_document_retrieval.Open full source frame
The tool panel marks document grading as the selected call.

Accumulated history is useful but also creates another opportunity for tool-selection errors. On a follow-up question, the same history may contain enough evidence to answer directly without retrieving again. That is a plausible multi-turn behavior, but the workshop evaluation covers single-turn tasks; Martin has not supplied a multi-turn evaluation tutorial here.

37:5037:56
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

37:50 · section reference included

Prefer crisp feedback before adding another agent

Automatically generating tests during a request adds a second source of uncertainty: the model's solution may be wrong, but so may the tests judging it. AlphaCodium explores generated tests alongside existing ones. Martin has not tested that online approach himself and is cautious about putting it into production. His concrete ChatLangChain example is narrower: use a function call to extract imports from an answer's code block, test those imports in isolation, and return any error to the model for repair. Relevant documentation can be supplied during the retry.

An audience suggestion offers a middle ground: prepare known-good tests before generating the implementation. Martin extends that idea to a bank of curated tests selected dynamically by question type or documentation area. This can supply task-specific feedback while preserving a trustworthy test definition, and the same test bank can support regression checking on larger projects.

Replacing a single grading prompt with a sophisticated checker agent raises a similar trade-off. Martin has not tried the audience's cited frameworks. He sees room for offline experiments, but favors lightweight checks for latency-sensitive production use: the extra planning, tool use, and coordination would need to justify their cost.

45:1745:40
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

45:17 · section reference included

Record the path as application data

The evaluation section of rag-agent-testing.ipynb, also provided in Colab, uses two acceptable paths: retrieve, grade, generate; or retrieve, grade, web search, generate. A bounded cyclic application could enumerate additional steps, but the custom exercise itself does not cycle. The evaluator compares the observed sequence with those accepted sequences.

Each custom graph node appends its step name to state["steps"]. Evaluation then reads that list. The following Python expresses the same exact-path check using the tool names from the exercise:

python

ACCEPTED_PATHS = {
    (
        "retrieve_documents",
        "grade_document_retrieval",
        "generate_answer",
    ),
    (
        "retrieve_documents",
        "grade_document_retrieval",
        "web_search",
        "generate_answer",
    ),
}


def evaluate_trajectory(steps: list[str]) -> dict[str, int | str]:
    return {
        "key": "accepted_trajectory",
        "score": int(tuple(steps) in ACCEPTED_PATHS),
    }

For a ReAct agent, the record already exists in message history. The notebook's helper extracts tool calls from those messages instead of requiring every operation to maintain a separate step list.

51:4752:12
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

51:47 · section reference included

Give routing its own decision point

With two vector stores, A and B, a ReAct agent needs enough information to distinguish them. That information comes from the overall agent prompt and the descriptions of the retrieval tools: one store contains subject X, the other subject Y. The model uses those descriptions, the question, and the current context to choose a call.

The local RAG graph makes that decision explicit at entry. Its router chooses between web search and a vector store; the same pattern can extend to more destinations. Requiring routing to run first removes the separate problem of persuading an open-ended agent to choose the routing operation.

History-aware routing needs one more input. Pass conversation history through state and include it in the routing prompt beside the new question. The router can then decide whether to retrieve, search elsewhere, or answer from retained evidence. Martin corrects a detail while inspecting the example: its routing function is implemented as an edge rather than a node. The essential boundary is still the same—a specific decision reads the state before execution proceeds.

56:0256:25
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

56:02 · section reference included

Follow a document-grading state update

The custom RAG graph uses dictionary-like state with five fields: question, generation, search, documents, and steps. Each node receives that state, reads the values it needs, and returns updates. This keeps the question, retrieved material, generated answer, routing flag, and execution record available across the run.

Jupyter notebook showing a highlighted GraphState class with five typed fields beneath its explanatory docstring.Open full source frame
GraphState defines the question, generation, search, documents, and steps fields.

The grading node reads the question and documents, records its step, and grades each document yes/no. It retains relevant documents. If any document is irrelevant, it sets search to "yes" so the next routing decision can request supplemental web search. The update changes the evidence and the next-step flag; it does not itself execute a search.

For a concrete example, suppose the question is “What does graph state store?” The retrieved set contains doc-a, “Graph state stores the question and retrieved documents,” and doc-b, “The stadium opens at noon.” With grades yes and no, the node retains doc-a, removes doc-b, and marks search as pending. This Python keeps the grading decision injectable while expressing the state operation:

python

from typing import Callable, Literal, TypedDict


class Document(TypedDict):
    id: str
    text: str


class GraphState(TypedDict):
    question: str
    generation: str
    search: Literal["yes", "no"]
    documents: list[Document]
    steps: list[str]


def grade_documents(
    state: GraphState,
    judge: Callable[[str, str], Literal["yes", "no"]],
) -> GraphState:
    kept: list[Document] = []
    search: Literal["yes", "no"] = "no"
    for document in state["documents"]:
        if judge(state["question"], document["text"]) == "yes":
            kept.append(document)
        else:
            search = "yes"
    return {
        **state,
        "documents": kept,
        "search": search,
        "steps": [*state["steps"], "grade_document_retrieval"],
    }


state: GraphState = {
    "question": "What does graph state store?",
    "generation": "",
    "search": "no",
    "documents": [
        {"id": "doc-a", "text": "Graph state stores the question and retrieved documents."},
        {"id": "doc-b", "text": "The stadium opens at noon."},
    ],
    "steps": ["retrieve_documents"],
}

grades: dict[str, Literal["yes", "no"]] = {
    "Graph state stores the question and retrieved documents.": "yes",
    "The stadium opens at noon.": "no",
}
updated = grade_documents(state, lambda question, text: grades[text])

Named keys also make access straightforward: a node reads state["question"] instead of locating the appropriate message in a conversation stack.

Grading filters evidence and schedules search

Constructed example: The question, document IDs and texts, and assigned yes/no grades are teaching values. The operation and state fields follow the demonstrated grading mechanism.

Question — unchanged
What does graph state store?

Operation: Apply grades yes to doc-a and no to doc-b, retain relevant evidence, and mark supplemental search as pending.

question

Before: Before document grading
What does graph state store?
After: After grading, before web search · Unchanged
What does graph state store?

documents: doc-a

Before: Before document grading
Graph state stores the question and retrieved documents.
After: After grading, before web search · Unchanged
Graph state stores the question and retrieved documents.

documents: doc-b

Before: Before document grading
The stadium opens at noon.
After: After grading, before web search · Removed
Not present

search

Before: Before document grading
no
After: After grading, before web search · Changed
yes

steps

Before: Before document grading
["retrieve_documents"]
After: After grading, before web search · Changed
["retrieve_documents", "grade_document_retrieval"]

generation

Before: Before document grading
""
After: After grading, before web search · Unchanged
""
The grading node preserves relevant evidence and the question while preparing a later web-search branch.
1:02:461:03:03
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:02:46 · section reference included

Reduce the model's burden, then expand its evidence

The local notebook defaults to Llama 3 through Ollama. Martin reports that the 8B model runs adequately on his M2 with 32 GB of memory, while an attendee cautions against their experience running 70B on an M1. For Martin's setup, 8B tool calling is near the reliability boundary. The custom graph asks for constrained structured outputs rather than repeated selection among tools, which helps explain why he finds it more dependable locally than ReAct.

Document chunking presents a different burden. Small, semantically coherent chunks help a retriever match a question, but passing only those chunks to the generator can omit necessary context. Martin's preferred technique is to decouple the retrieval unit from the generation context: index small chunks, retain a link to their parent document in a document store, and fetch the fuller document after a chunk matches.

Context strategyAdvantageCost or risk
Only narrow matching chunksCompact inputMissing surrounding evidence
Parent documents selected through chunksBroader context around a matchExtra tokens
Entire corpus in contextMinimal retrieval machineryLatency, cost, access control, and auditability

Full documents can still be wasteful when they are huge. Martin argues for a middle ground: use retrieval to identify a useful scope, then avoid making generation depend on an excessively narrow chunk or a fragile choice of k.

1:06:221:06:33
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:06:22 · section reference included

Distinguish available evidence from evidence the model uses

An audience question asks whether the model actually uses information throughout its context window. Martin first approaches the issue from retrieval: does the supplied document contain the answer, and how much irrelevant material accompanies it? In his example, a 100-page document contains the answer on page 55. He calls its answer-presence recall one, while its precision is low because most of the document is unrelated. That describes the evidence supplied to the model; it does not establish that the model successfully used it.

When extra context is affordable, Martin favors retaining enough material to avoid missing the answer. A system that prioritizes minimizing irrelevant context may make a different trade-off. The parent-document approach gives developers a way to widen the evidence available for generation without abandoning retrieval entirely.

Context placement depends on whether the evidence changes. If every question concerns one fixed document, Martin suggests placing that document in the system prompt and dispensing with retrieval. He also anticipates context caching as a way to reduce repeated processing of fixed material. His ten-million-token caching and billing example is a hypothetical expectation in the recording, not an established capacity or pricing contract.

Dynamic evidence needs an index, potentially with routing across several stores. On later turns, retain retrieved documents in state or message history and let the router inspect them alongside the new question. If they already support the answer, generation can proceed without another retrieval. The useful distinction is between needing new evidence and needing a new response from evidence already available.

1:12:271:12:38
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:12:27 · section reference included

What belongs in the model, the graph, or an API?

The discussion then moves from supplying context to training a model to use it. Martin initially distinguishes RAG's in-context learning from fine-tuning weights to encode knowledge. The audience clarifies a different goal: fine-tune behavior so the model uses supplied evidence appropriately, regardless of whether it appears in a system prompt or a user message. This avoids conflating learning facts with learning how to handle context.

Martin is wary of repeatedly fine-tuning changing facts, but sees behavioral tuning as promising. RAFT provides a relevant example: learning to answer from retrieved material while ignoring distractor documents. He has not tried it himself. That learned filtering behavior overlaps with the workshop's relevance checks, without replacing the entire routing and correction graph. His practical concern is maintenance: adopting a new base model may require repeating the specialized tuning, whereas explicit orchestration can retain the same application structure.

The audience pushes the argument further: providers could learn these behaviors from the long tail of tool and context usage, much as a data engine improves a deployed model. Martin cites OpenAI's Rockset acquisition as a possible signal toward managed retrieval, then considers what might disappear behind an API. That is a forecast in this discussion. Managed functionality can reduce implementation work, but developers may still need transparency and control over application-specific relevance criteria. Multimodality is his example of functionality many teams already accept behind an API; retrieval policy is harder to generalize.

1:20:121:20:25
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:20:12 · section reference included

A large window does not guarantee multi-fact retrieval

The long-context concern becomes concrete in Martin's Multi Needle in a Haystack study with Greg Kamradt. Using GPT-4 Turbo in a single turn, they insert one, three, or ten secret pizza ingredients into Paul Graham essays, with contexts extending to 120,000 tokens. The model must recover the ingredients; another condition adds reasoning to retrieval. Martin reports roughly 60% retrieval with ten needles in this GPT-4 Turbo study.

The heatmap explores context lengths from about 1,000 to 120,000 tokens and the positions of the inserted facts. Martin observes failures toward the beginning of longer contexts. He proposes recency bias as an explanation: recent tokens are often informative for next-token prediction, potentially encouraging attention to recent material. The observed positional failures are evidence; that causal explanation remains a hypothesis.

Slide titled “Retrieval is not guaranteed, reasoning harder than retrieval,” with grouped green and red bars and a retrieval heatmap.Open full source frame
Multi-fact retrieval results compare retrieval with retrieval plus reasoning.

The study changes how to read a clean single-needle chart. Difficulty depends on how many facts must be recovered, whether the task also requires reasoning, and how distinguishable the facts are from their surroundings. Pizza ingredients embedded in essays are conspicuous; closely related facts can be harder to separate. Martin therefore cautions against treating a large advertised context window as assurance that an entire corpus can be supplied and reliably used.

1:28:141:28:18
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:28:14 · section reference included

Reduce the tool-selection problem before scaling it

A large tool inventory creates another selection problem. Martin reports difficulty even with small sets and does not establish a numerical cutoff. He outlines three ways to reduce the burden:

  • Explicit graph routing: Put tools inside known nodes and route to them through application logic.
  • Specialist agents: Give each sub-agent a narrower task and a smaller tool set.
  • Semantic tool retrieval: Embed detailed natural-language tool descriptions and retrieve a relevant subset for the current request.

The third option applies the retrieval pattern to tools rather than documents.

For semantic tool retrieval, Martin proposes verbose, high-quality summaries of what each tool does, indexed for similarity search against the query. He has not tested that approach yet. Likewise, his advice on specialist agents points to LangGraph's multi-agent notebooks and their cited papers; his own experimentation is limited. He mentions Devin and OpenDevin tentatively as further software-agent examples, rather than presenting a verified architecture comparison.

Before adding any of this, ask whether the application needs variable control flow. A static RAG chain remains a reasonable starting point. Routing among vector stores, bypassing an irrelevant index for an out-of-domain question, and correcting poor retrieval are concrete reasons to add agentic decisions. Martin's preference for constrained graphs reflects these needs and his experience at LangChain, rather than a requirement that every RAG application become an agent.

1:32:491:32:56
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:32:49 · section reference included

Grow the evaluation set from real failures

A domain-specific evaluation set rarely appears fully formed. Start with canonical questions the application should answer. To bootstrap additional examples, Martin suggests giving one document at a time to a capable model such as Sonnet or GPT-4o and requesting three high-quality question-answer pairs, then auditing them. Restricting generation to one document makes the proposed examples easier to inspect. A small useful set can reveal failures before a comprehensive one exists.

Once the application is serving users, connect online monitoring to offline testing:

  1. Run reference-free retrieval, relevance, and grounding checks on production runs.
  2. Flag cases where the application appears to perform poorly.
  3. Audit the failure and establish a corrected reference answer.
  4. Add the resulting question-answer pair to the offline evaluation set.
  5. Use that expanded set when assessing the next application change.

The failure becomes durable test data only after curation; an online judge's flag is the start of that work.

Martin connects this loop to Karpathy's Tesla data-engine framing and Operation Vacation. In model training, labeled failures become training examples. In this application workflow, they expand the test set used to guide improvements. Both systems turn production experience into a more demanding definition of success.

1:41:581:42:04
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:41:58 · section reference included

Apply the same boundaries to SQL and reports

A text-to-SQL application can produce more rows than a model can use in a final explanation. In the SQL-agent tutorial Martin opens, attributed to teammate Ankush, the prompt instructs the model to limit results—the shown example uses LIMIT 5—and a query-check node reviews the proposed SQL before execution. This places a check between query generation and a potentially oversized result.

The follow-up question concerns a different issue: how a WHERE clause identifies the intended value in a high-cardinality column. Martin does not offer a specific value-resolution technique. He limits his advice to prompting and query review and acknowledges that detailed text-to-SQL behavior is outside his expertise. A row limit bounds output size; it does not resolve which value a user meant.

For long-form generation, Martin opens the STORM example for wiki articles. The workflow develops related topics and uses editor and expert roles, with experts conducting web research before article production. The notebook includes the graph and a generated wiki. He also offers a simpler starting point: retrieve evidence, then use a distillation prompt with explicit instructions for the final report's structure and format. The elaborate multi-agent workflow is one option, not a prerequisite for producing a useful document.

1:46:271:46:35
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:46:27 · section reference included

Separate clarification, model capability, and persistence

A report or chat workflow may discover partway through that essential domain information is missing. Passing control back to a user or reviewer requires both a mechanism for human intervention and a decision that intervention is needed. Martin locates LangGraph's human-in-the-loop documentation and recommends the team's customer-support example for multi-turn interaction. He has not implemented this himself and does not demonstrate a detector for missing information.

Asked about models trained specifically for agentic reasoning, Martin describes the pattern he has encountered: high-capacity generalist models, accurate tool calling, and task-specific prompts. ReAct is an orchestration and prompting approach with an interchangeable underlying model. He is less familiar with fine-tuning for a particular agent architecture.

Checkpointing addresses another part of the system: preserving state and returning to a saved point. Martin describes state that can persist across sessions and checkpoints from which an agent can be revisited. Broader deployment support is still previewed as forthcoming in the recording; this exchange explains the purpose of persistence without walking through a deployed implementation.

1:53:101:53:31
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:53:10 · section reference included

Distinguish execution errors from wrong decisions

An audience example asks what happens if an agent chooses YouTube upload when download was intended. Martin explains the ReAct error path: a tool node with fallbacks returns an execution error into message history, and the assistant can use that feedback to correct its next call. He inspects the notebook but does not find a failing trace to demonstrate live. This mechanism handles errors that a tool actually reports; it does not establish detection of a valid upload call that was simply the wrong action.

Structured-output validation is a narrower, more directly detectable case. An attendee describes Python, Pydantic, and Jason Liu's Instructor for validating outputs and retrying after schema errors. Martin agrees that malformed structured data is well suited to this feedback loop, while the general tool fallback also handles other execution failures. Schema validity and choosing the right operation remain separate properties.

The integration boundary is the node, not the model wrapper. LangGraph and LangSmith do not require LangChain. Martin proposes replacing the retrieval grader's with_structured_output call and grade schema with Instructor, then keeping the surrounding graph unchanged. It is offered as a workshop experiment rather than shown as a completed integration.

2:01:112:01:22
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:01:11 · section reference included

Relevance also depends on document authority and scope

The remaining bottleneck in ChatLangChain is often retrieval itself. Its documentation includes canonical guides and a long tail of integration pages with uneven depth. Raw semantic similarity may return a strong how-to guide alongside several incidental passages. All can match the question's wording while differing substantially in how useful they are for answering it.

Martin describes a design under consideration: retrieve a larger initial candidate set, then rerank for document importance. A general question about agents should favor a well-developed overview over an integration page that happens to mention building an agent. Larger context windows ease the problem of supplying enough text, but they do not decide which sources deserve priority.

The proposed ranking uses the documentation's structure. Tutorials, how-to guides, and conceptual guides receive higher importance for general questions. An integration-specific request—such as one about the Zep Cloud retriever—should still promote the corresponding integration page. A router could classify the question as general or specific, while heuristic quality or importance scores travel in indexed chunk metadata. Martin presents these as planned choices, not deployed results.

Follow-up questions can expose a scope problem even when the original retrieval was reasonable. An attendee rewrites a question to request a deeper explanation but gets the same chunks and nearly the same answer. Martin reframes the task: if the first request retrieved a page and the next needs a chapter, change the retrieval operation. Use chapter or section metadata to fetch the larger unit, retrieve the entire section, or increase k. Rewriting the query alone does not ensure broader evidence.

2:06:482:06:59
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:06:48 · section reference included

Test the branch—and recognize the missing interaction

The local exercise's sports questions serve a deliberate purpose. The vector store contains three blog posts about agents, prompting, and adversarial examples, so the two sports questions fall outside its domain and are intended to force the web-search branch. They test whether the application adapts when its local evidence cannot answer the request. A late participant exchange also describes much faster local execution after switching from 70B to 8B, reinforcing the practical importance of matching the exercise to available hardware.

The final question sharpens a distinction that ordinary chat history does not resolve. ReAct can retain follow-up messages, and the custom graph is primarily a single-turn question-answering example. But the attendee wants the agent to initiate a clarification question when details are missing. Neither supplied example implements that behavior. Martin points again to the more complex customer-support agent and commits to adding a simpler tutorial. The next capability is therefore specific: deciding that the available information is insufficient, asking for what is missing, and continuing the task after the user responds.

2:16:442:16:57
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:16:44 · section reference included

Resources

From the talk

Updates since the talk

Transcript download
Full transcript

Loading transcript…