AI Engineer Summit 2023
Building Context-Aware Reasoning Applications with LangChain and LangSmith
Loading slides…
Read the talk
Building Context-Aware Reasoning Applications with LangChain and LangSmith
A useful assistant needs more than a model: it needs relevant context, an appropriate execution structure, and tools for inspecting, evaluating, and improving the whole system.
From a talk by Harrison Chase
A weather question exposes the missing system
Ask ChatGPT about the current weather in San Francisco, and the opening example returns an explanation that it cannot access real-time weather data. The question is simple; the missing capability sits outside the model. A standalone language model does not inherently fetch current information, execute the code it generates, or retain previous interactions. This distinction matters in the talk’s 2023 setting: ChatGPT plugins had already introduced browsing and code execution as surrounding product capabilities. The limitation concerns what the model can do by itself.
Useful assistants require language models to operate inside a larger system. That system must supply the relevant context, enable reasoning about it, and support an appropriate response. A seemingly magical product experience therefore depends on substantial engineering underneath. Closing that gap between the experience and its implementation is the goal Harrison Chase sets for LangChain.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Tell the model, show it, and give it evidence
Context is the information a model needs to decide what to do. Greater model capability cannot compensate for information that the application never supplies. The first form is instruction prompting: tell the model how to respond to particular inputs or situations. Think of an employee handbook handed to someone on their first day. It describes expected behavior before the employee encounters each scenario. Chase expects stronger models to make these zero-shot instructions increasingly effective.
When behavior is difficult to describe, few-shot examples show it directly. A particular tone may be easier to demonstrate than to capture in a verbal specification. Structured output offers another case: a simple format can be described, but increasingly complicated formats benefit from concrete examples of valid responses. Instructions and demonstrations both communicate how the model should behave; examples are especially useful where the description becomes cumbersome.
Retrieval-augmented generation supplies a different kind of context: the material on which the answer should be based. Its basic sequence is:
- Receive the user’s question.
- Use a retrieval strategy to find relevant material.
- Pass that material and the question to the model.
- Instruct the model to answer from the supplied context.
This resembles an open-book test. Behavioral instructions establish how to answer; the retrieved text provides the reference material for answering.
Fine-tuning moves learning from examples into updates to the model’s weights. In Chase’s account, this was an emerging practice with useful overlap with few-shot prompting, particularly for tone and structured-data parsing. His contrast between three examples in a prompt and ten thousand training examples illustrates the difference in how many demonstrations can shape behavior; it is not an experiment or a recommended training-set size. When a specific output behavior is easier to show repeatedly than to describe, fine-tuning offers another way to teach it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From generating an output to choosing a step
Once the model has context, the next design question is how much control it should have over execution. Ordinary code specifies the steps to run. Adding a single language-model call changes who determines the output: the application supplies context, the model generates a response, and the application returns it to the user. The model does not yet select actions.
A chain extends this into a known sequence of calls. It might connect several model calls, or run a model call, an API call, and another model call. This allows a complex task to be divided into smaller components. It also allows knowledge to enter partway through execution: an earlier model output can inform what the application retrieves before a later model call. The information changes dynamically, but the sequence remains prescribed in advance.
A router gives the model a new responsibility: choosing which step to take. The available branches can differ in several ways:
- Prompt selection: Choose a prompt suited to a math problem or one suited to writing an English essay.
- Model selection: Choose a model according to the question’s demands. Chase uses Claude for long context and GPT-4 for reasoning as contemporary examples, not as a measured comparison between specified model versions.
- Tool selection: Choose which tool to call and generate its input.
The defining boundary here is that routing has no cycle. The model selects a branch, but it does not repeatedly return to the same decision point.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Loops add repeated decisions; autonomy removes scaffolding
Adding a loop produces a pattern commonly called an agent. Conceptually, it is a while loop in which the model chooses steps and decides whether to continue. Ending the loop finishes the task and returns a result to the user; continuing sends execution around again. The model now determines both outputs and successive actions.
The more autonomous style Chase associates with the original Auto-GPT removes further constraints on the available action sequences. One possibility is to expand the available tools during execution. Voyager, his example, accumulates reusable executable skills over time. Those skills enlarge what the agent can do; accumulating them is not a model-weight update.
Another change concerns the structure around the loop. A scaffolded system can explicitly separate plan, execute, and validate: ask the model for a plan, carry it out, then check the result, potentially with another model call. A less scaffolded agent must decide for itself when to plan, when to act, and when to validate. Those transitions become implicit model responsibilities instead of distinct sequences encoded in the application.
The architecture table brings these distinctions together: control shifts from code-defined outputs and sequences toward model-selected actions and, eventually, model-influenced action possibilities. Its labels distinguish a scaffolded state machine from the less constrained agent style, even though agent is also commonly used for the broader looping pattern. Chase treats this taxonomy as provisional in a rapidly changing field, rather than a permanent classification.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Choose the architecture, then inspect what actually ran
The weather assistant becomes an engineering problem when its internal execution is exposed. The displayed debugging interface contains an expanded AgentExecutor trace, a get_weather tool call, the weather-question input, a completed response, and a Success status. Seeing both the execution tree and the result makes the hidden work behind an assistant response inspectable.
The first challenge is orchestration: selecting the appropriate structure for the application. More model control is not automatically better.
| Structure | Useful property |
|---|---|
| Chain | Control over the sequence of steps |
| Agent | Dynamic responses to unexpected inputs and edge cases |
The choice depends on the task, and experimenting with alternative structures helps expose their tradeoffs. LangChain’s initial role was to make that prototyping easier.
LangSmith addresses the accompanying visibility problem. As an application becomes more complex, its final answer tells you less about how it reached that answer. Inspecting the exact sequence of tools and model calls makes it possible to understand and debug the execution itself.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Inspect the data and the fully assembled prompt
Supplying the right context entails familiar data-engineering work: loading, transforming, and transporting data, then observing what moved where. LangChain provides modules for loading and transformation; LangSmith helps reveal what the data looks like when it reaches the model. Two debugging questions remain distinct: did the vector store return the right documents, and did the application transform and format those documents so their contents are clear to the model? Correct retrieval can still be undermined by poor presentation.
Prompt engineering likewise requires looking at the actual input, not just the template. A fully assembled prompt may combine system instructions, few-shot examples, retrieved context, chat history, and previous agent steps. The first call is relatively easy to inspect. After several intermediate steps, the prompt depends on accumulated application state, making it harder to reconstruct by looking at the initial inputs alone.
The playground workflow makes a particular call available for direct experimentation:
- Locate the model call inside the execution trace.
- Open that call in the playground with its assembled prompt.
- Edit the instructions and inspect the resulting response.
- Swap model providers to investigate whether another model handles that input better.
This lets prompt iteration begin at the point where behavior needs investigation, including deep inside a multistep application.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A working MVP still needs evaluation data
Evaluation encounters two shortages: data and useful metrics. Traditional machine-learning development generally begins with a dataset, leaving examples available when evaluation starts. A language-model application can reach a working MVP through zero-shot behavior without collecting an application dataset at all. That makes initial development easier while leaving a gap in the evaluation process.
Building an evaluation dataset therefore becomes deliberate work. Examples can come from manual labeling, selected production traffic, or generation with language models. The dataset is something to build up and iterate on as the application develops, rather than something the MVP necessarily supplied.
Metrics are difficult because large, unstructured responses do not fit many traditional quantitative measures well. Chase sees qualitative inspection—often called a vibe check—as useful for developing intuition about behavior. Exposing model inputs alongside outputs helps explain what happened, even when that inspection does not yield an objective score.
For more systematic evaluation, Chase points to model-assisted judgments and production feedback:
- LLM-assisted evaluation: Use a language model to evaluate generated outputs.
- Direct feedback: Record explicit user judgments such as thumbs up or thumbs down.
- Indirect feedback: Observe behavior such as clicking a suggested link or responding with confusion. These signals may indicate success or failure, but they are suggestive rather than definitive.
Tracking feedback over time and comparing application variants with conventional A/B testing software provides an online view of how the system performs for users.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The engineering work crosses professional boundaries
As these systems grow, their construction spans AI engineers, data engineers, data scientists, and product managers. The right mix of skills for the emerging AI engineer role remains unsettled. Context delivery often needs data-engineering expertise, while the desired reasoning behavior is often expressed through prompts. Product managers and subject matter experts can be especially effective at specifying precisely how an application should behave. Enabling those technical and nontechnical contributors to work together is itself a tooling challenge—one Chase presents as open, not solved.
New model capabilities will continue changing the design space. Chase points to the conference’s GPT-4V demonstration as a sign of that change. The closing weather-assistant illustration returns to the original problem with a richer response: it supplies weather information and offers a ride and a coffee order. Those offers depict possible assistance, not evidence that a ride was booked or an order placed.
Moving from that illustrated experience to a dependable application still requires engineering: connecting capabilities, supplying context, controlling execution, and making the system understandable enough to improve. Supporting that work is the enduring role Chase assigns to the surrounding tools as models continue to change.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
The original paper on a Minecraft agent that accumulates reusable executable skills.
AutoGPT's project repository, with the original standalone agent retained under classic/ alongside the newer platform.
Further reading
The 2023 introduction to tracing, playground debugging, evaluation datasets and production feedback.
Implementation and setup instructions for Voyager's Minecraft environment and skill library.
Chase's subsequent explanation of chains, routers, state machines and agent loops, including their control tradeoffs.
Updates since the talk
A practical guide to creating datasets, defining evaluators and comparing experiments with the LangSmith SDK.
Full transcript
Loading transcript…