AI Engineer Summit 2023

Building Context-Aware Reasoning Applications with LangChain and LangSmith

Loading slides…

Read the talk

Building Context-Aware Reasoning Applications with LangChain and LangSmith

A useful assistant needs more than a model: it needs relevant context, an appropriate execution structure, and tools for inspecting, evaluating, and improving the whole system.

From a talk by Harrison Chase

A weather question exposes the missing system

Ask ChatGPT about the current weather in San Francisco, and the opening example returns an explanation that it cannot access real-time weather data. The question is simple; the missing capability sits outside the model. A standalone language model does not inherently fetch current information, execute the code it generates, or retain previous interactions. This distinction matters in the talk’s 2023 setting: ChatGPT plugins had already introduced browsing and code execution as surrounding product capabilities. The limitation concerns what the model can do by itself.

ChatGPT conversation asking about San Francisco weather, with a response explaining that it cannot access real-time weather data.Open full source frame
ChatGPT cannot answer a question about current San Francisco weather.

Useful assistants require language models to operate inside a larger system. That system must supply the relevant context, enable reasoning about it, and support an appropriate response. A seemingly magical product experience therefore depends on substantial engineering underneath. Closing that gap between the experience and its implementation is the goal Harrison Chase sets for LangChain.

0:160:33
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:16 · section reference included

Tell the model, show it, and give it evidence

Context is the information a model needs to decide what to do. Greater model capability cannot compensate for information that the application never supplies. The first form is instruction prompting: tell the model how to respond to particular inputs or situations. Think of an employee handbook handed to someone on their first day. It describes expected behavior before the employee encounters each scenario. Chase expects stronger models to make these zero-shot instructions increasingly effective.

When behavior is difficult to describe, few-shot examples show it directly. A particular tone may be easier to demonstrate than to capture in a verbal specification. Structured output offers another case: a simple format can be described, but increasingly complicated formats benefit from concrete examples of valid responses. Instructions and demonstrations both communicate how the model should behave; examples are especially useful where the description becomes cumbersome.

Retrieval-augmented generation supplies a different kind of context: the material on which the answer should be based. Its basic sequence is:

  1. Receive the user’s question.
  2. Use a retrieval strategy to find relevant material.
  3. Pass that material and the question to the model.
  4. Instruct the model to answer from the supplied context.

This resembles an open-book test. Behavioral instructions establish how to answer; the retrieved text provides the reference material for answering.

Fine-tuning moves learning from examples into updates to the model’s weights. In Chase’s account, this was an emerging practice with useful overlap with few-shot prompting, particularly for tone and structured-data parsing. His contrast between three examples in a prompt and ten thousand training examples illustrates the difference in how many demonstrations can shape behavior; it is not an experiment or a recommended training-set size. When a specific output behavior is easier to show repeatedly than to describe, fine-tuning offers another way to teach it.

1:491:57
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:49 · section reference included

From generating an output to choosing a step

Once the model has context, the next design question is how much control it should have over execution. Ordinary code specifies the steps to run. Adding a single language-model call changes who determines the output: the application supplies context, the model generates a response, and the application returns it to the user. The model does not yet select actions.

A chain extends this into a known sequence of calls. It might connect several model calls, or run a model call, an API call, and another model call. This allows a complex task to be divided into smaller components. It also allows knowledge to enter partway through execution: an earlier model output can inform what the application retrieves before a later model call. The information changes dynamically, but the sequence remains prescribed in advance.

A router gives the model a new responsibility: choosing which step to take. The available branches can differ in several ways:

  • Prompt selection: Choose a prompt suited to a math problem or one suited to writing an English essay.
  • Model selection: Choose a model according to the question’s demands. Chase uses Claude for long context and GPT-4 for reasoning as contemporary examples, not as a measured comparison between specified model versions.
  • Tool selection: Choose which tool to call and generate its input.

The defining boundary here is that routing has no cycle. The model selects a branch, but it does not repeatedly return to the same decision point.

5:175:28
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:17 · section reference included

Loops add repeated decisions; autonomy removes scaffolding

Adding a loop produces a pattern commonly called an agent. Conceptually, it is a while loop in which the model chooses steps and decides whether to continue. Ending the loop finishes the task and returns a result to the user; continuing sends execution around again. The model now determines both outputs and successive actions.

The more autonomous style Chase associates with the original Auto-GPT removes further constraints on the available action sequences. One possibility is to expand the available tools during execution. Voyager, his example, accumulates reusable executable skills over time. Those skills enlarge what the agent can do; accumulating them is not a model-weight update.

Another change concerns the structure around the loop. A scaffolded system can explicitly separate plan, execute, and validate: ask the model for a plan, carry it out, then check the result, potentially with another model call. A less scaffolded agent must decide for itself when to plan, when to act, and when to validate. Those transitions become implicit model responsibilities instead of distinct sequences encoded in the application.

The architecture table brings these distinctions together: control shifts from code-defined outputs and sequences toward model-selected actions and, eventually, model-influenced action possibilities. Its labels distinguish a scaffolded state machine from the less constrained agent style, even though agent is also commonly used for the broader looping pattern. Chase treats this taxonomy as provisional in a rapidly changing field, rather than a permanent classification.

Six-row table comparing Code, LLM Call, Chain, Router, State Machine, and Agent across output decisions, step selection, and available step sequences, using code and LLM icons.Open full source frame
Code and LLM calls begin a progression toward greater model control.
5:178:01
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:01 · section reference included

Choose the architecture, then inspect what actually ran

The weather assistant becomes an engineering problem when its internal execution is exposed. The displayed debugging interface contains an expanded AgentExecutor trace, a get_weather tool call, the weather-question input, a completed response, and a Success status. Seeing both the execution tree and the result makes the hidden work behind an assistant response inspectable.

Dark debugging interface with an expanded execution tree including get_weather, a weather-question input, a completed response, and a Success status.Open full source frame
An AgentExecutor trace shows the weather tool call and resulting response.

The first challenge is orchestration: selecting the appropriate structure for the application. More model control is not automatically better.

StructureUseful property
ChainControl over the sequence of steps
AgentDynamic responses to unexpected inputs and edge cases

The choice depends on the task, and experimenting with alternative structures helps expose their tradeoffs. LangChain’s initial role was to make that prototyping easier.

LangSmith addresses the accompanying visibility problem. As an application becomes more complex, its final answer tells you less about how it reached that answer. Inspecting the exact sequence of tools and model calls makes it possible to understand and debug the execution itself.

10:2110:32
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:21 · section reference included

Inspect the data and the fully assembled prompt

Supplying the right context entails familiar data-engineering work: loading, transforming, and transporting data, then observing what moved where. LangChain provides modules for loading and transformation; LangSmith helps reveal what the data looks like when it reaches the model. Two debugging questions remain distinct: did the vector store return the right documents, and did the application transform and format those documents so their contents are clear to the model? Correct retrieval can still be undermined by poor presentation.

Prompt engineering likewise requires looking at the actual input, not just the template. A fully assembled prompt may combine system instructions, few-shot examples, retrieved context, chat history, and previous agent steps. The first call is relatively easy to inspect. After several intermediate steps, the prompt depends on accumulated application state, making it harder to reconstruct by looking at the initial inputs alone.

The playground workflow makes a particular call available for direct experimentation:

  1. Locate the model call inside the execution trace.
  2. Open that call in the playground with its assembled prompt.
  3. Edit the instructions and inspect the resulting response.
  4. Swap model providers to investigate whether another model handles that input better.

This lets prompt iteration begin at the point where behavior needs investigation, including deep inside a multistep application.

12:2312:32
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

12:23 · section reference included

A working MVP still needs evaluation data

Evaluation encounters two shortages: data and useful metrics. Traditional machine-learning development generally begins with a dataset, leaving examples available when evaluation starts. A language-model application can reach a working MVP through zero-shot behavior without collecting an application dataset at all. That makes initial development easier while leaving a gap in the evaluation process.

Building an evaluation dataset therefore becomes deliberate work. Examples can come from manual labeling, selected production traffic, or generation with language models. The dataset is something to build up and iterate on as the application develops, rather than something the MVP necessarily supplied.

Metrics are difficult because large, unstructured responses do not fit many traditional quantitative measures well. Chase sees qualitative inspection—often called a vibe check—as useful for developing intuition about behavior. Exposing model inputs alongside outputs helps explain what happened, even when that inspection does not yield an objective score.

For more systematic evaluation, Chase points to model-assisted judgments and production feedback:

  • LLM-assisted evaluation: Use a language model to evaluate generated outputs.
  • Direct feedback: Record explicit user judgments such as thumbs up or thumbs down.
  • Indirect feedback: Observe behavior such as clicking a suggested link or responding with confusion. These signals may indicate success or failure, but they are suggestive rather than definitive.

Tracking feedback over time and comparing application variants with conventional A/B testing software provides an online view of how the system performs for users.

14:2214:32
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

14:22 · section reference included

The engineering work crosses professional boundaries

As these systems grow, their construction spans AI engineers, data engineers, data scientists, and product managers. The right mix of skills for the emerging AI engineer role remains unsettled. Context delivery often needs data-engineering expertise, while the desired reasoning behavior is often expressed through prompts. Product managers and subject matter experts can be especially effective at specifying precisely how an application should behave. Enabling those technical and nontechnical contributors to work together is itself a tooling challenge—one Chase presents as open, not solved.

New model capabilities will continue changing the design space. Chase points to the conference’s GPT-4V demonstration as a sign of that change. The closing weather-assistant illustration returns to the original problem with a richer response: it supplies weather information and offers a ride and a coffee order. Those offers depict possible assistance, not evidence that a ride was booked or an order placed.

Blue slide with a weather question, a hand holding a phone, sparkles, and a response giving San Francisco weather and offering an Uber and a coffee order.Open full source frame
A weather assistant example adds suggestions for a ride and coffee.

Moving from that illustrated experience to a dependable application still requires engineering: connecting capabilities, supplying context, controlling execution, and making the system understandable enough to improve. Supporting that work is the enduring role Chase assigns to the surrounding tools as models continue to change.

17:0117:13
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

17:01 · section reference included

Resources

From the talk

Updates since the talk

Transcript download
Full transcript

Loading transcript…