AI Engineer World's Fair 2026

Context Engineering in 2026: Compaction, Memory & Cost

Loading slides…

Read the talk

Context Engineering in 2026: Compaction, Memory & Cost

An AI tutor’s compaction experiments show how cached history, retrieval, and hardware limits change what belongs in the context window—and when removing it becomes necessary.

From a talk by Louis-François Bouchard, Omar Solano and Samridhi Vaid

Before you start: Familiarity with LLM tool calls, token-based pricing, and basic retrieval-augmented generation will help you follow the architecture and experiments.

The instruction was there. Why did the agent ignore it?

An agent receives an explicit instruction not to push to main. Later in the same session, it pushes anyway. The user responds furiously, hoping another correction will finally stick. Louis-François Bouchard opens with this familiar failure: as a session accumulates context, the agent can become worse at following instructions that were clear at the start. Switching models or coding harnesses may leave the underlying problem untouched.

Slide titled “Every long agent session, ever.” shows an instruction against pushing to main, the agent later doing so, the user's reaction, and the conclusion that its context deteriorated.Open full source frame
A long agent session ends with a push to main despite an explicit prohibition.

For the Towards AI Academy team, that frustration has a direct product consequence. Its AI tutor helps students work through lessons; an unhelpful debugging session can become a refund request. The team therefore built experiments around improving the tutor’s answers while reducing its operating cost. The experiment Space and open-source tutor provide the accompanying material. Bouchard introduces the context techniques, Omar Solano walks through the application and initial evaluations, and Samridhi Vaid extends the tests to cheaper models, longer inputs, and local inference.

The tutor serves technical courses, so a useful answer must satisfy several requirements at once:

  • Grounding: answer from the course material rather than relying solely on model knowledge.
  • Student and lesson scope: use the course and lesson relevant to the current student.
  • Session continuity: sustain a long debugging or iterative help session.
  • Code support: work with code and the errors students encounter.
  • Responsiveness: keep latency low enough that the tutor remains pleasant to use.

Real student interactions provide the baseline for testing these requirements.

0:340:55
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:34 · section reference included

One window, several kinds of context

Two constraints create different engineering problems. First, the context window is finite: instructions, lessons, code, and conversation all compete for the same space. Second, a model call is stateless unless the surrounding application supplies the state. Reopening the tutor does not automatically restore what a student discussed previously. Context management handles information within a session; memory carries information across sessions. The workshop’s experiments concentrate on getting the first problem right.

“Problems…” slide lists a finite context window and a stateless model, then distinguishes context management within a session from memory across sessions.Open full source frame
Finite context and stateless models create two distinct management problems.

A tutor request contains much more than the latest question. It includes the system prompt, tool definitions, chat history, previous tool outputs, retrieved course chunks, and the student’s new message. That message may itself contain substantial code or error logs. In this application, old tool outputs are the main source of growth: retrieved chunks and tool-call/result pairs accumulate as the conversation proceeds.

Bouchard connects this growth to context rot, suggesting that training a model to recover facts inserted into long corpora does not necessarily teach it to understand the whole evolving conversation. Quality is only one concern, however. Each follow-up resends earlier input, creating repeated token charges, and processing a longer prompt can increase time to first token, or TTFT. The experiments therefore ask about quality, spending, and responsiveness together.

5:145:33
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:14 · section reference included

Start with inexpensive compaction

Compaction aims to retain the smallest context that still supports the answer. Everything else can be dropped or stored outside the active window. This does not require a language model. Consider a tool that usually returns ten lines but suddenly emits three hundred. Preserve the beginning and end, insert an explicit truncation marker, and let the agent request more information if it needs the omitted material.

A small Python function captures that head-and-tail policy. Here, keeping five lines at each end is a tunable choice; the marker tells the agent that the middle is missing.

python

def truncate_tool_output(text: str, head: int = 5, tail: int = 5) -> str:
    if head < 1 or tail < 1:
        raise ValueError("head and tail must both be positive")

    lines = text.splitlines()
    if len(lines) <= head + tail:
        return text

    omitted = len(lines) - head - tail
    marker = (
        f"[Truncated {omitted} lines; call the tool again "
        "with a narrower request if needed.]"
    )
    return "\n".join(lines[:head] + [marker] + lines[-tail:])

Two other inexpensive policies are a sliding window, retaining the last N user turns, and tool-specific clearing, removing outputs whose contents are no longer useful. The appropriate window and clearing rules depend on what students ask and which tools the tutor exposes.

Model-assisted methods spend tokens now to reduce later input. A smaller model—even one running locally on a MacBook—may be sufficient for this work.

MethodWhat changes
Selective retentionKeep or discard material according to the conversation’s direction
Continuous summarizationReplace earlier turns with a running summary
Summary and resetSummarize the conversation at a limit, then restart from that summary
Delta summarizationUpdate an existing summary with newly learned information

Bouchard uses Claude Code as an example of summarizing and resetting at a context limit, then describes delta summaries as useful when subagents pass their findings back to a main agent. The tutor itself uses one agent: its workload did not require the additional coordination.

8:519:02
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:51 · section reference included

Keep the source outside the window

Offloading provides another option: retain information in files or a retrieval system and load it when needed. Ordinary retrieval-augmented generation, or RAG, already supports this pattern. The team also compared GraphRAG with its existing RAG pipeline. Bouchard reports tied results on their real-user evaluations, with greater setup cost for GraphRAG. A larger corpus with important relationships between topics might justify the graph, but this tutor did not show that benefit.

Saving the underlying conversation locally or on a server makes offloading reversible: information leaves the active prompt without disappearing. The tutor’s approach draws on Karpathy’s LLM Wiki. Cross-linked chunks point back to raw data files, and an index lists the chunks with short descriptions. Bouchard estimates that the index exposed to the agent is approximately 450 tokens.

Retrieval proceeds in stages:

  1. Inspect the index when a question appears related to stored student information.
  2. Read the relevant chunk.
  3. Answer from that chunk if it is sufficient.
  4. Follow its pointer to the raw data if the task requires more detail.

This is progressive disclosure: a simple question pulls little context, while a complex question can pull more. The same principle applies to skills. Small, precise skills can reference one another and load individually; when subagents are useful, each can receive only the skill context its task requires.

11:5512:08
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:55 · section reference included

A shorter prompt can cost more

Compaction becomes less straightforward when the provider caches prompt computation. Instead of processing every previously submitted token from scratch, the provider can reuse stored state, including the KV cache. Reused input receives a discounted rate, while new input is billed at the uncached rate. In the workshop’s historical DeepSeek example, cached input was 50 times cheaper than uncached input. That compares DeepSeek’s own two input rates, not DeepSeek against Gemini; it is not a current price quote.

Summarizing a conversation changes the tokens that would otherwise be reused. The rewritten portion must be processed as new input, although an unchanged earlier prefix may remain reusable. The economic comparison is therefore between a long discounted history and a short full-price replacement—not simply between their token counts.

The workshop’s compression threshold follows from a simplified one-request calculation. Let H be the retained history’s token count, S the replacement summary’s token count, and p the uncached input price per token:

Ccached history=Hp50Cnew summary=SpCnew summary<Ccached history  ⟺  S<H50.\begin{aligned} C_{\text{cached history}} &= H\frac{p}{50} \\ C_{\text{new summary}} &= Sp \\ C_{\text{new summary}} < C_{\text{cached history}} &\iff S < \frac{H}{50}. \end{aligned}

Under those assumptions, the replacement must be less than one-fiftieth as long to reduce that request’s input cost. This assumes all retained history is cached and the replacement is uncached; summary generation and later reuse are outside the calculation.

Bouchard describes caching savings of around 90% as common in the APIs he discusses, with larger discounts in some cases. Reusing computation can also reduce response latency. That makes routine summarization a potential trap: a policy designed to save input tokens can discard cheaper reusable context. Coding harnesses such as Claude Code and Codex combine caching with compaction, and provider APIs expose different ways to manage both.

“Caching: a game-changer” slide contrasts implicit and explicit caching, notes cost and latency benefits, and warns that summarization can throw the cache away.Open full source frame
Rewriting the prompt prefix can discard the cache.
15:3215:40
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

15:32 · section reference included

Choose what changes, then measure it

A cache-aware policy need not preserve every token forever. A substantial topic change may justify a fresh session. File access can use progressive disclosure, stale tool outputs can be removed selectively, and compaction can run when a specific constraint requires it. The aim is to make those decisions while preserving useful cache hits, rather than repeatedly rewriting the whole history. Provider-specific cache controls continue to change; Bouchard also points viewers to a prompt-caching article linked in the workshop slides.

Model routing offers another cost control: different tasks need not use the same model. But neither routing nor compaction is easy to assess without telemetry. Bouchard suggests Opik for tracking cache-hit rate, signals of user frustration, unusually long outputs, and other behavior a small model could flag. Context engineering is the decision about what the model sees on every call—what stays available for caching, what is removed or compacted, and when the policy acts. The experiment Space makes those choices available for follow-along inspection.

18:5519:08
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

18:55 · section reference included

One agent, middleware, and two tools

Solano’s architecture starts with a single ReAct agent looping through reasoning and tool calls. LangChain’s create_agent constructs the agent, and InMemorySaver retains the current conversation state. In the demonstrated implementation, the tutor does not restore past conversations. LangChain middleware changes behavior at runtime: it can summarize history, clear tool outputs, and apply the student’s source preferences.

The first tool retrieves tutor context through hybrid semantic and keyword search. The second lets the agent browse a filesystem knowledge base organized with wiki navigation. A FastAPI application wraps the agent behind an endpoint, and a Next.js interface exposes it to students. These layers separate the student interface, request handling, agent policy, and knowledge access without introducing multiple cooperating agents.

“One request, end to end” diagram shows Next.js UI feeding FastAPI and an agent containing middleware, retrieval and knowledge-base browsing tools, and Gemini 3.5 Flash.Open full source frame
The tutor connects a Next.js interface and FastAPI to one agent with middleware and two tools.
21:5622:16
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

21:56 · section reference included

Retrieve a few useful chunks from a large corpus

The knowledge base combines roughly two years of course lessons with documentation for LangChain, LlamaIndex, the OpenAI API, Codex, and Claude Code. Solano reports a corpus exceeding eight million tokens. The tutor cannot simply place all of it in one request, so the student’s source selection first narrows the searchable material.

The retrieval pipeline then follows a fixed sequence:

  1. Search with a Cohere embedding model for semantic similarity and BM25 for keyword matches, using a top-30 candidate setting.
  2. Merge the semantic and keyword results.
  3. Rerank the merged candidates and retain the top five chunks.
  4. Enforce a 100,000-token return cap by dropping trailing chunks if necessary.

Those settings came from configuration experiments, rather than intuition alone. The team measured whether the retrieved material included the correct knowledge-base page and selected settings that improved that recall.

24:4324:53
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

24:43 · section reference included

Give browsing a bounded filesystem

A targeted retrieval result may be enough for a lesson question, but a broader request could require several documents: how should a student use both Codex and Claude Code effectively? Solano points to research on agents browsing knowledge bases and introduces a second tool that runs Bash commands over the tutor’s files.

The browsable material has three layers:

FolderContents
rawOriginal Markdown lessons and documentation
generatedAutomatically collected document titles
wikiConcise topic and framework pages linking back to sources

Claude Code builds the wiki from the raw files. A fine-tuning topic page, for example, gathers pointers to the relevant source documents so the runtime agent can discover them without reading the entire corpus.

Deployment makes this browsing tool read-only. An allowlist restricts commands, and a sandbox confines access to the knowledge-base directory. Commands have an eight-second timeout, outputs are capped at 40,000 characters, and each turn permits at most 20 commands. If a lesson exceeds the output cap, the agent can issue a follow-up command for the remaining portion. Solano says the command-count limit had not been reached in their observations. The files and wiki are built offline, refreshed when material such as a new course is added, then deployed for reading.

With the demonstrated prompt, browsing ran on almost 90% of turns, although the team had not optimized that frequency. Disabling it preserved retrieval recall; Solano reports that adding browsing made the tutor 50% slower. Their explanation is straightforward: the first retrieval tool already found the relevant information, and browsing added calls. The real student questions may not have been complex enough to benefit from the extra navigation. The later evaluation log flags latency comparisons as confounded and distinguishes first answer text from first reasoning, tool, or text output, so the spoken speed result should be read as an observation from that setup.

27:0727:19
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

27:07 · section reference included

Turn an intuitive configuration into an observable system

The tutor’s initial context policy looked reasonable before anyone measured it:

PolicyInitial setting
Clear tool outputsTrigger after 5,000 tokens; retain at least the last five outputs
Summarize historyTrigger after 30,000 tokens; retain at least the last 20 messages
Apply source preferencesRestrict answers to the student’s selected sources

Keeping recent messages verbatim was intended to preserve their accuracy while reducing older context. Solano explicitly calls these defaults unproven: appearing fast and useful was not evidence that they were the best configuration.

The live demonstration uses a standalone interface in the Hugging Face Space; students also have an embedded chat bubble in their lessons. A sidebar enables or disables knowledge sources. The inspected interface shows a question about building a RAG pipeline with LlamaIndex and an answer containing setup instructions and code.

AI Tutor interface with a source-selection sidebar, a question about building a RAG pipeline with LlamaIndex, and an answer containing setup instructions and a code block.Open full source frame
The live tutor displays a RAG answer alongside selectable knowledge sources.

Solano then sends a request using Gemini 3.5 Flash, with reasoning and tool activity visible in the interface. After an apparent network delay, the completed answer’s activity reports four tool calls, four thinking steps, and ten sources. The open-source implementation exposes how these pieces fit together.

Every student turn produces telemetry: input tokens, output tokens, cached tokens, cost, TTFT, tool-call count, and whether summarization occurred. These records make it possible to compare context policies against actual behavior. A technique that sounds sensible can still make the tutor more expensive when it interferes with API caching.

32:1732:34
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

32:17 · section reference included

Test answers, retained facts, and whether the policy ran

The evaluation harness uses four terms. A preset is a tutor configuration, such as a summarization threshold or sliding-window policy. A task is either a single-turn question or a multi-turn session. A run applies a preset to a task. A bundle is the saved JSON artifact containing its results and metrics.

For single-turn tasks, Solano used Codex to collect student questions and staff answers from the course website. Cleaning removed duplicates and questions tied to unsuitable older library versions, leaving 60 question-answer pairs. Code checks whether the tutor retrieved the correct lesson; an LLM judge assesses whether the answer contains the right facts and the right kind of response. Solano suggests using Claude Code or Codex subscription workflows for grading as a cheaper alternative to API calls in their setup.

Multi-turn sessions test whether an early fact survives intervening conversation:

  1. Plant a fact: the synthetic student wants to learn RAG evaluation.
  2. Grow the history: insert filler messages to increase the conversation length.
  3. Probe later: ask what topic the student should learn today.
  4. Grade the answer: expect RAG evaluation, with topics such as hit rate and mean reciprocal rank, or MRR.
  5. Check the gate: confirm that the policy being tested actually triggered.

That last step matters. A successful answer in a summarization preset says little about summarization if the conversation never became long enough for a summary to be created.

“One session, start to finish” slide shows Turn 0 plant, Turns 1 to 10 grow, Turn 11 probe, Grade, and Gate, with expected RAG evaluation, hit rate, and MRR answers.Open full source frame
A session test plants a fact, grows the conversation, probes recall, and checks that compaction ran.

The run-task and run-battery functions execute the tests. Grading combines code checks with an LLM judge—in this case, Claude Code—and trigger checks determine whether the intended compaction occurred. Saved artifacts support later grading and reports on latency, TTFT, and other metrics without paying to rerun the tutor each time the evaluator changes.

36:2336:27
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

36:23 · section reference included

Full history beats the initial defaults

The initial comparison ran eleven presets. Untouched full history and the existing production defaults served as reference points for six technique families, including sliding windows, prompt compression, and selective retention. The model, prompt, tools, and dataset stayed fixed while the context policy changed. Solano reports that the evaluation spend exceeded $500, which helped motivate the cheaper-model experiments that followed.

The surprising winner for recovering facts across messages was leaving the context untouched. The production defaults performed worse. These were small initial screens: one experiment used one trial and another used two, with three session templates documented in the evaluation log. Solano emphasizes the ordering of the techniques rather than treating the exact scores as precise estimates.

Solano reports production costs of nearly $0.50 per single-turn response and $0.24 per turn in multi-turn sessions. Single-turn performance was high across techniques because a single response generally did not accumulate enough context to trigger them. The production preset’s reported multi-turn recall fell to 38%. Full history produced better recall and was reported as cheaper and faster than those defaults, subject to the latency-measurement limitation noted earlier.

Clearing tool results created work the tutor had already done. Once an earlier retrieved chunk disappeared, a later question could force the agent to retrieve it again. The extra calls added tokens and latency, while deleting history also removed information useful for recall. These initial Gemini experiments used sessions of only 11–13 turns, leaving open what would happen with a different model or substantially longer conversations.

42:2542:31
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

42:25 · section reference included

Cheaper cached inference preserves the same advantage

Vaid describes the first evaluation bill as almost $600 and broadens the question: when does compaction actually matter? The follow-up work separates cached chats, pasted documents and tool outputs, and local or larger-scale operation. These inputs create different problems. Conversation details can become buried; a pasted document can exceed the available window; tools can flood it with logs. A single context policy need not solve all three.

The team first switches to DeepSeek V4 Flash as a cheaper comparison. Vaid attributes much of its cost advantage to the historical cached-input discount described earlier. Keeping the full context again gives the best reported performance. Changing the model reduces the bill without reversing the initial result about retaining history.

The next probes ask about exact details from student chats: their setup, the errors they encountered, and fixes they had already tried. Vaid reports 95% success on those memory probes with full history, versus 32% after summarization or compaction. A summary can preserve the general subject while dropping the specific detail a later question needs. These percentages measure the particular memory probes, not general answer accuracy; the subsequent evaluation audit identified recycled-filler confounds in that battery.

On DeepSeek, the full-history setup sent the most tokens yet was the cheapest reported setup; Vaid says 97% of its tokens were cached. Summarization instead introduces fresh reads and writes as it transforms the history. Vaid also describes a 36-turn test associated with approximately 1.78 million tokens in which full history still came out ahead. The token total’s accounting is not specified clearly enough to treat it as either a simultaneous context size or a verified cumulative total, but the reported ordering remains the same: cached reuse outweighed the reduction in raw token count.

47:1047:24
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

47:10 · section reference included

Long-context recall and service cost are different questions

Longer tests ask whether the model can still retrieve a specific detail as the input grows. Vaid reports consistent retrieval of distinctive facts up to 800,000 tokens, while performance on ambiguous facts fell to roughly half that level. The distinction matters: finding a clearly identifiable fact is a narrower task than resolving ambiguous history or understanding an entire long conversation. The result supports retaining useful history for those probes without establishing that context rot has disappeared.

A low per-turn price can still become a large service bill. For a hypothetical 100,000–1,000,000 question turns per day, Vaid projects approximately $18,000–$180,000 per month on DeepSeek. The team was not serving that volume. These are historical workload projections rather than actual invoices; their complete token-mix assumptions are not supplied. Even so, the exercise motivates asking whether local inference could deliver similar results without per-token API charges.

“At scale, compaction is economics” chart plots monthly API bills against turns per day for Gemini and DeepSeek, with a highlighted cost band.Open full source frame
Projected API costs rise with daily tutor usage.
53:2453:34
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

53:24 · section reference included

Caching cannot make an oversized input fit

Local models can cache too, but the team’s MacBook tests were limited to a 32,000-token context window. Some individual lessons already exceeded that size. Once the required input does not fit, cache economics stop being the deciding factor: the application must compress the content or retrieve only the relevant portions. Increasing model size from roughly 7B or 8B parameters to 32B did not remove the tested context-window constraint. Parameter count and usable context capacity are separate properties, both constrained by the local serving setup.

Document retrieval worked much better than trying to preserve an oversized conversation. Vaid reports 100% accuracy on the local document-retrieval test, with processing times of 25–65 seconds. Retrieving relevant passages let the model answer from a manageable input. By contrast, an overstuffed long-conversation test took approximately 340 seconds and returned only one token. These observations concern the tested MacBook workloads; they show why choosing what enters the window can be a capacity requirement, even when preserving history is attractive in the cloud.

55:3655:49
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

55:36 · section reference included

Combine semantic retrieval with keyword search

Retrieval quality also depends on the search method. Dense retrieval matches meaning, but a specific buried fact may depend on preserving an exact term. Vaid reports the following results from the document tests:

Retrieval methodTested inputReported recall
Dense semantic retrieval50,000–200,000 tokensAbout 80%
Dense semantic retrieval400,000 tokens; facts buried in the middle0%
BM25 keyword retrievalThe corresponding tested facts100%

These workload-specific results motivate the tutor’s hybrid design: semantic similarity and keyword matching provide different routes to the relevant source. The supplied results do not include complete configurations or sample sizes for treating those percentages as a general retrieval benchmark.

“Retrieve, and make it hybrid” chart shows dense RAG results labeled 80% and 0%, and a green keyword/BM25 bar labeled 100%.Open full source frame
The retrieval chart contrasts dense RAG results with keyword/BM25 retrieval.

Local document retrieval success did not translate into equivalent chat memory. Vaid reports approximately 92–95% for cloud full-history chat, versus 33% locally under the context-window constraint. Running on already-owned hardware removes the per-token API bill, but throughput remains limited; it does not make serving capacity unlimited or hardware free. Retrieval can still make local inference useful when the task allows the application to select a small portion of a larger document.

58:3158:49
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

58:31 · section reference included

Name the constraint before choosing compaction

The closing synthesis connects three findings: detailed memory probes favored full history, distinctive facts remained retrievable in very long inputs, and caching made the setup sending the most tokens the cheapest in the reported comparison. A separate illustration for 1,000 students estimates about $40,000 per month on Gemini versus $1,900 on DeepSeek. That is another historical projection, with a different workload framing from the daily-turn chart. Vaid’s resulting rule is to identify the actual constraint—cost, capacity, recall, or throughput—before selecting a compaction method.

For the tutor’s current student volume and hardware constraints, the team chooses cached DeepSeek V4 Flash in the cloud, with hybrid retrieval. The stated policy keeps full history until a 30,000-token default compaction threshold. That threshold qualifies the broader recommendation to preserve history: the workshop does not end with an unlimited-history deployment, nor does it establish that this particular threshold is optimal.

The practical configuration follows the workload: preserve reusable conversation while it fits the chosen policy, retrieve course material selectively, and compact when a named limit requires it. Vaid closes by directing readers who want to build the tutor or explore more of the evaluations to Towards AI Academy’s full-stack AI engineering course; the presentation covers only part of that work.

1:00:331:00:43
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:00:33 · section reference included

Resources

From the talk

Updates since the talk

Transcript download
Full transcript

Loading transcript…