AI Engineer World's Fair 2026
Context Engineering in 2026: Compaction, Memory & Cost
Loading slides…
Read the talk
Context Engineering in 2026: Compaction, Memory & Cost
An AI tutor’s compaction experiments show how cached history, retrieval, and hardware limits change what belongs in the context window—and when removing it becomes necessary.
From a talk by Louis-François Bouchard, Omar Solano and Samridhi Vaid
Before you start: Familiarity with LLM tool calls, token-based pricing, and basic retrieval-augmented generation will help you follow the architecture and experiments.
The instruction was there. Why did the agent ignore it?
An agent receives an explicit instruction not to push to main. Later in the same session, it pushes anyway. The user responds furiously, hoping another correction will finally stick. Louis-François Bouchard opens with this familiar failure: as a session accumulates context, the agent can become worse at following instructions that were clear at the start. Switching models or coding harnesses may leave the underlying problem untouched.
For the Towards AI Academy team, that frustration has a direct product consequence. Its AI tutor helps students work through lessons; an unhelpful debugging session can become a refund request. The team therefore built experiments around improving the tutor’s answers while reducing its operating cost. The experiment Space and open-source tutor provide the accompanying material. Bouchard introduces the context techniques, Omar Solano walks through the application and initial evaluations, and Samridhi Vaid extends the tests to cheaper models, longer inputs, and local inference.
The tutor serves technical courses, so a useful answer must satisfy several requirements at once:
- Grounding: answer from the course material rather than relying solely on model knowledge.
- Student and lesson scope: use the course and lesson relevant to the current student.
- Session continuity: sustain a long debugging or iterative help session.
- Code support: work with code and the errors students encounter.
- Responsiveness: keep latency low enough that the tutor remains pleasant to use.
Real student interactions provide the baseline for testing these requirements.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
One window, several kinds of context
Two constraints create different engineering problems. First, the context window is finite: instructions, lessons, code, and conversation all compete for the same space. Second, a model call is stateless unless the surrounding application supplies the state. Reopening the tutor does not automatically restore what a student discussed previously. Context management handles information within a session; memory carries information across sessions. The workshop’s experiments concentrate on getting the first problem right.
A tutor request contains much more than the latest question. It includes the system prompt, tool definitions, chat history, previous tool outputs, retrieved course chunks, and the student’s new message. That message may itself contain substantial code or error logs. In this application, old tool outputs are the main source of growth: retrieved chunks and tool-call/result pairs accumulate as the conversation proceeds.
Bouchard connects this growth to context rot, suggesting that training a model to recover facts inserted into long corpora does not necessarily teach it to understand the whole evolving conversation. Quality is only one concern, however. Each follow-up resends earlier input, creating repeated token charges, and processing a longer prompt can increase time to first token, or TTFT. The experiments therefore ask about quality, spending, and responsiveness together.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Start with inexpensive compaction
Compaction aims to retain the smallest context that still supports the answer. Everything else can be dropped or stored outside the active window. This does not require a language model. Consider a tool that usually returns ten lines but suddenly emits three hundred. Preserve the beginning and end, insert an explicit truncation marker, and let the agent request more information if it needs the omitted material.
A small Python function captures that head-and-tail policy. Here, keeping five lines at each end is a tunable choice; the marker tells the agent that the middle is missing.
python
def truncate_tool_output(text: str, head: int = 5, tail: int = 5) -> str:
if head < 1 or tail < 1:
raise ValueError("head and tail must both be positive")
lines = text.splitlines()
if len(lines) <= head + tail:
return text
omitted = len(lines) - head - tail
marker = (
f"[Truncated {omitted} lines; call the tool again "
"with a narrower request if needed.]"
)
return "\n".join(lines[:head] + [marker] + lines[-tail:])
Two other inexpensive policies are a sliding window, retaining the last N user turns, and tool-specific clearing, removing outputs whose contents are no longer useful. The appropriate window and clearing rules depend on what students ask and which tools the tutor exposes.
Model-assisted methods spend tokens now to reduce later input. A smaller model—even one running locally on a MacBook—may be sufficient for this work.
| Method | What changes |
|---|---|
| Selective retention | Keep or discard material according to the conversation’s direction |
| Continuous summarization | Replace earlier turns with a running summary |
| Summary and reset | Summarize the conversation at a limit, then restart from that summary |
| Delta summarization | Update an existing summary with newly learned information |
Bouchard uses Claude Code as an example of summarizing and resetting at a context limit, then describes delta summaries as useful when subagents pass their findings back to a main agent. The tutor itself uses one agent: its workload did not require the additional coordination.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keep the source outside the window
Offloading provides another option: retain information in files or a retrieval system and load it when needed. Ordinary retrieval-augmented generation, or RAG, already supports this pattern. The team also compared GraphRAG with its existing RAG pipeline. Bouchard reports tied results on their real-user evaluations, with greater setup cost for GraphRAG. A larger corpus with important relationships between topics might justify the graph, but this tutor did not show that benefit.
Saving the underlying conversation locally or on a server makes offloading reversible: information leaves the active prompt without disappearing. The tutor’s approach draws on Karpathy’s LLM Wiki. Cross-linked chunks point back to raw data files, and an index lists the chunks with short descriptions. Bouchard estimates that the index exposed to the agent is approximately 450 tokens.
Retrieval proceeds in stages:
- Inspect the index when a question appears related to stored student information.
- Read the relevant chunk.
- Answer from that chunk if it is sufficient.
- Follow its pointer to the raw data if the task requires more detail.
This is progressive disclosure: a simple question pulls little context, while a complex question can pull more. The same principle applies to skills. Small, precise skills can reference one another and load individually; when subagents are useful, each can receive only the skill context its task requires.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A shorter prompt can cost more
Compaction becomes less straightforward when the provider caches prompt computation. Instead of processing every previously submitted token from scratch, the provider can reuse stored state, including the KV cache. Reused input receives a discounted rate, while new input is billed at the uncached rate. In the workshop’s historical DeepSeek example, cached input was 50 times cheaper than uncached input. That compares DeepSeek’s own two input rates, not DeepSeek against Gemini; it is not a current price quote.
Summarizing a conversation changes the tokens that would otherwise be reused. The rewritten portion must be processed as new input, although an unchanged earlier prefix may remain reusable. The economic comparison is therefore between a long discounted history and a short full-price replacement—not simply between their token counts.
The workshop’s compression threshold follows from a simplified one-request calculation. Let H be the retained history’s token count, S the replacement summary’s token count, and p the uncached input price per token:
Under those assumptions, the replacement must be less than one-fiftieth as long to reduce that request’s input cost. This assumes all retained history is cached and the replacement is uncached; summary generation and later reuse are outside the calculation.
Bouchard describes caching savings of around 90% as common in the APIs he discusses, with larger discounts in some cases. Reusing computation can also reduce response latency. That makes routine summarization a potential trap: a policy designed to save input tokens can discard cheaper reusable context. Coding harnesses such as Claude Code and Codex combine caching with compaction, and provider APIs expose different ways to manage both.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Choose what changes, then measure it
A cache-aware policy need not preserve every token forever. A substantial topic change may justify a fresh session. File access can use progressive disclosure, stale tool outputs can be removed selectively, and compaction can run when a specific constraint requires it. The aim is to make those decisions while preserving useful cache hits, rather than repeatedly rewriting the whole history. Provider-specific cache controls continue to change; Bouchard also points viewers to a prompt-caching article linked in the workshop slides.
Model routing offers another cost control: different tasks need not use the same model. But neither routing nor compaction is easy to assess without telemetry. Bouchard suggests Opik for tracking cache-hit rate, signals of user frustration, unusually long outputs, and other behavior a small model could flag. Context engineering is the decision about what the model sees on every call—what stays available for caching, what is removed or compacted, and when the policy acts. The experiment Space makes those choices available for follow-along inspection.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
One agent, middleware, and two tools
Solano’s architecture starts with a single ReAct agent looping through reasoning and tool calls. LangChain’s create_agent constructs the agent, and InMemorySaver retains the current conversation state. In the demonstrated implementation, the tutor does not restore past conversations. LangChain middleware changes behavior at runtime: it can summarize history, clear tool outputs, and apply the student’s source preferences.
The first tool retrieves tutor context through hybrid semantic and keyword search. The second lets the agent browse a filesystem knowledge base organized with wiki navigation. A FastAPI application wraps the agent behind an endpoint, and a Next.js interface exposes it to students. These layers separate the student interface, request handling, agent policy, and knowledge access without introducing multiple cooperating agents.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Retrieve a few useful chunks from a large corpus
The knowledge base combines roughly two years of course lessons with documentation for LangChain, LlamaIndex, the OpenAI API, Codex, and Claude Code. Solano reports a corpus exceeding eight million tokens. The tutor cannot simply place all of it in one request, so the student’s source selection first narrows the searchable material.
The retrieval pipeline then follows a fixed sequence:
- Search with a Cohere embedding model for semantic similarity and BM25 for keyword matches, using a top-30 candidate setting.
- Merge the semantic and keyword results.
- Rerank the merged candidates and retain the top five chunks.
- Enforce a 100,000-token return cap by dropping trailing chunks if necessary.
Those settings came from configuration experiments, rather than intuition alone. The team measured whether the retrieved material included the correct knowledge-base page and selected settings that improved that recall.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Give browsing a bounded filesystem
A targeted retrieval result may be enough for a lesson question, but a broader request could require several documents: how should a student use both Codex and Claude Code effectively? Solano points to research on agents browsing knowledge bases and introduces a second tool that runs Bash commands over the tutor’s files.
The browsable material has three layers:
| Folder | Contents |
|---|---|
raw | Original Markdown lessons and documentation |
generated | Automatically collected document titles |
wiki | Concise topic and framework pages linking back to sources |
Claude Code builds the wiki from the raw files. A fine-tuning topic page, for example, gathers pointers to the relevant source documents so the runtime agent can discover them without reading the entire corpus.
Deployment makes this browsing tool read-only. An allowlist restricts commands, and a sandbox confines access to the knowledge-base directory. Commands have an eight-second timeout, outputs are capped at 40,000 characters, and each turn permits at most 20 commands. If a lesson exceeds the output cap, the agent can issue a follow-up command for the remaining portion. Solano says the command-count limit had not been reached in their observations. The files and wiki are built offline, refreshed when material such as a new course is added, then deployed for reading.
With the demonstrated prompt, browsing ran on almost 90% of turns, although the team had not optimized that frequency. Disabling it preserved retrieval recall; Solano reports that adding browsing made the tutor 50% slower. Their explanation is straightforward: the first retrieval tool already found the relevant information, and browsing added calls. The real student questions may not have been complex enough to benefit from the extra navigation. The later evaluation log flags latency comparisons as confounded and distinguishes first answer text from first reasoning, tool, or text output, so the spoken speed result should be read as an observation from that setup.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn an intuitive configuration into an observable system
The tutor’s initial context policy looked reasonable before anyone measured it:
| Policy | Initial setting |
|---|---|
| Clear tool outputs | Trigger after 5,000 tokens; retain at least the last five outputs |
| Summarize history | Trigger after 30,000 tokens; retain at least the last 20 messages |
| Apply source preferences | Restrict answers to the student’s selected sources |
Keeping recent messages verbatim was intended to preserve their accuracy while reducing older context. Solano explicitly calls these defaults unproven: appearing fast and useful was not evidence that they were the best configuration.
The live demonstration uses a standalone interface in the Hugging Face Space; students also have an embedded chat bubble in their lessons. A sidebar enables or disables knowledge sources. The inspected interface shows a question about building a RAG pipeline with LlamaIndex and an answer containing setup instructions and code.
Solano then sends a request using Gemini 3.5 Flash, with reasoning and tool activity visible in the interface. After an apparent network delay, the completed answer’s activity reports four tool calls, four thinking steps, and ten sources. The open-source implementation exposes how these pieces fit together.
Every student turn produces telemetry: input tokens, output tokens, cached tokens, cost, TTFT, tool-call count, and whether summarization occurred. These records make it possible to compare context policies against actual behavior. A technique that sounds sensible can still make the tutor more expensive when it interferes with API caching.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Test answers, retained facts, and whether the policy ran
The evaluation harness uses four terms. A preset is a tutor configuration, such as a summarization threshold or sliding-window policy. A task is either a single-turn question or a multi-turn session. A run applies a preset to a task. A bundle is the saved JSON artifact containing its results and metrics.
For single-turn tasks, Solano used Codex to collect student questions and staff answers from the course website. Cleaning removed duplicates and questions tied to unsuitable older library versions, leaving 60 question-answer pairs. Code checks whether the tutor retrieved the correct lesson; an LLM judge assesses whether the answer contains the right facts and the right kind of response. Solano suggests using Claude Code or Codex subscription workflows for grading as a cheaper alternative to API calls in their setup.
Multi-turn sessions test whether an early fact survives intervening conversation:
- Plant a fact: the synthetic student wants to learn RAG evaluation.
- Grow the history: insert filler messages to increase the conversation length.
- Probe later: ask what topic the student should learn today.
- Grade the answer: expect RAG evaluation, with topics such as hit rate and mean reciprocal rank, or MRR.
- Check the gate: confirm that the policy being tested actually triggered.
That last step matters. A successful answer in a summarization preset says little about summarization if the conversation never became long enough for a summary to be created.
The run-task and run-battery functions execute the tests. Grading combines code checks with an LLM judge—in this case, Claude Code—and trigger checks determine whether the intended compaction occurred. Saved artifacts support later grading and reports on latency, TTFT, and other metrics without paying to rerun the tutor each time the evaluator changes.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Full history beats the initial defaults
The initial comparison ran eleven presets. Untouched full history and the existing production defaults served as reference points for six technique families, including sliding windows, prompt compression, and selective retention. The model, prompt, tools, and dataset stayed fixed while the context policy changed. Solano reports that the evaluation spend exceeded $500, which helped motivate the cheaper-model experiments that followed.
The surprising winner for recovering facts across messages was leaving the context untouched. The production defaults performed worse. These were small initial screens: one experiment used one trial and another used two, with three session templates documented in the evaluation log. Solano emphasizes the ordering of the techniques rather than treating the exact scores as precise estimates.
Solano reports production costs of nearly $0.50 per single-turn response and $0.24 per turn in multi-turn sessions. Single-turn performance was high across techniques because a single response generally did not accumulate enough context to trigger them. The production preset’s reported multi-turn recall fell to 38%. Full history produced better recall and was reported as cheaper and faster than those defaults, subject to the latency-measurement limitation noted earlier.
Clearing tool results created work the tutor had already done. Once an earlier retrieved chunk disappeared, a later question could force the agent to retrieve it again. The extra calls added tokens and latency, while deleting history also removed information useful for recall. These initial Gemini experiments used sessions of only 11–13 turns, leaving open what would happen with a different model or substantially longer conversations.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Cheaper cached inference preserves the same advantage
Vaid describes the first evaluation bill as almost $600 and broadens the question: when does compaction actually matter? The follow-up work separates cached chats, pasted documents and tool outputs, and local or larger-scale operation. These inputs create different problems. Conversation details can become buried; a pasted document can exceed the available window; tools can flood it with logs. A single context policy need not solve all three.
The team first switches to DeepSeek V4 Flash as a cheaper comparison. Vaid attributes much of its cost advantage to the historical cached-input discount described earlier. Keeping the full context again gives the best reported performance. Changing the model reduces the bill without reversing the initial result about retaining history.
The next probes ask about exact details from student chats: their setup, the errors they encountered, and fixes they had already tried. Vaid reports 95% success on those memory probes with full history, versus 32% after summarization or compaction. A summary can preserve the general subject while dropping the specific detail a later question needs. These percentages measure the particular memory probes, not general answer accuracy; the subsequent evaluation audit identified recycled-filler confounds in that battery.
On DeepSeek, the full-history setup sent the most tokens yet was the cheapest reported setup; Vaid says 97% of its tokens were cached. Summarization instead introduces fresh reads and writes as it transforms the history. Vaid also describes a 36-turn test associated with approximately 1.78 million tokens in which full history still came out ahead. The token total’s accounting is not specified clearly enough to treat it as either a simultaneous context size or a verified cumulative total, but the reported ordering remains the same: cached reuse outweighed the reduction in raw token count.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Long-context recall and service cost are different questions
Longer tests ask whether the model can still retrieve a specific detail as the input grows. Vaid reports consistent retrieval of distinctive facts up to 800,000 tokens, while performance on ambiguous facts fell to roughly half that level. The distinction matters: finding a clearly identifiable fact is a narrower task than resolving ambiguous history or understanding an entire long conversation. The result supports retaining useful history for those probes without establishing that context rot has disappeared.
A low per-turn price can still become a large service bill. For a hypothetical 100,000–1,000,000 question turns per day, Vaid projects approximately $18,000–$180,000 per month on DeepSeek. The team was not serving that volume. These are historical workload projections rather than actual invoices; their complete token-mix assumptions are not supplied. Even so, the exercise motivates asking whether local inference could deliver similar results without per-token API charges.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Caching cannot make an oversized input fit
Local models can cache too, but the team’s MacBook tests were limited to a 32,000-token context window. Some individual lessons already exceeded that size. Once the required input does not fit, cache economics stop being the deciding factor: the application must compress the content or retrieve only the relevant portions. Increasing model size from roughly 7B or 8B parameters to 32B did not remove the tested context-window constraint. Parameter count and usable context capacity are separate properties, both constrained by the local serving setup.
Document retrieval worked much better than trying to preserve an oversized conversation. Vaid reports 100% accuracy on the local document-retrieval test, with processing times of 25–65 seconds. Retrieving relevant passages let the model answer from a manageable input. By contrast, an overstuffed long-conversation test took approximately 340 seconds and returned only one token. These observations concern the tested MacBook workloads; they show why choosing what enters the window can be a capacity requirement, even when preserving history is attractive in the cloud.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Combine semantic retrieval with keyword search
Retrieval quality also depends on the search method. Dense retrieval matches meaning, but a specific buried fact may depend on preserving an exact term. Vaid reports the following results from the document tests:
| Retrieval method | Tested input | Reported recall |
|---|---|---|
| Dense semantic retrieval | 50,000–200,000 tokens | About 80% |
| Dense semantic retrieval | 400,000 tokens; facts buried in the middle | 0% |
| BM25 keyword retrieval | The corresponding tested facts | 100% |
These workload-specific results motivate the tutor’s hybrid design: semantic similarity and keyword matching provide different routes to the relevant source. The supplied results do not include complete configurations or sample sizes for treating those percentages as a general retrieval benchmark.
Local document retrieval success did not translate into equivalent chat memory. Vaid reports approximately 92–95% for cloud full-history chat, versus 33% locally under the context-window constraint. Running on already-owned hardware removes the per-token API bill, but throughput remains limited; it does not make serving capacity unlimited or hardware free. Retrieval can still make local inference useful when the task allows the application to select a small portion of a larger document.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Name the constraint before choosing compaction
The closing synthesis connects three findings: detailed memory probes favored full history, distinctive facts remained retrievable in very long inputs, and caching made the setup sending the most tokens the cheapest in the reported comparison. A separate illustration for 1,000 students estimates about $40,000 per month on Gemini versus $1,900 on DeepSeek. That is another historical projection, with a different workload framing from the daily-turn chart. Vaid’s resulting rule is to identify the actual constraint—cost, capacity, recall, or throughput—before selecting a compaction method.
For the tutor’s current student volume and hardware constraints, the team chooses cached DeepSeek V4 Flash in the cloud, with hybrid retrieval. The stated policy keeps full history until a 30,000-token default compaction threshold. That threshold qualifies the broader recommendation to preserve history: the workshop does not end with an unlimited-history deployment, nor does it establish that this particular threshold is optimal.
The practical configuration follows the workload: preserve reusable conversation while it fits the chosen policy, retrieve course material selectively, and compact when a named limit requires it. Vaid closes by directing readers who want to build the tutor or explore more of the evaluations to Towards AI Academy’s full-stack AI engineering course; the presentation covers only part of that work.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Source code and setup instructions for the tutor, retrieval system and evaluation harness.
Workshop Space hosting the experiment showcase and access to the tutor.
Karpathy's pattern for accumulating knowledge in an interlinked, LLM-maintained wiki backed by original sources.
Updates since the talk
- AI tutor evaluation methodology and findingsDocumentation
Rolling record of experiment configurations, metrics, findings and subsequent corrections.
Bouchard's workshop companion, including additional experiments and the team's subsequent production configuration.
Current provider documentation for cache persistence and prefix matching.
- DeepSeek current pricingDocumentation
Current model limits and separate cached-input, uncached-input and output rates.
Full transcript
Loading transcript…