AI Engineer Code 2025
Context Platform Engineering to Reduce Token Anxiety — Val Bercovici and Callan Fox, WEKA
Loading slides…
Read the talk
Context Platform Engineering to Reduce Token Anxiety
Agent context is reusable only if the inference platform retains it and retrieves it quickly enough. WEKA’s toolkit explores how cache lifetime, memory tiers and workload shape affect that reuse.
From a talk by Val Bercovici and Callan Fox
Before you start: Familiarity with LLM context windows, tool-calling agents and the distinction between prompt processing and token generation will help.
How do you keep an agent’s context available?
How do you build an inference platform that can serve agent swarms without repeatedly processing the context they already supplied? WEKA opens with a toolkit for investigating that problem. Callan Fox’s load generator models agents and subtasks with configurable service-level objectives, deterministic or random prompt cycles, model parallelism, aggregated or disaggregated prefill and decode, and different memory tiers. These are connected choices: the workload determines what context is reused, while the platform determines whether that reuse actually happens.
Val Bercovici invites developers to download, modify and contribute to the open-source toolkit. Its motivation follows Manus’s context-engineering lessons, which identify KV-cache hit rate as the most important metric for their production agents. A KV cache holds attention keys and values computed for earlier tokens; reusing that state avoids repeating the corresponding prefill work. Context platform engineering concerns the infrastructure that makes those hits possible.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The prediction problem behind prompt-cache pricing
Token anxiety starts with hitting limits while trying to get work done. Bercovici’s ambition is to engineer platforms that remove those constraints; the immediate alternative is what he calls prompt-cache arbitrage. Developers must balance ordinary input and output prices against separate cache-write and cache-read prices. Buying retention only pays off if enough subsequent requests reuse the cached material.
The Anthropic example makes the prediction concrete: should you pay to retain context for five minutes or an hour? The answer depends on how many reads arrive before the cache expires. The pricing slide highlights Claude Sonnet 4.5 across those separate categories. Cache writes must also be distinguished from ordinary input: current Claude documentation maintains that distinction, so an ordinary-input versus cache-read comparison should not be treated as a cache-write quote. The platform-engineering goal is to make useful retention less dependent on developers guessing future request timing.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Two feedback loops, one storage obligation
Human feedback and agent activity run at different speeds. A person may pause to review a result while parallel agents and subtasks keep working, generating and consuming context in the background. Much of that context could be reused, but its availability depends on what survives in the platform’s cache between requests. This turns a token-pricing question into a token-storage problem.
Subscription commitments and explicit cache-write or cache-read choices become infrastructure obligations. The platform must translate its service-level agreements into operational objectives, then provision and manage KV caches and memory tiers to meet them. Fox’s interpretation is that purchasing a token tier or paying for cache retention effectively purchases slots in token storage. This is a useful model of provider economics, rather than a disclosed specification of how every subscription is enforced. The engineering task is to optimize the infrastructure so its SLOs meet the promised service as consistently as possible.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Inside a coding-agent conversation
Fox begins his trace walkthrough with successive requests drawn as columns. Salmon represents new tokens, gray represents tokens that could be cached, and blue represents output tokens. Blue dots beneath the columns mark actual user responses. Context accumulates until it reaches a high watermark imposed by the model or inference provider. The agent then summarizes and begins another cycle. That compression can lose details and reduce the agent’s fidelity; raising the usable watermark would let it carry more of the original context forward.
Looking inside the gray region reveals why user-message counts are a poor description of the workload. In the Claude Code example, system instructions and user text occupy a relatively small share. Tool calls and tool results dominate: the agent runs a Bash command, consumes its result, and decides what to do next. Each step adds material that can become part of the next request’s reusable context.
Fox reports median request intervals of roughly 10–15 seconds in the agent data he describes, with the timing strongly dependent on whether humans approve each tool call. He says the mean interval can reach minutes or even hours because human responses take much longer. His observations span billions of tokens. The median captures the rapid tool loop; the mean exposes the long pauses that a retention policy must also handle.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Longer retention trades memory for less prefill
Columns of successive requests hide elapsed time. Fox therefore switches to a temporal view: solid lines use the left axis to show the working set, the number of tokens held in cache, while dotted lines use the right axis to show cache hit rate. The red one-minute retention policy repeatedly rises and falls when requests arrive more than a minute apart. A few requests may hit, then the cached state expires and the next request has to rebuild it.
| Retention | Effect in the illustrated conversation | Storage consequence |
|---|---|---|
| One minute, red | Expires across relatively short pauses | Repeated cache refreshes |
| Five minutes, blue | Bridges more request gaps | Retains context longer |
| One hour | Preserves reuse across longer pauses | Holds more tokens for longer |
The five-minute policy improves reuse but still misses longer gaps. Extending retention to an hour increases the working set, eventually substantially. Better reuse therefore requires memory tiers with enough capacity to preserve the context across those pauses.
Hit-rate percentages can be hard to internalize, so Fox offers another measure: how many times does the platform prefill the same chunk of tokens? For the illustrated conversation, Fox reports roughly 15–16 prefills of the same token chunk at a one-minute TTL, with longer retention bringing the average toward one. This makes the waste tangible: the platform repeatedly computes state it could have kept.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The provider needs a productive hit-rate band
Fox next asks the audience to imagine operating its own inference service, whether self-hosted or hosted on dedicated infrastructure. He forecasts that arrangement becoming more common from 2026 onward. From the operator’s perspective, the question is how much useful output a given system can produce as context length and cache reuse change.
The relationship between cache hit rate and output-token production is nonlinear. Its shape depends on context length, accelerator choice and the arrangement of prefill and decode, including disaggregation. Fox labels three operating regions A, B and C, with C representing the desirable region; he characterizes A and B as unprofitable or poor value. Providers consequently have an incentive to keep users in a productive cache-hit band. That is the economic reasoning behind his description of subscriptions as cache allotments: agent workloads that continually recompute old context can consume the GPU capacity needed to serve useful output.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Token storage needs capacity and transfer speed
A useful token-storage tier must satisfy three requirements together:
- Capacity: retain enough of the working set to reach the region where additional cache begins yielding diminishing returns.
- Store performance: accept KV state quickly enough that entries are not dropped before reaching the tier and GPUs do not stall waiting to offload them.
- Fetch performance: return cached state quickly enough that GPUs do not stall waiting to use it.
The accelerator remains the resource the rest of the system is organized to keep productive. A large cache that cannot move data at the required rate does not solve the problem.
Keeping every session in high-bandwidth memory, or HBM, would be attractive, but Fox describes it as impractical, partly because of batching. DRAM is the common additional tier: it offers acceptable performance, but its capacity is limited and tightly coupled to compute. Expanding memory can therefore require expanding the surrounding system. Fox also criticizes available pooled-DRAM approaches for imposing performance penalties.
WEKA’s Augmented Memory Grid connects inference systems to the company’s existing storage product, previously used in AI training and HPC. Fox describes it as a supported, optimized connector backed by NVMe. He claims roughly 1,000-fold greater density, but does not specify the comparison basis. Capacity is only one side of the comparison: Fox also describes an alternative storage system delivering 50–60 GB/s, yet still too slow for the token-storage demands under discussion. The relevant test is whether a tier can preserve enough state and deliver it fast enough to benefit inference.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Exercise the working set, not just the server
The load generator behaves like a provider managing incoming demand. It can constrain load using optional SLOs, which Fox names as time to first token and output tokens per request. The latter is the wording used in the presentation; its time unit is not specified. Disabling the controls drives load without those constraints. A test can hold the number of coding-agent users constant or increase it over time, progressively exercising more of the memory hierarchy.
Fox points to his testing methodology and GitHub tooling for details, then distinguishes two access patterns:
| Pattern | Workload | What it exposes |
|---|---|---|
| Sequential | Traverse an initial working set in order | A sharp performance cliff when the tier overflows |
| Random, growing pool | Increase the user pool and sample prompts randomly | A blend of hits across memory tiers |
Sequential traversal is deliberately deterministic. Even a small overflow can produce a large performance drop. Random sampling is the fairer comparison in Fox’s account because some requests still hit HBM while others hit a second tier such as DRAM. It measures the resulting mixture rather than forcing every access through the same overflow pattern.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
What changes when the workload outgrows DRAM?
The decode-focused results compare three configurations:
| Plot color | Configuration |
|---|---|
| Purple | HBM plus WEKA |
| Orange | HBM plus DRAM |
| Orange-pink | HBM plus DRAM plus another POSIX storage system |
The dotted line tracks the growing user pool. In the initial shaded region, all three configurations benefit primarily from HBM cache hits. As the pool grows, the working set exceeds what the DRAM tier can hold, and both the orange and orange-pink configurations decline sharply.
The WEKA configuration also declines as fewer accesses benefit from HBM. The benchmarking tool automatically pulls back concurrency, and the configurations eventually settle into steadier operation. WEKA Labs reports that its configuration sustains more users and more output tokens at steady state than the two alternatives in this decode-focused comparison. The presentation does not give numerical values for that advantage.
Fox reports a larger advantage in the disaggregated prefill comparison, attributing it to GPU efficiency with large prefill batches and a single decode. Here the configurations help separate two bottlenecks. Purple versus orange demonstrates the benefit of having more cache capacity and therefore more opportunities to hit. The orange-pink configuration has enough capacity for all the possible hits, but cannot deliver the cached state into the GPUs quickly enough to realize that benefit.
A cache hit must arrive in time to be useful. Fox characterizes WEKA’s result as combining greater capacity with DRAM-speed access, allowing the benefit to persist for longer periods. The final comparison completes the earlier working-set argument: extending retention reduces repeated prefill only when the storage tier can also sustain the writes and fetches needed to keep the accelerators working.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Put the workload assumptions to the test
Bercovici closes by inviting developers to download, test, fork and improve the toolkit. The Inference Server Cache Performance Testing Suite provides the practical entry point for exploring these workloads. The repository and quick-start slide accompanies that invitation, followed by a QR code for more information and a promise of links to the referenced blogs. The intended outcome is less token anxiety and less prompt-cache arbitrage through platforms that preserve and serve context more effectively.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Tools for benchmarking KV-cache reuse, varying working sets, and replaying agent traces against inference servers.
Callan Fox's testing guide covers baseline checks, offload overhead, memory-tier isolation, and blended working-set workloads.
Manus's production lessons on stable prefixes, append-only context, cache breakpoints, and agent design.
Further reading
Fox's companion explanation of agent context, retention, provider economics, and WEKA's three-tier benchmark comparisons.
Visual explanations of context growth, subagents, and missed cache reuse in coding-agent sessions.
Updates since the talk
Current API guidance for prompt caching, retention options, token accounting, and pricing.
Full transcript
Loading transcript…