AI Engineer Code 2025

Context Platform Engineering to Reduce Token Anxiety — Val Bercovici and Callan Fox, WEKA

Loading slides…

Read the talk

Context Platform Engineering to Reduce Token Anxiety

Agent context is reusable only if the inference platform retains it and retrieves it quickly enough. WEKA’s toolkit explores how cache lifetime, memory tiers and workload shape affect that reuse.

From a talk by Val Bercovici and Callan Fox

Before you start: Familiarity with LLM context windows, tool-calling agents and the distinction between prompt processing and token generation will help.

How do you keep an agent’s context available?

How do you build an inference platform that can serve agent swarms without repeatedly processing the context they already supplied? WEKA opens with a toolkit for investigating that problem. Callan Fox’s load generator models agents and subtasks with configurable service-level objectives, deterministic or random prompt cycles, model parallelism, aggregated or disaggregated prefill and decode, and different memory tiers. These are connected choices: the workload determines what context is reused, while the platform determines whether that reuse actually happens.

Diagram showing a load generation and replay engine, coding agents, an inference API, prefill and decode compute, and two memory tiers.Open full source frame
WEKA’s benchmarking methodology connects load generation, coding agents, inference and memory tiers.

Val Bercovici invites developers to download, modify and contribute to the open-source toolkit. Its motivation follows Manus’s context-engineering lessons, which identify KV-cache hit rate as the most important metric for their production agents. A KV cache holds attention keys and values computed for earlier tokens; reusing that state avoids repeating the corresponding prefill work. Context platform engineering concerns the infrastructure that makes those hits possible.

0:090:26
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:09 · section reference included

The prediction problem behind prompt-cache pricing

Token anxiety starts with hitting limits while trying to get work done. Bercovici’s ambition is to engineer platforms that remove those constraints; the immediate alternative is what he calls prompt-cache arbitrage. Developers must balance ordinary input and output prices against separate cache-write and cache-read prices. Buying retention only pays off if enough subsequent requests reuse the cached material.

The Anthropic example makes the prediction concrete: should you pay to retain context for five minutes or an hour? The answer depends on how many reads arrive before the cache expires. The pricing slide highlights Claude Sonnet 4.5 across those separate categories. Cache writes must also be distinguished from ordinary input: current Claude documentation maintains that distinction, so an ordinary-input versus cache-read comparison should not be treated as a cache-write quote. The platform-engineering goal is to make useful retention less dependent on developers guessing future request timing.

Claude Docs pricing table with the Claude Sonnet 4.5 row highlighted across input, cache-write, cache-hit and output columns.Open full source frame
Claude pricing separates input tokens, cache writes, cache hits and output tokens.
2:272:48
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:27 · section reference included

Two feedback loops, one storage obligation

Human feedback and agent activity run at different speeds. A person may pause to review a result while parallel agents and subtasks keep working, generating and consuming context in the background. Much of that context could be reused, but its availability depends on what survives in the platform’s cache between requests. This turns a token-pricing question into a token-storage problem.

Subscription commitments and explicit cache-write or cache-read choices become infrastructure obligations. The platform must translate its service-level agreements into operational objectives, then provision and manage KV caches and memory tiers to meet them. Fox’s interpretation is that purchasing a token tier or paying for cache retention effectively purchases slots in token storage. This is a useful model of provider economics, rather than a disclosed specification of how every subscription is enforced. The engineering task is to optimize the infrastructure so its SLOs meet the promised service as consistently as possible.

3:514:12
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:51 · section reference included

Inside a coding-agent conversation

Fox begins his trace walkthrough with successive requests drawn as columns. Salmon represents new tokens, gray represents tokens that could be cached, and blue represents output tokens. Blue dots beneath the columns mark actual user responses. Context accumulates until it reaches a high watermark imposed by the model or inference provider. The agent then summarizes and begins another cycle. That compression can lose details and reduce the agent’s fidelity; raising the usable watermark would let it carry more of the original context forward.

Looking inside the gray region reveals why user-message counts are a poor description of the workload. In the Claude Code example, system instructions and user text occupy a relatively small share. Tool calls and tool results dominate: the agent runs a Bash command, consumes its result, and decides what to do next. Each step adds material that can become part of the next request’s reusable context.

Fox reports median request intervals of roughly 10–15 seconds in the agent data he describes, with the timing strongly dependent on whether humans approve each tool call. He says the mean interval can reach minutes or even hours because human responses take much longer. His observations span billions of tokens. The median captures the rapid tool loop; the mean exposes the long pauses that a retention policy must also handle.

5:365:50
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:36 · section reference included

More agents create more opportunities—and more demand

A multi-agent workflow adds an orchestrator and specialized subagents. Some subagents discard their context between activations; others retain it. Giving each subagent context tailored to its task helps concentrate attention on the relevant problem, but it also increases total context consumption. Fox recolors the previously gray regions to expose shared material across agents and show how those relationships evolve before work returns to the orchestrator and another agent wakes. The exact pattern varies across Codex, Claude Code and other systems.

Potentially reusable context is not the same as a cache hit. In the next view, yellow regions identify material that must be refreshed instead of reused. Fox says an uncached input can cost potentially ten times as much as a cached read. The historical Claude Sonnet example in Manus’s post supports that ordinary-input comparison; cache-write charges are a separate category. For flat-rate subscribers, Fox argues that missed reuse can instead appear as earlier rate limits. Reducing those misses benefits both the user and the provider doing the repeated computation.

8:188:31
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:18 · section reference included

Longer retention trades memory for less prefill

Columns of successive requests hide elapsed time. Fox therefore switches to a temporal view: solid lines use the left axis to show the working set, the number of tokens held in cache, while dotted lines use the right axis to show cache hit rate. The red one-minute retention policy repeatedly rises and falls when requests arrive more than a minute apart. A few requests may hit, then the cached state expires and the next request has to rebuild it.

RetentionEffect in the illustrated conversationStorage consequence
One minute, redExpires across relatively short pausesRepeated cache refreshes
Five minutes, blueBridges more request gapsRetains context longer
One hourPreserves reuse across longer pausesHolds more tokens for longer

The five-minute policy improves reuse but still misses longer gaps. Extending retention to an hour increases the working set, eventually substantially. Better reuse therefore requires memory tiers with enough capacity to preserve the context across those pauses.

Time-series chart with solid working-set lines, dashed cache-hit-rate lines, and annotations identifying both axes and short-TTL inefficiencies.Open full source frame
Working-set usage and cache hit rates across different cache retention times.

Hit-rate percentages can be hard to internalize, so Fox offers another measure: how many times does the platform prefill the same chunk of tokens? For the illustrated conversation, Fox reports roughly 15–16 prefills of the same token chunk at a one-minute TTL, with longer retention bringing the average toward one. This makes the waste tangible: the platform repeatedly computes state it could have kept.

10:4510:53
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:45 · section reference included

The provider needs a productive hit-rate band

Fox next asks the audience to imagine operating its own inference service, whether self-hosted or hosted on dedicated infrastructure. He forecasts that arrangement becoming more common from 2026 onward. From the operator’s perspective, the question is how much useful output a given system can produce as context length and cache reuse change.

The relationship between cache hit rate and output-token production is nonlinear. Its shape depends on context length, accelerator choice and the arrangement of prefill and decode, including disaggregation. Fox labels three operating regions A, B and C, with C representing the desirable region; he characterizes A and B as unprofitable or poor value. Providers consequently have an incentive to keep users in a productive cache-hit band. That is the economic reasoning behind his description of subscriptions as cache allotments: agent workloads that continually recompute old context can consume the GPU capacity needed to serve useful output.

13:4613:55
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:46 · section reference included

Token storage needs capacity and transfer speed

A useful token-storage tier must satisfy three requirements together:

  • Capacity: retain enough of the working set to reach the region where additional cache begins yielding diminishing returns.
  • Store performance: accept KV state quickly enough that entries are not dropped before reaching the tier and GPUs do not stall waiting to offload them.
  • Fetch performance: return cached state quickly enough that GPUs do not stall waiting to use it.

The accelerator remains the resource the rest of the system is organized to keep productive. A large cache that cannot move data at the required rate does not solve the problem.

Keeping every session in high-bandwidth memory, or HBM, would be attractive, but Fox describes it as impractical, partly because of batching. DRAM is the common additional tier: it offers acceptable performance, but its capacity is limited and tightly coupled to compute. Expanding memory can therefore require expanding the surrounding system. Fox also criticizes available pooled-DRAM approaches for imposing performance penalties.

WEKA’s Augmented Memory Grid connects inference systems to the company’s existing storage product, previously used in AI training and HPC. Fox describes it as a supported, optimized connector backed by NVMe. He claims roughly 1,000-fold greater density, but does not specify the comparison basis. Capacity is only one side of the comparison: Fox also describes an alternative storage system delivering 50–60 GB/s, yet still too slow for the token-storage demands under discussion. The relevant test is whether a tier can preserve enough state and deliver it fast enough to benefit inference.

16:0516:25
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

16:05 · section reference included

Exercise the working set, not just the server

The load generator behaves like a provider managing incoming demand. It can constrain load using optional SLOs, which Fox names as time to first token and output tokens per request. The latter is the wording used in the presentation; its time unit is not specified. Disabling the controls drives load without those constraints. A test can hold the number of coding-agent users constant or increase it over time, progressively exercising more of the memory hierarchy.

Fox points to his testing methodology and GitHub tooling for details, then distinguishes two access patterns:

PatternWorkloadWhat it exposes
SequentialTraverse an initial working set in orderA sharp performance cliff when the tier overflows
Random, growing poolIncrease the user pool and sample prompts randomlyA blend of hits across memory tiers

Sequential traversal is deliberately deterministic. Even a small overflow can produce a large performance drop. Random sampling is the fairer comparison in Fox’s account because some requests still hit HBM while others hit a second tier such as DRAM. It measures the resulting mixture rather than forcing every access through the same overflow pattern.

18:5119:02
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

18:51 · section reference included

What changes when the workload outgrows DRAM?

The decode-focused results compare three configurations:

Plot colorConfiguration
PurpleHBM plus WEKA
OrangeHBM plus DRAM
Orange-pinkHBM plus DRAM plus another POSIX storage system

The dotted line tracks the growing user pool. In the initial shaded region, all three configurations benefit primarily from HBM cache hits. As the pool grows, the working set exceeds what the DRAM tier can hold, and both the orange and orange-pink configurations decline sharply.

The WEKA configuration also declines as fewer accesses benefit from HBM. The benchmarking tool automatically pulls back concurrency, and the configurations eventually settle into steadier operation. WEKA Labs reports that its configuration sustains more users and more output tokens at steady state than the two alternatives in this decode-focused comparison. The presentation does not give numerical values for that advantage.

Fox reports a larger advantage in the disaggregated prefill comparison, attributing it to GPU efficiency with large prefill batches and a single decode. Here the configurations help separate two bottlenecks. Purple versus orange demonstrates the benefit of having more cache capacity and therefore more opportunities to hit. The orange-pink configuration has enough capacity for all the possible hits, but cannot deliver the cached state into the GPUs quickly enough to realize that benefit.

A cache hit must arrive in time to be useful. Fox characterizes WEKA’s result as combining greater capacity with DRAM-speed access, allowing the benefit to persist for longer periods. The final comparison completes the earlier working-set argument: extending retention reduces repeated prefill only when the storage tier can also sustain the writes and fetches needed to keep the accelerators working.

20:2620:36
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

20:26 · section reference included

Put the workload assumptions to the test

Bercovici closes by inviting developers to download, test, fork and improve the toolkit. The Inference Server Cache Performance Testing Suite provides the practical entry point for exploring these workloads. The repository and quick-start slide accompanies that invitation, followed by a QR code for more information and a promise of links to the referenced blogs. The intended outcome is less token anxiety and less prompt-cache arbitrage through platforms that preserve and serve context more effectively.

Slide showing the Inference Server Cache Performance Testing Suite quick start beside the callanfox/kv-cache-tester GitHub file listing.Open full source frame
The context platform engineering toolkit’s GitHub repository and quick-start instructions.
23:0723:18
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

23:07 · section reference included

Resources

From the talk

Updates since the talk

Transcript download
Full transcript

Loading transcript…