# Context Platform Engineering to Reduce Token Anxiety — Val Bercovici and Callan Fox, WEKA

Val Bercovici · Callan Fox, WEKA

AI Engineer Code 2025

[Talk page](https://ai.engineer/talks/NTBX-wxUhHs-context-platform-engineering-reduce-token-anxiety) · [Video](https://www.youtube.com/watch?v=NTBX-wxUhHs)

## Context Platform Engineering to Reduce Token Anxiety

Agent context is reusable only if the inference platform retains it and retrieves it quickly enough. WEKA’s toolkit explores how cache lifetime, memory tiers and workload shape affect that reuse.

Familiarity with LLM context windows, tool-calling agents and the distinction between prompt processing and token generation will help.

### How do you keep an agent’s context available?

[0:09](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=9s)

How do you build an inference platform that can serve agent swarms without repeatedly processing the context they already supplied? WEKA opens with a toolkit for investigating that problem. Callan Fox’s load generator models agents and subtasks with configurable **service-level objectives**, deterministic or random prompt cycles, model parallelism, aggregated or disaggregated prefill and decode, and different memory tiers. These are connected choices: the workload determines what context is reused, while the platform determines whether that reuse actually happens.

![Diagram showing a load generation and replay engine, coding agents, an inference API, prefill and decode compute, and two memory tiers.](https://ai.engineer/api/talk-essay-media/19ff46bf1c17940c347a490a695fa26d09262921905f9664fbd3b68cface3fc8)

WEKA’s benchmarking methodology connects load generation, coding agents, inference and memory tiers.

Val Bercovici invites developers to download, modify and contribute to the open-source toolkit. Its motivation follows [Manus’s context-engineering lessons](https://manus.im/blog/Context-Engineering-for-AI-Agents-Lessons-from-Building-Manus), which identify **KV-cache hit rate** as the most important metric for their production agents. A KV cache holds attention keys and values computed for earlier tokens; reusing that state avoids repeating the corresponding prefill work. Context platform engineering concerns the infrastructure that makes those hits possible.

### The prediction problem behind prompt-cache pricing

[2:27](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=147s)

Token anxiety starts with hitting limits while trying to get work done. Bercovici’s ambition is to engineer platforms that remove those constraints; the immediate alternative is what he calls prompt-cache arbitrage. Developers must balance ordinary input and output prices against separate cache-write and cache-read prices. Buying retention only pays off if enough subsequent requests reuse the cached material.

The Anthropic example makes the prediction concrete: should you pay to retain context for five minutes or an hour? The answer depends on how many reads arrive before the cache expires. The pricing slide highlights **Claude Sonnet 4.5** across those separate categories. Cache writes must also be distinguished from ordinary input: [current Claude documentation](https://platform.claude.com/docs/en/build-with-claude/prompt-caching) maintains that distinction, so an ordinary-input versus cache-read comparison should not be treated as a cache-write quote. The platform-engineering goal is to make useful retention less dependent on developers guessing future request timing.

![Claude Docs pricing table with the Claude Sonnet 4.5 row highlighted across input, cache-write, cache-hit and output columns.](https://ai.engineer/api/talk-essay-media/a55c9b68d67c0067515148b02a82747387cfa71893c63abb9564d072ccd7c8a2)

Claude pricing separates input tokens, cache writes, cache hits and output tokens.

### Two feedback loops, one storage obligation

[3:51](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=231s)

Human feedback and agent activity run at different speeds. A person may pause to review a result while parallel agents and subtasks keep working, generating and consuming context in the background. Much of that context could be reused, but its availability depends on what survives in the platform’s cache between requests. This turns a token-pricing question into a **token-storage problem**.

Subscription commitments and explicit cache-write or cache-read choices become infrastructure obligations. The platform must translate its service-level agreements into operational objectives, then provision and manage KV caches and memory tiers to meet them. Fox’s interpretation is that purchasing a token tier or paying for cache retention effectively purchases slots in token storage. This is a useful model of provider economics, rather than a disclosed specification of how every subscription is enforced. The engineering task is to optimize the infrastructure so its SLOs meet the promised service as consistently as possible.

### Inside a coding-agent conversation

[5:36](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=336s)

Fox begins his trace walkthrough with successive requests drawn as columns. Salmon represents new tokens, gray represents tokens that could be cached, and blue represents output tokens. Blue dots beneath the columns mark actual user responses. Context accumulates until it reaches a high watermark imposed by the model or inference provider. The agent then summarizes and begins another cycle. That compression can lose details and reduce the agent’s fidelity; raising the usable watermark would let it carry more of the original context forward.

Looking inside the gray region reveals why user-message counts are a poor description of the workload. In the Claude Code example, system instructions and user text occupy a relatively small share. Tool calls and tool results dominate: the agent runs a Bash command, consumes its result, and decides what to do next. Each step adds material that can become part of the next request’s reusable context.

Fox reports median request intervals of roughly 10–15 seconds in the agent data he describes, with the timing strongly dependent on whether humans approve each tool call. He says the mean interval can reach minutes or even hours because human responses take much longer. His observations span billions of tokens. The median captures the rapid tool loop; the mean exposes the long pauses that a retention policy must also handle.

### More agents create more opportunities—and more demand

[8:18](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=498s)

A multi-agent workflow adds an orchestrator and specialized subagents. Some subagents discard their context between activations; others retain it. Giving each subagent context tailored to its task helps concentrate attention on the relevant problem, but it also increases total context consumption. Fox recolors the previously gray regions to expose shared material across agents and show how those relationships evolve before work returns to the orchestrator and another agent wakes. The exact pattern varies across Codex, Claude Code and other systems.

**Potentially reusable context is not the same as a cache hit.** In the next view, yellow regions identify material that must be refreshed instead of reused. Fox says an uncached input can cost potentially ten times as much as a cached read. The historical Claude Sonnet example in Manus’s post supports that ordinary-input comparison; cache-write charges are a separate category. For flat-rate subscribers, Fox argues that missed reuse can instead appear as earlier rate limits. Reducing those misses benefits both the user and the provider doing the repeated computation.

### Longer retention trades memory for less prefill

[10:45](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=645s)

Columns of successive requests hide elapsed time. Fox therefore switches to a temporal view: solid lines use the left axis to show the **working set**, the number of tokens held in cache, while dotted lines use the right axis to show cache hit rate. The red one-minute retention policy repeatedly rises and falls when requests arrive more than a minute apart. A few requests may hit, then the cached state expires and the next request has to rebuild it.

| Retention | Effect in the illustrated conversation | Storage consequence |
|---|---|---|
| One minute, red | Expires across relatively short pauses | Repeated cache refreshes |
| Five minutes, blue | Bridges more request gaps | Retains context longer |
| One hour | Preserves reuse across longer pauses | Holds more tokens for longer |

The five-minute policy improves reuse but still misses longer gaps. Extending retention to an hour increases the working set, eventually substantially. Better reuse therefore requires memory tiers with enough capacity to preserve the context across those pauses.

![Time-series chart with solid working-set lines, dashed cache-hit-rate lines, and annotations identifying both axes and short-TTL inefficiencies.](https://ai.engineer/api/talk-essay-media/2095b09de86457f4592e6488f426da44f5d688f971b5aaaaf720f3d08384ef45)

Working-set usage and cache hit rates across different cache retention times.

Hit-rate percentages can be hard to internalize, so Fox offers another measure: how many times does the platform prefill the same chunk of tokens? For the illustrated conversation, Fox reports roughly 15–16 prefills of the same token chunk at a one-minute TTL, with longer retention bringing the average toward one. This makes the waste tangible: the platform repeatedly computes state it could have kept.

### The provider needs a productive hit-rate band

[13:46](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=826s)

Fox next asks the audience to imagine operating its own inference service, whether self-hosted or hosted on dedicated infrastructure. He forecasts that arrangement becoming more common from 2026 onward. From the operator’s perspective, the question is how much useful output a given system can produce as context length and cache reuse change.

The relationship between cache hit rate and output-token production is nonlinear. Its shape depends on context length, accelerator choice and the arrangement of prefill and decode, including disaggregation. Fox labels three operating regions A, B and C, with C representing the desirable region; he characterizes A and B as unprofitable or poor value. Providers consequently have an incentive to keep users in a productive cache-hit band. That is the economic reasoning behind his description of subscriptions as cache allotments: agent workloads that continually recompute old context can consume the GPU capacity needed to serve useful output.

### Token storage needs capacity and transfer speed

[16:05](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=965s)

A useful token-storage tier must satisfy three requirements together:

- **Capacity:** retain enough of the working set to reach the region where additional cache begins yielding diminishing returns.
- **Store performance:** accept KV state quickly enough that entries are not dropped before reaching the tier and GPUs do not stall waiting to offload them.
- **Fetch performance:** return cached state quickly enough that GPUs do not stall waiting to use it.

The accelerator remains the resource the rest of the system is organized to keep productive. A large cache that cannot move data at the required rate does not solve the problem.

Keeping every session in high-bandwidth memory, or **HBM**, would be attractive, but Fox describes it as impractical, partly because of batching. DRAM is the common additional tier: it offers acceptable performance, but its capacity is limited and tightly coupled to compute. Expanding memory can therefore require expanding the surrounding system. Fox also criticizes available pooled-DRAM approaches for imposing performance penalties.

WEKA’s [Augmented Memory Grid](https://www.weka.io/product/augmented-memory-grid/) connects inference systems to the company’s existing storage product, previously used in AI training and HPC. Fox describes it as a supported, optimized connector backed by NVMe. He claims roughly 1,000-fold greater density, but does not specify the comparison basis. Capacity is only one side of the comparison: Fox also describes an alternative storage system delivering 50–60 GB/s, yet still too slow for the token-storage demands under discussion. The relevant test is whether a tier can preserve enough state and deliver it fast enough to benefit inference.

### Exercise the working set, not just the server

[18:51](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=1131s)

The load generator behaves like a provider managing incoming demand. It can constrain load using optional SLOs, which Fox names as time to first token and output tokens per request. The latter is the wording used in the presentation; its time unit is not specified. Disabling the controls drives load without those constraints. A test can hold the number of coding-agent users constant or increase it over time, progressively exercising more of the memory hierarchy.

Fox points to his [testing methodology](https://medium.com/@callan.j.fox/evaluating-management-of-kv-cache-within-an-inference-system-2d7c3d266c3a) and GitHub tooling for details, then distinguishes two access patterns:

| Pattern | Workload | What it exposes |
|---|---|---|
| Sequential | Traverse an initial working set in order | A sharp performance cliff when the tier overflows |
| Random, growing pool | Increase the user pool and sample prompts randomly | A blend of hits across memory tiers |

Sequential traversal is deliberately deterministic. Even a small overflow can produce a large performance drop. Random sampling is the fairer comparison in Fox’s account because some requests still hit HBM while others hit a second tier such as DRAM. It measures the resulting mixture rather than forcing every access through the same overflow pattern.

### What changes when the workload outgrows DRAM?

[20:26](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=1226s)

The decode-focused results compare three configurations:

| Plot color | Configuration |
|---|---|
| Purple | HBM plus WEKA |
| Orange | HBM plus DRAM |
| Orange-pink | HBM plus DRAM plus another POSIX storage system |

The dotted line tracks the growing user pool. In the initial shaded region, all three configurations benefit primarily from HBM cache hits. As the pool grows, the working set exceeds what the DRAM tier can hold, and both the orange and orange-pink configurations decline sharply.

The WEKA configuration also declines as fewer accesses benefit from HBM. The benchmarking tool automatically pulls back concurrency, and the configurations eventually settle into steadier operation. WEKA Labs reports that its configuration sustains more users and more output tokens at steady state than the two alternatives in this decode-focused comparison. The presentation does not give numerical values for that advantage.

Fox reports a larger advantage in the disaggregated prefill comparison, attributing it to GPU efficiency with large prefill batches and a single decode. Here the configurations help separate two bottlenecks. Purple versus orange demonstrates the benefit of having more cache capacity and therefore more opportunities to hit. The orange-pink configuration has enough capacity for all the possible hits, but cannot deliver the cached state into the GPUs quickly enough to realize that benefit.

**A cache hit must arrive in time to be useful.** Fox characterizes WEKA’s result as combining greater capacity with DRAM-speed access, allowing the benefit to persist for longer periods. The final comparison completes the earlier working-set argument: extending retention reduces repeated prefill only when the storage tier can also sustain the writes and fetches needed to keep the accelerators working.

### Put the workload assumptions to the test

[23:07](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=1387s)

Bercovici closes by inviting developers to download, test, fork and improve the toolkit. The [Inference Server Cache Performance Testing Suite](https://github.com/callanjfox/kv-cache-tester) provides the practical entry point for exploring these workloads. The repository and quick-start slide accompanies that invitation, followed by a QR code for more information and a promise of links to the referenced blogs. The intended outcome is less token anxiety and less prompt-cache arbitrage through platforms that preserve and serve context more effectively.

![Slide showing the Inference Server Cache Performance Testing Suite quick start beside the callanfox/kv-cache-tester GitHub file listing.](https://ai.engineer/api/talk-essay-media/ffd95e4e79c39590aea946a64c8756abf2a4e52fe3b806bd5f287c7afa36ae29)

The context platform engineering toolkit’s GitHub repository and quick-start instructions.

## Resources

- [Inference Server Cache Performance Testing Suite](https://github.com/callanjfox/kv-cache-tester) — Tools for benchmarking KV-cache reuse, varying working sets, and replaying agent traces against inference servers.
- [Evaluating management of KV Cache within an inference system](https://medium.com/@callan.j.fox/evaluating-management-of-kv-cache-within-an-inference-system-2d7c3d266c3a) — Callan Fox's testing guide covers baseline checks, offload overhead, memory-tier isolation, and blended working-set workloads.
- [Importance of Context Platform Engineering](https://www.linkedin.com/pulse/importance-context-platform-engineering-callan-fox-i81wc/) — Fox's companion explanation of agent context, retention, provider economics, and WEKA's three-tier benchmark comparisons.
- [A Visual Guide to How Agents Consume AI Inference](https://www.linkedin.com/pulse/visual-guide-how-ai-agents-use-inference-inside-llm-callan-fox-q9brc) — Visual explanations of context growth, subagents, and missed cache reuse in coding-agent sessions.
- [Context Engineering for AI Agents: Lessons from Building Manus](https://manus.im/blog/Context-Engineering-for-AI-Agents-Lessons-from-Building-Manus) — Manus's production lessons on stable prefixes, append-only context, cache breakpoints, and agent design.
- [Claude prompt caching documentation](https://platform.claude.com/docs/en/build-with-claude/prompt-caching) — Current API guidance for prompt caching, retention options, token accounting, and pricing.

## Slides

### Context Platform Engineering (CPE) @ AIE CODE

Val Bercovici and Callan Fox introduce benchmarking agents across KV cache memory tiers and announce that WEKA is open sourcing its Context Platform Engineering toolkit.

[0:17](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=17s) [Title slide about benchmarking agents across KV cache memory tiers, with presenter names and portraits.](https://assets.ai.engineer/sites/aiecode2025/sha256/f8144fc30cfe8c0bfa6b9283ade6df3bd038d18aa48a39c343a4b662ea08fe42.webp)

Context Platform Engineering (CPE) @ AIE CODE

Benchmarking and testing Agents across KV Cache memory tiers

Callan Fox
Principal Product Manager

Val Bercovici
Chief AI Officer

© WEKA® 2025
1
WEKA®

### Open Sourcing: WEKA Benchmarking and Testing Methodology

A load generator replays coding-agent sessions against an inference API, tracking TTFT and output tokens per second per request across prefill, decode and tiered memory configurations.

[0:24](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=24s) [Diagram showing a load generator, coding agents, an inference API, prefill and decode compute, and memory tiers.](https://assets.ai.engineer/sites/aiecode2025/sha256/d346692e65a2eae30037bccb3c3e179f6e88d3d1c7a16a085c622245f5b053d9.webp)

Open Sourcing: WEKA Benchmarking and testing methodology

Load Generation and Replay Engine
Sessions chosen randomly out of the pool of users

Maintains adherence to SLOs of TTFT and Output Tokens/s per Request
TTFT
Output Token/s per Req

Coding Agent 1
Coding Agent 2
Coding Agent 3
Coding Agent 4
Coding Agent 5
Coding Agent N

Continue to expand pool of available sessions every couple of minutes

Users
Input Tokens
Output Tokens
Inference API
Dynamo
Hit
Miss
Compute Prefill (vLLM)
Compute Decode (vLLM)
Store
Memory Tier 1
Memory Tier 2
Memory hierarchy with various configurations

© WEKA® 2025
2
WEKA

### KV Cache Offload Performance Test Suite

The open-source toolkit benchmarks inference servers and KV-cache offload. Its quick start covers single prompts, cache hit rates, and memory tiers; the speakers invite feedback and contributions.

[1:00](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=60s) [Slide showing the test suite's quick-start instructions and repository file list.](https://assets.ai.engineer/sites/aiecode2025/sha256/5703a31f75ad1a91c5f6861c602da7c5f8fbd1e876e2f7cd174197fadbb781b6.webp)

GitHub Repo: KV Cache offload performance test suite

Inference Server Cache Performance Testing Suite

A toolkit for testing and benchmarking Large Language Model (LLM) inference servers and their KV Cache offload performance for optimal cache hit rates. There is an accompanying guide on how I use this coming soon to a public link.

Quick Start

# Install dependencies
pip install -r requirements.txt

# Run a simple single-prompt test
python single_prompt_tester.py \
--api-endpoint http://localhost:8000 \
--min-tokens 1000 \
--max-tokens 128000 \
--output-dir results

# Test various cache hit rates
python cache_rate_tester.py \
--api-endpoint http://localhost:8000 \
--context-sizes 8000 32000 64000 \
--working-set-size 2000000 \
--output-dir output

# Test performance across different memory tiers
python working_set_tester.py \
--api-endpoint http://localhost:8000 \
--context-sizes 30000 \
--min-working-set-size 100000 \
--max-working-set-size 5000000 \
--working-set-increments 10

Interface text:
callanjfox / kv-cache-tester  Public
Code  Issues  Pull requests  Actions  Projects  Security  Insights
master  2 Branches  0 Tags  Go to file
callanjfox and claude  Fix graph layout: move summary statistics to title area  80e5672 · 3 days
launcher  Initial vLLM engine launcher with modular architecture
.gitignore  Initial vLLM engine launcher with modular architecture
CHANGELOG.md  Fix --strict-time-window, duplicate entries crash, and res...
README.md  Add sustained mode to working_set_tester.py
cache_rate_tester.py  Initial vLLM engine launcher with modular architecture
combine_graphs.py  Initial commit: KV Cache Tester
generate_index.py  Add sustained mode to working_set_tester.py
kv_cache_size.py  Initial commit: KV Cache Tester
regenerate_graphs.py  Initial commit: KV Cache Tester
requirements.txt  Initial commit: KV Cache Tester
single_prompt_tester.py  Initial commit: KV Cache Tester
working_set_tester.py  Fix graph layout: move summary statistics to title area

© WEKA 2025
3
WEKA

### CPE Simplifies Max KV Cache Hit Rates

Manus calls KV-cache hit rate its most important metric for a production-stage AI agent because it affects latency and cost. The speaker says CPE simplifies reaching maximum hit rates.

[1:21](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=81s) [Slide highlighting a passage that describes KV cache hit rate as important to agent latency and cost.](https://assets.ai.engineer/sites/aiecode2025/sha256/a9fe998c002c866bcc149f011f94045782c8442d6b8215653eabd0def569311a.webp)

CPE simplifies max KV Cache hit rates

Design Around the KV-Cache

If I had to choose just one metric, I'd argue that the KV-cache hit rate is the single most important metric for a production-stage AI agent. It directly affects both latency and cost. To understand why, let's look at how a typical agent operates:

© WEKA® 2025
4
WEKA

Interface text:
manus.im/blog/Context-Engineering-for-AI-Agents-Lessons-from-Buildin...
4
1

### CPE Solves Token Anxiety

The chart shows nearly 40,000 rate-limited requests on Jun 13. The speaker says context platform engineering can eliminate token rate limits.

[2:27](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=147s) [Slide showing a rate-limited requests chart overlaid with an illustration of stressed developers.](https://assets.ai.engineer/sites/aiecode2025/sha256/9f9244821a773fed047a3a5fe3016d036183866f013beb7e537a42856fe62278.webp)

CPE Solves Token Anxiety

Rate-limited requests
The number of requests that were blocked due to rate limits

Y-axis ticks: 0, 10,000, 20,000, 30,000, 40,000
Timeline: Jun 01, Jun 07, Jun 13
Legend:
Claude Sonnet 4 - Input Tokens per Minute Cache Aware
Claude Opus 4 - Input Tokens per Minut[illegible]
Claude Sonnet 4 - Output Tokens per Min[illegible]

Illustration:
DEVS WEEP AS ANTHROPIC ANNOUNCES RATE LIMITS
100%
AI
USAGE
AI
[illegible]

© WEKA® 2025
5
WEKA®

### Claude token and cache pricing

The highlighted Claude Sonnet 4.5 row shows different prices for input, cache writes, cache hits and output. The speaker describes balancing write costs against predicted cache hits over 5-minute or 1-hour intervals.

[2:45](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=165s) [Claude documentation pricing table comparing token and cache rates across models.](https://assets.ai.engineer/sites/aiecode2025/sha256/19ec03d70bec782f658d0f261dae477c286d7d1d7ac5cb40447519460919f19b.webp)

Model | Base Input Tokens | 5m Cache Writes | 1h Cache Writes | Cache Hits & Refreshes | Output Tokens
Claude Opus 4.1 | $15 / MTok | $18.75 / MTok | $30 / MTok | $1.50 / MTok | $75 / MTok
Claude Opus 4 | $15 / MTok | $18.75 / MTok | $30 / MTok | $1.50 / MTok | $75 / MTok
Claude Sonnet 4.5 | $3 / MTok | $3.75 / MTok | $6 / MTok | $0.30 / MTok | $15 / MTok
Claude Sonnet 4 | $3 / MTok | $3.75 / MTok | $6 / MTok | $0.30 / MTok | $15 / MTok
Claude Sonnet 3.7 (deprecated) | $3 / MTok | $3.75 / MTok | $6 / MTok | $0.30 / MTok | $15 / MTok
Claude Haiku 4.5 | $1 / MTok | $1.25 / MTok | $2 / MTok | $0.10 / MTok | $5 / MTok
Claude Haiku 3.5 | $0.80 / MTok | $1 / MTok | $1.6 / MTok | $0.08 / MTok | $4 / MTok
Claude Opus 3 (deprecated) | $15 / MTok | $18.75 / MTok | $30 / MTok | $1.50 / MTok | $75 / MTok
Claude Haiku 3 | $0.25 / MTok | $0.30 / MTok | $0.50 / MTok | $0.03 / MTok | $1.25 / MTok

Interface text:
Claude Docs
English
Search... ⌘K
Console
Home
Developer Guide
API Reference
Model Context Protocol (MCP)
Resources
Release Notes
Model deprecations
Pricing
Build with Claude
Features overview
Using the Messages API
Context windows
Prompting best practices
Capabilities
Prompt caching
Context editing
Extended thinking
Streaming Messages

### User–agent cadence mismatch

Human feedback moves slowly, while agents run faster inference loops and consume tokens in the background, many of which may be cacheable.

[3:49](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=229s) [Diagram contrasting a slow human feedback loop with fast agentic inference loops.](https://assets.ai.engineer/sites/aiecode2025/sha256/e9ce50b292b72e68ca271120b9e7a32e30e357f67378d6205399ef9ac380ced6.webp)

User – Agent cadence mismatch

Instruction
Slooowwww Human Feedback
Actual Human
AI Agent
Feedback

Input
FAST Agentic Loops
Understand
Reason
Execute
Feedback
AI Inference
Response

Lets visualize this, with each loop as a column showing possible cache savings

© WEKA® 2025
7
WEKA

### Basics of prompt cache arbitrage

Cache hits draw on token storage; misses go through prefill, which stores tokens and transfers KV to decode. Users pay different rates for cache reads, cache creation and output tokens.

[4:24](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=264s) [Inference architecture showing routing, caching, prefill, decode and token storage.](https://assets.ai.engineer/sites/aiecode2025/sha256/b40342f45915b9e443bd51535158ba0d8dafcd532c77837e2fea038fd6289178.webp)

Basics of an inference system Prompt Cache Arbitrage

User 1
Input Tokens
Output Tokens
Inference API
Routing, Caching Layer
Hit
Miss
Compute Prefill
Compute Decode
Transfer KV to decode nodes
Store
Token Storage

Users are charged different rates for different token usage
These are "Cache read input tokens"
There are "Cache creation input tokens"
These are "Output tokens"
Prefill and decode are separated as they have different optimizations

Service Level Agreements (SLA)
Service Level Objectives (SLO)

© WEKA® 2025
8
WEKA

### Token storage and prompt cache arbitrage

Token storage holds user cache allotments for reuse. As caches age out, users can be evicted and their sessions need prefill again; the slide connects this cycle to agentic workload performance.

[4:54](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=294s) [Inference diagram expands token storage to show user cache allotments, expiration and eviction.](https://assets.ai.engineer/sites/aiecode2025/sha256/72eb12a72aaa1a5279e50f36ffd9748cfe289cf12c8affcdecd7e1258c70bdd4.webp)

Token storage is the Prompt Cache Arbitrage landscape

User 1
Input Tokens
Output Tokens
Inference API
Routing, Caching Layer
Hit
Miss
Compute Prefill
Compute Decode
Store
Token Storage
User 1
User 2
User 3
User 4
User N

When a user buys a subscription, what they are buying is an allotment of cache in the “memory hierarchy”

Token storage has a set TTL to allow this budget to constantly reset if not used
SLA
Users are eventually evicted and these sessions require prefill again

For agentic workloads, overall system performance is heavily influenced by token storage
Different strategies but basically that over time users cache is aged out
SLO

© WEKA® 2025
9
WEKA

### Agent cache observability

Context grows across prompts until a high watermark triggers summarization and a new cycle. Salmon shows new input tokens, grey shows potentially cached input, and blue dots mark user responses.

[5:53](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=353s) [Stacked columns show new and potentially cached input tokens across prompts, followed by summarization and a new cycle.](https://assets.ai.engineer/sites/aiecode2025/sha256/8bc6063443bdcd2503414e51fd2c2d214a65083b438f8e5f2ef88419406751fb.webp)

Vibe coded agent KC Cache observability proxy (Plotly graph)

Context Length (Tokens)
Prompts

Salmon is new input tokens
Blue is output tokens, very thin due to scale
Grey is possible cached input tokens
High watermark of context to trigger “summarization”
Summarization of previous cycle
Start of a new cycle
Initial input, and agent asking for feedback
Next, lets look at what these requests consist of

© WEKA® 2025
11
WEKA®

### Components of request context

In this Claude Code example, user text is a small part of growing request context; tool use and results make up much of the rest. Agent cycles can take seconds, while human cycles take minutes.

[6:57](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=417s) [Stacked prompt chart separates cached request components, cache misses and output as context grows.](https://assets.ai.engineer/sites/aiecode2025/sha256/521df03b7fb85829090f63539fc1c9ce05256bec84f651e9d8d340c2a0207c65.webp)

What the context of each request consists of

Request are made up of many components, varies by use case

Agent cycles could be in seconds, while humans cycles in minutes

Y-axis: Context Length (Tokens)
X-axis: Prompts

Legend:
Cache Miss
tool_result (cached)
tool_use (cached)
assistant_text (cached)
user_text (cached)
system (cached)
tools (cached)
Output

© WEKA® 2025
12
WEKA®

### Agent with sub-agents and context consumption

An orchestrator returns between research, architecture, coding, and testing tasks. Each sub-agent’s context grows as it works, targeting more context at a particular part of the problem.

[8:17](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=497s) [Chart shows context growing through research, architecture, coding and testing, with orchestrator steps between tasks.](https://assets.ai.engineer/sites/aiecode2025/sha256/b46b256d5d95774263afcfe108b0a4a2d57e3170b0ca10cdc4497ebdaf08c6f0.webp)

Agent with sub-agents and context consumption

Context Length (Tokens)
Prompts

Initialization and research
Orchestrator
Architecture
Orchestrator
Coding
Orchestrator
Testing and Integration
Orchestrator

© WEKA® 2025
13
WEKA®

### Relationships of context usage between agents and sub-agents

Each sub-agent shares some context and has its own. Context grows during research, architecture, coding, and testing; the orchestrator resumes between stages and wakes the next agent.

[9:07](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=547s) [Color-coded context chart illustrates common and unique context across agents and orchestrator steps.](https://assets.ai.engineer/sites/aiecode2025/sha256/a5c68f5373fe1571efde186225730850cd7236fc7b8d83d7db9e4d1dedf70b6b.webp)

Relationships of context usage between agents and sub-agents

Each sub-agent has some common context, and its own unique context

Y-axis: Context Length (Tokens)
X-axis: Prompts

Initialization and research
Orchestrator
Architecture
Orchestrator
Coding
Orchestrator
Testing and Integration
Orchestrator

© WEKA® 2025
14
WEKA®

### Realities of caching by inference providers

Gray bars show context that could be cached, while yellow marks cache misses that require recomputation. The speaker says API users pay input-token costs again for those misses.

[9:30](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=570s) [Agent context chart marks cache misses in yellow, indicating tokens recomputed unnecessarily.](https://assets.ai.engineer/sites/aiecode2025/sha256/f3fc0a9cde97a0b45547049ad7f7053264ba3ee6a74c99d685f5bf6328a6a49e.webp)

Realities of caching by inference providers (yellow cache miss)

These are tokens that needed to be recomputed unnecessarily (lack of cache, time of human feedback, etc)

Y-axis: Context Length (Tokens)
X-axis: Prompts

Initialization and research
Orchestrator
Architecture
Orchestrator
Coding
Orchestrator
Testing and Integration
Orchestrator

© WEKA® 2025
15
WEKA

### Caching inefficiencies from the limited size of many caches

A one-minute TTL lets cached tokens expire between requests. Five minutes improves the hit rate, while one hour holds more tokens to bridge longer gaps.

[10:44](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=644s) [Time-series chart compares working-set cache usage and hit rates at different cache durations, noting inefficiencies during conversation pauses.](https://assets.ai.engineer/sites/aiecode2025/sha256/49a2746edb80f5fe62b53451f5978be757883c4d69bf5f05e73355b489cef0af.webp)

Caching inefficiencies from the limited size of many caches

Cache hit rate (secondary Axis)
Working Set Cache Usage (Primary Axis)
Shorter cache TTL unable to ride out downtime in conversations leading to inefficiencies

Left axis: Working Set (Tokens in cache)
Left-axis ticks: 0, 50k, 100k, 150k, 200k, 250k, 300k, 350k
Right axis: Cache Hit Rate (%)
Right-axis ticks: 50, 60, 70, 80, 90, 100
Horizontal axis: Time
Timeline: Oct 4, 2025; 10:00, 11:00, 12:00, 13:00, 14:00, 15:00

Legend:
1 minute (Working Set)
5 minutes (Working Set)
15 minutes (Working Set)
1 hour (Working Set)
1 day (Working Set)
1 minute (Cache Hit Rate)
5 minutes (Cache Hit Rate)
15 minutes (Cache Hit Rate)
1 hour (Cache Hit Rate)
1 day (Cache Hit Rate)

© WEKA® 2025
16
WEKA

### KV cache hit rate and GPU prefills versus TTL

In this conversation, longer cache TTL raises the hit rate and reduces repeated GPU prefills. At one minute, the same tokens are prefilled about 15–16 times; with longer TTLs, the count approaches one.

[12:50](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=770s) [As cache TTL increases, cache hit rate rises and the number of GPU prefills falls.](https://assets.ai.engineer/sites/aiecode2025/sha256/17d5687573169d7af228f583007dc205bfb68bbf86f87cc841867a3d6350aeec.webp)

KV Cache hit rate vs (redundant) GPU Prefills vs TTL (1pYd)

KV Cache hit rate vs (redundant) GPU Prefills vs TTL

Left axis: Cache Hit Rate (%)
Left-axis ticks: 80, 85, 90, 95, 100
Right axis: Number of GPU Prefills
Right-axis ticks: 2, 4, 6, 8, 10, 12, 14, 16
X-axis: TTL
X-axis ticks: 1m, 2m, 3m, 4m, 5m, 10m, 15m, 30m, 1h, 2h, 4h, 8h, 1d

© WEKA® 2025
17
WEKA®

### Lessons from WEKA Labs: Context Platform Engineering

The talk turns to lessons from WEKA Labs on context platform engineering.

[13:46](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=826s) [Section title slide for lessons from WEKA Labs on context platform engineering.](https://assets.ai.engineer/sites/aiecode2025/sha256/f8e0b9d4276fc1b10002b5cc62c0269295e54f631d052210bba09b7d28e6ad6f.webp)

WEKA Augmented Memory Grid

Lessons from WEKA Labs:
Context Platform Engineering

© WEKA® 2025
18
WEKA®

### Prefill becomes dominant at larger context lengths

Output throughput rises sharply at high cache hit rates in region C. The speaker notes that the curve changes with context length and the accelerators used.

[13:53](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=833s) [A chart shows output throughput rising sharply at high cache hit rates, with regions A, B and C marked.](https://assets.ai.engineer/sites/aiecode2025/sha256/7a1105487044fb271205d16068cb3f9bd7b46e98ee132d77ac1d0f5790ec321f.webp)

CPE: Prefill becomes dominant at larger context lengths

Output Throughput Comparison Across Context Lengths

This is unearthing the ratio of time in prefill to time in decode

Y-axis: Output Tokens/s
Y-axis ticks: 500, 1000, 1500, 2000, 2500, 3000
X-axis: Cache Hit Rate (%)
X-axis ticks: 0, 20, 40, 60, 80, 100
Regions: A, B, C

As an inference provider, you obviously want to live in “C”

© WEKA® 2025
19
WEKA

### Token storage memory hierarchies

The diagram connects routing and prefill to token storage. Fast writes and retrieval help avoid blocking GPUs, while capacity supports cached context for more concurrent users.

[16:03](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=963s) [A diagram connects an inference API, routing and caching, prefill and decode compute, and token storage, with notes on performance and capacity.](https://assets.ai.engineer/sites/aiecode2025/sha256/774abcdc01e32d756c8ca60414cac02887516d301134bf8dc7c9e98534b6cf99.webp)

CPE to maximize token storage memory hierarchies (1pYd)

User 1
Input Tokens
Output Tokens
Inference API
Routing, Caching Layer
Hit
Miss
Compute Prefill
Compute Decode
GPU
GPU
Store
Token Storage
User 1
User 2
User 3
User 4
User N

Extreme retrieve performance to minimize TTFT while saving recomputing tokens

Integration with frameworks for disaggregated prefill and decode

Extreme store performance to not block concurrency, or have KV cache evicted before save

Integration of multiple "memory tiers" all working together as one

Enough capacity support a larger number of concurrent users, with prioritization of context

© WEKA® 2025
21
WEKA

### Different types of memory tiers

WEKA positions AMG near DRAM latency and storage capacity, while pooled local DRAM offers more capacity at higher latency. The slide claims AMG is 1000x denser than DRAM.

[17:00](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=1020s) [A capacity-versus-latency diagram positions HBM, DRAM, pooled local DRAM, WEKA AMG and storage.](https://assets.ai.engineer/sites/aiecode2025/sha256/961998f2ac913f29934773a8ec1714811f041bf625992fe3de1605943c2f2d2b.webp)

Different types of memory tiers

Y-axis: Latency (log scale), Less to More
X-axis: Memory Tier Capacity, Less to More

HBM
DRAM
Pooled Local DRAM
Storage (POSIX)
WEKA AMG (E-W network)

Tightly coupled with compute
Has the capacity but not the latency
DRAM speed at NVMe scale with NeuralMesh as Token Warehouse
1000x denser than DRAM

© WEKA® 2025
22
WEKA

### Open Source: WEKA CPE Toolkit

The toolkit replays coding-agent sessions against an inference system. User counts can stay fixed or grow to compare memory tiers, with optional TTFT and output tokens/s per request SLOs.

[18:43](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=1123s) [A load generation and replay engine supplies coding agent sessions to an inference system with prefill, decode and memory tiers.](https://assets.ai.engineer/sites/aiecode2025/sha256/2fbde00daf216d638a78629a4e7cda51af3f943d0abb0a22f05bc37bcce03762.webp)

Open Source: WEKA CPE Toolkit

Load Generation and Replay Engine
Sessions chosen randomly out of the pool of users
Maintains adherence to SLOs of TTFT and Output Tokens/s per Request
TTFT
Output Token/s per Req

Users
Coding Agent 1
Coding Agent 2
Coding Agent 3
Coding Agent 4
Coding Agent 5
Coding Agent N
Continue to expand pool of available sessions every couple of minutes

Input Tokens
Output Tokens
Inference API
Dynamo
Hit
Miss
Compute Prefill (vLLM)
Compute Decode (vLLM)
Store
Memory Tier 1
Memory Tier 2
Memory hierarchy with various configurations

© WEKA® 2025
23
WEKA

### Testing with parallelism and deterministic prompt cycling

The test cycles a working set of unique prompts in order across parallel GPUs. The speaker says even a slight memory-tier overflow causes a large performance drop.

[19:29](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=1169s) [GPU cache space and an external memory tier are shown alongside a working set divided into prompts cycled in order.](https://assets.ai.engineer/sites/aiecode2025/sha256/e0f2700dda514d80efaa64715d5ff4a8dbeae27e86bbc1ce0cc87e18ab744925.webp)

Testing with parallelism and deterministic prompt cycling

Total KV Cache Space

TP N
GPU_0: Model Weights | KV Batch | Spare
GPU_1: Model Weights | KV Batch | Spare
GPU_2: Model Weights | KV Batch | Spare
GPU_N: Model Weights | KV Batch | Spare

“External” Memory Tier

Needs to be larger than spare, but not exceed the size of “external”
Working Set Size
Headroom
Divided into unique prompts of N context length
Prompts cycled in deterministic order by default

© WEKA® 2025
24
WEKA

### Adding growing working sets and prompt random cycling

The test grows the working set and samples prompts randomly, exercising GPU cache and external memory tiers as the pool expands.

[20:00](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=1200s) [GPU cache space connects to two external memory tiers, with bars illustrating a working set growing during a test.](https://assets.ai.engineer/sites/aiecode2025/sha256/7edc9d5bf1cdc21066fcc7d81e32a5d2d5bad900fdfb54d2597da700e56caa3e.webp)

Adding growing working sets and prompt random cycling

Total KV Cache Space

TP N
GPU_0: Model Weights | KV Batch | Spare
GPU_1: Model Weights | KV Batch | Spare
GPU_2: Model Weights | KV Batch | Spare
GPU_N: Model Weights | KV Batch | Spare

“External” Memory Tier 1
“External” Memory Tier 2

Allows working set size to increase throughout the test and show the other memory tiers being exercised

© WEKA® 2025
25
WEKA

### Test results of WEKA Augmented Memory Grid

The talk turns to test results for the WEKA Augmented Memory Grid.

[20:25](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=1225s) [Results section title above a decorative grid with purple and pink squares.](https://assets.ai.engineer/sites/aiecode2025/sha256/0d8523c0a4ccf618b74956f626586486888fe48a722136cbaf7b0a28eff6c8fa.webp)

WEKA Augmented Memory Grid

Test results of WEKA Augmented Memory Grid

© WEKA® 2025
26
WEKA®

### Decode role, random user selection

As the user selection pool grows, HBM+WEKA settles around 300 output tokens/s, while the two DRAM configurations fall toward 100 tokens/s.

[20:29](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=1229s) [Chart compares output tokens per second across three memory configurations as the user selection pool grows.](https://assets.ai.engineer/sites/aiecode2025/sha256/f39f8b7f106f099f7108e5d7f9ca415b6006933eae54e339bb614f049479ae44.webp)

Decode role, random user selection,

Left axis: Output Token/s
Left-axis ticks: 0, 100, 200, 300, 400, 500, 600
Right axis: Users in Selection Pool
Right-axis ticks: 0, 20, 40, 60, 80, 100, 120, 140

Legend:
HBM+WEKA
HBM+DRAM
HBM+DRAM+FS
Users in Selection Pool

WEKA

### Prefill role, random user selection

At the largest selection pools, HBM+WEKA sustains roughly 280,000–300,000 input tokens/s while the other configurations fall toward 50,000. The speaker attributes its advantage to cache capacity at DRAM speed.

[22:00](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=1320s) [Chart compares input tokens per second across three memory configurations as the user selection pool grows.](https://assets.ai.engineer/sites/aiecode2025/sha256/e81351e4337071eb68dcc0941f30a0ea2ed3c9a02b5f3eeedb3a5f2885e852ab.webp)

Prefill role, random user selection,

Left axis: Input Token/s
Left-axis ticks: 0, 100000, 200000, 300000, 400000, 500000, 600000
Right axis: Users in Selection Pool
Right-axis ticks: 0, 20, 40, 60, 80, 100, 120, 140

Legend: HBM+WEKA; WEKA+DRAM; HBM+DRAM+FS; Users in Selection Pool

WEKA

### Augmented memory for context platform engineering

Scan the QR code for more information on augmented memory for context platform engineering.

[23:33](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=1413s) [QR code pointing to more information about augmented memory for context platform engineering.](https://assets.ai.engineer/sites/aiecode2025/sha256/7c7fc9a2a17cb29305b3a5aa61d79d4aa6a7fcebdc991df85e4d8fe5f1969819.webp)

More Information

Augmented Memory For Context Platform Engineering

WEKA

## Full transcript

[0:00](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=0s) This is Val Bercovici, WEKA's Chief AI Officer, and I am joined by

[0:05](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=5s) Callan Fox, head of the product management team here at WEKA.

[0:09](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=9s) And we're both thrilled to present Context Platform Engineering to you at the AI.Engineering Code Summit. Now, let's kick this off with, uh, an announcement we're making. We're actually open sourcing our Context Platform Engineering toolkit.

[0:26](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=26s) And this toolkit, uh, features a really cool load generator that Callan wrote that lets you configure agent swarms, uh, and agent subtasks with very specific SLOs, being able to cycle through deterministic and random prompt cycles, and engineer context platforms with all sorts of model parallelism options, disaggregated or aggregated prefill and decode options, and some really important memory

[0:51](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=51s) tiering options we're gonna be discussing here. So if we advance the next slide, we'll see that this is an open source toolkit that's already available to you on GitHub.

[1:02](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=62s) So Callan and I really encourage you to just get on GitHub, download this, play with it, and give us your feedback. Let us know what you need changed. Feel free to contribute and fork the project, uh, and advance the, the field of context platform engineering, which we're gonna be introducing to you later today.

[1:19](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=79s) So moving on, one of the key requirements for context platform engineering really relates to the context engineering, uh, insight that our friends at Manus shared with us earlier this summer in their c-- their pretty infamous now context engineering blog.

[1:36](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=96s) And they highlighted the fact that KV cache hit rate is the single most important metric for production grade AI agents. And the reason context platform engineering is so important is it dramatically simplifies reaching maximum KV cache hit rates, as we're about to show you.

[1:54](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=114s) On a more personal level, if we think about token anxiety, I know that each and every one of us, you know, feel that anxiety. The reason context platform engineering is so important is shared by the context engineering blog from Manus earlier this summer, where they particularly emphasize KV cache hit rates are the single most important metrics for

[2:16](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=136s) production grade AI agents. And context platform engineering quite simply maximizes KV cache hit rates in a very straightforward manner.

[2:27](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=147s) On a more personal note, if you think about t-the concept of token anxiety, as we all regularly hit token rate limits, context platform engineering helps you engineer platforms that eliminate token rate limits, uh, and help us be more productive with regards to developing our software.

[2:48](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=168s) Now, in the absence of context platform engineering, we often resort to context financial engineering, and that's fundamentally prompt cache arbitrage, where we balance the needs of pricing between the bookends of input and output tokens with these new token pricing categories that have appeared in the landscape over the past few months, focusing on cache writes and cache reads.

[3:09](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=189s) And we've got to be somewhat clairvoyant when we're doing this arbitrage to figure out how many cache writes we want to invest in for either five-minute time to live, in some cases, with Anthropic, for example, uh, we can do one hour time to live, and that's all against, balanced against the predictions we need to make on how

[3:27](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=207s) many cache reads and cache hits we think we're gonna have during those intervals. This becomes very, very tricky to be clairvoyant and predict the future, and I think it's much better to apply context platform engineering techniques to overcome token anxiety and prompt cache arbitrage than to continue to, to do the arbitrage in context financial engineering.

[3:47](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=227s) And so one of the ways we're gonna be doing that

[3:51](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=231s) is looking at, and C- and Callan's gonna dive into this deeply, the cadence mismatch between the relatively slow human feedback loops for agents, and then the agent swarms and the agent subtasks themselves that iterate at much higher cadence, often in parallel, waiting on humans, but conducting a lot of really cool work in the background, consuming a lot

[4:12](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=252s) of tokens in the background, many of which are cacheable, but we just never know how the platform is able to respond. And that's one thing we're gonna be diving into here, is the fact that if we go to the next slide, we're looking at fundamentally a token storage problem.

[4:28](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=268s) And what we're gonna be doing is explaining how the service-level agreements we sign up to when we subscribe to our various, uh, you know, token tiers, or we actually commit in our instructions and our agentic instructions a specific token cache writes and cache reads, how those SLAs convert to service-level objectives delivered by the context platform itself.

[4:51](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=291s) And more particularly, one of the insights that Callan reached from his research at WEKA Labs is that what we're doing when we actually subscribe to our token tiers or we actually pay for particular token writes, is we're really purchasing cache, KV cache slots in token storage.

[5:10](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=310s) So there's definitely a whole science around the context platform engineering to how context platforms take those SLA requirements, optimize infrastructure, optimize KV caching and memory tiers, and deliver specific SLOs to try and meet those SLAs as much as possible.

[5:27](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=327s) So with that, let me actually hand it over to Callan for, uh, actual research findings and lab and, and test results from WEKA Labs.

[5:36](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=336s) Thanks, Val. So look, what I wanna do is just go back to one of the slides that Val showed earlier. And what I'm gonna do from now on is I'm gonna focus on that right-hand loop, and the first thing I'm gonna do is I'm gonna start by visualizing what that loop actually looks like, and then we're gonna

[5:50](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=350s) go into a little bit more detail So if you, if you think about that loop as a column, and I've got a graph here that shows a very, very common, uh, pattern that happens in agents.

[6:04](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=364s) So the, the salmon color is showing new tokens that the system's being exposed to. The gray is something that could be cached, again, within a limited amount of cache.

[6:12](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=372s) We'll get into that shortly. The blue is the output tokens, and these blue dots down the bottom are showing when the user is actually giving responses in this particular case.

[6:21](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=381s) This is a really common example you get, where basically you start off, you consume context all the way up until you hit a, um, a high, a high watermark set by either the model maximum length or by the inference provider itself.

[6:35](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=395s) There's a summarization, um, phase, and then you start a new cycle. And everybody knows that summarization phase where sometimes, you know, the, the agent loses a little bit of its fidelity, a bit of its intelligence, and, uh, and that's why we're trying to, you know, uh, get more context engineering to a larger set of platforms, and we

[6:53](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=413s) can, we can raise that watermark. So if we go into this in a little bit more detail, the question I often get is, "Okay, well, the... what is that?

[7:01](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=421s) If that's all gray, what's, what's that made out of?" So here I'm able to, um, get the data and actually look at individual prompts and what actually makes them up.

[7:11](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=431s) So when you look at agentic data, especially agentic coding, the actual user input is only a really small part of it. And you can kind of see it here just visually, that if you just scan across the, the lighter watercolors, uh, the, um, the system prompt and the user text itself, and the rest of it is tool

[7:30](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=450s) use and tool responses. So, uh, this is, this one in particular is from Claude Code, where you're spending a lot of time, um, where the, the system is, you know, doing like, for example, a, a Bash command.

[7:42](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=462s) It's grabbing something, it's getting a result, and then it's doing something else. So where, where, where this really shows out in the data is if you actually look at the median time between requests, it may be something for, for a conversation that looks like that.

[7:54](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=474s) We have data for billions and billions and billions and billions of tokens. Um, the median time is ten seconds, fifteen seconds maybe. Um, that heavily depends on whether a human's involved in checking every single, uh, tool use.

[8:08](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=488s) But the mean time is in the minutes because the human... or even hours, because the human time to respond is much, much, much higher. And that's what we were showing before of the two sides of the loop.

[8:18](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=498s) So the other thing that's interesting and, and something that's very common today is, is, uh, is multi-agent. So you might have a core agent, which I've shown here as the orchestrator, and then you've got these sub-agents that are like spun up to do individual tasks.

[8:31](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=511s) And depending on the type of agentic, uh, coding, um, or just any agentic software in general, these agents or these sub-agents may be short-lived, as in their context does not endure between one wake-up and the next, or there are some, some when they do endure.

[8:49](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=529s) And it's really important to use our agents because it allows us to create, to effectively target more context at very particular parts of what the problem you're trying to solve.

[8:59](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=539s) But as a result, you do actually end up using more context, and I'll explain that very shortly. But if you visualize this gray section a different way and I show you the colors, you can kind of see how there's this common relationship of the common context between all of them.

[9:14](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=554s) Again, this is, varies a little bit depending on Codex versus Claude Code versus, versus others. But you can see how it changes over time and how the agents, um, relate to each other and have this common understanding, and then back to the orchestrator to, to wake up the next agent.

[9:32](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=572s) The, the, the, the thing that we're here to talk about today, though, mainly is that like while there's a lot of gray that could be cached, the reality is very different.

[9:40](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=580s) So if you send this to an inference provider, what ends up happening is you don't actually get one hundred percent of the cache hits that you could, um, that you could get.

[9:50](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=590s) Now, why does this matter? Well, there's two ways to look at this. If you're paying for API tokens, uh, you're literally... it's literally costing you more money because every time you see a yellow here, and this is just a simple example, you're paying input token costs.

[10:06](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=606s) So you're re, you're refreshing your cache and you're paying a full hit for that, so potentially ten times more than, than what you were if it was cached. If you're a subscription user and you're thinking, "Well, I don't care about the cost.

[10:18](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=618s) I don't pay for that, I pay a flat rate," that is true, but you're still, like we said before, you're paying for a subscription and that subscription is rate limited due, due to your cache usage, and, um, you may actually hit rate limits further, uh, or quicker.

[10:34](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=634s) So that's something that we want to be able to do, and we work with a lot of providers today to, to remove as much of this as possible. That's good for the user experience and it's also good for the provider.

[10:45](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=645s) So why does this happen? Well, I mean, it, it... If you think about the last graph where I show the columns, they're, they're not, they don't take into account time.

[10:53](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=653s) They're just one after the other after the other. But there's obviously, um, a, a, a temporal, uh, way to look at this. So this is the way that I like to think about it, and I know this is a little bit more of a complex graph to look at, but bear with me for a second.

[11:07](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=667s) So on the left-hand side, I'm talking about working set. So that's the number of tokens that the cache system is holding in its memory based on different time to lives of the co- of the actual cache itself.

[11:19](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=679s) And then the, the bit at the top, the dotted lines based on the right-hand secondary axis is showing the cache hit rate as a result. So the red is showing one minute time to live, and what you can see is there's prompts here at the start on the left where the, um, it's thrashing up and down.

[11:38](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=698s) And the reason it's doing that is the time between requests at that period is, is longer than one minute. So you're getting a period where you might, uh, take the cache, get a hit or two, and then drop the cache, and then you get another one, you've got to refresh it.

[11:53](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=713s) So it, it just, it doesn't really make sense, right? You go to five minutes, which is the blue, and you can now ride out more and more of those cache hits.

[12:01](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=721s) And as a result, you get a higher cache hit rate. You can see it at that very start, um, up there, uh, comparing the two. But then you're still missing many others.

[12:11](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=731s) There's still many times where the, the time between a request is even larger. So the next one up is showing one hour. And while that requires the cache system to hold, uh, you know, a little bit more tokens in cache and eventually quite a fair bit more tokens in cache, it's got to hold it for a longer

[12:29](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=749s) period of time. But the result to the end user is a better, um, actual experience and to the enterpr- to the, uh, inference provider, which I'll show very shortly, it's a much better experience for them as well.

[12:41](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=761s) The problem, though, is to do that, you need to be able to k- hold a lot of tokens in cache, and you need good memory tiers to support that.

[12:49](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=769s) Um, so the next thing I want to go into is that a lot of people think of... cache hit rate isn't really something that a human's able to really internalize well.

[13:01](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=781s) So another way that I can visualize it is by thinking about it in terms of the number of times on average that a chunk of, of tokens, which is a group of tokens, is refreshed.

[13:11](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=791s) So in this particular conversation that we're looking at here, you can see that there's this, this is showing the relationship of as I increase the time to live, how that affects my cache hit rate.

[13:21](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=801s) But it also shows, based on the secondary axis, that at one minute I'm literally rep- re- uh, prefilling like fifteen, sixteen times the same tokens. And over time we can get that all the way down to approaching one, um, and, um, make significant differences to, again, the experience of both the user and the inference provider.

[13:46](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=826s) So with that, what I'd like to do now is go into the, the context engineering side of it, some of the lessons we learned and, um, just sort of really drive this home.

[13:55](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=835s) So now I want you to think about, uh, what I think will, will be common in twenty twenty-six and onwards of people s- hosting their own or having their own dedicated systems hosting for them.

[14:06](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=846s) So imagine you being an inference provider now. Okay, so now what I want you to think of is think of yourself as an inference provider. Uh, maybe you've, um, you've, you know, worked with us or one of our partners to build our own, your own self-hosted instance.

[14:19](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=859s) Um, and, uh, you want to get the most out of it. What this graph is showing you is, uh, a relationship between a certain context length and the cache hit rate and how many output tokens you get as a result of that cache hit rate.

[14:33](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=873s) Now, the first thing you'll see is it's not linear, and, uh, it, it... and the, the shape of this curve will change based on the context length, based on the accelerators you use.

[14:43](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=883s) Ba- there's lots of things that come into it, how you do pis- uh, disagg and prefill. Uh, th- there's a lot of stuff that comes into it, but the c- the, the curve is more or less the same.

[14:54](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=894s) And if I asked you as an inference provider, where do you want to be? You'd obviously say C, and if you're in A or B, you're def- you're not making money or you're not getting enough value out of the system.

[15:05](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=905s) And inference providers that we work with, that they, that they have the same answer obviously. So the question is, well, how do they keep in C? And this is where it goes back to a, a slide that, um, Val showed earlier, where what they're doing is they're incentivizing users to stay within C.

[15:24](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=924s) And this is where we c- we came to the realization that a lot of the times, because of how much cache hit rate imp- uh, impacts your actual output, that's why s- you're buying cache allotments in storage when you're actually buying subscription services, because it is so important to them that you stay in a certain cache hit

[15:43](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=943s) rate band, especially for agentic workflows. Otherwise they literally... you'll just melt the GPU clusters that they have. Um, and I, and I think it's a really powerful thing to, to have in your head about how that works.

[15:57](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=957s) So what we're going to do now is go through and think about, okay, well, what, what makes up this token storage?

[16:05](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=965s) So when you think about the token storage, there's lots of aspects that, uh, the memory tiers that support the token storage need to be able to do. But to really make it really, really simple, it's literally as, as, a- as simple as you need enough capacity in these memory tiers so that you can hold a optimal amount

[16:25](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=985s) of cache. Uh, if you think back to the, the slides I just showed, there's this point where having more cache helps you a little bit, but it kind of gets to a point of diminishing returns.

[16:36](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=996s) Um, w- you need to get at least to that point, and you need to be able to store extremely fast into it, because if you can't, you're going to be dropping KVs before they're in the memory tier or you're going to be blocking GPUs, which is probably even worse.

[16:49](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=1009s) And then the other way you need to do it is you need to be able to fetch from that token storage very, very rapidly so that you can, again, not block the GPUs.

[16:57](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=1017s) They're the primary first-class citizen of this whole system.

[17:02](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=1022s) So what does it look like? So there's a few different types of memory tiers. The most common obviously is HBM and, uh, Val and I would love it if all our sessions were in HBM at all times.

[17:10](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=1030s) It's just not reasonable. Um, there's many reasons for this around how the batch works, but which we're not going to go into today. But the point is, is that the, the, the main common way that this is done today is DRAM.

[17:22](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=1042s) And there's nothing really wrong with DRAM as such. It, it, it's sort of a means to an end, but it's quite limited in size. It's, it's okay in terms of performance.

[17:31](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=1051s) But the other thing is, is it's tightly coupled with the compute. So if you want to expand your DRAM, there's not really many good ways to do that. There are some technologies out there that kind of do this, but the way they're implemented, they kind of just hurt your performance.

[17:45](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=1065s) And that's what I'm showing with pooled DRAM. You could pool more together, but it's, you know, it's kind of a, uh, uh, it, it doesn't help that much. So what we at WEKA, um, did is we took all the durable advantages of our product, which has been, you know, tried and tested in AI training and HPC environments,

[18:06](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=1086s) and Augmented Memory Grid is basically a, a supported, um, optimized connector between the inference systems and our, um, existing product. And because we're backed by NVMe, we- we're, we're much denser.

[18:21](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=1101s) We're like a thousand times, depending on how you look at it, denser. It's quite significant. And then I show another example of, uh, a storage at the top there where, you know, not, not something sluggish, something that can still get fifty, sixty gigabytes a second, but, uh, and it has the capacity, but still relative to what we're

[18:39](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=1119s) talking about is, is still quite slow. Okay. So then moving on to how do we test this? So again, um, we, we talked about how we're, we've open sourced this.

[18:51](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=1131s) Um, basically, um, Val already covered the, the main part of it and that the fact that it, it, it acts like it's an inference provider. It's trying to keep the load within two SLOs if you enable them.

[19:02](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=1142s) You actually don't have to enable them, and it'll just go as hard as it can regardless of, of an SLO being time to first token or output tokens per request.

[19:12](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=1152s) But the main thing that it can do is you can either set a static number of coding agent users, or you can, um, increase the number of those users over time so that you can slowly utilize more of the memory tiers and be able to compare different configurations.

[19:30](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=1170s) So there's two ways that it works. Um, I'll just be quick through these sections 'cause you can read about this. I have a blog that explains how I do with the testing that goes through all of this in detail, and there's obviously the GitHub as well.

[19:43](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=1183s) But basically it can do the initial working set and then sequentially go through those prompts. So this will be very, very, very deterministic 'cause as soon as you over- overflow the memory tier even the slightest bit, you'll see a massive drop-off in performance.

[19:59](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=1199s) But the other way that it can be done, and realistically the more fairer way that it can be done, is you can ex- increase the size over time, so the amount of concurrent users that you're accessing out of a pool, and you can randomly sample where in that sample set you'll get that, uh, prompt from.

[20:16](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=1216s) So sometimes you might be hitting HBM, sometimes you might be hitting your, your memory tier two, let's say it's that, let's say that's DRAM, and you get a really nice blended number.

[20:26](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=1226s) So with that, let's go in and tell you, show you some results and just sort of explain and, and show why we're so excited about what we're talking about today.

[20:36](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=1236s) So this is showing three comparisons. Comparison number one is HBM with WEKA, that's the purple. Uh, there's orange, which is HBM and DRAM. And there's the, you know, orangey pinky color with, uh, HBM plus DRAM plus that, uh, other, uh, POSIX system that I talked about earlier.

[20:54](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=1254s) The dotted line is showing, uh, concurrent users, so the amount, the amount of users that are in a pool, and that's increasing over time. So in the initial shaded area, you can see that all three of them get an advantage of HBM.

[21:08](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=1268s) The primary, uh, hit out of, uh, cache hit rate is coming out of HBM. But then over time, as we increase the users more and more and more, you're overflowing what the DRAM system, what the DRAM memory tier can do, and both orange and the pinky color start to drop off quite dramatically.

[21:27](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=1287s) Um, we also, from a WEKA perspective, also drop off 'cause we get less and less advantage from HBM, so we have to, uh, pull back our concurrency a little bit.

[21:37](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=1297s) The system does automatically, the, uh, the benchmarking tool. But then once we've sort of got down to the steady state, all three start to like, um, level out a little bit.

[21:48](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=1308s) But the main difference is, is that once you get down to that steady state, we can maintain that at a much higher amount of users at a much higher amount of output tokens.

[22:00](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=1320s) The other way that you look at this is, um, that was a decode-focused role. Um, if you look at a prefill-focused role, if you're doing disagg prefill, um, then the prefill's actually even better result for us because the systems, the GPUs are so much more efficient when you're doing large, um, batches of prefill tokens with a single

[22:21](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=1341s) decode. Um, then we, we can basically saturate things more fairly and, um, and it continues. Now the main difference between pink and orange is that we, uh, sorry, purple and orange, is that we have a lot more cache, so we can hit a lot more.

[22:39](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=1359s) The interesting thing about the orangey pinky color is that it also has the ability to hit every single thing that it's possible, but it's not fast enough to get it into the GPU for it to make a difference.

[22:51](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=1371s) And that's why we're sort of showing the difference between these three, 'cause with purple you're getting the advantage of capacity but at DRAM speed, so you can maintain that benefit for longer periods of time.

[23:03](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=1383s) And then maybe, Val, I'll hand back to you.

[23:07](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=1387s) Absolutely. No, that was a great walkthrough, Callan, of all of your research and benchmark results in WEKA Labs. So once again, we're thrilled to be announcing the open sourcing of this context platform engineering toolkit today.

[23:18](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=1398s) Please do download it, use it, give us your feedback. Again, feel free to fork it and improve it yourself. And we look forward together to just contributing to less token anxiety overall, less prompt cache arbitrage, and more context and context platform engineering in the future.

[23:33](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=1413s) A nice QR code for you to find out even more information. And at the end of this video in, um, in the actual transcript section and so forth, there'll be links to all the blogs we referenced here.

[23:44](https://www.youtube.com/watch?v=NTBX-wxUhHs&t=1424s) So thank you for joining us today, and we look forward to carrying on the context platform engineering conversation with you in the future.
