AI Engineer World's Fair 2026
Your Agent Is Wasting Tokens and You Don't Know It
Loading slides…
Read the talk
Your Agent Is Wasting Tokens and You Don’t Know It
Repeated prompts, oversized tool results, unchecked loops, and growing histories all add to an agent’s bill. Five controls target those costs at different points in execution.
From a talk by Erik Hanchett
Before you start: You should be familiar with system prompts, model context windows, and the way an agent alternates between model calls and tool execution.
Cache repeated prompts
Your agent sends a large system prompt on its first call. On the next call, the instructions are unchanged: why pay to process them from scratch again? This is the opening example in the speaker’s five controls for agent token costs. the speaker introduces himself as a senior developer advocate at AWS and demonstrates the controls using Strands Agents, an SDK that supports different model providers.
The demonstration places cache_prompt="default" on BedrockModel and supplies the repeated instructions through the agent’s system_prompt. The following Python preserves that configuration pattern:
python
from strands import Agent
from strands.models import BedrockModel
SYSTEM_PROMPT = """
You are a support assistant. Use the supplied product documentation
when answering questions. Ask for missing details before diagnosing
an issue, and distinguish documented behavior from your suggestions.
"""
agent = Agent(
model=BedrockModel(cache_prompt="default"),
system_prompt=SYSTEM_PROMPT,
)
The short prompt makes the wiring visible; actual cache eligibility depends on the model’s minimum prefix length. the speaker describes subsequent calls as sending a much smaller system prompt. More precisely, Bedrock prompt caching reuses computation for an eligible, unchanged prefix and discounts cache reads; it does not necessarily reduce the prompt transmitted. Cache expiration and possible cache-write premiums affect the savings. The demonstrated cache_prompt setting belongs to the Bedrock provider and is now deprecated in favor of cache_config.
System instructions are only the first candidate. Repeated tool definitions and stable message content can also be cached where the provider supports it. Cache the content that stays the same across calls so repeated work can benefit from reuse.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Route tasks by difficulty
A demanding task may justify a newer frontier model. A simpler task may not need that expense. the speaker’s comparison uses Claude Haiku for cheaper, simpler work and Claude Sonnet for more difficult work, without specifying model versions or a price ratio.
| Task | Model choice in the example |
|---|---|
| Simpler work | Claude Haiku |
| More difficult work | Claude Sonnet |
The useful distinction is the capability the task requires. An agent does not need to send every operation to its most expensive model.
Routing can begin with an ordinary if statement. This function accepts two already configured Strands agents and chooses between them using a difficulty decision supplied by the application:
python
from strands import Agent
def route_task(
task: str,
difficult: bool,
cheap_agent: Agent,
capable_agent: Agent,
):
if difficult:
return capable_agent(task)
return cheap_agent(task)
Alternatively, a separate inexpensive model can classify the task and select the destination. In either case, the routing decision becomes an explicit part of the agent’s workflow, allowing different use cases to reach different models.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Offload large tool results
A tool can return far more information than the model needs for its next decision. the speaker introduces a manual Strands approach: store the large result locally or in cloud storage, then supply a summary. He also mentions supporting APIs without naming them.
The cost appears repeatedly when that large result remains in the context carried through the agent loop. The model can receive the same bulky output again on each subsequent call, even after the useful facts have already been extracted.
- Persist the full tool result outside the model context.
- Summarize the information needed for the next decision.
- Carry that compact representation into subsequent calls instead of repeatedly including the full result.
Offloading separates retaining the data from repeatedly paying to include it in context. Keeping a copy in storage alone does not achieve the goal if the full result still travels through every model call.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Cap loops and inspect tool behavior
Summarizing tool output reduces the size of repeated calls. The next control limits how often those calls happen. the speaker reports encountering agents that invoke the same tool over and over. He warns that an uncapped tool loop might run ten or twenty times, or continue indefinitely. These are failure examples, not measured workload results.
Set an explicit maximum iteration count so the agent cannot continue without a bound. the speaker supplies no recommended limit; the appropriate budget depends on the work the agent needs to complete.
Before deployment, inspect each tool with observability tooling:
- Call duration: How long does each execution take?
- Repetition: How many times does the agent call the tool or cycle through the loop?
Inspecting both reveals whether the expense comes from slow individual calls, repeated calls, or both. Those observations give you a basis for judging tool efficiency and changing the workflow.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Trim history while preserving a summary
A multi-turn conversation creates another source of repeated input. Each exchange grows the history, and subsequent model calls can resend everything that came before. the speaker describes whole-history resending as consuming hundreds or thousands of tokens as conversations grow. The issue is cumulative: an old exchange can keep contributing to the input long after it stops helping the current task.
A SlidingWindowConversationManager bounds the recent history retained for the model. the speaker uses a configurable ten-message window as his example. Set that value explicitly to express the example in Python:
python
from strands import Agent
from strands.agent.conversation_manager import (
SlidingWindowConversationManager,
)
agent = Agent(
conversation_manager=SlidingWindowConversationManager(
window_size=10,
),
)
Ten messages is the talk’s example, not the current SDK default: the current Python reference defaults to 40 and accounts for valid tool-use/result relationships during reduction.
The trade-off appears when the window drops the beginning of the conversation. Earlier requirements or decisions may disappear along with less useful exchanges. the speaker’s remedy is to summarize the older history and put that smaller representation into the context when the window boundary is reached. The retained context then combines recent detail with a compact account of what came before. A sliding window alone does not supply that summary; summarization is an additional step.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Use observations to guide the next change
The closing recap brings the controls back to the agent as a whole: reuse stable prompt content, choose models by difficulty, and keep bulky tool data out of repeated context. the speaker then returns to measurement—inspect tool durations and loop counts, and iterate from what they show. For longer conversations, keep checking what history gets carried forward as new messages arrive. The practical work is to examine the repeated model call: which content still belongs there, which model should handle it, and whether another iteration is justified.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Python reference for configuring retained conversation messages while preserving valid tool-call relationships.
Further reading
Explains reusable prompt prefixes, cache eligibility, expiration, and cache-read versus cache-write billing.
Examples of retaining recent messages and summarizing older conversation context.
Lifecycle hooks with examples for modifying tool results and enforcing per-tool invocation limits.
Updates since the talk
Current Bedrock provider configuration, including cache_config and the deprecated cache_prompt option.
A plugin that moves large tool outputs into storage and keeps previews and retrieval references in context.
Full transcript
Loading transcript…