AI Engineer World's Fair 2026

Your Agent Is Wasting Tokens and You Don't Know It

Loading slides…

Read the talk

Your Agent Is Wasting Tokens and You Don’t Know It

Repeated prompts, oversized tool results, unchecked loops, and growing histories all add to an agent’s bill. Five controls target those costs at different points in execution.

From a talk by Erik Hanchett

Before you start: You should be familiar with system prompts, model context windows, and the way an agent alternates between model calls and tool execution.

Cache repeated prompts

Your agent sends a large system prompt on its first call. On the next call, the instructions are unchanged: why pay to process them from scratch again? This is the opening example in the speaker’s five controls for agent token costs. the speaker introduces himself as a senior developer advocate at AWS and demonstrates the controls using Strands Agents, an SDK that supports different model providers.

The demonstration places cache_prompt="default" on BedrockModel and supplies the repeated instructions through the agent’s system_prompt. The following Python preserves that configuration pattern:

python

from strands import Agent
from strands.models import BedrockModel

SYSTEM_PROMPT = """
You are a support assistant. Use the supplied product documentation
when answering questions. Ask for missing details before diagnosing
an issue, and distinguish documented behavior from your suggestions.
"""

agent = Agent(
    model=BedrockModel(cache_prompt="default"),
    system_prompt=SYSTEM_PROMPT,
)

The short prompt makes the wiring visible; actual cache eligibility depends on the model’s minimum prefix length. the speaker describes subsequent calls as sending a much smaller system prompt. More precisely, Bedrock prompt caching reuses computation for an eligible, unchanged prefix and discounts cache reads; it does not necessarily reduce the prompt transmitted. Cache expiration and possible cache-write premiums affect the savings. The demonstrated cache_prompt setting belongs to the Bedrock provider and is now deprecated in favor of cache_config.

Slide titled “Cache the System Prompt” shows Agent and BedrockModel code with cache_prompt="default" and system_prompt=BIG_SYSTEM_PROMPT, alongside a presenter inset.Open full source frame
Cache the system prompt: the code example sets cache_prompt="default".

System instructions are only the first candidate. Repeated tool definitions and stable message content can also be cached where the provider supports it. Cache the content that stays the same across calls so repeated work can benefit from reuse.

0:000:18
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:00 · section reference included

Route tasks by difficulty

A demanding task may justify a newer frontier model. A simpler task may not need that expense. the speaker’s comparison uses Claude Haiku for cheaper, simpler work and Claude Sonnet for more difficult work, without specifying model versions or a price ratio.

TaskModel choice in the example
Simpler workClaude Haiku
More difficult workClaude Sonnet

The useful distinction is the capability the task requires. An agent does not need to send every operation to its most expensive model.

Routing can begin with an ordinary if statement. This function accepts two already configured Strands agents and chooses between them using a difficulty decision supplied by the application:

python

from strands import Agent


def route_task(
    task: str,
    difficult: bool,
    cheap_agent: Agent,
    capable_agent: Agent,
):
    if difficult:
        return capable_agent(task)
    return cheap_agent(task)

Alternatively, a separate inexpensive model can classify the task and select the destination. In either case, the routing decision becomes an explicit part of the agent’s workflow, allowing different use cases to reach different models.

1:011:15
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:01 · section reference included

Offload large tool results

A tool can return far more information than the model needs for its next decision. the speaker introduces a manual Strands approach: store the large result locally or in cloud storage, then supply a summary. He also mentions supporting APIs without naming them.

The cost appears repeatedly when that large result remains in the context carried through the agent loop. The model can receive the same bulky output again on each subsequent call, even after the useful facts have already been extracted.

  1. Persist the full tool result outside the model context.
  2. Summarize the information needed for the next decision.
  3. Carry that compact representation into subsequent calls instead of repeatedly including the full result.

Offloading separates retaining the data from repeatedly paying to include it in context. Keeping a copy in storage alone does not achieve the goal if the full result still travels through every model call.

1:512:00
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:51 · section reference included

Cap loops and inspect tool behavior

Summarizing tool output reduces the size of repeated calls. The next control limits how often those calls happen. the speaker reports encountering agents that invoke the same tool over and over. He warns that an uncapped tool loop might run ten or twenty times, or continue indefinitely. These are failure examples, not measured workload results.

Set an explicit maximum iteration count so the agent cannot continue without a bound. the speaker supplies no recommended limit; the appropriate budget depends on the work the agent needs to complete.

Before deployment, inspect each tool with observability tooling:

  • Call duration: How long does each execution take?
  • Repetition: How many times does the agent call the tool or cycle through the loop?

Inspecting both reveals whether the expense comes from slow individual calls, repeated calls, or both. Those observations give you a basis for judging tool efficiency and changing the workflow.

2:413:00
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:41 · section reference included

Trim history while preserving a summary

A multi-turn conversation creates another source of repeated input. Each exchange grows the history, and subsequent model calls can resend everything that came before. the speaker describes whole-history resending as consuming hundreds or thousands of tokens as conversations grow. The issue is cumulative: an old exchange can keep contributing to the input long after it stops helping the current task.

A SlidingWindowConversationManager bounds the recent history retained for the model. the speaker uses a configurable ten-message window as his example. Set that value explicitly to express the example in Python:

python

from strands import Agent
from strands.agent.conversation_manager import (
    SlidingWindowConversationManager,
)

agent = Agent(
    conversation_manager=SlidingWindowConversationManager(
        window_size=10,
    ),
)

Ten messages is the talk’s example, not the current SDK default: the current Python reference defaults to 40 and accounts for valid tool-use/result relationships during reduction.

The trade-off appears when the window drops the beginning of the conversation. Earlier requirements or decisions may disappear along with less useful exchanges. the speaker’s remedy is to summarize the older history and put that smaller representation into the context when the window boundary is reached. The retained context then combines recent detail with a compact account of what came before. A sliding window alone does not supply that summary; summarization is an additional step.

3:373:42
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:37 · section reference included

Use observations to guide the next change

The closing recap brings the controls back to the agent as a whole: reuse stable prompt content, choose models by difficulty, and keep bulky tool data out of repeated context. the speaker then returns to measurement—inspect tool durations and loop counts, and iterate from what they show. For longer conversations, keep checking what history gets carried forward as new messages arrive. The practical work is to examine the repeated model call: which content still belongs there, which model should handle it, and whether another iteration is justified.

Recap slide lists Cache the System Prompt, Route by Difficulty, Offload Big Tool Results, Cap Your Tool Loops, and Trim the History, with a presenter inset at lower right.Open full source frame
“Same Agent. Smaller Bill.” recaps five fixes for agent token costs.
4:515:05
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:51 · section reference included

Resources

From the talk

Updates since the talk

Transcript download
Full transcript

Loading transcript…