Without Caching, Your AI Agent Costs 10x More - George Cameron, Artificial Analysis
Loading slides…
Read the talk
Why a $21 agent task can cost more than $200 without prompt caching
George Cameron of Artificial Analysis traces the cost of a knowledge-work assignment across 165 model calls, then shows how to decide whether a more capable model will produce work worth paying for.
From a talk by George Cameron
At a glance
Ideas worth remembering
For each completed assignment, retain per-call input volume, cached input, reasoning and answer charges, and the number of turns. Those measurements let you reconstruct the bill and price the effect of cache misses.
Test reasoning-effort settings on complete assignments. Compare deliverable quality and total cost across the full history of calls.
Use a known correct result to set a capability threshold for bounded tasks. For open-ended analysis, compare the improvement in the work against the additional spend.
Use public evaluations to shortlist models, then measure the candidates with your own tasks, tools and inference setup.
Choosing a model also means choosing how to run it
George Cameron, co-founder of the independent benchmarking company Artificial Analysis, starts with a decision that a model leaderboard leaves open. Suppose you know which model you want for your agent. How will you access it? The inference provider affects how much you pay, how fast you get a response and how often repeated input qualifies for a cache discount. You can keep the model and the assignment fixed and still change the economics of delivering the result.
That relationship explains the breadth of the company’s benchmarking. Artificial Analysis evaluates coding agents and other agents, language models, and models that generate images, video, speech and music. It also measures the hardware and inference layers that serve them. Among the references to its work from OpenAI, Anthropic, Google and NVIDIA, Cameron singles out the two-axis charts. A single bar can tell a developer which model scored highest. Plotting capability against another quantity, such as cost, helps answer a question that often arrives between getting a prototype working and rolling it out in production.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A general intelligence score and a closer look at spreadsheets
The Artificial Analysis Intelligence Index combines ten evaluations into a single measure of general language-model capability. Its coverage includes knowledge work, scientific reasoning, coding, terminal use, knowledge and hallucination. The choice of evaluations reflects a practical ambition: help developers judge models for applications they can build now and the ones they expect to build next. Measuring progress toward AGI would be a different undertaking.
At the time of Cameron’s comparison, Anthropic’s Opus 5.5 led the index. Artificial Analysis listed more than 500 models, though, and it also published individual evaluation results. Those results give an engineer a way to narrow the field according to what an agent actually has to do. A difference in general intelligence can matter less to a particular application than a difference in one of the capabilities the index combines.
For the New York audience, Cameron picks AA Analyst Agent, a finance and accounting benchmark. One demand it places on an agent is navigating complex spreadsheets. The Gemini series performed particularly well there, as did Anthropic’s models. An engineer building a spreadsheet agent has a concrete reason to try Gemini even if the initial search began with the overall index leader.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
New releases keep changing the field, including for open weights
Cameron next follows the leading model from each of roughly twelve labs over time. Each release adds a point. The growing density of points captures one reason developers find it exhausting to keep up, while the scores themselves suggest to Cameron that capability gains are accelerating. A model selection made a few releases ago can predate a meaningful change in what an agent can accomplish.
The historical comparison also puts open-weights models within roughly three to nine months of leading proprietary models in capability. That interval matters when a team needs a level of intelligence the frontier reached some time ago. An open-weights model can enter consideration for that workload, bringing a different set of inference and pricing choices with it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The price of one task spans cents to dollars
By mid-to-late 2026, Cameron says, models could cost several dollars to complete a single assignment, a change from a year earlier. In his comparison, Opus and Fable each cost more than $5 per task, while some models cost 5 cents or less. That hundredfold spread makes model selection consequential even before an engineer multiplies the bill by production volume.
GPT-6 Luna came in at about 7 cents per task. Cameron rates it as substantially more capable than the original GPT-4 and further ahead of GPT-3.5 Turbo, the model he credits with opening up this wave of language-model applications. The inexpensive end of the comparison therefore includes candidates that would have looked exceptionally capable to someone building just a few model generations earlier.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Hold the intelligence target fixed and track its falling price
Artificial Analysis grouped models by the Intelligence Index score they could reach. For a threshold such as 30 or 50, it began when a model first reached that score, then tracked the cheapest model capable of reaching it at each later point. That method lets the identity of the cheapest qualifying model change while preserving the capability requirement.
A score of 30 initially cost a little over a dollar in this analysis. About a year later it cost roughly 10 cents, and by Cameron’s presentation it was in the 1-to-10-cent range. Across the tiers he describes, the cost of accessing a given level of intelligence fell about tenfold in roughly a year. To illustrate the planning implications, he imagines a large agent workload that costs $1 million today. If the pattern continued, the same level of intelligence might be accessible for $100,000 a year later and potentially $10,000 after two years. Those last two figures are projections from the observed decline.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Build the $21 Briefcase bill from one call and 165 steps
Reasoning models had already complicated an estimate based on the length of the returned answer. A model could generate billable reasoning tokens on its way to producing that answer. An agent adds another dimension. It can make many model calls before the user gets a deliverable, and the input to a later call can include work from earlier ones.
Cameron uses the AA Briefcase Agentic Knowledge Work Benchmark to put numbers on that process. An agent works in an environment containing thousands of files and receives an assignment such as creating a report, a PowerPoint presentation or a marketing video. Opus did the work well in Cameron’s assessment. The bill he wants to explain is the average across assignments, about $21 for one finished task.
Start with a single language-model call. Cameron corrects his estimate for the answer-token component to about 2 cents. Reasoning tokens add somewhat more. The model also processes input, whose cost depends on both its length and how much of it hits the prompt cache. Counting those components together, a call averaged about 13 cents. Now put the call back into the assignment. Opus took an average of 165 steps. Multiplying the rounded figures gives $21.45, close to the reported $21. Reading the final report and counting its words would reveal little about either factor in that multiplication.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Repeated input makes the cache hit rate visible on the bill
Within a Briefcase call, Cameron identifies input volume as the largest cost driver. The next question is what fraction of those input tokens hit the cache. A hit in his example receives roughly a tenfold discount on the ordinary input-token price. Message history makes the distinction especially important for an agent. Earlier material can accompany a new request, so the agent can process a substantial volume of input even when the latest addition to the conversation is small.
To reconstruct the charge, separate the cached input tokens from the remaining input on each call. Multiply each count by its applicable price, add the reasoning and answer charges, and sum the calls through completion. Cameron identifies the number of turns as the biggest driver of the full task bill in this example. Input volume and cache hit percentage explain much of the cost of one call. The turn count tells you how many times the agent incurs a call charge before it delivers the work.
He then takes the same Briefcase example and sets the cache hit percentage to zero. The estimated task price rises from about $21 to over $200. One way to encounter that failure is to run inference with vLLM and leave prompt caching disabled. The calculation also gives a team comparing inference providers a reason to examine the hit rate it can actually achieve alongside the advertised token prices. Artificial Analysis had begun publishing cache hit percentage benchmarks for that comparison.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Reasoning effort changes the cost of subsequent turns
Opus 5.5 at low reasoning effort cost about $1 per Briefcase task. At maximum effort it cost about $21. One component of that difference is straightforward: changing the setting changes how many reasoning tokens the model generates. Two further effects emerge across the assignment. The effort setting can change how many turns the agent takes, and reasoning carried forward in message history contributes to the input processed on later turns. A token generated as reasoning on one call can thus contribute to input charges downstream.
The price comparison alone leaves open how much the deliverable improved. It does establish that choosing Opus 5.5 still leaves an important configuration decision. Cameron then compares it with GLM 5.3 Flash, a cheaper open-weights model he considers highly capable. Across models, all the components of the bill can change. For long-running agents, he again singles out input-token prices and the proportion of input that hits the cache as major reasons two models can cost very different amounts to complete the work.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Read the cost–intelligence frontier in either direction
Knowing that GLM 5.3 Flash is cheaper than Opus 5.5, or that Luna is cheaper than Astra, still leaves the builder’s decision unresolved. Cameron starts with general intelligence plotted against cost per completed task. The Pareto frontier picks out models for which no other candidate in the comparison offers at least as much intelligence at a lower price, or more intelligence at the same price.
With a fixed budget, look along that frontier for the highest intelligence score within the spending limit. With a fixed capability requirement, start at the score the application needs and find the cheapest model that reaches it. The general index gives both searches a starting point. Deciding which direction to search requires knowing what a better answer to the user’s particular question would look like.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A cost analysis and an $80 subscription total reward different capabilities
Cameron calls an assignment an unlimited intelligence task when additional intelligence can keep improving the result. His example is to analyze what drives costs in a business and identify how to optimize them. An agent can get the figures right and still leave room for a better explanation of the costs or a better proposal for reducing them. For that assignment, the budget question is how much the business is willing to spend for a better analysis.
A limited intelligence task has a ceiling. Ask how much you spent on subscriptions last month. Given the right MCPs to access the relevant records, Luna would return $80 in Cameron’s example. Astra would also return $80. Once the agent can reliably find the records and calculate that total, a more capable model has no better number to offer. Here the engineer can establish the capability needed to get the answer right, then shop for the least expensive way to obtain it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Benchmark the shortlist inside the agent you intend to run
Cameron’s selection process moves from the public landscape to the team’s own work. Identify the metrics that matter, compare models on those priorities and make a shortlist. For a small prototype, try the candidates in the actual agent with its tools and context. For a production rollout, he recommends building a benchmark around the workload and running a shortlist of perhaps five models through it. Compare the quality of the completed deliverable with the full cost of obtaining it.
Artificial Analysis offers that process through a product called Optima. A team uploads its context, and Artificial Analysis works with the team using the Optima agent to generate a benchmark. The output is a set of trade-off curves based on the team’s definition of intelligence for its task. That gives the engineer a way to answer the question left open by the $1 and $21 Briefcase runs: on the assignment this team actually needs done, what improvement does the extra spend buy?
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Full transcript
Loading transcript…