# Without Caching, Your AI Agent Costs 10x More - George Cameron, Artificial Analysis

George Cameron, Artificial Analysis

[Talk page](https://ai.engineer/talks/prGP0huECPo-without-caching-your-ai-agent-costs-10x) · [Video](https://www.youtube.com/watch?v=prGP0huECPo)

## Why a $21 agent task can cost more than $200 without prompt caching

George Cameron of Artificial Analysis traces the cost of a knowledge-work assignment across 165 model calls, then shows how to decide whether a more capable model will produce work worth paying for.

### Choosing a model also means choosing how to run it

[0:12](https://www.youtube.com/watch?v=prGP0huECPo&t=12s)

George Cameron, co-founder of the independent benchmarking company Artificial Analysis, starts with a decision that a model leaderboard leaves open. Suppose you know which model you want for your agent. How will you access it? The inference provider affects how much you pay, how fast you get a response and how often repeated input qualifies for a cache discount. You can keep the model and the assignment fixed and still change the economics of delivering the result.

That relationship explains the breadth of the company’s benchmarking. Artificial Analysis evaluates coding agents and other agents, language models, and models that generate images, video, speech and music. It also measures the hardware and inference layers that serve them. Among the references to its work from OpenAI, Anthropic, Google and NVIDIA, Cameron singles out the two-axis charts. A single bar can tell a developer which model scored highest. Plotting capability against another quantity, such as cost, helps answer a question that often arrives between getting a prototype working and rolling it out in production.

### A general intelligence score and a closer look at spreadsheets

[2:47](https://www.youtube.com/watch?v=prGP0huECPo&t=167s)

The Artificial Analysis Intelligence Index combines ten evaluations into a single measure of general language-model capability. Its coverage includes knowledge work, scientific reasoning, coding, terminal use, knowledge and hallucination. The choice of evaluations reflects a practical ambition: help developers judge models for applications they can build now and the ones they expect to build next. Measuring progress toward AGI would be a different undertaking.

At the time of Cameron’s comparison, Anthropic’s Opus 5.5 led the index. Artificial Analysis listed more than 500 models, though, and it also published individual evaluation results. Those results give an engineer a way to narrow the field according to what an agent actually has to do. A difference in general intelligence can matter less to a particular application than a difference in one of the capabilities the index combines.

For the New York audience, Cameron picks AA Analyst Agent, a finance and accounting benchmark. One demand it places on an agent is navigating complex spreadsheets. The Gemini series performed particularly well there, as did Anthropic’s models. An engineer building a spreadsheet agent has a concrete reason to try Gemini even if the initial search began with the overall index leader.

![Bars show the percentage of 80 spreadsheet tasks each model solved on all five attempts. Gemini 3.7 Flash leads at 60.0%, followed by Claude Sonnet 5.5 and Claude Fable 5.1 at 57.5% each.](https://ai.engineer/api/talk-essay-media/eeabb74c3f0d1970e99b6087a19c1404ab8cac8d2f23b33449f206c91ebd7e9f)

AA-AnalystAgent scores on finance spreadsheet tasks

### New releases keep changing the field, including for open weights

[5:25](https://www.youtube.com/watch?v=prGP0huECPo&t=325s)

Cameron next follows the leading model from each of roughly twelve labs over time. Each release adds a point. The growing density of points captures one reason developers find it exhausting to keep up, while the scores themselves suggest to Cameron that capability gains are accelerating. A model selection made a few releases ago can predate a meaningful change in what an agent can accomplish.

The historical comparison also puts open-weights models within roughly three to nine months of leading proprietary models in capability. That interval matters when a team needs a level of intelligence the frontier reached some time ago. An open-weights model can enter consideration for that workload, bringing a different set of inference and pricing choices with it.

### The price of one task spans cents to dollars

[6:53](https://www.youtube.com/watch?v=prGP0huECPo&t=413s)

By mid-to-late 2026, Cameron says, models could cost several dollars to complete a single assignment, a change from a year earlier. In his comparison, Opus and Fable each cost more than $5 per task, while some models cost 5 cents or less. That hundredfold spread makes model selection consequential even before an engineer multiplies the bill by production volume.

![Stacked bars show weighted average cost in US dollars per task, broken down by token type. Totals increase from a displayed $0.00 for K2 Horizon to $7.63 for Claude Fable 5.1.](https://ai.engineer/api/talk-essay-media/690ee0dd9443796cc81b633819f2091af49e7d74d1f3c4dd4fa52c6eb0da35db)

Cost per Intelligence Index Task by model and token type

GPT-6 Luna came in at about 7 cents per task. Cameron rates it as substantially more capable than the original GPT-4 and further ahead of GPT-3.5 Turbo, the model he credits with opening up this wave of language-model applications. The inexpensive end of the comparison therefore includes candidates that would have looked exceptionally capable to someone building just a few model generations earlier.

### Hold the intelligence target fixed and track its falling price

[8:53](https://www.youtube.com/watch?v=prGP0huECPo&t=533s)

Artificial Analysis grouped models by the Intelligence Index score they could reach. For a threshold such as 30 or 50, it began when a model first reached that score, then tracked the cheapest model capable of reaching it at each later point. That method lets the identity of the cheapest qualifying model change while preserving the capability requirement.

![Six colored stair-step lines plot cost per task against model release date. Each line steps downward as the cost of reaching its Intelligence Index threshold falls. The vertical axis uses a logarithmic dollar scale.](https://ai.engineer/api/talk-essay-media/2355e2085425b4b71470ebe1069f5dcdb43e406cfea36d386bf7226d418430ad)

Language model cost per task over time at six Intelligence Index thresholds

A score of 30 initially cost a little over a dollar in this analysis. About a year later it cost roughly 10 cents, and by Cameron’s presentation it was in the 1-to-10-cent range. Across the tiers he describes, the cost of accessing a given level of intelligence fell about tenfold in roughly a year. To illustrate the planning implications, he imagines a large agent workload that costs $1 million today. If the pattern continued, the same level of intelligence might be accessible for $100,000 a year later and potentially $10,000 after two years. Those last two figures are projections from the observed decline.

### Build the $21 Briefcase bill from one call and 165 steps

[10:18](https://www.youtube.com/watch?v=prGP0huECPo&t=618s)

Reasoning models had already complicated an estimate based on the length of the returned answer. A model could generate billable reasoning tokens on its way to producing that answer. An agent adds another dimension. It can make many model calls before the user gets a deliverable, and the input to a later call can include work from earlier ones.

Cameron uses the AA Briefcase Agentic Knowledge Work Benchmark to put numbers on that process. An agent works in an environment containing thousands of files and receives an assignment such as creating a report, a PowerPoint presentation or a marketing video. Opus did the work well in Cameron’s assessment. The bill he wants to explain is the average across assignments, about $21 for one finished task.

Start with a single language-model call. Cameron corrects his estimate for the answer-token component to about 2 cents. Reasoning tokens add somewhat more. The model also processes input, whose cost depends on both its length and how much of it hits the prompt cache. Counting those components together, a call averaged about 13 cents. Now put the call back into the assignment. Opus took an average of 165 steps. Multiplying the rounded figures gives $21.45, close to the reported $21. Reading the final report and counting its words would reveal little about either factor in that multiplication.

![A waterfall chart builds up the cost of a model call from answer, reasoning and input tokens. Prompt caching reduces the average call from $1.27 to $0.128. An average of 165 calls brings the task cost to $21.05.](https://ai.engineer/api/talk-essay-media/e2a466984925bd0bdd37f8f7994213b5f4d5095fe9d09f98e34e8f124f88ac5b)

How Claude Opus 5.5 (Max) reaches $21.05 per AA-Briefcase task

### Repeated input makes the cache hit rate visible on the bill

[11:48](https://www.youtube.com/watch?v=prGP0huECPo&t=708s)

Within a Briefcase call, Cameron identifies input volume as the largest cost driver. The next question is what fraction of those input tokens hit the cache. A hit in his example receives roughly a tenfold discount on the ordinary input-token price. Message history makes the distinction especially important for an agent. Earlier material can accompany a new request, so the agent can process a substantial volume of input even when the latest addition to the conversation is small.

To reconstruct the charge, separate the cached input tokens from the remaining input on each call. Multiply each count by its applicable price, add the reasoning and answer charges, and sum the calls through completion. Cameron identifies the number of turns as the biggest driver of the full task bill in this example. Input volume and cache hit percentage explain much of the cost of one call. The turn count tells you how many times the agent incurs a call charge before it delivers the work.

He then takes the same Briefcase example and sets the cache hit percentage to zero. The estimated task price rises from about $21 to over $200. One way to encounter that failure is to run inference with vLLM and leave prompt caching disabled. The calculation also gives a team comparing inference providers a reason to examine the hit rate it can actually achieve alongside the advertised token prices. Artificial Analysis had begun publishing cache hit percentage benchmarks for that comparison.

### Reasoning effort changes the cost of subsequent turns

[13:18](https://www.youtube.com/watch?v=prGP0huECPo&t=798s)

Opus 5.5 at low reasoning effort cost about $1 per Briefcase task. At maximum effort it cost about $21. One component of that difference is straightforward: changing the setting changes how many reasoning tokens the model generates. Two further effects emerge across the assignment. The effort setting can change how many turns the agent takes, and reasoning carried forward in message history contributes to the input processed on later turns. A token generated as reasoning on one call can thus contribute to input charges downstream.

![Four bar charts show turns, context per turn, input tokens and cost rising from Low to Max effort. From High to Max, cost rises from $6.27 to $21.05 per task and AA-Briefcase Elo rises from 1,688 to 1,807.](https://ai.engineer/api/talk-essay-media/daa1c979f45ed277d3eebf64404452a2b081181043d812c002ec3fb1876f5bbf)

Claude Opus 5.5 cost, token usage and AA-Briefcase Elo across five effort levels

The price comparison alone leaves open how much the deliverable improved. It does establish that choosing Opus 5.5 still leaves an important configuration decision. Cameron then compares it with GLM 5.3 Flash, a cheaper open-weights model he considers highly capable. Across models, all the components of the bill can change. For long-running agents, he again singles out input-token prices and the proportion of input that hits the cache as major reasons two models can cost very different amounts to complete the work.

### Read the cost–intelligence frontier in either direction

[15:30](https://www.youtube.com/watch?v=prGP0huECPo&t=930s)

Knowing that GLM 5.3 Flash is cheaper than Opus 5.5, or that Luna is cheaper than Astra, still leaves the builder’s decision unresolved. Cameron starts with general intelligence plotted against cost per completed task. The Pareto frontier picks out models for which no other candidate in the comparison offers at least as much intelligence at a lower price, or more intelligence at the same price.

![A scatterplot places higher-scoring models toward the top and cheaper models toward the left. The upper-left region is shaded green. A dotted Pareto line rises through points including MiMo-V2.6-Pro and GPT-6.1 Sol (max) toward Claude Opus 5.5 (max with fallback).](https://ai.engineer/api/talk-essay-media/08f0792a2f47ee4d89747c444e91e775c8f43cfb63f3181a50347c5727ca6461)

The Pareto frontier for Intelligence Index score versus cost per task

With a fixed budget, look along that frontier for the highest intelligence score within the spending limit. With a fixed capability requirement, start at the score the application needs and find the cheapest model that reaches it. The general index gives both searches a starting point. Deciding which direction to search requires knowing what a better answer to the user’s particular question would look like.

### A cost analysis and an $80 subscription total reward different capabilities

[17:00](https://www.youtube.com/watch?v=prGP0huECPo&t=1020s)

Cameron calls an assignment an *unlimited intelligence* task when additional intelligence can keep improving the result. His example is to analyze what drives costs in a business and identify how to optimize them. An agent can get the figures right and still leave room for a better explanation of the costs or a better proposal for reducing them. For that assignment, the budget question is how much the business is willing to spend for a better analysis.

A *limited intelligence* task has a ceiling. Ask how much you spent on subscriptions last month. Given the right MCPs to access the relevant records, Luna would return $80 in Cameron’s example. Astra would also return $80. Once the agent can reliably find the records and calculate that total, a more capable model has no better number to offer. Here the engineer can establish the capability needed to get the answer right, then shop for the least expensive way to obtain it.

### Benchmark the shortlist inside the agent you intend to run

[18:12](https://www.youtube.com/watch?v=prGP0huECPo&t=1092s)

Cameron’s selection process moves from the public landscape to the team’s own work. Identify the metrics that matter, compare models on those priorities and make a shortlist. For a small prototype, try the candidates in the actual agent with its tools and context. For a production rollout, he recommends building a benchmark around the workload and running a shortlist of perhaps five models through it. Compare the quality of the completed deliverable with the full cost of obtaining it.

Artificial Analysis offers that process through a product called Optima. A team uploads its context, and Artificial Analysis works with the team using the Optima agent to generate a benchmark. The output is a set of trade-off curves based on the team’s definition of intelligence for its task. That gives the engineer a way to answer the question left open by the $1 and $21 Briefcase runs: on the assignment this team actually needs done, what improvement does the extra spend buy?

## Key takeaways

- For each completed assignment, retain per-call input volume, cached input, reasoning and answer charges, and the number of turns. Those measurements let you reconstruct the bill and price the effect of cache misses.
- Test reasoning-effort settings on complete assignments. Compare deliverable quality and total cost across the full history of calls.
- Use a known correct result to set a capability threshold for bounded tasks. For open-ended analysis, compare the improvement in the work against the additional spend.
- Use public evaluations to shortlist models, then measure the candidates with your own tasks, tools and inference setup.

## Slides

### Artificial Analysis benchmarks across the AI stack

[0:23](https://www.youtube.com/watch?v=prGP0huECPo&t=23s) [Four panels pair benchmark labels with a coding-agent bar chart, a language-model timeline, an image comparison showing food trucks, and a hardware throughput chart.](https://assets.ai.engineer/sites/aiecode2025/sha256/224fc2f8be20d349298d1eca01c40820c912be6256549636f1a9e1ab73ea5033.webp)

Artificial Analysis benchmarks intelligence, performance & cost across the AI stack

Agents
Artificial Analysis Coding Agent Index
Coding Agent Index
6+ other agent categories
Performance and cost

Language model intelligence
Frontier Language Model Intelligence, Over Time
Intelligence Index
GDPval-AA
Performance and cost

Image, video, speech & music
Image Arena
Vote to reveal your model preference
Arenas
Speech benchmarks
Performance and cost

Hardware & inference
System Output Throughput vs. Output Speed per Query
AA-AgentPerf
AA-SLT (System Load Test)
NVIDIA, AMD, Cerebras, Groq

Artificial Analysis
02

### Artificial Analysis Intelligence Index scores by model

[2:59](https://www.youtube.com/watch?v=prGP0huECPo&t=179s) [A descending bar chart ranks 27 models. Claude Opus 5.5 leads at 58. Claude Sonnet 5.5 and Muse Wonder Full both score 56. A checkered fill marks Muse Wonder Full as having no publicly available API.](https://assets.ai.engineer/sites/aiecode2025/sha256/ad3a34268aab0f364bbf8cd9fbd2355c6f545cc8c8f5eefcfe8dcede4bff77a8.webp)

Intelligence Index is the best single number for understanding the frontier.

Artificial Analysis Intelligence Index ↗
Artificial Analysis Intelligence Index v4.3.2 incorporates 10 evaluations: AA-Briefcase v1.1, GDPval-AA v2.1, AutomationBench-AA, Terminal-Bench 4.0, SciCode, Humanity’s Last Exam, GDP.pdf, CritPt, AA-Omniscience, AA-LCR v1.1

No publicly available API

Claude Opus 5.5 (max with fallback)  58
Claude Sonnet 5.5 (max with fallback)  56
Muse Wonder Full (max)  56
Claude Fable 5.1 (max with fallback)  53
GPT-6 Astra (max)  53
Gemini 4 Argon (high)  53
Muse Wonder (max)  52
GPT-6.1 Sol (max)  52
Muse Spark 1.3 (max)  48
Grok 4.7 (xhigh)  46
MiMo-V2.6-Pro  46
Qwen3.8 Max  45
GLM-5.3 (max)  45
Step 5 Preview  44
Kimi K3 (max)  44
Claude Haiku 5.5 (max)  43
GLM-5.3-Flash  42
Gemini 3.8 Flash (high)  41
DeepSeek V4.1 Flash (max)  39
Mistral Large 4 Preview  38
GPT-6 Luna (max)  38
Qwen3.8 27B (xhigh)  34
K2 Horizon 375B A23B  31
MiniMax-M3  29
Inkling (xhigh)  25
Nemotron 3 Ultra  23
Muse Glimmer (high)  17

Artificial Analysis
07

### AA-AnalystAgent scores on finance spreadsheet tasks

[4:37](https://www.youtube.com/watch?v=prGP0huECPo&t=277s) [Bars show the percentage of 80 spreadsheet tasks each model solved on all five attempts. Gemini 3.7 Flash leads at 60.0%, followed by Claude Sonnet 5.5 and Claude Fable 5.1 at 57.5% each.](https://assets.ai.engineer/sites/aiecode2025/sha256/eeabb74c3f0d1970e99b6087a19c1404ab8cac8d2f23b33449f206c91ebd7e9f.webp)

We also publish individual eval results. AA-AnalystAgent focuses on finance tasks by asking models to complete challenging tasks in complex Excel spreadsheets.

AA-AnalystAgent: Score
Share of tasks solved on all 5 of 5 attempts · 80 tasks, 5 attempts per task · Independently benchmarked by Artificial Analysis

Gemini 3.7 Flash (high)  60.0%
Claude Sonnet 5.5 (max with fallback)  57.5%
Claude Fable 5.1 (max with fallback)  57.5%
Claude Opus 5.5 (max with fallback)  56.3%
Claude Haiku 5.5 (max)  53.8%
Grok 4.7 (xhigh)  52.5%
GPT-6 Astra (max)  51.2%
GPT-6.1 Sol (max)  50.0%
Gemini 4 Argon (high)  48.8%
Gemini 3.8 Flash (high)  46.3%
Qwen3.8 Max  45.0%
GLM-5.3 (max)  38.8%
Kimi K3 (max)  38.8%
Muse Spark 1.3 (max)  37.5%
MiMo-V2.6-Pro  37.5%
Step 5 Preview  35.0%
DeepSeek V4.1 Flash (max)  35.0%
GLM-5.3-Flash  25.0%
Inkling (xhigh)  23.8%
Muse Glimmer (high)  20.0%
MiniMax-M3  10.0%
Nemotron 3 Ultra  6.3%

Artificial Analysis
08

### Frontier Language Model Intelligence, Over Time

[5:23](https://www.youtube.com/watch?v=prGP0huECPo&t=323s) [A step chart tracks Intelligence Index scores by release date for 15 model developers. Scores rise slowly through 2024 and more steeply in 2025 and 2026. Anthropic and Meta have the two highest endpoints.](https://assets.ai.engineer/sites/aiecode2025/sha256/9703e6a4715d186d93b16e93f15dd24edd409622cc98f92109ef8e833718cfb5.webp)

The race to deliver the best ‘intelligence on tap’ is more competitive than ever.

Frontier Language Model Intelligence, Over Time
Artificial Analysis Intelligence Index v4.3.2 incorporates 10 evaluations: AA-Briefcase v1.1, GDPval-AA v2.1, AutomationBench-AA, Terminal-Bench 4.0, SciCode, Humanity’s Last Exam, GDP.pdf, CritPt, AA-Omniscience, AA-LCR v1.1

Anthropic   Meta   OpenAI   Google   SpaceXAI   Xiaomi   Alibaba   Z AI   StepFun   Kimi   DeepSeek   Mistral   Institute of Foundation Models   MiniMax
Thinking Machines

Artificial Analysis Intelligence Index
60   45   30   15   0
Release Date
Nov 22   May 23   Nov 23   May 24   Oct 24   Apr 25   Oct 25   Mar 26   Oct 26

Artificial Analysis
09

### Progress in Open Weights vs. Proprietary Intelligence

[6:23](https://www.youtube.com/watch?v=prGP0huECPo&t=383s) [Black and blue step lines trace the highest proprietary and open-weights Intelligence Index scores by release date. The blue line reaches comparable scores later, ending near 46 while the black line ends near 58.](https://assets.ai.engineer/sites/aiecode2025/sha256/27a1d8552bc3f687dbe0b34cf895bc942d1ff1591681628ef059031508d8556a.webp)

Open weights models continue to trail the proprietary frontier by three to nine months.

Progress in Open Weights vs. Proprietary Intelligence
Artificial Analysis Intelligence Index v4.3.2 incorporates 10 evaluations: AA-Briefcase v1.1, GDPval-AA v2.1, AutomationBench-AA, Terminal-Bench 4.0, SciCode, Humanity’s Last Exam, GDP.pdf, CritPt, AA-Omniscience, AA-LCR v1.1

Proprietary   Open Weights

Artificial Analysis Intelligence Index
60   45   30   15   0
Release Date
Nov 22   May 23   Nov 23   May 24   Oct 24   Apr 25   Oct 25   Mar 26   Sep 26

Artificial Analysis
10

### Cost per Intelligence Index Task by model and token type

[7:14](https://www.youtube.com/watch?v=prGP0huECPo&t=434s) [Stacked bars show weighted average cost in US dollars per task, broken down by token type. Totals increase from a displayed $0.00 for K2 Horizon to $7.63 for Claude Fable 5.1.](https://assets.ai.engineer/sites/aiecode2025/sha256/690ee0dd9443796cc81b633819f2091af49e7d74d1f3c4dd4fa52c6eb0da35db.webp)

Cost per Task differences span orders of magnitude.

Cost per Intelligence Index Task
Weighted average cost (USD) per Intelligence Index task · Segmented by token type · Lower is better

Answer   Reasoning   Cache Write   Cache Hit   Input

K2 Horizon 375B A23B  $0.00
Muse Glimmer (high)  $0.05
GPT-6 Luna (max)  $0.07
MiMo-V2.6-Pro  $0.13
Claude Haiku 5.5 (max)  $0.21*
DeepSeek V4.1 Flash (max)  $0.27
GLM-5.3-Flash  $0.28
MiniMax-M3  $0.51
Nemotron 3 Ultra  $0.60
  $0.48
GPT-6.1 Sol (max)  $0.72
Qwen3.8 27B (xhigh)  $1.01
Step 5 Preview  $1.03
  $0.62
Mistral Large 4 Preview  $1.13
  $0.57
Gemini 3.8 Flash (high)  $1.24
  $0.56
Muse Spark 1.3 (max)  $1.60
  $0.48   $0.84
Gemini 4 Argon (high)  $1.99
  $0.98
Kimi K3 (max)  $2.00
  $0.49   $0.80
GLM-5.3 (max)  $2.01
  $1.52
GPT-6 Astra (max)  $3.26
  $0.53   $0.83   $0.70   $0.98
Grok 4.7 (xhigh)  $3.74
  $1.69   $1.53
Muse Wonder (max)  $4.52
  $0.51   $0.57   $1.92   $1.44
Qwen3.8 Max  $5.41
  $1.76   $2.96
Claude Sonnet 5.5 (max with fallback)  $5.46
  $0.51   $1.47   $1.23   $2.21
Claude Opus 5.5 (max with fallback)  $5.98
  $0.70   $1.68   $1.07   $2.42
Claude Fable 5.1 (max with fallback)  $7.63
  $1.54   $2.36   $1.86   $1.60

Artificial Analysis
12

### Language model cost per task over time at six Intelligence Index thresholds

[8:51](https://www.youtube.com/watch?v=prGP0huECPo&t=531s) [Six colored stair-step lines plot cost per task against model release date. Each line steps downward as the cost of reaching its Intelligence Index threshold falls. The vertical axis uses a logarithmic dollar scale.](https://assets.ai.engineer/sites/aiecode2025/sha256/2355e2085425b4b71470ebe1069f5dcdb43e406cfea36d386bf7226d418430ad.webp)

The Cost per Task to reach a given level of intelligence falls 10–100× within a year.

Language Model Cost per Intelligence Index Task, by Intelligence Index Threshold, Over Time
Cost in USD per Intelligence Index task, averaged across evaluations by index weight.
Thresholds use Artificial Analysis Intelligence Index v4.3.2.

Intelligence Index ≥ 10    Intelligence Index ≥ 20    Intelligence Index ≥ 30    Intelligence Index ≥ 40    Intelligence Index ≥ 50    Intelligence Index ≥ 60

Artificial Analysis
Cost per Intelligence Index Task (USD)
$10    $1    $0.1    $0.01    $0.001
Jan 25    Mar 25    May 25    Jul 25    Sep 25    Nov 25    Jan 26    Mar 26    Apr 26    Jun 26    Aug 26    Oct 26
Release Date

Artificial Analysis    13

### How Claude Opus 5.5 (Max) reaches $21.05 per AA-Briefcase task

[10:15](https://www.youtube.com/watch?v=prGP0huECPo&t=615s) [A waterfall chart builds up the cost of a model call from answer, reasoning and input tokens. Prompt caching reduces the average call from $1.27 to $0.128. An average of 165 calls brings the task cost to $21.05.](https://assets.ai.engineer/sites/aiecode2025/sha256/e2a466984925bd0bdd37f8f7994213b5f4d5095fe9d09f98e34e8f124f88ac5b.webp)

Cost per Task build-up: Opus 5.5 cost $21 on average to complete an AA-Briefcase task, # turns and input token cost are the biggest drivers.
Cost build-up for an average AA-Briefcase task · Claude Opus 5.5 (Max)

Cost (USD, log scale)
$100    $10    $1    $0.10    $0.01

Answer tokens
$0.016
822 tokens
× $20/M output

+ Reasoning tokens
×2.5    $0.041
1,229 tokens
× $20/M output

+ Input tokens
×31    $1.27
307K-token context
× $4/M input

Prompt caching
÷10    $0.128
98.3% read from cache
at $0.20/M (misses $5/M)

One model call
$0.128
average turn
in the trajectory

× Turns per task
×165    $21.05
165 model calls
to complete the task

Cost per Task
$21.05
AA-Briefcase
Claude Opus 5.5 (Max)

Artificial Analysis    14

### Claude Opus 5.5 cost, token usage and AA-Briefcase Elo across five effort levels

[13:31](https://www.youtube.com/watch?v=prGP0huECPo&t=811s) [Four bar charts show turns, context per turn, input tokens and cost rising from Low to Max effort. From High to Max, cost rises from $6.27 to $21.05 per task and AA-Briefcase Elo rises from 1,688 to 1,807.](https://assets.ai.engineer/sites/aiecode2025/sha256/daa1c979f45ed277d3eebf64404452a2b081181043d812c002ec3fb1876f5bbf.webp)

Different effort levels for the same model drive vast differences in cost. Effort levels drive turns and token consumption.
Claude Opus 5.5 on AA-Briefcase by reasoning effort setting · averages per task

Turns per task
High → Max: ×2.3
Low 20    Medium 56    High 70    Xhigh 110    Max 165

Average context per turn
High → Max: ×1.8
Low 52K    Medium 135K    High 171K    Xhigh 260K    Max 307K

Input tokens per task
High → Max: ×4.2
Low 1.0M    Medium 7.5M    High 12M    Xhigh 29M    Max 51M

Cost per Task
High → Max: ×3.4
Low $1.15    Medium $4.40    High $6.27    Xhigh $12.27    Max $21.05

AA-Briefcase Elo
Low 1,279    Medium 1,627    High 1,688    Xhigh 1,766    Max 1,807

Source: Artificial Analysis AA-Briefcase (91 tasks), Claude Opus 5.5 at Low / Medium / High / Xhigh / Max effort (Default Fallback). Cost per Task uses first-party pricing and representative cache hit rates. Input tokens per task = turns × average context per turn.

Artificial Analysis    15

### How token usage and prices explain the cost gap between Opus 5.5 and GLM-5.3-Flash

[14:38](https://www.youtube.com/watch?v=prGP0huECPo&t=878s) [A descending waterfall compares average AA-Briefcase task costs. Substituting GLM-5.3-Flash usage at Opus prices lowers the cost from $21.05 to $10.40. Applying GLM input and output prices lowers it to $0.98.](https://assets.ai.engineer/sites/aiecode2025/sha256/f602c4b797f8fd1d4ad0c9bcd67858033250616ad5359c6edd4a9c418125f098.webp)

Illustrating why model selection matters, GLM-5.3-Flash costs an average of 21× less than Opus 5.5.
Bridge in AA-Briefcase Cost per Task, Claude Opus 5.5 (Max) to GLM-5.3-Flash (max), one driver at a time

Cost per Task (USD)
Token usage: ÷2.0
measured at Opus 5.5 prices, $21.05 → $10.40
Token prices: ÷10.6
$10.40 → $0.98

$20    $15    $10    $5    $0

Claude Opus 5.5 (Max)
Cost per Task    $21.05

Turns per task    −$2.34
165 → 147

Input tokens per turn    −$4.53
307K → 198K

Output tokens per turn    −$3.37
2,051 → 901

Share of input read from cache    −$0.41
98.3% → 98.6%

Input token prices    −$6.84
cached $0.20 → $0.03
uncached $5.00 → $0.15

Output token price    −$2.57
$20.00 → $0.50

GLM-5.3-Flash (max)
Cost per Task    $0.98

Source: Artificial Analysis AA-Briefcase (91 tasks); first-party pricing, representative cache hit rates. Usage effects are measured at Opus 5.5 prices, then price effects. Applying prices first attributes ÷11.7 to price and ÷1.8 to usage; the product is the same 21.4×.

Artificial Analysis    16

### The Pareto frontier for Intelligence Index score versus cost per task

[15:56](https://www.youtube.com/watch?v=prGP0huECPo&t=956s) [A scatterplot places higher-scoring models toward the top and cheaper models toward the left. The upper-left region is shaded green. A dotted Pareto line rises through points including MiMo-V2.6-Pro and GPT-6.1 Sol (max) toward Claude Opus 5.5 (max with fallback).](https://assets.ai.engineer/sites/aiecode2025/sha256/08f0792a2f47ee4d89747c444e91e775c8f43cfb63f3181a50347c5727ca6461.webp)

Optimizing AI is about making trade-offs. The starting point should always be the Intelligence vs Cost per Task Pareto frontier.

Intelligence Index vs. Cost per Intelligence Index Task
Artificial Analysis Intelligence Index · Weighted average cost (USD) per Artificial Analysis Intelligence Index task

Most attractive quadrant    Pareto line
StepFun    Z AI    Xiaomi    MiniMax    OpenAI    Anthropic    Mistral    Meta    SpaceXAI    NVIDIA    Google    Alibaba    DeepSeek    Kimi

Artificial Analysis
Artificial Analysis Intelligence Index
65    60    55    50    45    40    35    30    25    20    15    10

Muse Glimmer (high)
GPT-6 Luna (max)
MiMo-V2.6-Pro
Claude Haiku 5.5 (max)*
GLM-5.3-Flash
DeepSeek V4.1 Flash (max)
MiniMax-M3
Nemotron 3 Ultra
GPT-6.1 Sol (max)
Qwen3.8 27B (xhigh)
Step 5 Preview
Mistral Large 4 Preview
Gemini 3.8 Flash (high)
Kimi K3 (max)
Muse Spark 1.3 (max)
Gemini 4 Argon (high)
GLM-5.3 (max)
Claude Sonnet 5.5 (max with fallback)
GPT-6 Astra (max)
Grok 4.7 (xhigh)
Muse Wonder (max)
Qwen3.8 Max (0902)
Claude Opus 5.5 (max with fallback)
Claude Fable 5.1 (max with fallback)

$0.06    $0.08    $0.1    $0.2    $0.3    $0.4    $0.5    $0.6    $0.8    $1    $2    $3    $4    $5    $6    $7    $8    $10    $20
Cost per Task (USD, Log Scale)

Artificial Analysis    18

### Unlimited and limited intelligence tasks illustrated with two business-cost questions

[17:01](https://www.youtube.com/watch?v=prGP0huECPo&t=1021s) [Two panels plot task value against model intelligence. The curve for analyzing business costs keeps rising. The curve for totaling last month’s subscription spending reaches a marked threshold and then stays flat.](https://assets.ai.engineer/sites/aiecode2025/sha256/88316e7ba87e1df0933979cc5f1c0cdf35f29b17b565e0ebe1133c6017709e81.webp)

Tasks can be allocated into two buckets: Unlimited Intelligence Tasks and Limited Intelligence Tasks, which have a ceiling.

Unlimited Intelligence Tasks
Value
More intelligence →
THE QUESTION
What is my budget, and how much latency, cost, complexity or other trade-offs are ‘worth it’?
EXAMPLE
“Analyze drivers of costs in my business area and how I should optimize them”

Limited Intelligence Tasks
Value
More intelligence →
THE QUESTION
How much intelligence is needed? Then find the cheapest option within that.
EXAMPLE
“How much did I spend on subscriptions last month?”

Artificial Analysis    19

### Enterprises use Artificial Analysis to select models based on their priorities.

[18:10](https://www.youtube.com/watch?v=prGP0huECPo&t=1090s) [Four columns show an intelligence bar chart, the Model Recommender, a three-model shortlist with intelligence scores, and Optima for building a custom benchmark.](https://assets.ai.engineer/sites/aiecode2025/sha256/089df4349b55d1dc2a5ede5ac95d34af9ce3f828c3564bd4cdd04d44a72b9f24.webp)

Enterprises use Artificial Analysis to select models based on their priorities.

STEP 1
Understand the landscape
Independent benchmarks of intelligence, speed and cost across ~700 models.

Intelligence
Updated
Artificial Analysis Intelligence Index · Higher is better
Not publicly available
56  53  53  53  52  48  46  46  45  39

STEP 2
Compare models based on priorities
Model Recommender turns your priorities into a ranked top 10.

Model Recommender
Get personalized model recommendations that optimize for your priorities across intelligence, speed, and cost

STEP 3
Shortlist models
Test the finalists side by side, including on your own prompts.

Top recommendations                         Intelligence
1  Claude Sonnet 5.5                         56
   Max effort · Anthropic
2  GPT-6.1 Sol                               51
   Xhigh · OpenAI
3  Claude Opus 5.5                           56
   Xhigh effort · Anthropic
Example: agentic use case where cost matters

STEP 4                                      ADVANCED USERS
Build a custom benchmark
Optima benchmarks models on your own data and tasks to find the best fit.

Optima
by Artificial Analysis
NEW
Build your own custom benchmark

Artificial Analysis                                      20

## Full transcript

[0:12](https://www.youtube.com/watch?v=prGP0huECPo&t=12s) Hey, everyone. Uh, thanks for coming. I'm George. I'm one of the co-founders of Artificial Analysis. Um, today I'll be talking through optimizing the cost of intelligence. To give a quick background to who we are at Artificial Analysis, we're a independent AI benchmarking company. Uh, we benchmark across the AI stack, so the agent layer with our coding agent benchmarks and other agent benchmarks, language models for their intelligence, uh, image, video,

[0:43](https://www.youtube.com/watch?v=prGP0huECPo&t=43s) speech, and music models, um, as well as the hardware and inference layers so that we can, uh, benchmark the trade-offs between all these layers. When you're making decisions, you're not just choosing a model, you're also choosing... uh, you're making an inference deci-decision. How do I access the model? How much am I paying for it? What's the speed that I'm, that I'm getting? What's the cache hit rate? We believe strongly that all of the kinda layers of the stack are interrelated. We're all making decisions across them, and so that's why we benchmark the full stack of

[1:13](https://www.youtube.com/watch?v=prGP0huECPo&t=73s) AI.

[1:15](https://www.youtube.com/watch?v=prGP0huECPo&t=75s) Kinda leading companies, um, kinda reference Artificial Analysis in our results, I think. But what we're most proud of here, when we take a look at these references from OpenAI, from Anthropic, uh, from Google, and, uh, and NVIDIA, they're not just referencing our raw intelligence benchmark numbers. It's not just the bar chart. We, um, we're big fans of our intelligence index, but we're kinda most interested

[1:45](https://www.youtube.com/watch?v=prGP0huECPo&t=105s) in talking through the trade-offs that we're making between these technologies. Um, and so that's why we're proud that all of these are kinda two by two, uh, charts showing trade-offs, um, between, uh, choices that developers need to make. Um, sometimes not when they're, when they're getting things working, uh, but really when they're rolling it out in production.

[2:11](https://www.youtube.com/watch?v=prGP0huECPo&t=131s) We're also quite widely referenced, um, by kinda traditional and, uh, and new media. We are based in San Francisco. We've got a team of fifty, um, and we are hiring. Um, so we're hiring across, uh, for deployed engineers, um, AI research staff as part of our member of technical staff team, and ML engineers. Um, and so, you know, catch me after or, or please reach out if you, if you might be interested,

[2:41](https://www.youtube.com/watch?v=prGP0huECPo&t=161s) um, in joining the, the world of AI benchmarking.

[2:47](https://www.youtube.com/watch?v=prGP0huECPo&t=167s) I think before we understand cost, we need to understand the kinda landscape of AI and the metrics to, to understand and to optimize. And so starting with understanding the AI landscape, I'll first introduce our intelligence index. It's... We view it as the best single number to understand the generalist... general intelligence, uh, of language models. It's a synthesis metric of ten different evaluations

[3:17](https://www.youtube.com/watch?v=prGP0huECPo&t=197s) covering different capability areas of AI. That includes k- knowledge work, it includes scientific reasoning, uh, it includes coding and terminal use, uh, and it includes other capabilities, including kinda knowledge, uh, hallucination, and other relevant capabilities and considerations, um, regarding intelligence that people use when, when choosing between language models. What we

[3:47](https://www.youtube.com/watch?v=prGP0huECPo&t=227s) say is for the current use cases being built and the next use cases. It's not necessarily to measure and,  and benchmark progress towards AGI. It's really to help developers make decisions for the next use cases that they're building, uh, using AI. And so of course, Opus 5.5 from Anthropic is currently leading the index. Um, I think to many in this room, that won't be a big surprise. Um, but also to discuss today is that there's, as you

[4:17](https://www.youtube.com/watch?v=prGP0huECPo&t=257s) can see here, many other models. We have over five hundred models on the Artificial Analysis website, um, and many kinda current and, and, and presented on this chart. And I think what we'll be talking through today is a bit that, that kinda takeaway that there's also reason to use models other than the most intelligent model in the world.

[4:39](https://www.youtube.com/watch?v=prGP0huECPo&t=279s) We also public individual eval results, so we released AA Analyst Agent, um, which is a finance and accounting focused, focused benchmark. Um, thought I'd share that, this being New York. And we share the individual results for those wanting to go to the next layer of specificity when thinking about their model selection and, and selection of other technologies, um, because it's not always the case that models exactly follow, um, the generalist

[5:09](https://www.youtube.com/watch?v=prGP0huECPo&t=309s) view of the intelligence index. Um, and so here we kinda highlight that the Gemini series of models are actually very good at navigating complex, um, complex spreadsheets. Um, of course, the Anthropic series models is, uh, is of course as well.

[5:25](https://www.youtube.com/watch?v=prGP0huECPo&t=325s) To step back a bit, looking at our intelligence index and the leading model over time from the top twelve or so labs here, um, firstly, I think big takeaway, it's kind of hectic. There's many dots, and I think probably that resonates for a lot of us in the room who have kinda felt this over the past few years, where there've been huge number of model, uh, releases, and the number of model releases in-is increasing. And in a sense, kind of the, the pace of progress, um, in our

[5:55](https://www.youtube.com/watch?v=prGP0huECPo&t=355s) view, is also accelerating, and I think that is shown, uh, in this chart here. And so I think the takeaway here is that AI is moving quickly, progress is moving quickly, and so kinda we might feel exhausted trying to keep up with it. But, um, it makes sense to, to, to invest that time and energy because if you wanna stay o-on top of AI and what AI is capable of, um, then staying on top of releases, um, is a, a big part of that.

[6:25](https://www.youtube.com/watch?v=prGP0huECPo&t=385s) And next view, won't deep dive. This could be a whole talk, but open-weights models continue to keep within three to nine months of the leading proprietary models. And so open-weights models remain an option. Uh, and so I think that's, that's the takeaway for the talk today and, and it flows through to a lot of, um, kinda cost implications, uh, which sometimes open-weight models can have favorable, uh, characteristics.

[6:53](https://www.youtube.com/watch?v=prGP0huECPo&t=413s) Next, let's dive into cost and how to think about cost. And I want to discuss think-- talking about cost from the perspective of being kinda mid, late twenty twenty-six. Because I think if we were doing this... if I was doing this talk even six or nine months ago, I think it would be a different talk. AI has moved a long way, and so we'll go into the drivers. But to kinda start with a little bit kinda the answer of why we're so conscious of cost is because these

[7:23](https://www.youtube.com/watch?v=prGP0huECPo&t=443s) models are costing many dollars, um, to complete individual tasks. That wasn't the case, uh, twelve months ago. Um, people are spending more than ever on AI and can spend more than ever on a single task, um, on AI. And so here we present Opus and Fable costing above five dollars to complete a task. But on the other side, um, there's also models that cost five cents and less to complete a task.

[7:53](https://www.youtube.com/watch?v=prGP0huECPo&t=473s) And so what we're seeing is there's orders of magnitude, multiple orders of magnitude, uh, difference in cost, uh, between the, between the models. And that means that the cost decision and the model decision, uh, matters. And what I would say is that, yes, you might say, "Oh, the, the, uh, the models on the left here are less intelligent," and that's t-totally true. But they're still very good. Uh, a Luna here, GPT five-- um, sorry, GPT six Luna is seven cents,

[8:23](https://www.youtube.com/watch?v=prGP0huECPo&t=503s) uh, per task. Um, and it's much better than, than GPT four was, um, originally, right? Which kind of... And, and, and way better than GPT three point five Turbo, which really kind of like opened up, um, th-this wave of kinda language models. And so I would say that there's two orders of magnitude difference between the leading proprietary models and much cheaper, but still very good, very capable models.

[8:53](https://www.youtube.com/watch?v=prGP0huECPo&t=533s) And next way to kinda consider looking at cost is, uh, from a time perspective. And what we've done is we've bucketed models by their intelligence and looked at when that intelligence tier was first reached, and then how did the kinda cost of trying to access the intelligence with the cheapest model develop over time? And what we see here is within one year, accessing that level of intelligence, a thirty on the intelligence index or a fifty on the intelligence index,

[9:25](https://www.youtube.com/watch?v=prGP0huECPo&t=565s) essentially reduces by an order of magnitude, kinda ten X, within around a year. So picking the, the light purple here, um, once we reached a thirty, cost a bit over a dollar. A year later, it fell to around ten cents. And then now we're well below, uh, the ten cent. We're in the kinda, um, one cent to ten cents territory. And so I think it's-- there's also a time dimension here whereby we should be, um, you know,

[9:55](https://www.youtube.com/watch?v=prGP0huECPo&t=595s) making choices for now, but also conscious that that is going to change very quickly in terms of the intelligence that we can achieve. And, and that maybe one million dollars, um, spent for a big agentic workload can, y-you know, is likely we could access that level of intelligence at a hundred k and then in two years, potentially around ten k.

[10:18](https://www.youtube.com/watch?v=prGP0huECPo&t=618s) Now, here we have a build-up of the cost per task. This is a lot more complex than where we were twelve months and definitely twenty-four months ago. I think twelve months ago, we were talking about reasoning models. We were saying that we have to pay for reasoning tokens. And, um, y-you know, the, the, the labs were just talking about the output token price. Um, and so we needed to do an analysis, um, as a, as a community to really understand the costs because we had to count reasoning tokens. Now with agents, it's become way more complex.

[10:48](https://www.youtube.com/watch?v=prGP0huECPo&t=648s) Way more complex. So to understand the build-up of a cost per task, here we use our AA Briefcase Agentic Knowledge Work Benchmark. It looks like agents in an environment of thousands of files completing knowledge work tasks like creating reports, creating PowerPoints, creating marketing videos. And what we can see here is, on average, Opus is costing twenty-one dollars to complete a task. And it does a great job. No arguments there. Focusing on cost, though, what we can see here is

[11:18](https://www.youtube.com/watch?v=prGP0huECPo&t=678s) that answer tokens, you know, cost around sixteen cents. Reasoning tokens, um... Oh, sorry, uh, it's like two cents.  Um, and then we, then we have reasoning tokens, a bit more. But then really where the, you know, building up to the single language model call, the first kinda column that hits the ground, or the second there. Um, so each call is thirteen cents, right? But with agentic trajectories, we're having to think about the number of steps, of course. And so it took a hundred

[11:48](https://www.youtube.com/watch?v=prGP0huECPo&t=708s) and sixty-five steps to complete the task. And so that's where-- that's a new dimension that we need to think about when it comes to agents. And then thinking to kinda what goes into a single language model call, you can see there's answer tokens, reasoning tokens, input tokens. And then with input tokens, you need to think about how many of those input tokens are kinda cache hits, um, and so get that roughly ten X, uh, discount on an input token price. And what we can see with the build-up is the biggest

[12:18](https://www.youtube.com/watch?v=prGP0huECPo&t=738s) driver- Of the kind of input into what a single language model calls is the number of input tokens, and then secondly, the amount or, or proportion of those input tokens that are cache hits. And then the third biggest driver after that is, of course, um, well, it's actually the biggest, but is, is the number of turns per task. And so we were talking a year ago around the number of output tokens, how much reasoning, the reasoning verbosity. I think today we really need to be talking about input tokens,

[12:49](https://www.youtube.com/watch?v=prGP0huECPo&t=769s) uh, cache hit percentage, um, and then the number of turns per task is the key metrics that we should be thinking about when thinking about the cost, um, of running, of running agents. And I think just to really emphasize the number of cache hits, if we don't have that, then... and, and we don't have any cache hits, right? Cache hit percentage zero, caching isn't working. Maybe we've got our own setup with vLLM. We haven't enabled prompt caching. Um, then instead of twenty-one dollars per task, this

[13:18](https://www.youtube.com/watch?v=prGP0huECPo&t=798s) costs over two hundred dollars. So that's just to really emphasize how much caching matters here when we're talking about agents and also when talking about inference providers, how much you really need to, to care about that. We started recently publishing benchmarks around cache hit percentage on our website. And next consideration is, um, you know, different reasoning efforts. And so here we have, um, kind of column, um, column charts showing the different reasoning efforts, um, for Opus Five

[13:48](https://www.youtube.com/watch?v=prGP0huECPo&t=828s) point five and the implications on these metrics. And so I think pretty much the key takeaway here is when we're talking about reasoning efforts, we're not just talking about, um, uh, kind of reasoning, the number of reasoning tokens, um, for an ag- single agentic call. It does impact the number of turns. Um, and of course, then you have reasoning tokens, and then those reasoning tokens feed throughout the agentic trajectory into kind of much more input tokens, um,

[14:18](https://www.youtube.com/watch?v=prGP0huECPo&t=858s) because of the way that like message histories work, uh, when it comes to, comes to using agents. And so even with it-- when using the same model, very important to think about the reasoning effort, because here we can see that at low Opus Five can cost-- Five point five can cost a dollar per task, and at max it can cost twenty-one dollars.

[14:41](https://www.youtube.com/watch?v=prGP0huECPo&t=881s) And here's kind of, um, a breakdown going from Opus Five point five to GLM Five point three Flash, a much cheaper but still extremely good, very capable model, um, that's, that's open weights. And I think when we look at this, when we look at this breakdown, this is really just to show that, one, there's big differences between, as we saw, between the leading models and the, and much cheaper models. And two is that all of the different drivers that I was talking about play a

[15:11](https://www.youtube.com/watch?v=prGP0huECPo&t=911s) role in driving cost differences, um, between models. And then thirdly, of course, that input token prices and your cache hit percentage, um, are the biggest driver, um, of cost when doing long, uh, horizon agent tasks.

[15:30](https://www.youtube.com/watch?v=prGP0huECPo&t=930s) So we've talked about the cost, and I think there's no points for knowing that GLM Five point three Flash is cheaper than Opus Five point five. I think a lot of people in this room know that Luna is cheaper than, than Astra. Great. So the challenge for us as builders is making the trade-off decision. That's the hard thing. How do we understand our trade-off curves for our tasks? And I think a starting point that we think is useful to

[16:00](https://www.youtube.com/watch?v=prGP0huECPo&t=960s) use is the generalist, um, view of intelligence versus cost per task and starting with the Pareto. So starting with, okay, my task, um, where does it sit a-a-across this, uh, this intelligence Pareto in terms of the model that's needed? If I'm able to spend more or wanting leading intelligence, I wanna look to the right of the Pareto. If I want cheaper, then I should let-- look to the left of the

[16:30](https://www.youtube.com/watch?v=prGP0huECPo&t=990s) Pareto. The Pareto being this, um, th-this line, uh, depicted in this, in this chart here. And when choosing and wanting a cheaper model, let's take a look at the, at the Pareto and seeing, okay, I'm willing to, uh, uh... my budget is less, and so what's the model that at that point offers the best intelligence, uh, for that, that dollar? Or from the other perspective of, I want this level of intelligence, uh, what's the cheapest way I can get it? How to think about navigating that

[17:00](https://www.youtube.com/watch?v=prGP0huECPo&t=1020s) decision and asking yourself those questions. A way that we think is helpful is to bucket tasks into two categories. So first is unlimited intelligence tasks, whereby more intelligence can always deliver a better result. An example being analyze drivers of the costs in my business and how do I optimize them. Kinda smarter will always deliver a better result there. Next, the second bucket is limited intelligence tasks, and here is when

[17:29](https://www.youtube.com/watch?v=prGP0huECPo&t=1049s) more intelligence won't give you a better result. So the question, how much did I spend on subscriptions last month? I think a cheaper like Luna will answer that question, um, as long as it's set up with the right MCPs. Uh, currently, it'll tell me eighty dollars. Using Astra will also tell me eighty doll- eight- eighty dollars. I'm not getting any benefit from more intelligence. Um, and so then I should focus on what's the cheapest way... what's the level of intelligence needed, the ceiling, and what's the cheapest way that I can get that level of

[17:59](https://www.youtube.com/watch?v=prGP0huECPo&t=1079s) intelligence. So that's limited intelligence, unlimited intelligence. The question is, what's my budget? How much am I willing to pay for increased intelligence?

[18:12](https://www.youtube.com/watch?v=prGP0huECPo&t=1092s) In terms of how we suggest using Artificial Analysis, we suggest to kind of... and how to navigate these decisions, I think it's understand the landscape, the metrics, then compare models based on priorities, shortlist models, and then for kinda smaller prototype-y workloads, use the shortlist, test on your own, uh, you know, in your own context and your own agents. For more advanced users, particularly when rolling out in production, we recommend creating your own

[18:42](https://www.youtube.com/watch?v=prGP0huECPo&t=1122s) benchmark to take that shortlist of five models and try and make, um, the decision, optimize for the decision, um, because of the kind of magnitudes of cost that we're dealing with now. Um, we recently released a product called Optima, um, that allows users to come to Artificial Analysis, upload their context, and we will, um, together with you, using a lot of our, um, knowledge that we've distilled into this kinda Optima agent, we generate you a benchmark, um, and give you your

[19:12](https://www.youtube.com/watch?v=prGP0huECPo&t=1152s) own trade-off curves between your view of intelligence for your task, um, and the cost to help you, help you make these trade-offs. So that's the cost of intelligence. Um, thanks very much for, for listening. Come check us out, our website at artificialanalysis.ai. And um, as mentioned at the start, we're, uh, we're hiring quite broadly across roles. So thank you very much.
