[
  {
    "startMs": 12860,
    "text": "Hey, everyone. Uh, thanks for coming. I'm George. I'm one of the co-founders of Artificial Analysis. Um, today I'll be talking through optimizing the cost of intelligence. To give a quick background to who we are at Artificial Analysis, we're a independent AI benchmarking company. Uh, we benchmark across the AI stack, so the agent layer with our coding agent benchmarks and other agent benchmarks, language models for their intelligence, uh, image, video,"
  },
  {
    "startMs": 43020,
    "text": "speech, and music models, um, as well as the hardware and inference layers so that we can, uh, benchmark the trade-offs between all these layers. When you're making decisions, you're not just choosing a model, you're also choosing... uh, you're making an inference deci-decision. How do I access the model? How much am I paying for it? What's the speed that I'm, that I'm getting? What's the cache hit rate? We believe strongly that all of the kinda layers of the stack are interrelated. We're all making decisions across them, and so that's why we benchmark the full stack of"
  },
  {
    "startMs": 73120,
    "text": "AI."
  },
  {
    "startMs": 75880,
    "text": "Kinda leading companies, um, kinda reference Artificial Analysis in our results, I think. But what we're most proud of here, when we take a look at these references from OpenAI, from Anthropic, uh, from Google, and, uh, and NVIDIA, they're not just referencing our raw intelligence benchmark numbers. It's not just the bar chart. We, um, we're big fans of our intelligence index, but we're kinda most interested"
  },
  {
    "startMs": 105700,
    "text": "in talking through the trade-offs that we're making between these technologies. Um, and so that's why we're proud that all of these are kinda two by two, uh, charts showing trade-offs, um, between, uh, choices that developers need to make. Um, sometimes not when they're, when they're getting things working, uh, but really when they're rolling it out in production."
  },
  {
    "startMs": 131860,
    "text": "We're also quite widely referenced, um, by kinda traditional and, uh, and new media. We are based in San Francisco. We've got a team of fifty, um, and we are hiring. Um, so we're hiring across, uh, for deployed engineers, um, AI research staff as part of our member of technical staff team, and ML engineers. Um, and so, you know, catch me after or, or please reach out if you, if you might be interested,"
  },
  {
    "startMs": 161960,
    "text": "um, in joining the, the world of AI benchmarking."
  },
  {
    "startMs": 167440,
    "text": "I think before we understand cost, we need to understand the kinda landscape of AI and the metrics to, to understand and to optimize. And so starting with understanding the AI landscape, I'll first introduce our intelligence index. It's... We view it as the best single number to understand the generalist... general intelligence, uh, of language models. It's a synthesis metric of ten different evaluations"
  },
  {
    "startMs": 197600,
    "text": "covering different capability areas of AI. That includes k- knowledge work, it includes scientific reasoning, uh, it includes coding and terminal use, uh, and it includes other capabilities, including kinda knowledge, uh, hallucination, and other relevant capabilities and considerations, um, regarding intelligence that people use when, when choosing between language models. What we"
  },
  {
    "startMs": 227520,
    "text": "say is for the current use cases being built and the next use cases. It's not necessarily to measure and,  and benchmark progress towards AGI. It's really to help developers make decisions for the next use cases that they're building, uh, using AI. And so of course, Opus 5.5 from Anthropic is currently leading the index. Um, I think to many in this room, that won't be a big surprise. Um, but also to discuss today is that there's, as you"
  },
  {
    "startMs": 257440,
    "text": "can see here, many other models. We have over five hundred models on the Artificial Analysis website, um, and many kinda current and, and, and presented on this chart. And I think what we'll be talking through today is a bit that, that kinda takeaway that there's also reason to use models other than the most intelligent model in the world."
  },
  {
    "startMs": 279440,
    "text": "We also public individual eval results, so we released AA Analyst Agent, um, which is a finance and accounting focused, focused benchmark. Um, thought I'd share that, this being New York. And we share the individual results for those wanting to go to the next layer of specificity when thinking about their model selection and, and selection of other technologies, um, because it's not always the case that models exactly follow, um, the generalist"
  },
  {
    "startMs": 309540,
    "text": "view of the intelligence index. Um, and so here we kinda highlight that the Gemini series of models are actually very good at navigating complex, um, complex spreadsheets. Um, of course, the Anthropic series models is, uh, is of course as well."
  },
  {
    "startMs": 325660,
    "text": "To step back a bit, looking at our intelligence index and the leading model over time from the top twelve or so labs here, um, firstly, I think big takeaway, it's kind of hectic. There's many dots, and I think probably that resonates for a lot of us in the room who have kinda felt this over the past few years, where there've been huge number of model, uh, releases, and the number of model releases in-is increasing. And in a sense, kind of the, the pace of progress, um, in our"
  },
  {
    "startMs": 355680,
    "text": "view, is also accelerating, and I think that is shown, uh, in this chart here. And so I think the takeaway here is that AI is moving quickly, progress is moving quickly, and so kinda we might feel exhausted trying to keep up with it. But, um, it makes sense to, to, to invest that time and energy because if you wanna stay o-on top of AI and what AI is capable of, um, then staying on top of releases, um, is a, a big part of that."
  },
  {
    "startMs": 385800,
    "text": "And next view, won't deep dive. This could be a whole talk, but open-weights models continue to keep within three to nine months of the leading proprietary models. And so open-weights models remain an option. Uh, and so I think that's, that's the takeaway for the talk today and, and it flows through to a lot of, um, kinda cost implications, uh, which sometimes open-weight models can have favorable, uh, characteristics."
  },
  {
    "startMs": 413180,
    "text": "Next, let's dive into cost and how to think about cost. And I want to discuss think-- talking about cost from the perspective of being kinda mid, late twenty twenty-six. Because I think if we were doing this... if I was doing this talk even six or nine months ago, I think it would be a different talk. AI has moved a long way, and so we'll go into the drivers. But to kinda start with a little bit kinda the answer of why we're so conscious of cost is because these"
  },
  {
    "startMs": 443140,
    "text": "models are costing many dollars, um, to complete individual tasks. That wasn't the case, uh, twelve months ago. Um, people are spending more than ever on AI and can spend more than ever on a single task, um, on AI. And so here we present Opus and Fable costing above five dollars to complete a task. But on the other side, um, there's also models that cost five cents and less to complete a task."
  },
  {
    "startMs": 473640,
    "text": "And so what we're seeing is there's orders of magnitude, multiple orders of magnitude, uh, difference in cost, uh, between the, between the models. And that means that the cost decision and the model decision, uh, matters. And what I would say is that, yes, you might say, \"Oh, the, the, uh, the models on the left here are less intelligent,\" and that's t-totally true. But they're still very good. Uh, a Luna here, GPT five-- um, sorry, GPT six Luna is seven cents,"
  },
  {
    "startMs": 503860,
    "text": "uh, per task. Um, and it's much better than, than GPT four was, um, originally, right? Which kind of... And, and, and way better than GPT three point five Turbo, which really kind of like opened up, um, th-this wave of kinda language models. And so I would say that there's two orders of magnitude difference between the leading proprietary models and much cheaper, but still very good, very capable models."
  },
  {
    "startMs": 533660,
    "text": "And next way to kinda consider looking at cost is, uh, from a time perspective. And what we've done is we've bucketed models by their intelligence and looked at when that intelligence tier was first reached, and then how did the kinda cost of trying to access the intelligence with the cheapest model develop over time? And what we see here is within one year, accessing that level of intelligence, a thirty on the intelligence index or a fifty on the intelligence index,"
  },
  {
    "startMs": 565300,
    "text": "essentially reduces by an order of magnitude, kinda ten X, within around a year. So picking the, the light purple here, um, once we reached a thirty, cost a bit over a dollar. A year later, it fell to around ten cents. And then now we're well below, uh, the ten cent. We're in the kinda, um, one cent to ten cents territory. And so I think it's-- there's also a time dimension here whereby we should be, um, you know,"
  },
  {
    "startMs": 595560,
    "text": "making choices for now, but also conscious that that is going to change very quickly in terms of the intelligence that we can achieve. And, and that maybe one million dollars, um, spent for a big agentic workload can, y-you know, is likely we could access that level of intelligence at a hundred k and then in two years, potentially around ten k."
  },
  {
    "startMs": 618239,
    "text": "Now, here we have a build-up of the cost per task. This is a lot more complex than where we were twelve months and definitely twenty-four months ago. I think twelve months ago, we were talking about reasoning models. We were saying that we have to pay for reasoning tokens. And, um, y-you know, the, the, the labs were just talking about the output token price. Um, and so we needed to do an analysis, um, as a, as a community to really understand the costs because we had to count reasoning tokens. Now with agents, it's become way more complex."
  },
  {
    "startMs": 648640,
    "text": "Way more complex. So to understand the build-up of a cost per task, here we use our AA Briefcase Agentic Knowledge Work Benchmark. It looks like agents in an environment of thousands of files completing knowledge work tasks like creating reports, creating PowerPoints, creating marketing videos. And what we can see here is, on average, Opus is costing twenty-one dollars to complete a task. And it does a great job. No arguments there. Focusing on cost, though, what we can see here is"
  },
  {
    "startMs": 678600,
    "text": "that answer tokens, you know, cost around sixteen cents. Reasoning tokens, um... Oh, sorry, uh, it's like two cents.  Um, and then we, then we have reasoning tokens, a bit more. But then really where the, you know, building up to the single language model call, the first kinda column that hits the ground, or the second there. Um, so each call is thirteen cents, right? But with agentic trajectories, we're having to think about the number of steps, of course. And so it took a hundred"
  },
  {
    "startMs": 708580,
    "text": "and sixty-five steps to complete the task. And so that's where-- that's a new dimension that we need to think about when it comes to agents. And then thinking to kinda what goes into a single language model call, you can see there's answer tokens, reasoning tokens, input tokens. And then with input tokens, you need to think about how many of those input tokens are kinda cache hits, um, and so get that roughly ten X, uh, discount on an input token price. And what we can see with the build-up is the biggest"
  },
  {
    "startMs": 738240,
    "text": "driver- Of the kind of input into what a single language model calls is the number of input tokens, and then secondly, the amount or, or proportion of those input tokens that are cache hits. And then the third biggest driver after that is, of course, um, well, it's actually the biggest, but is, is the number of turns per task. And so we were talking a year ago around the number of output tokens, how much reasoning, the reasoning verbosity. I think today we really need to be talking about input tokens,"
  },
  {
    "startMs": 769084,
    "text": "uh, cache hit percentage, um, and then the number of turns per task is the key metrics that we should be thinking about when thinking about the cost, um, of running, of running agents. And I think just to really emphasize the number of cache hits, if we don't have that, then... and, and we don't have any cache hits, right? Cache hit percentage zero, caching isn't working. Maybe we've got our own setup with vLLM. We haven't enabled prompt caching. Um, then instead of twenty-one dollars per task, this"
  },
  {
    "startMs": 798724,
    "text": "costs over two hundred dollars. So that's just to really emphasize how much caching matters here when we're talking about agents and also when talking about inference providers, how much you really need to, to care about that. We started recently publishing benchmarks around cache hit percentage on our website. And next consideration is, um, you know, different reasoning efforts. And so here we have, um, kind of column, um, column charts showing the different reasoning efforts, um, for Opus Five"
  },
  {
    "startMs": 828724,
    "text": "point five and the implications on these metrics. And so I think pretty much the key takeaway here is when we're talking about reasoning efforts, we're not just talking about, um, uh, kind of reasoning, the number of reasoning tokens, um, for an ag- single agentic call. It does impact the number of turns. Um, and of course, then you have reasoning tokens, and then those reasoning tokens feed throughout the agentic trajectory into kind of much more input tokens, um,"
  },
  {
    "startMs": 858164,
    "text": "because of the way that like message histories work, uh, when it comes to, comes to using agents. And so even with it-- when using the same model, very important to think about the reasoning effort, because here we can see that at low Opus Five can cost-- Five point five can cost a dollar per task, and at max it can cost twenty-one dollars."
  },
  {
    "startMs": 881484,
    "text": "And here's kind of, um, a breakdown going from Opus Five point five to GLM Five point three Flash, a much cheaper but still extremely good, very capable model, um, that's, that's open weights. And I think when we look at this, when we look at this breakdown, this is really just to show that, one, there's big differences between, as we saw, between the leading models and the, and much cheaper models. And two is that all of the different drivers that I was talking about play a"
  },
  {
    "startMs": 911284,
    "text": "role in driving cost differences, um, between models. And then thirdly, of course, that input token prices and your cache hit percentage, um, are the biggest driver, um, of cost when doing long, uh, horizon agent tasks."
  },
  {
    "startMs": 930504,
    "text": "So we've talked about the cost, and I think there's no points for knowing that GLM Five point three Flash is cheaper than Opus Five point five. I think a lot of people in this room know that Luna is cheaper than, than Astra. Great. So the challenge for us as builders is making the trade-off decision. That's the hard thing. How do we understand our trade-off curves for our tasks? And I think a starting point that we think is useful to"
  },
  {
    "startMs": 960324,
    "text": "use is the generalist, um, view of intelligence versus cost per task and starting with the Pareto. So starting with, okay, my task, um, where does it sit a-a-across this, uh, this intelligence Pareto in terms of the model that's needed? If I'm able to spend more or wanting leading intelligence, I wanna look to the right of the Pareto. If I want cheaper, then I should let-- look to the left of the"
  },
  {
    "startMs": 990024,
    "text": "Pareto. The Pareto being this, um, th-this line, uh, depicted in this, in this chart here. And when choosing and wanting a cheaper model, let's take a look at the, at the Pareto and seeing, okay, I'm willing to, uh, uh... my budget is less, and so what's the model that at that point offers the best intelligence, uh, for that, that dollar? Or from the other perspective of, I want this level of intelligence, uh, what's the cheapest way I can get it? How to think about navigating that"
  },
  {
    "startMs": 1020004,
    "text": "decision and asking yourself those questions. A way that we think is helpful is to bucket tasks into two categories. So first is unlimited intelligence tasks, whereby more intelligence can always deliver a better result. An example being analyze drivers of the costs in my business and how do I optimize them. Kinda smarter will always deliver a better result there. Next, the second bucket is limited intelligence tasks, and here is when"
  },
  {
    "startMs": 1049904,
    "text": "more intelligence won't give you a better result. So the question, how much did I spend on subscriptions last month? I think a cheaper like Luna will answer that question, um, as long as it's set up with the right MCPs. Uh, currently, it'll tell me eighty dollars. Using Astra will also tell me eighty doll- eight- eighty dollars. I'm not getting any benefit from more intelligence. Um, and so then I should focus on what's the cheapest way... what's the level of intelligence needed, the ceiling, and what's the cheapest way that I can get that level of"
  },
  {
    "startMs": 1079924,
    "text": "intelligence. So that's limited intelligence, unlimited intelligence. The question is, what's my budget? How much am I willing to pay for increased intelligence?"
  },
  {
    "startMs": 1092644,
    "text": "In terms of how we suggest using Artificial Analysis, we suggest to kind of... and how to navigate these decisions, I think it's understand the landscape, the metrics, then compare models based on priorities, shortlist models, and then for kinda smaller prototype-y workloads, use the shortlist, test on your own, uh, you know, in your own context and your own agents. For more advanced users, particularly when rolling out in production, we recommend creating your own"
  },
  {
    "startMs": 1122284,
    "text": "benchmark to take that shortlist of five models and try and make, um, the decision, optimize for the decision, um, because of the kind of magnitudes of cost that we're dealing with now. Um, we recently released a product called Optima, um, that allows users to come to Artificial Analysis, upload their context, and we will, um, together with you, using a lot of our, um, knowledge that we've distilled into this kinda Optima agent, we generate you a benchmark, um, and give you your"
  },
  {
    "startMs": 1152204,
    "text": "own trade-off curves between your view of intelligence for your task, um, and the cost to help you, help you make these trade-offs. So that's the cost of intelligence. Um, thanks very much for, for listening. Come check us out, our website at artificialanalysis.ai. And um, as mentioned at the start, we're, uh, we're hiring quite broadly across roles. So thank you very much."
  }
]
