Same Model, Different Speed: Why Your Inference Provider Matters — FriendliAI

All right, let's get started. So, hello, everyone. My name is Yunmo. I'm a founding engineer at FriendliAI, uh, building a inference engine with our team to optimize inference performance. Uh, so today, I'm gonna talk about how we built the fast and reliable inference API for frontier open weight models. So . Oh, thank you . Okay, so before we dive in, let's do a quick show of

hands. So who here uses closed weight frontier models like GPT or Claude? Okay, most of you are already using it. And then who here mainly use open weight models?

Yeah, yeah, fewer than the closed weight models. So personally, I recently using a mix of both these days, but I'm gradually migrating more of my workload to open weight models. The reason is that open weight models, uh, have crossed the threshold these days. For example, in coding, reasoning, and agentic benchmarks, models like GLM five point two are competing in the same tier as the leading proprietary models. So as you can see from

this Code Arena re-leaderboard, uh, GLM five point two is ranked even higher than Claude Opus four point eight, so it is right, uh, below the Claude, uh, Fable five. So similarly, across various coding terminal and tool use benchmarks, uh, GLM five point two and MiniMax-M3 shows strong results. So the question is no longer, uh, only whether open weight models are good enough. For many workloads, now they are. So the

next question is, how fast and reliably we can run them in production. So benchmark numbers alone, uh, don't really give a clear sense of how well things actually works. So instead, I ran a, a simple hands-on experiment to see how these models actually perform. So I asked both GLM five point two and Opus four point eight, uh, four point eight to build a tower defense game like this. So as you can see, both models produce working

implementation with similar structure and quality. So both one has a map, and the turrets are shooting the, the enemies within the range. So yeah, both looks pretty great considering I made it with a si-uh, single prompt. But when we look at the cost, the difference becomes very clear. So GLM five point two was more than five point six times cheaper for this task. And in agent sly-style workflows where, uh, you need to iterate multiple times,

the, the, the gap compounds quickly. So this is not just about model quality anymore. It is about what you can afford to run at scale. So after the model quality and cost, the next que-next question is provider quality. And provider quality is really about performance. So even if the model weights are the same, latency, throughput, and reliability can vary significantly depending on where you run it. Uh, so this one has a OpenRouter snapshot,

which shows that, uh, these differences are measurable. So if you're building an AI product, you should not only ask which model to use, but you should also ask which providers gives you the best performance, reliability, and pricing. So I will show you another example. So here I use the same model as GLM five point two and the same coding agent workflow. So the task is to build a simple Candy Crush style mobile game, which we call Model Crush. So

when you see this, the left side is FriendliAI. It generates the, the, the game within two minutes. Another provider needs, uh, more than three minutes. So the model is the same, the agent ev-ev is the same, but the provider was different. So model developer tools and coding agent, this i- this expe- this is the experience that users actually feels. So from this simple example, we saw that even with the same model

and the same task, the experience can feel very different depending on the provider. The difference comes from the entire inference stack. So the model architecture, cache system, routing policy, scheduler, quantization strategy, GPU kernels, all those things come together to shape what the developer actually experiences. So this is the, uh, core of FriendliAI's value. So we are not just exposing a model API via, uh... So FriendliAI is built as a full stack

inference, uh, engine for GenAI models. This matters, uh, because fast and reliable inference is not just one optimization. It is batching, scheduling, quantization, spec decoding, mo- and kernels, all, all, everything should work together. So let me briefly explain, uh, how we built that and how we make, uh, the inference performance, uh, to the cutting-edge level. So in the

modern agent applications, inputs are often very long. It easily exceeds, uh, ten k tokens, while outputs are relatively short. And more importantly, a large portion of the inputs are repeated across requests. In many real workloads, like fifty percent to eighty to ninety percents of the tokens are identical across different requests. So from a system perspective, this means, uh, most of the compute cost is in the prefill stage, where

the model processes the entire inputs and build a KV cache. So if we recompute this k- uh, shared prefix every time, we are wasting both latency and GPU cycles. So when a new request share, uh, a prefix with the previous one, we can reuse those tensor and skip the re- corres- corresponding prefix computation and directly reduce the TTFT, and, uh, we can avoid recomputing large chunk of the input. And but however, making this work at scale requires two key che-

uh, mechanisms. So first one is cache-aware routing. So in the distributed system, KV cache is not centralized. It lives on individual replicas. So the router must send incoming request to the replica that already holds the relevant prefix in its cache. Otherwise, we ru- we lose the benefit of reuse. The second thing is cache hierarchy. So KV cache is extremely memory-intensive. For large models and long context, KV cache size can exceed

model weight. So, and as you know, GPU VRAM is the fastest, but it's also very limited resources. So, uh, we treat cache as a hierarchy, like hot cache stays in GPU memory, warm cache is in, uh, CPU RAM, and co- colder cache can be offloaded to NVMe or, uh, the disk spaces. The system dynamically offloads less used blocks and prefetches them when, uh, when they need it. So by combining cache elra- cache-aware routing and

hierarchical memory management, we significantly reduce the prefix cost, lower TTFT, and making, uh, long context agent workload much more efficient. So the next thing is interactive speed. So standard autoregressive decoding, it generates one tokens per forward pass that makes decode, uh, inherently a sequencer, and during the decode, the GPU is often underutilized. And speculative decoding changes the shape of this problem. So

instead of generating one token and stopping, it proposes multiple candidates tokens ahead. Then the target model validates those, uh, candidates in a single pass, and if the tokens are accepted, they are free. So this improves time per output token, uh, without changing model's, uh, output quality. And there are different ways to draft candidates. So one way is model-based speculation. It uses a small

draft model to predict the future tokens, and then, and the benefit of it is it can achieve higher accep- acceptance rate, but the cost is the, uh, extra compute. And another method is model-free speculation. It uses pattern from a prompt or prior output, so it has lower overhead, but it works best only when the output resembles the existing context. So the important part is the trade-off. So

speculation can improve per request latency, but validation adds overhead to the batch, so the best strategy is not static. And at FriendliAI, we take a hybrid approach, so the runtime can dynamically select between model-based and model-free speculation, uh, based on the workload and request, uh, characteristic. So we also think about speculation as something we can improve over time. So with self-evolving speculation, uh, draft model

can adapt based on the real serving traffic. So the speculator continuously improves its prediction of the workload, uh, actually running in the, in your, uh, production. So the next thing I want to mention is reliability. So speed alone is not enough. So production AI application, uh, reliability is part of speed. So if an API is fast in average, but frequently hits timeout, like five,

5XX error, 429 rate limit or fail, or that kind of failure, the user experiences, uh, speed bre- breaks. So this becomes even more important for agentic workloads. As you know, agent may need multiple model calls to complete one task, and if just one call fails in the middle, the entire workflow can be invalidated. So that's why we do not optimize only for the latency or throughput. We optimize the full API

experiences, uh, including the error rate, uptime, all those things together. And this OpenAI snapshot for, uh, GM- GLM 5.1 Pro, uh, shows that FriendliAI is not only the fast, but it's also high reliable with over ninety-nine point nine percent uptime. So what I want to say is many inference provider out here, of course, in this venue as well, they will tell you that they are fast, and some point of them,

uh, they are, but on cer- on certain benchmark, under certain condition. But for your application, the only benchmark that really matters is your own workload. So your prompt, uh, your token lengths, your traffic patterns, and your latency expectation. So our recommendation is simple. So please bring your workload to us and test the pu- free experience with us in terms of, like, speed, reliability, cost, and quality. And of

course, um, we build our inference service for exactly these production workloads, and we, and we believe that we can give you the best experience. So we have a booth in this venue, so bring your hardest problem to us, and let's solve it together. So to close, I want to share a testimonial from one of our customer. It's a Kilo Code. So Kilo Code is one of the largest coding agent platform, processing, like, several trillions of tokens every week.

So they have tested many different inference providers over past years, and in their speed test for, uh, GLM models, they were especially impressed by FriendliAI. So compared with many other inference provider, FriendliAI was over, like, f- seven time faster and had a sig- significantly lower error rate. So I believe you can get the same kind of experience with FriendliAI. So if you're building your frontier open weight models, building with your fr- frontier open weight

models, come try them on FriendliAI and bring your real workload, and let us show you what fast and reliable inference can do. So this is the end of my talk, and if you have any question, feel free to ask. Thank you.