AI Engineer World's Fair 2026
The Frontier AI Inference Cloud for Agents — Byung-Gon (Gon) Chun, FriendliAI
Read the talk
The Frontier AI Inference Cloud for Agents
Byung-Gon (Gon) Chun explains how FriendliAI serves agents by reusing growing context, preserving cache locality, and scheduling model calls around the time needed to finish a task.
From a talk by Byung-Gon (Gon) Chun
At a glance
Ideas worth remembering
End-to-end task latency captures the agent’s actual wait: model calls alternate with tool execution, context grows, and the number of calls depends on the input.
Prefix caching reuses computed KV state and processes the new suffix. Memory management, hierarchical caching, and reuse across replicas keep that state available.
Cache-aware routing preserves locality while balancing load. Evenly distributing requests can send an agent away from its cached prefix and repeat prefill.
Agent-level context can inform preemption, speculative prefill, and cache eviction, allowing a decision now to improve a later step.
Keep the comparisons separate: $1.50 versus 27 cents concerns the tower defense model comparison; 2× faster completion concerns the same-model mobile-game demonstration; seven times faster concerns Kilo’s customer split test.
One tower defense game, two different bills
Byung-Gon (Gon) Chun, founder and CEO of FriendliAI, introduces agentic inference from the serving side. FriendliAI grew out of a research team at Seoul National University; Chun credits that team with inventing continuous batching and describes its Orca work as an influence on vLLM. The question now is how inference infrastructure should change when agents become its customers. 0:41
Chun sees two developments coming together: agents spreading through software, operations, and knowledge work, and open-weight models becoming capable enough to make those workflows economical. A coding example gives that claim a concrete test. The same agent receives the same assignment—build a tower defense game—with an open-weight GLM 5.2 model served by FriendliAI and an Anthropic Opus 4.8 model. Chun describes both results as clearly usable, without claiming that the games are identical. GLM 5.2 and Opus 4.8 are the model labels Chun reports for these demonstrated runs. Exact checkpoints and evaluation conditions are not supplied, so this is a speaker-reported demonstration rather than an independently reproduced benchmark. 2:02
The bills differ substantially: about $1.50 for the Opus 4.8 run and 27 cents for the GLM 5.2 run on FriendliAI, a reported 5.6-fold cost difference. For this assignment, each result clears the practical quality threshold, making the cheaper run useful. That is the promise of open weights in an agent workflow. But the model bill is only part of the story: the serving system also determines how quickly and reliably the agent reaches its finished result.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The latency clock runs until the task finishes
Chat gives an inference service a simple unit of work: a person asks a question, the model answers, and the person reads the response. Request latency measures that wait. An agent’s assignment can contain many model calls and tool calls, continuing autonomously between them. Its user is waiting for completed work, so the main performance measure becomes end-to-end task latency. A fast response helps insofar as it advances the task. 3:46
The task typically runs through a plan–act–observe loop. Planning invokes the LLM; acting may invoke a tool; observing appends the tool’s result to the context before the next planning call. This creates gaps between inference calls while tools execute. The agent can also create sub-agents that run in parallel. Requests therefore arrive from an ongoing program whose next move depends on earlier results, rather than from a fixed sequence of independent questions.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Every observation adds context—and repeats old work
FriendliAI’s internal coding-agent runs show long inputs that grow as the task progresses. Each observation joins the next model input, so consecutive steps share a large beginning, or prefix, followed by newly appended material. Processing the entire input again means spending compute on context the service has already processed. In Chun’s words, it is work “we already did.” 5:31
A long-horizon research task makes the accumulation easier to understand: an agent is asked to explain a decoding framework in vLLM. The work has multiple stages, each containing sub-agents that make inference calls and tool calls. Tasks of this kind can involve tens or hundreds of inference steps over minutes or hours, while shared context continues through the work. Repeatedly processing that history adds cost and waiting time throughout the chain.
Three properties complicate serving: context grows, tool execution interrupts the sequence of model calls, and the number of calls depends on the input. A fixed request-rate assumption cannot capture the whole workload. The inference service needs to help an evolving task finish, including preserving useful state between calls when the agent is busy elsewhere.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Compute the prefix once, then keep it available
FriendliAI organizes its serving stack around four pillars: prefix caching, key-value cache management, cache-aware routing, and agent-aware optimization. These sit above model-level work such as sparse attention for long context, error reduction, and resilient serving. The shared objective is to reduce the time to finished work by recognizing the structure of agent workloads. 7:27
Prefix caching stores the key-value, or KV, state computed for the shared beginning of an input. On a later step, the service reuses that state and processes only the new suffix. Prefill is the input-processing work that would otherwise happen again before generation begins. Reading cached state is cheaper than recomputing that prefill, improving time to first token and reducing repeated compute. A longer task offers more opportunities to reuse the initial work. 8:08
The saving depends on cached state fitting somewhere and moving efficiently when needed. FriendliAI describes complementary mechanisms:
- GPU memory management: Pack more active context into GPU memory and reduce the KV memory footprint, allowing more useful state to remain available.
- Hierarchical caching: Use GPU memory, host memory, and disks to extend cache capacity beyond GPU limits.
- Reuse across replicas: Make a prefix usable across serving replicas, rather than restricting its value to one instance.
These mechanisms address capacity and availability; the talk does not develop the specific memory-reduction technique or policies for moving state between tiers.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Route task A back to its cached prefix
At cluster scale, a cache can exist and still go unused. A load balancer that spreads requests evenly may send a later agent call to a destination without its prefix. That destination performs cold prefill, repeating the computation that caching was meant to save. A cache-aware router considers where the required state already resides. 9:26
Follow task A in the talk’s routing example. Its first request goes through path one, where the prefix state is computed and cached. The agent then acts, observes the result, and calls the model again with the shared prefix plus appended context. Sending the second request through path one preserves locality: the service reuses the prefix and processes the suffix. Sending it somewhere without that state requires processing the prefix again. The destination changes the work needed to answer the agent’s next call.
What does routing the second request back to path one save? The diagram separates the agent’s progression from the location of its cached state. The warm route brings the new request and existing KV state together; the cold route repeats prefix computation. Locality still has to share the decision with load balancing, because repeatedly favoring one cache-rich path can make it a hotspot.
Processes the prefix through path one.
The second request can reuse prefix state on path one. A destination without that state must recompute it; the router must also consider load.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Schedule a call as part of a longer program
Agent-aware optimization extends this reasoning beyond routing. Scheduling every LLM call independently discards information about the longer program that produced it. Agent-level context could change which work proceeds now, what input gets prepared next, and which cached state remains available. 10:07
Chun identifies three possible decisions:
- Preemption: Interrupt the right work when knowledge of the larger agent task suggests a better scheduling choice.
- Speculative prefill: Prepare context for a likely next step before its model call arrives.
- Cache eviction: Use agent-level context to decide which state to discard.
These are presented as the next frontier, with examples rather than a complete scheduling policy. Their benefit may appear in a later call, which is why optimizing the whole task changes how the service judges a decision.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Measure the finished game, then choose how to deploy
The closing demonstration holds the model constant and changes the inference provider. Kilo Code uses the same GLM 5.2 model to create a simple mobile game through FriendliAI’s API and another provider’s API. Chun reports that FriendliAI finishes the task end to end 2× faster and attributes the result to its agent-focused cloud design. This measures the combined serving stack on that task; it does not isolate how much each caching, routing, or scheduling mechanism contributes. 10:57
The proposed production stack starts with an agent the builder already uses, adds a capable open-weight model, and serves it through infrastructure designed for task performance. Chun names MiniMax and Kimi among the model options and identifies Kilo and LG as production customers. Quality, speed, reliability, and cost belong in the same decision: lower token costs become valuable when the combined system completes useful work.
A Kilo customer testimonial supplies a separate comparison. Its split test of GLM5 usage against other third-party providers and direct model-lab access describes FriendliAI as consistently seven times faster, with a significantly lower error rate, and as a core component of Kilo’s stack. The testimonial does not specify the timing metric or quantify the error-rate difference. Its multiplier therefore belongs to that customer split test, separate from the GLM 5.2 model and 2× result in the mobile-game demonstration. 12:45
The same serving stack is offered through three deployment choices:
- Model API: A serverless API for calling frontier open-weight models, presented as the quickest way to begin.
- Dedicated endpoints: An isolated deployment with guaranteed service-level agreements for production workloads.
- BYOG—bring your own GPU: Friendli inference running on the customer’s own infrastructure.
The choice concerns how to consume the stack: through a shared API service, an isolated deployment, or infrastructure the customer supplies. 13:20
Chun closes by inviting builders to try frontier open-weight models in an existing agent stack. The practical question is whether the combination completes the builder’s tasks at an acceptable quality, cost, and speed. The tower defense comparison demonstrates a lower model bill; the mobile-game comparison demonstrates faster completion with the model held constant. Together they explain why economical agents require attention to both model choice and the inference system carrying the loop.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
Related talks
- KV Cache-Aware Routing and P/D Disaggregation on Kubernetes — Yuchen Fama & Ashish Kamra, Red Hat
Continues the serving-infrastructure topic with cache-aware routing and prefill/decode topology on Kubernetes.
- Context Engineering for Complex Codebases: Research, Plan, Implement
Pairs the serving-side explanation of repeated context with the application-side problem of organizing context for coding tasks.
Read the complete timestamped transcript
- 0:12
Let's, uh, get started. Uh, hi, everyone. Thank you for c-coming. Uh, this is the l-late afternoon in the last day, um, so I really appre-appreciate it. I'm Gon, founder and CEO of FriendliAI. Today, I wanna talk about agentic inference. So I'll first walk through what changed, uh, why it matters, and how we rebuilt the inference cloud for agents.
- 0:41
Before we go deeper, let me briefly introduce FriendliAI. FriendliAI is the frontier AI inference cloud for agents. So we run inference for agents at scale, uh, faster, cheaper, and more reliably. We are born from a research team at Seoul National University, and those research roots still define us. We are the team that invented continuous batching, the inference optimization that is now standard across the industry,
- 1:11
and our Orca work inspired vLLM, a widely used open source framework. Today, we operate globally, headquartered in San Francisco, with a team in Seoul to scale frontier inference.
- 1:25
As you know, 2026 is the year agents go into massive production, and it's driven by two trends coming together. First, agents are going exponential. AI an-agents are driving explosive adoption across software, operations, and knowledge work. Second, open-weight models have reached the frontier and make agents economical. They now rival closed frontier models in capability, which
- 1:55
means you can run frontier quality agents on open models with much lower token cost.
- 2:02
Let me make the open-weight model part concrete. Open-weight models are now strong enough for these types of real agentic workflows. Here, we gave the exact same task, building a tower defense game with a coding agent, to two models. On the left is GLM 5.2, an open-weight model running on FriendliAI. On the right is Anthropic's Opus 4.8. The important point is not that the outputs are identical.
- 2:32
The point is that both complete the task at a level that is clearly usable. For many agentic workflows, open-weight models have crossed the quality threshold.
- 2:46
But the economics are very different. For the same te-task, Opus 4.8 costs about $1.50, GLM 5.2 on FriendliAI costs 27 cents, about 5.6 times cheaper. So this is the promise I mentioned earlier. Open-weight models give you frontier quality agents at a fraction of the cost. But model cost is only one part of the story. To make
- 3:16
agent actually fast and reliable, the inference stack itself has to change. So let's look at what actually happens inside an agentic workload. So first, let's look at changes in the workload. In the past, the dominant usage was chat. The basic unit was a request. A person asks a question, the model answers, and the person reads it. Latency meant how fast did I get one response.
- 3:46
Agents are different. The basic unit is a task. A task may involve many model calls, many tool calls, and they may run autonomously for a while. So the user does not really care about the latency of one individual request. The user cares about when the whole task is completed. That means we have to optimize for tasks, not just individual requests.
- 4:16
Let's look at agentic workloads more closely. An agent really runs a session made up of tasks. Each task typically runs in a loop. First it plans, which usually means an LLM call. Then it acts, maybe by calling a tool. Then it observes the result and adds the, that back into the context, and it repeats this until the task is done. So we are constantly alternating between LLM inference
- 4:45
and one or more non-LLM tool executions. So there is a gap between LLM calls. An agent can also create sub-agents and run them in parallel.
- 5:01
Agent inputs also look very different from chat. The graph here shows the prompt and completion length dis-distributions of our internal coding agent runs with GLM 5.2, which we use day to day. They are much longer. They grow as the task progresses since every observation gets appended back into the context. There is an important pattern here. Consecutive agent steps usually share a
- 5:31
huge prefix. If we recompute the same prefix every time, we are burning a lot of compute on work we already did. So this is one of the biggest opportunities in agentic inference.
- 5:46
So how token hungry are agents? Now let's look at a long-horizon task example like deep research. We ran Explain the Spec decoding framework in vLLM using code code with GLM 5.2 on FriendliAI. There are multiple stages, and each stage is composed of sub-agents which run multiple inferences and tool calls. So it might run tens or even hundreds of inference steps,
- 6:17
sometimes over minutes or hours, and the shared context keeps going the whole time. For the user, what matters is not the latency of a single token or one call, what matters is when is my task completed.
- 6:35
So agentic inference is not just chat with more requests. It's a different problem. The context grows over time. Tool work is interleaved between model calls. The number of model calls depends on the input, so you can't really plan around a fixed request rate plan. And the real metric is end-to-end task latency, not a single request latency.
- 7:03
This is where FriendliAI comes in. We rebuilt the frontier inference cloud specifically for agentic workloads around the challenges I just walked through. And we set one goal: optimize end-to-end task latency, uh, the task, not just the request. So how do we do that? Let me show you the key engineering behind it.
- 7:27
Here's the engineering map for how we think about it. We built the stack layer by layer around agentic workloads. There are four big pillars I'm gonna cover today: prefix caching, key value, in short, KV, cache management, cache-aware routing, agent-aware optimization. And of course, underneath, we need model layer optimization like sparse attention for long context, techniques to
- 7:56
reduce errors, fast corners, resilient serving, and more. In this talk, I'm gonna focus on the four pillars.
- 8:08
Let's start with prefix caching. Since agent steps share a large prefix, we compute key value for the prefix once and cache it. Then on later steps, we reuse the cache key value and only process the new suffix. Reading from cache is much cheaper than recomputing prefill, so this improves time to first token and reduces compute on every step. And the longer the task runs in agents, the more valuable this
- 8:38
becomes.
- 8:41
But caching only works if the k- KV cache actually fits and can move around efficiently. So we need strong KV cache management. We use frugal memory management to pack more active context onto each GPU memory. We use KV contiguation to reduce the memory footprint. We u- we use hierarchical caching across GPU memory, host memory, and disks, so
- 9:11
we can go beyond GPU limits. And we also use disputed caching, so one prefix can be served across replicas, not just inside one instance.
- 9:26
At global cluster scale, routing becomes really important. A naive load balancer may spread requests evenly across GPU clusters, but it can destroy cache locality. A cache-aware router at a global scale does something smarter. It sends a request to the path that already has the right prefix cached, turning a cold prefill into a warm cache hit. At the same time, it still has to balance load, so one path doesn't become a
- 9:56
hotspot. In this example, the two requests of task A go to the same path one for cache locality.
- 10:07
The next piece is agent-aware optimization, and this is the next frontier of agentic inference. Today, most systems schedule each LLM call as if it were independent. They don't really understand that this call is part of a l- longer agent program. But if the optimizer knows the agent-level context, it can make better decisions. For example, preempting the right work,
- 10:37
speculatively prefilling context for a likely next step, or making a better cache eviction decision based on agent-level context. So the goal is to reduce end-to-end task latency, not just make one call look fast.
- 10:57
When we put all of this together, this is the payoff. We are using the same model, GLM5.2, with Kilo Code to create a simple mobile game. We ran the same task with model APIs of FriendliAI and another well-known inference provider. As you can see, FriendliAI completes the same task end to end 2x faster thanks to our agentic-centric cloud design.
- 11:27
So what does this unlock in practice? A stronger production agent stack. Take an agent you already like. Now plug in open-weight frontier models like GLM5.2, MiniMax, and Kimi served on FriendliAI. The model gives you frontier quality capability and better economics. FriendliAI gives you the speed, reliability, and end-to-end task performance needed in production.
- 11:57
That combination, quality, speed, reliability, and cost, is what makes agents actually useful and economical in production
- 12:09
FriendliAI is currently powering teams in production from AI native startups to global enterprises. I'd like to highlight a couple here. Kilo is a hugely popular agentic AI coding tool serving millions of users.
- 12:28
LG is a global enterprise whose businesses range from electronics to healthcare to energy. Very different companies, but they all need the same thing: fast, reliable, cost-effective agentic inference.
- 12:45
This testimonial from our client Kilo says it all. Over the past year, Kilo Code has tested several inference providers hosting both open and closed models. In a split test of GLM5 usage compared against other third-party providers and direct usage from the model lab g.ai, FriendliAI was consistently seven times faster with a significantly lower error rate. Today, FriendliAI is a core
- 13:15
component of the Kilo stack.
- 13:20
And you can consume this however fits your stack. Model API is the fastest way to start. Call Frontier open-weight models through our serverless API, dedicated endpoints give you your own isolated deployment with guaranteed SLAs for production workloads. And BYOG, bring your own GPU, lets you run Friendli inference on your own infrastructure. Same stack, three ways to deploy.
- 13:49
To wrap up, there are three things to remember. First, Frontier open-weight models make production agents economically scalable. Second, agents are not just chat with more calls. Agentic inference requires optimizing end-to-end task latency with the challenges I mentioned. Third, FriendliAI is built as an inference cloud for that world. Fast, reliable, cost-effective agentic
- 14:19
inference. Thank you for attending my session. If you're building agents, give, uh, Frontier open-weight models a try on FriendliAI today. You can get started at friendli.ai in minutes. And, uh, thank you. I'll be around after the session. Um, thank you.