# Your Self-Hosted LLM Needs to Say No - Harshul Jain & Parv Khatri, Audible & SAP

Harshul Jain, Audible · Parv Khatri, SAP

[Talk page](https://ai.engineer/talks/ntVpbyDTYaQ-your-self-hosted-llm-needs-say-no) · [Video](https://www.youtube.com/watch?v=ntVpbyDTYaQ)

## Your Self-Hosted LLM Needs to Say No

One request to fix a failing test can turn into dozens of model calls. Harshul Jain and Parv Khatri show how to decide which calls reach the GPU, how an agent should handle a refusal, and how to measure the fraction of arriving requests that get an answer on time.

### The prompt arrives before anyone knows the cost of the answer

[0:05](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=5s)

Ask a model to fix a failing test. The serving system can count the input tokens when the prompt arrives. The answer takes shape one token at a time, with each token becoming part of the context for the next. By the time the system knows the output length, it has already generated the answer. Admission control has to decide whether to start much earlier, while the GPU time this request will need is still uncertain.

Follow the request through a self-hosted deployment. The API gateway checks the key and counts calls. A router picks a replica, one running copy of the model. Its inference engine loads the weights onto the GPU and batches requests so several answers can advance together. Each active request also accumulates a key-value (KV) cache, the model state it keeps for the context it has already processed. A longer answer adds more state.

How much work the deployment can take depends on what the product promises. In interactive chat, people notice both a poor answer and a long wait for the next token. An asynchronous agent can tolerate delay, but a poor answer may mean rerunning an expensive task, so quality and throughput matter. Autocomplete needs to return many suggestions quickly and can use a less expensive model. Suppose a deployment can sustain eight concurrent requests while meeting its service-level objective (SLO). What happens when 40 arrive in two seconds?

Two incidents in Jain and Khatri's review show what can give way first. On November 8, 2023, heavy traffic at OpenAI exhausted memory on routing servers. Health checks began failing. The team added replicas, rolled them back, then cut incoming traffic and restored it gradually. An Anthropic model tier slowed down during a January 21, 2026 incident and began returning errors over roughly 90 minutes. The reported cause was insufficient compute for demand. OpenAI's follow-up emphasized shedding load gracefully. Anthropic's emphasized detecting the capacity limit earlier.

### What the engine commits when it starts a request

[8:49](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=529s)

Before a request takes GPU memory, an admission layer can start it, defer it or refuse it. Starting it commits the engine to two phases of work. During **prefill**, the engine reads the prompt and builds its KV cache. During **decode**, it extends the answer and the cache one token at a time. That memory stays occupied as long as the request is running.

What does a short request inherit from the calls ahead of it? Consider A, which produces a short answer, followed by B, which produces a long one. When short request C arrives, B is still in memory and its cache can keep growing. C may need little work of its own and still miss its latency target. The decision about C depends partly on a commitment the engine made to B.

If active caches fill memory, the engine can evict work, swap a request out to wait, or return an error after spending resources on it. Any of those can cost the service its latency target. Teams that self-host for privacy or cost own those decisions along with the GPUs.

A person chatting with a model leaves gaps to read the replies. A coding agent can spend those same seconds issuing calls, retrying them and starting more work. In Jain's test-fix example, the agent might read about 14 files and run about 10 tests. Across the task, tool use and follow-up reasoning can generate 40–50 calls, some quick and some long-running. At the cluster, one user's task has become a burst of related requests with different costs.

![Stacks of KV-cache blocks approach a dashed GPU memory limit over time. Three panels describe evicting the cache and recomputing later, swapping it to CPU or disk, and preempting the request with an error.](https://ai.engineer/api/talk-essay-media/ba7b9db401783022ab333e247e8ffa0ff791a9356eb07eba5a72aa4aa702c5cf)

Generation adds KV pages after admission, so the memory needed by a running request can continue to grow.

### Bring the request and the engine state together at ingress

[15:32](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=932s)

Which component can decide whether there is room? The agent sees its own call and controls retries. The gateway knows the API key and its rate limit. The router knows which replicas are healthy. The engine sees its batch and cache and can decide what to run or evict. In this arrangement, the component with the best view of pressure discovers the cost after the request has entered.

Placement changes both visibility and maintenance cost. A proxy at **ingress** can inspect the incoming request and whatever metrics the engine exports. At the **queue boundary**, the scheduler can intervene as it chooses the next batch. **During decode**, it can pause a running request or reclaim its cache halfway through an answer. Those later interventions have more information, but changing the scheduler can leave the team maintaining an engine fork through upstream releases. Jain and Khatri choose ingress so the policy can run against a stock engine.

What reaches that proxy, and which decision sends work onward? The request supplies input length, tenant, agent session and, when available, a deadline. The engine supplies cache occupancy, the age of the oldest queued request, prompt-cache hits and load across replica pools. The proxy can now ask whether the deployment has room for *this* caller's work. Only an admission sends the request onward to acquire GPU memory.

These inputs suggest different actions. Defer a prompt that exceeds current memory headroom or the extra calls from a session with too much work in flight. Refuse a tenant that exceeds its share, or a request whose deadline would expire in the queue. Spare KV capacity favors admission. A long wait for the oldest queued request favors deferral. A cached prompt can go first because some of its processing is already paid for. When every pool is full, refuse the arrival. For any of these rules, the observation has to be timely. A one-minute average can wash out 40 calls in two seconds.

![Arrows point into a policy that can admit, defer, or reject. Request inputs are input length, client or tenant, agent session, and an optional deadline. Engine inputs are KV cache use, queue age, cache hit rate, and load per pool.](https://ai.engineer/api/talk-essay-media/99f12c76d7dd76cd5dbbeb70c5b040c772be9c7807eff3b049750485160e4696)

The ingress policy combines properties of the incoming request with live queue, cache and load signals.

Which path commits GPU memory?
Engine metrics travel back to ingress as information. Of the three decisions the proxy can make with that information, only admission sends work to the engine.
- Incoming request — Input length, tenant, session and deadline
- Engine metrics — KV occupancy, queue age, cache hits and replica load
- Ingress policy — Decides before the request acquires a KV cache
- Start on the engine — Allocate cache and join the running batch
- Try later — Give the client a delay while capacity returns
- Refuse this attempt — Return a reason the caller can act on
- Incoming request → Ingress policy (Request attributes)
- Engine metrics → Ingress policy (Exported state)
- Ingress policy → Start on the engine (Admit)
- Ingress policy → Try later (Defer)
- Ingress policy → Refuse this attempt (Reject)

### What the agent should do with each response

[22:08](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1328s)

**Admit.** A `200` opens a stream. The engine has committed cache for the prompt, and the request decodes alongside its batch, one token per step. On the client, distinguish the wait for the first token from time spent streaming. Prefill alone can take seconds under load, so Khatri starts the streaming timeout at the first token. If the agent abandons the answer, close the connection. A local timeout that leaves it open can leave the engine generating tokens and holding memory. A stream can also break after the headers have arrived. The agent then has a partial answer and must decide whether it can continue or needs another attempt.

**Defer.** For a temporary burst, the proposed server returns `503` with `Retry-After`. It can estimate the delay from when running work is likely to free enough cache. A rising queue age with capacity due back soon is a reason to ask the client to wait. Under sustained saturation, repeatedly issuing short deferrals just reschedules the same demand.

When do ten deferred clients come back? If all ten retry immediately, they create another spike while the server is still busy. Across the original burst, one retry per call can turn 40 attempts into 80. Honoring `Retry-After` gives the engine time to make progress. Adding a small random delay, or jitter, spreads the returning clients out even when they all received the same suggested wait.

**Reject.** The reason determines what the agent can change.

- **Caller over its share (`429`).** The policy may have seen a quota violation or too many concurrent calls from this tenant or session. Reduce the agent's load by capping its in-flight calls.
- **Provider overloaded (`529` in the example).** All replica pools are busy. Wait longer, switch to another model or provider, or park the task for later.

In either case, give retries a budget. Use exponential backoff with jitter and tell the user what happened once the budget runs out. Otherwise a refusal at ingress can immediately become another arrival.

![Client pseudocode waits for the Retry-After delay plus jitter before retrying. The diagram shows ten retries clustered together in the upper panel and spread across time in the lower panel.](https://ai.engineer/api/talk-essay-media/8817144544783d4554adcb80850a130da334708a6a1ef87941a8e30b0a7b554c)

Waiting exactly the suggested interval makes clients retry together. Jitter spreads their retries out.

### Twenty successful calls can still leave two tasks unfinished

[27:33](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1653s)

A fixed allowance per API key helps tenants share the service, but it stays fixed as GPU load changes. A proxy that caps concurrent requests per replica gets closer to the hardware. Its blind spot is cost. A quick lookup and a long planning call each occupy one slot, even though their token counts, cache use and lifetimes differ. Once the queue grows, the client's timeout can end up making the refusal. A cost-aware policy estimates the incoming work in tokens and KV memory against the engine's current state.

Specifying a policy takes four choices. The **unit** says what gets admitted, such as a request, an agent session or a tenant. The **signal** is what the policy observes as the system gets busy. The **rule** converts that observation into a decision and takes time to run. The **action** determines what happens to the caller. These choices constrain each other. A deadline rule, for instance, needs the caller to supply a deadline.

What changes if the unit is a session? Imagine two agent tasks with 20 calls each and room for about half the calls. Split admissions evenly and each task gets roughly ten calls through. Both still depend on calls that failed to get in. All the capacity has been used, and neither task can finish. In the session-level alternative, A gets its full set of calls and finishes. B gets one clear deferral. A count of successful calls would miss the difference between zero completed tasks and one.

Choose the signal by the failure you need to detect. KV occupancy can reveal impending cache exhaustion. Pair occupancy with cache-hit rate when the cost is recomputation, since even a full cache may be doing valuable work by reusing prompt prefixes. Queue depth exposes accumulating waits. Deadline slack tells you how much delay a request can still tolerate. An estimate of output length helps plan for long answers. Cache pressure, for example, will miss overload on a deployment where the cache never comes close to filling.

### Four ways to turn a signal into a decision

[34:01](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2041s)

The rule itself has to fit inside the time available for a decision. For the agent sending 40 calls in two seconds, a planner that takes five seconds can finish after the entire burst has arrived. Four research designs illustrate what that planning time buys and what cheaper rules can see.

- **Single threshold.** Compare KV occupancy with one limit for each incoming request. The example takes tens of microseconds per decision. It can respond quickly on each arrival, provided cache pressure actually rises under load.
- **Dual gate.** Require two signals, KV use and cache hits, to agree before acting. The example admits whole agent sessions and checks the signals in under 100 microseconds. It reported roughly four times SGLang's throughput in an offline test. That result leaves performance under live arrivals and retries open.
- **Fleet optimizer.** QLM plans across queues together to decide which requests to admit. It can account for work elsewhere in the fleet, at a reported cost of about five seconds per decision.
- **Formal deadline bound.** Calculate whether a request can still meet its time-to-first-token deadline in the worst case covered by the bound. The example makes a microsecond-scale decision and reported a drop in deadline misses from 99% to zero in its evaluation.

The papers used different models, engines and workloads, so their gains cannot establish a ranking. Jain compares them along two axes. Does the policy optimize deadlines or throughput? Does it admit individual requests or whole sessions? For a coding agent, Jain wants to admit sessions as live traffic arrives, using an engine he can deploy as shipped. The session-aware dual gate comes close because it can run without an engine fork. Its offline evaluation leaves that combination unproven.

Which rule would you try first on your own deployment? If requests carry deadlines, try a deadline bound. Otherwise, observe the deployment at peak load. If KV memory fills, try an occupancy threshold, or add a second gate when prefix reuse affects the cost of admission. If the cache has room while calls pile up, try a queue threshold. You can find out which branch applies before spending time tuning a threshold.

### Replay the burst and count every arrival

[38:35](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2315s)

Start by measuring the load the replicas can sustain with admission control disabled. That establishes what 1× capacity means on this deployment. Specify the candidate policy's inputs, limits and refusal behavior. Then replay the same traffic trace with the policy enabled and disabled, sweeping offered load from 0.5× to 2× measured capacity. Below capacity a policy can appear to work simply because it has never had to refuse anything.

What score makes the cost of a refusal visible? Khatri's **goodput under admission** divides the number of requests meeting the SLO by the number of requests that arrived. For the streaming SLO here, the answer must start on time and avoid excessive gaps between tokens. Suppose the policy refuses 95 of 100 requests and serves the other five perfectly. Attainment among admitted requests is five out of five, or 100%. Goodput under admission counts all 100 in the denominator. Five percent of all arriving requests met the latency target.

The trace needs to preserve the behavior that creates pressure in the first place. Keep agent fan-out, retries and session membership. Uniform random arrivals discard the timing and dependencies the policy must handle. Set the deadline relative to latency measured without load so readers can tell how much slowdown the experiment allowed.

Khatri asks for seven disclosures alongside an admission result.

- **Load.** Give offered load as a multiple of measured capacity.
- **Deadline.** Express it relative to unloaded latency.
- **Signal liveness.** Show whether the metric the policy watches actually moved under stress.
- **Variation.** Give the repeat count and how much the runs differed.
- **Parameters.** Identify which values were tuned locally, borrowed or left at their defaults.
- **Denominator.** Say whether refusals counted toward goodput.
- **Configuration.** Record the model, engine and version, hardware, workload, SLO and baseline.

### Benchmark the same decision function that handles live traffic

[43:51](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2631s)

Jain and Khatri built **AdmitPerf** to compare policies on a common deployment. Its ingress library sits in front of an inference engine such as vLLM or SGLang and makes a decision for each arrival. Its benchmark harness exercises a policy against an engine on a real GPU. Both call the same decision function, so a benchmarked rule can go into the serving path without being reimplemented. Among the 16 papers they compared on evaluation setup, no two shared the same combination of baseline, engine, workload and SLO.

The harness has five operations. Bring up the GPU-backed engine, smoke-test that it serves requests, run each policy several times, compare the results and the size of the differences, then tear the infrastructure down to stop paying for it. Provisioning sits outside the benchmark. Loading the model takes minutes, and reprovisioning between candidates would change the deployment being used as the control.

How can the harness tell an empty cache from a missing metric? Each policy declares the signals it needs. A KV-threshold policy requires the fraction of KV cache in use. If the engine cannot supply it, the harness stops at startup. Without that check, a missing value could read as zero. The policy would admit every request and produce a plausible-looking benchmark of the admit-all baseline.

For a run that proceeds, the harness records each decision with the state the policy saw. The run directory contains four JSON artifacts covering the manifest, summary, decisions and outcomes. An engineer can inspect what triggered a refusal on a particular arrival as well as what changed over the whole run.

### On the A10, the queue grew while the KV cache had room

[47:48](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2868s)

The GPU comparison used a half-billion-parameter model on an A10, with the engine capped at four simultaneous sequences. The arrival rate was 14 requests per second, with two repeats. With admission control disabled, p95 time to first token was about 2.5 seconds, with variation of roughly 700 milliseconds either way. That percentile describes the wait near the slow end of the observed requests.

What did a shorter queue buy? A policy allowing at most two waiting requests admitted about 63% of arrivals and shed about 37%. Its p95 time to first token fell to about one second. Goodput under admission stayed the same in this comparison. Admitted requests saw a shorter tail wait while the fraction of *all arrivals* served within the SLO held steady.

The engine metrics explain why queue depth worked here. KV occupancy peaked at about 0.5%. Four short active sequences left ample cache even as work accumulated outside the batch. A policy waiting for high KV occupancy would have kept admitting work. The waiting-request cap had an effect because the queue was where this deployment registered overload.

There is a laptop path for trying the mechanism. AdmitPerf includes a small fake engine that implements the three endpoints the harness uses and gets busy under load. A developer can exercise the policy there, then measure its behavior on the target GPU. AdmitPerf can also discover a policy packaged separately from its repository. The tool grew out of the *LLM Inference at Scale* handbook survey. Jain points readers to both projects on the [AdmitPerf source-code slide](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3084s).

## Key takeaways

- Make the admission decision before prefill commits GPU memory. A request that fits now can still put pressure on the batch as its answer and KV cache grow.
- Choose the admission unit around what must finish. Splitting capacity evenly across two dependent sets of calls can leave both agent tasks waiting.
- Give the agent an action it can take. `Retry-After` with jitter spreads returning traffic out. A caller-specific limit calls for lower concurrency. Pool-wide overload calls for a longer wait, another provider or parking the task.
- Check whether the signal moves under load, then score the policy across all arriving requests. On the A10, KV occupancy peaked at 0.5%. Capping the queue at two waiting requests cut p95 time to first token from about 2.5 seconds to about one second at the same reported goodput.

## Slides

### Saying No.

When a coding agent sends more requests than your self-hosted LLM can handle, admission control decides which requests get through.

[0:00](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=0s) [Title slide for “Saying No.” by Harshul Jain and Parv Khatri. The subtitle introduces admission control when a coding agent overwhelms a self-hosted LLM.](https://assets.ai.engineer/sites/aiecode2025/sha256/7a833c8fd7930b20ab68d449a0cf20190ccfdcd6357b48547f259463ea555bd5.webp)

Saying No.
Admission Control When Your Coding Agent Overwhelms Your Own LLM

Harshul Jain    Parv Khatri

AIE CODE SF 2026
45 MIN · 4 PARTS

01 Motivation
02 Admission control
03 Strategies
04 Benchmarking

### Four stops, one request’s journey

The orange card marks the starting question: why can the system admit only so much traffic? One request connects the next three stops: where the system decides whether to admit it, how it says no, and how to test whether the policy works.

[2:14](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=134s) [The orange Motivation card introduces the question of how much traffic infrastructure can admit. Three white cards follow for admission control, strategies and benchmarking.](https://assets.ai.engineer/sites/aiecode2025/sha256/dcca267ccbca07141816984a76f1aa9c01063d10158c35e3585a0ff2c16e0dcd.webp)

AGENDA
Four stops, one request’s journey

01 Motivation
Why infrastructure can only admit so much

02 Admission control
Where the decision lives and what it needs

03 Strategies
How a system says no

04 Benchmarking
How to test a policy

4 / 66

Interface text:
4 / 66  [illegible]

### Inference: a prompt goes in, an answer comes out

A user asks for help fixing a failing test. The model builds its reply one token at a time, using earlier tokens to choose the next, and streams back an explanation that the test expects a sorted list.

[2:20](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=140s) [A user asks for help fixing a failing test. The model generates “The,” “test” and “expects” in sequence. The streamed answer explains that the test expects a sorted list.](https://assets.ai.engineer/sites/aiecode2025/sha256/8661523d99329e84eeb60f53176c35440c285382546d12d7cbdcf5950d63130f.webp)

Inference: a prompt goes in, an answer comes out
STARTING POINT · LLM INFERENCE

1 · The prompt
someone asks
Fix this failing test

2 · The model
writes one token at a time
LLM
1  The
2  test
3  expects
4  next...
each word depends on the ones before

3 · The answer
streams back
The test expects a sorted list.
Fix line 42...

One request is one trip through the model. A product makes thousands of these trips a minute.
5 / 66

Interface text:
5 / 66  [illegible]

### Two public outages, two years apart

OpenAI cut incoming traffic after its routers hit memory limits. At Anthropic, Sonnet 4.5 slowed down before 5xx errors appeared. Both follow-ups point to deciding which requests to accept before they reach the engine.

[7:11](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=431s) [The OpenAI panel shows routers reaching a memory limit and names graceful load shedding as the follow-up. The Anthropic panel shows rising latency and server errors under insufficient compute and calls for earlier detection of capacity limits.](https://assets.ai.engineer/sites/aiecode2025/sha256/9926dabb263d3ce5a9014d13b2b212157f5bd59d092df07cc93ca18ee7194eed.webp)

TWO DAYS IT WENT WRONG
Two public outages, two years apart

OpenAI — Nov 8, 2023
router    router
memory limit
1  Heavy completion traffic all day
2  Routing nodes hit memory limits
3  Replicas added, then rolled back
4  Fix: cut inbound traffic, ramp slowly
Their follow-up: graceful load shedding.

Anthropic — Jan 21, 2026
latency
5xx
5xx
not enough compute
1  Sonnet 4.5 tier slowed down first
2  Errors followed, over about 90 minutes
3  Cause: not enough compute for demand
Their follow-up: detect capacity limits earlier.

Neither follow-up says a faster engine. Both ask for a decision earlier, before the engine.
More incidents in the appendix →
8 / 66

Interface text: [illegible]

### Both follow-ups point at the front door

Admission control sits between the client and the queue. Before a request takes GPU memory, it decides whether to start it now, queue it for later, or reject it. Here, a 429 tells the client to retry in 2 seconds.

[8:48](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=528s) [The pipeline highlights Admit between Client and Queue, followed by Route and Engine. Three panels show a request taking GPU memory immediately, waiting in a queue, or receiving a 429 response with a retry delay.](https://assets.ai.engineer/sites/aiecode2025/sha256/9b069dfc6fbc366244cd54673f6fb1bcb3a8750eb36118e287dfd49f3f54a64d.webp)

WORKING BACKWARD · WHERE THE FIX LIVES
Both follow-ups point at the front door

Client → Admit → Queue → Route → Engine

Admit    start now
GPU memory
Into service, takes GPU memory now.

Hold    wait in queue
queue · served later
Accepted, served later.

Reject    not now
429
at capacity
retry in 2 s
A defined error: 429, 503 or 529.

This decision is admission control. Why does it have to happen before the engine?
9 / 66

### A yes holds memory for the request’s whole life

Prefill builds the KV cache. Each output token grows it until the last token frees the memory. Request B’s longer answer holds memory longer than A’s or C’s. At admission, the engine can’t tell how long that will be.

[10:12](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=612s) [Three request timelines show a wait, a blue prefill block and a yellow decode segment that grows as tokens are generated. Brackets mark memory held from prefill through the last token. Request B has the longest decode and holds memory longest.](https://assets.ai.engineer/sites/aiecode2025/sha256/f9a6a14d840121491fa6d4123c6cc5671e24ef6d3dc0c6183ea82fe61b1a6581.webp)

REQUEST LIFECYCLE · PREFILL AND DECODE
A yes holds memory for the request’s whole life

Request A
short answer
prefill    decode
memory held    freed

Request B
long answer
prefill    decode
memory held    freed

Request C
short answer
prefill    decode
memory held    freed

How long will it run?
Unknown at the door, known only at the end

Waits: no memory used
Prefill: reads the prompt
Decode: one token at a time, cache grows

The KV cache grows until the last token. At the door, nobody knows how long that will be.
10 / 66

### Memory keeps growing after a request is admitted

As a request generates tokens, its KV cache grows. If GPU memory fills, the engine can drop the cache and rebuild it later, move it to CPU or disk, or cancel the request. Rejecting the request up front avoids wasting or delaying work.

[11:33](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=693s) [Stacks of KV-cache blocks approach a dashed GPU memory limit over time. Three panels describe evicting the cache and recomputing later, swapping it to CPU or disk, and preempting the request with an error.](https://assets.ai.engineer/sites/aiecode2025/sha256/ba7b9db401783022ab333e247e8ffa0ff791a9356eb07eba5a72aa4aa702c5cf.webp)

WHY THE ENGINE CANNOT FIX IT ALONE
Memory keeps growing after a request is admitted

KV cache in GPU memory
illustrative
!   GPU memory full   !
time →

Evict
drop the cache, recompute later

Swap
move the cache to CPU or disk

Preempt
cancel the request, return an error

All three throw away or delay work already done. Saying no at the door avoids all three.
11 / 66

Interface text: 11/66   [illegible]

### More teams now host their own models

Teams self-host to keep code and prompts in-house, make costs predictable at volume and choose which model to run. They also take over a decision the API provider used to make: which requests their GPUs will serve.

[12:28](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=748s) [Three panels give reasons for self-hosting models. Privacy keeps code and prompts inside the network. Cost reflects owned or reserved GPUs. Control lets the team choose the model, version and serving setup.](https://assets.ai.engineer/sites/aiecode2025/sha256/bc6fd767cabd2d69d090b546822017d6e6a8145b6da3ced4215b2b522d34f2ab.webp)

WHOSE PROBLEM THIS IS
More teams now host their own models

Privacy
code stays in-house
your code
Your code and prompts never leave your network.

Cost
predictable at volume
GPU
owned
You pay for GPUs you own or reserve, not per token.

Control
your choice of model
Models
Llama 3.3 70B
Qwen3 Coder
your fine-tune
You pick the model, the version and how it is served.

With a hosted API, the provider runs the front door. When you host it, that job is yours.
12 / 66

Interface text:
12 / 66
[illegible]

### Coding Agents changed what arrives

Serving stacks were tuned for people who pause to read between requests. A coding agent can make dozens of calls for one task, retry on timeout, and stall the whole plan while it waits for one slow call.

[13:17](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=797s) [A person at a laptop sends one request at a time and pauses to read. Beside it, a coding agent sends dozens of calls for one task and retries on timeout. A slow call stalls the agent’s plan.](https://assets.ai.engineer/sites/aiecode2025/sha256/8a52a50f82577d453d771b57a67a9c7853f4e50adef41be9f6f26e4314c28aa4.webp)

MOTIVATION · WHY NOW
Coding Agents changed what arrives

A person chatting
the traffic we planned for
one at a time
· One request at a time
· Seconds of reading in between
· A slow answer is an annoyance

A coding agent
the traffic we got
+ retries
· Dozens of calls for one task
· No pause, retries on timeout
· A slow call stalls the whole plan

Serving stacks were tuned for the left panel. Agents send the right one. Here is one task.
13 / 66

Interface text:
13 / 66  [illegible]

### Forty requests, eight slots, three options

When forty requests compete for eight slots, queueing makes new arrivals wait. Evicting a running request discards its partial answer. Admission control can refuse a request early with a 429 and tell the caller to retry in two seconds.

[15:01](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=901s) [For forty requests and eight slots, three panels show the tradeoffs. Queueing delays request starts, eviction discards a partial answer, and early refusal sends the caller a 429 response with a two-second retry delay.](https://assets.ai.engineer/sites/aiecode2025/sha256/afee6e21af0d82b09969a70f26d563cb9fb1fbaf69dc6abccdf2e932e7ded5bb.webp)

PUTTING IT TOGETHER
Forty requests, eight slots, three options

Queue it
everyone waits
The line grows and every request starts later.

Evict it
drop running work
partial answer lost
Work already done is thrown away.

Refuse it
admission control
429
retry in 2 s
The caller gets a clear no, early, and can retry.

Most engines do the first two by default. This talk is about the third.
15 / 66

### Four stops, one request’s journey

A checkmark closes the section on capacity limits. The orange card marks the second stop, admission control. Where does the system decide whether to accept a request, and what information does it need?

[15:30](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=930s) [Four agenda cards run left to right. Fundamentals is checked, Admission control is highlighted in orange, and Strategies and Benchmarking follow.](https://assets.ai.engineer/sites/aiecode2025/sha256/70caa5eec7b35b0b9d9dcd7f2355bf021ec19c789558c9ec301a051241852d0f.webp)

AGENDA
Four stops, one request’s journey

✓ Fundamentals
Why infrastructure can only admit so much

02 Admission control
Where the decision lives and what it needs

03 Strategies
How a system says no

04 Benchmarking
How to test a policy

17 / 66

Interface text:
17/66  [illegible]

### Admission control

Where in the serving stack should a system decide whether to accept a request? This section asks what information the system needs to make that call and what response the client receives.

[15:41](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=941s) [Section 02 asks where the decision to say no lives. It covers where the decision sits, what it returns to the client, and what it needs to see. A presenter tile appears on the right.](https://assets.ai.engineer/sites/aiecode2025/sha256/de313455328fe647f5b98eb3535ac7049a9d2af428dd90c4ef4e90bd2ea249ed.webp)

02
Admission control
Where does the decision to say no live?

COVERS
Where the decision sits, what it returns to the client, and what it needs to see.

18 / 66

Interface text:
Parv Khatri

### Three moments to decide, only one is in front of the engine

At ingress, a gateway can refuse a request before the engine accepts it. The scheduler sees more as it picks the next batch or reclaims KV cache during decode. Changing either decision means maintaining an engine fork.

[17:19](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1039s) [Arrows connect three panels. At ingress, a proxy or gateway can refuse a request. At the queue boundary, the engine scheduler picks the next batch. During decode, it can preempt work or reclaim KV cache.](https://assets.ai.engineer/sites/aiecode2025/sha256/d1ac10f3fac2c31bb650015bceb7af108fe31f8a50914c3ce6c2d2eb06a4fead.webp)

02 · ADMISSION CONTROL · WHEN TO DECIDE

Three moments to decide, only one is in front of the engine

Ingress
The admission layer: refuse before the request is accepted
READS
Request features and exposed engine metrics
NEEDS
A proxy or gateway. No engine change.

Queue boundary
Inside the engine: the scheduler picks the next batch
READS
Also the current batch state
NEEDS
Scheduler changes. A maintained fork.

During decode
Inside the engine: the scheduler preempts or reclaims KV
READS
Also decode progress
NEEDS
Scheduler changes. A maintained fork.

Later moments see more, but they are engine scheduling, not a separate layer. Ingress is the only one that needs no engine fork.

20 / 66

### The policy reads the request and the engine together

The gateway sees the request’s input length and any deadline. The engines see KV cache use and how long the oldest request has waited. The admission policy uses both views to admit, defer, or reject the request.

[19:35](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1175s) [Arrows point into a policy that can admit, defer, or reject. Request inputs are input length, client or tenant, agent session, and an optional deadline. Engine inputs are KV cache use, queue age, cache hit rate, and load per pool.](https://assets.ai.engineer/sites/aiecode2025/sha256/99f12c76d7dd76cd5dbbeb70c5b040c772be9c7807eff3b049750485160e4696.webp)

02 · ADMISSION CONTROL · WHAT IT MUST SEE
The policy reads the request and the engine together

From the request    gateway
• Input length
• Client or tenant
• Agent session
• Deadline, if it carries one

Policy
Admit
Defer
Reject

From the engines    engine
• KV cache use
• Queue age
• Cache hit rate
• Load per pool

The gateway has one view, the engine the other. Admission needs both.
22 / 66

Interface text: [illegible]

### Defer returns 503 with a Retry-After

During a burst, the engine estimates when a running batch will free enough KV cache for the request. Here the server returns 503 and tells the client to retry in two seconds. If capacity won’t return soon, the server rejects the request.

[23:59](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1439s) [An HTTP 503 Service Unavailable response specifies a two-second retry delay. The policy panel explains that a running batch is expected to free enough KV cache soon and that queue age is rising during a burst.](https://assets.ai.engineer/sites/aiecode2025/sha256/4c3bf29c39f8c3e7f23b8ebf385d8bcd319287ee48a54cd78bd8ac2b9eaef736.webp)

02 · ADMISSION CONTROL · DEFER
Defer returns 503 with a Retry-After

The response
HTTP

HTTP/1.1 503 Service Unavailable
Retry-After: 2
{"error":"at_capacity"}

When to defer
the policy

CAPACITY RETURNS SOON
A running batch will free enough KV within τ.

A BURST, NOT A TREND
Queue age is rising but not saturated.

WHERE T COMES FROM
The engine: time until enough memory frees.

Defer is for load that will fit shortly. If capacity is not coming back, reject.
26 / 66

Interface text:
26 / 66  [illegible]

### Handling a defer: wait the time, plus jitter

Without Retry-After, all ten agents retry together. When the server asks them to wait two seconds, each agent adds a small random delay to that wait. The server then receives the ten retries spread across time.

[24:53](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1493s) [Client pseudocode waits for the Retry-After delay plus jitter before retrying. The diagram shows ten retries clustered together in the upper panel and spread across time in the lower panel.](https://assets.ai.engineer/sites/aiecode2025/sha256/8817144544783d4554adcb80850a130da334708a6a1ef87941a8e30b0a7b554c.webp)

Handling a defer: wait the time, plus jitter
02 · ADMISSION CONTROL · HANDLING A DEFER

The response
server and agent

HTTP/1.1 503 Service Unavailable
Retry-After: 2
# agent
sleep(retry_after + jitter())
retry(call)

Ten retries
what the server sees

Without Retry-After
all ten retry together

With Retry-After and jitter
spread across the window
Timeline: now, +2 s

Without a time, every agent picks its own delay, often zero. Defer turns a retry storm into a schedule you control.

27 / 66

### Reject returns 429 or 529

A 429 tells the caller to reduce its own load because its tenant or session is using more than its share. A 529 means every replica is full, so the provider is rejecting requests from all callers.

[25:38](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1538s) [The 429 card describes a client exceeding its share, with per-client load, concurrent calls, and quota as signals. The 529 card describes saturation across all replicas, with pool-level KV use and queue age as signals.](https://assets.ai.engineer/sites/aiecode2025/sha256/13b03008795774acc4cbf7f0e6bf73dca0353f342cb9765c7bf9fe3096109c72.webp)

Reject returns 429 or 529
02 · ADMISSION CONTROL · REJECT

429    client over its share
Too many requests
MEANS
This tenant or session is using more than its share.
POLICY READS
Per-client load, concurrent calls, quota.

529    pool full
Provider overloaded
MEANS
Every replica is saturated. Nobody gets in.
POLICY READS
Pool-level KV use, queue age, no relief in sight.

A reject says why. The reason decides how the client should react.
28 / 66

### Strategies

The self-hosted LLM stack lacks a layer that checks capacity before accepting a request. Harshul Jain introduces admission control strategies for deciding when to turn a request away.

[27:30](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1650s) [Section 03 asks how a system can say no and on what basis. A recap identifies the need for a layer that checks available capacity. A presenter tile appears on the right.](https://assets.ai.engineer/sites/aiecode2025/sha256/eef1990d8eb2e0bfce54fdc56bba2eba020cfe59cb2d4ec21ccb728a39704506.webp)

03
Strategies
How can a system say no, and based on what?

SO FAR
No layer in today's stack decides whether there is capacity. That layer is ours to add.

9 min
30 / 66

Interface text:
harshul jain

### How to build a policy: four choices

First choose what to admit. A rule then uses a load signal to decide whether to admit the work, defer it with Retry-After, or reject it with 429. The unit constrains the signal: deadline slack only works for requests that carry a deadline.

[29:52](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1792s) [Four cards show how to choose the unit to admit, the load signal to watch, the rule that makes the decision, and the action the client sees. Actions include admitting, deferring with Retry-After, and rejecting with 429.](https://assets.ai.engineer/sites/aiecode2025/sha256/751ade166c858e7f555f26ce72af72bd6310121b7ed895f608dd1d89e91ae54e.webp)

How to build a policy: four choices
03 · STRATEGIES · BUILDING A POLICY

1 Unit
Request
Agent session
Tenant

WHAT GETS ADMITTED
One call, or the whole task

2 Signal
KV pressure
Queue depth
Deadline slack

WHAT IT WATCHES
Must move under load

3 Rule
Threshold
Dual gate
Optimizer
Formal bound

HOW IT DECIDES
Microseconds to seconds

4 Action
Admit
Defer with Retry-After
Reject with 429

WHAT THE CLIENT HEARS
A contract agents can follow

Each choice limits the next. Next: choice 1, what to admit.
32 / 66

Interface text:
32 / 66   [illegible]

### Admit single requests or whole sessions?

In this example, two agent tasks need 20 calls each. Admitting about half of each task’s calls leaves both waiting for the rest. Admitting whole sessions lets A finish and gives B one answer for all 20 calls: try again later.

[31:33](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1893s) [Orange squares mark admitted calls and gray squares mark deferred calls. In the per-request example, both sessions receive partial admission and neither finishes. With per-session admission, session A finishes and session B is deferred as a whole.](https://assets.ai.engineer/sites/aiecode2025/sha256/9330d4d3cc2452ee5529e702aad31d93674d5eb0de308fcdcab9aa6ae2ed97c9.webp)

Admit single requests or whole sessions?
03 · STRATEGIES · THE UNIT

Legend: Admitted, Deferred
ILLUSTRATIVE

Per request
About half of each task gets in. No task can finish.

Per session
Session A finishes. Session B gets one clear answer.

Session A · 20 calls
Session B · 20 calls

Coding agents send sessions. Next: choice 2, which signal to watch.
33 / 66

Interface text: 33 / 66 [illegible]

## Full transcript

[0:05](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=5s) Good morning everyone. My name is

[0:07](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=7s) Hershel Jan and I have here with me

[0:10](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=10s) Pearl. Today we are going to talk about

[0:13](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=13s) saying no. It's called as admission

[0:16](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=16s) control. How our system decides which

[0:20](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=20s) request to get in when your coding

[0:22](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=22s) agents overwhelm the LLMs you host

[0:25](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=25s) yourself.

[0:27](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=27s) Uh before we move on a quick intro about

[0:30](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=30s) us.

[0:32](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=32s) So I am a senior software engineer at

[0:34](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=34s) Audible where I lead some of the MLAI

[0:37](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=37s) infrastructure.

[0:38](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=38s) Uh I am also writing an open-source

[0:40](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=40s) handbook called as LLM inference at

[0:43](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=43s) scale. It has received over 300 plus

[0:45](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=45s) stars by now.

[0:47](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=47s) I am also maintaining an open-source

[0:49](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=49s) framework called agentship which helps

[0:51](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=51s) you ship reliable and secure agentic

[0:55](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=55s) harnesses in an ultra fast speed. Uh

[0:59](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=59s) perv would you like to introduce

[1:00](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=60s) yourself?

[1:01](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=61s) Sure. Hi everyone. I'm Perv. I'm a

[1:04](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=64s) senior software developer at SAP. I work

[1:06](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=66s) on like cloud application and LLM

[1:09](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=69s) infrastructure. I wrote an LLM fine-tune

[1:12](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=72s) guide. I contribute to agentship and I'm

[1:15](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=75s) like building a task tracker which is a

[1:18](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=78s) local post task board for you and your

[1:20](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=80s) agent.

[1:22](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=82s) Uh apart from this if you would like to

[1:24](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=84s) connect with us our LinkedIn QR codes

[1:27](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=87s) are right there on this slide. Now back

[1:29](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=89s) to you Hershel.

[1:32](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=92s) Great. So

[1:35](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=95s) here is the plan. We are going to start

[1:37](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=97s) with understanding the motivation why

[1:41](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=101s) any serving stack can admit so much of a

[1:45](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=105s) traffic.

[1:46](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=106s) Second, we try to dive into admission

[1:48](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=108s) control where the decision to say no

[1:51](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=111s) lives and what it needs to see. Third,

[1:55](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=115s) we will go through some of the proven

[1:57](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=117s) strategies, the different ways a system

[2:00](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=120s) can actually make the admission

[2:01](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=121s) decision. And fourth, we will try to

[2:05](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=125s) dive into the benchmarking where you

[2:07](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=127s) will learn how to prove that your policy

[2:11](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=131s) for admission is working or it's not

[2:13](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=133s) working.

[2:16](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=136s) So let's get started with the

[2:18](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=138s) motivation. Um

[2:21](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=141s) and for this

[2:26](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=146s) everything today uh let's try to

[2:28](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=148s) understand what is inference. Everything

[2:31](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=151s) today happens during this inference

[2:34](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=154s) itself. So inference is simply a moment

[2:37](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=157s) when someone sends a prompt to the model

[2:40](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=160s) and gets back an answer.

[2:43](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=163s) So the step one is like a user prompt

[2:45](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=165s) comes in. It's a sequence of words. Uh

[2:48](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=168s) for example, it could be like failing a

[2:51](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=171s) fix a failing test. The model which is

[2:55](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=175s) running on your GPUs writes the answer

[2:57](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=177s) one token at a time. A token is roughly

[3:00](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=180s) like a word as I mentioned. Uh it writes

[3:03](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=183s) the first word then uses that word to

[3:06](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=186s) pick the second uh then it uses both to

[3:10](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=190s) pick the third and it keeps on

[3:12](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=192s) continuing like that and it cannot write

[3:16](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=196s) all of them at once because every word

[3:18](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=198s) depends on the previous one. So those

[3:22](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=202s) once all the words are generated they

[3:24](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=204s) stream back to the user uh who ask for

[3:28](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=208s) the result and that volt prompt and the

[3:32](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=212s) answer out is like that one request. Now

[3:34](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=214s) I want you to hold on to one small

[3:37](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=217s) detail like model has no idea how long

[3:41](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=221s) this answer is going to be. It only

[3:44](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=224s) finds out whether it writes when it only

[3:49](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=229s) finds out when it writes the first last

[3:51](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=231s) token.

[3:53](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=233s) Keep that in mind. It will matter in the

[3:55](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=235s) few minutes.

[4:01](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=241s) So a model never sits on the internet

[4:05](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=245s) itself. Every request travels through a

[4:08](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=248s) short pipeline. It starts in your app or

[4:12](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=252s) your agent. Then it reaches your API

[4:15](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=255s) gateway

[4:16](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=256s) which checks your API key counts how

[4:18](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=258s) many requests you are sending. Then a

[4:21](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=261s) router picks a replica which which is

[4:23](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=263s) just one running copy of the model and

[4:26](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=266s) finally the requests land on the

[4:28](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=268s) inference engine. The common some of the

[4:31](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=271s) common inference engines are the VLMs,

[4:34](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=274s) SGLAN, Tensor RT etc. engine is what

[4:39](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=279s) actually hosts your model.

[4:42](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=282s) It actually serves your model uh and it

[4:45](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=285s) loads the model weights into your GPUs.

[4:48](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=288s) It groups many requests into one batch

[4:50](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=290s) so the GPU stays busy and for every

[4:53](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=293s) request it keeps something called as a

[4:56](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=296s) KV cache. In plain words, the KV cache

[4:58](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=298s) is the memory the request builds up when

[5:02](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=302s) it try when its answer is being written.

[5:05](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=305s) Longer the answer, bigger this memory

[5:07](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=307s) gets.

[5:08](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=308s) Keep this picture in your head. Every

[5:10](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=310s) box here makes some decision but one

[5:13](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=313s) decision is missing and we will find it

[5:16](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=316s) out in the few minutes.

[5:19](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=319s) First, how much an engine can actually

[5:22](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=322s) hold.

[5:26](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=326s) Every deployment has a ceiling and it

[5:28](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=328s) comes from the three targets that pull

[5:30](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=330s) against each other. Quality is how big

[5:33](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=333s) your and precise your model is. Latency

[5:37](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=337s) is how fast the first token arrives and

[5:40](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=340s) every token after it. Throughput is how

[5:43](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=343s) many requests your GPUs can serve at a

[5:45](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=345s) same time. You know the old saying good,

[5:49](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=349s) fast, cheap, but you have to pick two.

[5:51](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=351s) It's the same here. You cannot pick all

[5:53](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=353s) the three. You cannot prioritize all the

[5:55](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=355s) three. So you can't So every product has

[5:59](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=359s) to pick two. For an example like an

[6:01](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=361s) interactive chat, you would want to go

[6:04](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=364s) with the quality and the latency because

[6:06](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=366s) users are more sensitive to that. It's

[6:09](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=369s) quite obvious to the users if the

[6:11](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=371s) quality is down or the latency is high.

[6:14](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=374s) For something like an async agents, you

[6:16](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=376s) would want to go with a higher

[6:17](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=377s) throughput as well as quality. If

[6:20](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=380s) quality is less, you will have to redo

[6:23](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=383s) the task and redoing is very very

[6:25](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=385s) expensive because these are the

[6:27](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=387s) background and the asynchronous tasks.

[6:30](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=390s) And the high volume real-time systems

[6:32](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=392s) such as the autocompletes they would try

[6:35](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=395s) to favor latency as well as and the

[6:37](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=397s) throughput in favor of the quality

[6:39](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=399s) because you don't need an expensive

[6:41](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=401s) model for doing the autocompletes for

[6:43](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=403s) you. But whichever corner you give up

[6:46](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=406s) there is still a maximum load you can

[6:49](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=409s) carry while hitting your SLO

[6:52](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=412s) your service level objective. Let's say

[6:54](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=414s) that on this deployment it's eight

[6:57](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=417s) requests at once. In a normal minute,

[7:00](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=420s) you never notice that limit. Now,

[7:03](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=423s) picture the 40 requests arriving in the

[7:05](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=425s) same two seconds.

[7:08](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=428s) It's such a high load.

[7:15](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=435s) And this is not like hypothetical. It

[7:17](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=437s) has happened to like biggest providers

[7:19](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=439s) in the world. Here are the two public

[7:21](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=441s) postmortms two years apart.

[7:24](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=444s) On November 8, 2023, OpenAI after a day

[7:27](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=447s) of unusual heavy traffic, maybe not 40

[7:31](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=451s) requests, it would be some million

[7:33](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=453s) requests, their routing servers ran out

[7:36](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=456s) of the memory. They started failing

[7:37](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=457s) their health checks. They added more

[7:40](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=460s) replicas, then had to roll them back.

[7:42](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=462s) The way out was to cut incoming traffic

[7:45](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=465s) and ramp it up slowly. In January 21

[7:50](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=470s) 2026 enthropic the sonnet 4.5 tire gear

[7:55](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=475s) got slow first then it started returning

[7:59](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=479s) errors over about 90 minutes. The cause

[8:02](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=482s) they gave was simple not enough compute

[8:04](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=484s) for the demand.

[8:07](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=487s) And now let's look at what each team

[8:10](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=490s) decided as their next set of the action

[8:12](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=492s) item. Neither of them says we need a

[8:15](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=495s) faster engine.

[8:17](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=497s) Open AI's followup is graceful load

[8:20](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=500s) shedding. Enthropics is detecting

[8:24](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=504s) capacity limits earlier.

[8:26](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=506s) It's the same idea in the different

[8:29](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=509s) words which is decide on your request

[8:32](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=512s) sooner

[8:34](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=514s) to make sure which request gets in. And

[8:38](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=518s) there are more incidents which we have

[8:41](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=521s) linked in the appendix. If you are

[8:43](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=523s) interested definitely go and read

[8:44](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=524s) through them.

[8:49](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=529s) So

[8:52](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=532s) if you walk backward from those

[8:54](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=534s) follow-ups,

[8:55](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=535s) they both point at the same place, the

[8:57](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=537s) front door. Here is the pipeline again

[9:01](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=541s) with the missing step filled in.

[9:05](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=545s) We have a client, we have an admit

[9:08](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=548s) layer, the Q router, and then the

[9:10](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=550s) engine.

[9:12](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=552s) It's never the client talking directly

[9:14](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=554s) to your engine or client talking to your

[9:17](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=557s) queue directly. There is an admission

[9:20](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=560s) phase in between.

[9:22](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=562s) Admission is a decision made before a

[9:25](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=565s) request takes any GPU memory and it only

[9:28](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=568s) has three possible answers.

[9:33](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=573s) Admission decides

[9:36](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=576s) whether to start now or you want to

[9:42](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=582s) hold the request. You want to accept it

[9:44](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=584s) but make it wait. Third is the reject.

[9:47](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=587s) That means not right now. It can throw

[9:50](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=590s) different amount of errors here such as

[9:52](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=592s) 429s and we will dive into this bit more

[9:56](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=596s) deeper in the second part.

[10:00](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=600s) this that is the decision that was

[10:01](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=601s) missing from our pipeline.

[10:06](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=606s) So why does it have to happen before the

[10:09](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=609s) engine

[10:11](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=611s) and what would happen if you will

[10:13](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=613s) actually accept the request what would

[10:16](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=616s) be the cost of that yes

[10:19](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=619s) so let's try to take a look at this like

[10:22](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=622s) two phases of the inference like prefill

[10:24](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=624s) and the decode. So first comes the

[10:26](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=626s) prefill. That's the engine reading the W

[10:29](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=629s) prompt, building the request KV cache.

[10:31](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=631s) And then comes the decode. That's the

[10:33](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=633s) engine writing the answer one token at a

[10:35](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=635s) time. With every token, the cash grows a

[10:38](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=638s) little more. All of that memory stays

[10:40](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=640s) reserved until the request finishes.

[10:45](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=645s) So a yes is not a moment. It's a

[10:48](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=648s) commitment that lasts the request whole

[10:50](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=650s) life. And here you can look at the

[10:53](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=653s) example.

[10:55](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=655s) There is a request A, there is a request

[10:57](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=657s) B. Once you admit a request A which is a

[11:00](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=660s) short answer request and then you admit

[11:02](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=662s) a request B which is a long answer

[11:04](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=664s) request. Now when you admit a request C

[11:06](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=666s) which is again a short answer request,

[11:09](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=669s) we don't know how longer it would take

[11:11](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=671s) because there is a longer request which

[11:13](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=673s) was already in the memory. Should we

[11:16](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=676s) even take that request C or should we

[11:18](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=678s) not even take the request C. So there is

[11:20](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=680s) a cost associated of like admitting

[11:23](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=683s) every request because admission could

[11:26](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=686s) still result in missing out your SLOs's.

[11:34](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=694s) Another impact apart from the SLOs's is

[11:37](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=697s) your memory.

[11:40](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=700s) Here is what it does to your GPU. So

[11:43](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=703s) every running request it keeps growing

[11:45](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=705s) your cache as it writes. So your total

[11:47](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=707s) memory keeps climbing. When the memory

[11:49](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=709s) gets to the full engine has three ways

[11:51](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=711s) to cut. Either it will aait

[11:54](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=714s) or like throw away your request. Either

[11:57](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=717s) it will put it to your waiting state

[11:59](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=719s) that means it will swap or it will just

[12:03](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=723s) say okay I won't be able to serve and I

[12:05](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=725s) I will probably return you an error.

[12:09](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=729s) Every one of those throws I mean every

[12:12](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=732s) one of those causes a certain delay in

[12:16](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=736s) processing of the request. and missing

[12:19](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=739s) out your SLOs's again while also wasting

[12:21](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=741s) your memory.

[12:30](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=750s) So why is it like our problem like we

[12:33](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=753s) have con servers from open AI anthropic

[12:37](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=757s) bold teams which are serving their own

[12:39](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=759s) infrastructure

[12:41](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=761s) for the inference.

[12:44](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=764s) So it's a problem for us because

[12:48](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=768s) there are teams who want privacy. They

[12:51](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=771s) want to have their own models hosted on

[12:54](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=774s) their own GPUs

[12:57](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=777s) and they want it to be hosted on their

[12:59](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=779s) own GPUs due to the cost perspectives

[13:02](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=782s) but they have to also protect

[13:06](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=786s) the burndown on those GPUs.

[13:08](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=788s) So they need to effectively decide which

[13:11](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=791s) request to serve across their inference

[13:14](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=794s) ecosystem.

[13:22](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=802s) On top of

[13:24](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=804s) and on top of that like the traffic

[13:26](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=806s) arriving at the door has also changed.

[13:30](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=810s) Earlier there used to be a person that

[13:32](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=812s) was chatting. He would send one request

[13:34](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=814s) at a time. There would be a seconds of

[13:36](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=816s) reading in between and then there would

[13:38](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=818s) be a slow answer. But now coding agents

[13:40](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=820s) they have taken over. There are tons of

[13:42](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=822s) calls that these coding agents they send

[13:45](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=825s) along with the retries, pauses,

[13:49](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=829s) um

[13:51](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=831s) like the undo calls as well and it there

[13:54](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=834s) is such a overwhelming traffic that your

[13:57](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=837s) cluster has to handle and what that

[14:00](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=840s) means is cluster has a responsibility to

[14:03](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=843s) take a wise decision of which requests

[14:05](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=845s) to accept and which uh requests to

[14:08](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=848s) reject.

[14:15](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=855s) And this is exactly that visualization.

[14:17](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=857s) You ask your agent that fix an o test.

[14:21](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=861s) It runs all your it tries to run

[14:24](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=864s) multiple tool calls that is reading of

[14:27](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=867s) the files like around there could be

[14:29](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=869s) like 14 files. There could be like

[14:32](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=872s) around 10 tests that it has to run and

[14:35](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=875s) it would combine together to make 40 50

[14:38](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=878s) calls for just one request and all these

[14:42](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=882s) 40 requests they require like reaching

[14:44](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=884s) out to the LLM getting back the

[14:46](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=886s) response. Uh and in between some of

[14:50](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=890s) these requests would be long running

[14:52](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=892s) some of the requests would be short

[14:55](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=895s) prompt request. So we need to decide

[14:57](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=897s) effectively where which one to pick and

[15:00](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=900s) which one not to pick.

[15:09](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=909s) Yeah. And that was the like reason why

[15:12](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=912s) we need to worry about

[15:15](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=915s) admission control. Why we need to worry

[15:18](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=918s) about

[15:21](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=921s) handling the traffic right way across

[15:23](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=923s) our infancy ecosystem.

[15:32](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=932s) Okay. So this brings us to the admission

[15:36](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=936s) control.

[15:37](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=937s) Uh now moving on to the next slide. Uh

[15:42](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=942s) that is slide number 18. So in this

[15:45](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=945s) slide we will be covering three

[15:47](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=947s) questions. Where does this decision

[15:50](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=950s) from? which request to admit

[15:52](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=952s) and which to reject sits back at your

[15:55](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=955s) stack

[15:57](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=957s) and the another question is what does it

[16:00](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=960s) need to see to make a good call or

[16:03](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=963s) judgment on the request and what does it

[16:07](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=967s) returns back to the client. Moving on to

[16:10](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=970s) the next slide. So here uh we see the

[16:15](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=975s) pipeline again which each layer's jobs

[16:17](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=977s) is written out. The agent sees its own

[16:21](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=981s) call and decides when to retry.

[16:25](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=985s) The gateway sees API key and request

[16:28](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=988s) count and enforces a rate limit rate

[16:31](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=991s) limiter for each and the router which

[16:35](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=995s) sees replicas or healthy and which one

[16:37](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=997s) to pick. Now the engine sees its KV

[16:41](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1001s) cache and the current batch and decides

[16:44](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1004s) the order of request to run and what to

[16:47](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1007s) evict and print.

[16:50](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1010s) So every layer decides something. Now

[16:54](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1014s) ask each of them one question. Is the

[16:57](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1017s) room for this request?

[17:00](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1020s) The agent says not my job. The gateway

[17:03](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1023s) says I cannot see it. And the router say

[17:07](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1027s) not my job and the engine says I only

[17:09](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1029s) find out when I have already said yes.

[17:13](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1033s) The admission box is missing in in

[17:17](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1037s) between.

[17:20](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1040s) So now moving on to the next slide. Um

[17:23](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1043s) so where to put this decision change

[17:28](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1048s) what it can see and what it cost you to

[17:31](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1051s) run. So there are three possible

[17:34](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1054s) moments. The first is an invoice right

[17:38](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1058s) in front of the engine. Here the policy

[17:41](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1061s) sees the request itself plus whatever

[17:43](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1063s) metrics the engine exposes. All you need

[17:47](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1067s) is proxy or gateway. The engine stays

[17:50](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1070s) exactly as it is. The second one is the

[17:53](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1073s) Q boundary. Now the decision happens

[17:56](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1076s) inside the engineuler

[17:58](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1078s) as it picks the next batch to run.

[18:03](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1083s) The third is during decoding when the

[18:06](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1086s) scheduleuler can pause a request or take

[18:08](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1088s) back its cache halfway through an

[18:11](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1091s) answer. Those later movement can see

[18:14](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1094s) more but they mean changing the

[18:16](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1096s) engineuler.

[18:18](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1098s) That means keeping your own modified

[18:20](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1100s) copy of BLLM or SG lang and updating it

[18:24](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1104s) every time original changes forever. So

[18:28](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1108s) an ingress is the only movement that

[18:30](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1110s) works with the engine exactly as it

[18:33](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1113s) ships.

[18:34](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1114s) So that's where we will stay for the

[18:36](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1116s) rest of the talk. So now moving on to

[18:39](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1119s) the next slide.

[18:42](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1122s) Uh so what actually sits in ingress,

[18:46](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1126s) right? Uh so you you you can think of it

[18:49](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1129s) as a function. It can take in what it

[18:52](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1132s) knows about the request and current

[18:54](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1134s) state of the system and it returns one

[18:58](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1138s) of the three answers

[19:01](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1141s) like if we have to admit the request. So

[19:04](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1144s) the admit is the request enters the

[19:07](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1147s) engine now and memory is committed for

[19:09](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1149s) this moment. If it's deferred with a

[19:13](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1153s) time we can call toao hold on like uh

[19:16](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1156s) hold on and come back after some time

[19:19](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1159s) and there is another decision which is

[19:22](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1162s) with uh reject with a code that no here

[19:26](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1166s) is the reason why and until the policy

[19:29](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1169s) says admit the request cost you nothing.

[19:32](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1172s) It holds no GPU memory at all.

[19:37](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1177s) Now moving on to the next slide.

[19:43](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1183s) uh to make the call well the policy

[19:46](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1186s) needs two views at the same time from

[19:49](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1189s) the request and from the engine. So from

[19:53](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1193s) the request how long the input is, which

[19:56](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1196s) client or tenant sends it, which agent

[19:59](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1199s) session it belongs to, and a deadline.

[20:03](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1203s) And from the engine point of view, how

[20:06](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1206s) long uh KV cache is, how long the oldest

[20:10](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1210s) request has been waiting in the queue,

[20:12](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1212s) how often the prompt is already cached,

[20:15](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1215s) and how how loaded each pool of replica

[20:19](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1219s) is. Now look back at the ST. The gateway

[20:22](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1222s) has left side and the engine has the

[20:26](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1226s) right side. Nobody has the both.

[20:30](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1230s) And that's really why admission has to

[20:33](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1233s) be its own layer. Now moving on to the

[20:36](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1236s) next slide.

[20:39](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1239s) Here are those eight signals turned into

[20:41](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1241s) simple rule. Each one is compared to

[20:44](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1244s) threshold and when it crosses it pushes

[20:47](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1247s) towards the outcome in the colored pale

[20:51](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1251s) from the request side.

[20:53](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1253s) If the input is too big to fit in the

[20:56](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1256s) memory headroom just refer it. If a

[20:59](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1259s) client or tenant is over

[21:02](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1262s) it share reject it with 429.

[21:06](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1266s) If one agent session has too many calls

[21:09](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1269s) in flight defer with extras. If waiting

[21:13](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1273s) would make a request miss its deadline

[21:15](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1275s) anyway reject it from the engine side.

[21:19](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1279s) Uh you can see at the bottom half of

[21:23](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1283s) bottom half side. While KB cache is

[21:26](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1286s) under high water mark admit it. When the

[21:31](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1291s) oldest request has waited longer than

[21:33](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1293s) your target defer it. If the prompt is

[21:36](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1296s) already cached admit it first because

[21:39](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1299s) it's cheap to serve. And when every pool

[21:41](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1301s) is full, reject it with 529. One rule of

[21:45](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1305s) thumb, pick the signal that changes in

[21:48](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1308s) second, like a QH, not a one minute

[21:52](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1312s) average. A coding agent sends 40 calls

[21:55](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1315s) in 2 seconds.

[21:57](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1317s) A minute average won't even notice. So

[22:00](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1320s) that's what the policy reads. Now what

[22:03](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1323s) does it say back? Now moving on to the

[22:06](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1326s) next slide.

[22:08](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1328s) We will take the three answers one at a

[22:11](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1331s) time and look at each from both sides.

[22:15](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1335s) What the server sends and what your

[22:17](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1337s) agent should do with it. First, yes, the

[22:21](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1341s) server sends back a 200 and the response

[22:24](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1344s) starts streaming. At that moment, three

[22:26](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1346s) things happens inside the engine. It

[22:29](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1349s) commits KV cache from the whole prompt

[22:32](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1352s) and the request joins the running batch

[22:34](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1354s) and decode one token per step alongside

[22:37](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1357s) everyone else

[22:40](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1360s) and it holds that memory until it

[22:42](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1362s) finishes or until the client

[22:44](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1364s) disconnects.

[22:46](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1366s) That's the commitment from part one just

[22:49](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1369s) in the form of an HTTP response which is

[22:52](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1372s) why the admission decision has to happen

[22:54](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1374s) before 200 never after it.

[22:59](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1379s) Now the client side if we are writing

[23:02](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1382s) the agent here is how to handle 200.

[23:06](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1386s) Watch the stream not the status code.

[23:10](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1390s) First time is properly start your

[23:13](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1393s) timeout clock when the first token

[23:15](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1395s) arrives not when you send the request

[23:18](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1398s) under the heavy load prefill alone can

[23:21](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1401s) take few seconds. Second, if you decide

[23:24](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1404s) to give up, actually close the

[23:26](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1406s) connection. The server only frees the KV

[23:29](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1409s) cache when the request finishes or the

[23:31](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1411s) client disconnects. An agent that times

[23:34](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1414s) out but leaves the connection open is

[23:36](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1416s) still holding the GPU memory. And the

[23:39](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1419s) third one, uh, a stream can fail halfway

[23:43](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1423s) through, right? And that that's no

[23:46](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1426s) status for that, right? and you're

[23:49](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1429s) you're left with partial answer and you

[23:51](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1431s) have to decide whether to retry or carry

[23:53](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1433s) on. In short, 200 promise to start. It's

[23:58](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1438s) not a promise to finish. Right?

[24:03](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1443s) The second answer is dipper which

[24:05](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1445s) basically means not yet. The server

[24:07](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1447s) sends back 503 with retry header.

[24:11](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1451s) The header tells the client how many

[24:13](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1453s) seconds to wait. The policy picks that

[24:15](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1455s) number from the engine. For example, how

[24:18](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1458s) long until the running batch finishes

[24:20](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1460s) and freeze the enough cash

[24:22](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1462s) and you differ when capacity is coming

[24:25](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1465s) back soon? It's a burst, not so not a

[24:27](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1467s) trend, right? And for example, like if

[24:30](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1470s) you think about it like Q age is rising

[24:34](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1474s) but the system isn't saturated.

[24:39](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1479s) If the capacity isn't coming back

[24:41](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1481s) anytime soon, a dipper just kicks kicks

[24:44](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1484s) the can down the road, right? And in

[24:48](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1488s) that case, we can just reject it with

[24:50](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1490s) the honest answer.

[24:53](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1493s) Moving on to the next slide. On the

[24:55](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1495s) agent side, handling a defer is simple.

[24:59](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1499s) Sleep for retry after time and a little

[25:02](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1502s) random jitter, then retry again. The

[25:05](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1505s) jitter just means a small random extra

[25:07](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1507s) delay. So client don't like don't call

[25:11](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1511s) or like don't at all come back at the

[25:14](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1514s) exact same moment. They will have to

[25:16](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1516s) wait.

[25:19](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1519s) Look what the server sees from 10

[25:21](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1521s) retries without retry after every agent

[25:25](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1525s) picks its own delay. And that's usually

[25:29](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1529s) that's usually zero. So they all come

[25:32](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1532s) back at once. That's how 40 calls turns

[25:35](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1535s) into 80.

[25:38](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1538s) With retry after and jitter the retry

[25:41](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1541s) spreads out across the window serves

[25:43](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1543s) asked for.

[25:50](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1550s) Okay. So if if we see the uh reject

[25:55](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1555s) reject returns 429 or 529 the third

[25:59](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1559s) answer is no.

[26:01](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1561s) Uh and there there are two different

[26:03](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1563s) codes for it because there are two

[26:05](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1565s) different reasons. A 429s means too many

[26:08](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1568s) requests.

[26:10](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1570s) This particular client or agent session

[26:13](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1573s) is using more than its share. The policy

[26:16](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1576s) saw something like per client load, the

[26:18](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1578s) number of concurrent calls or a kota.

[26:21](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1581s) The problem is the caller and then the

[26:24](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1584s) 529 means the provider is overloaded.

[26:28](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1588s) Enthropic uses this code. Every replica

[26:31](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1591s) is sold. So nobody is getting in right

[26:34](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1594s) now.

[26:36](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1596s) The problem is pool not the caller.

[26:40](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1600s) Now uh handling a reject. So how should

[26:44](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1604s) agent react on 429? Reduce your own

[26:48](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1608s) load. Cap how many calls that agent has

[26:51](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1611s) in flight at once.

[26:54](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1614s) And on 529 retrying harder only makes

[26:57](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1617s) this worse. Waiting longer switching to

[27:00](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1620s) different model provider or park the

[27:02](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1622s) task for later. and one either and on

[27:07](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1627s) either one keep a retry budget. Use

[27:09](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1629s) expansion exponential back off with

[27:12](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1632s) jitter which means waiting longer after

[27:15](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1635s) each failed try and after few tries tell

[27:19](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1639s) the user what's going on because a

[27:21](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1641s) reject that gets retrieded at full speed

[27:24](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1644s) is how you end up in one of the postmodm

[27:26](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1646s) from the talk at we described at the

[27:29](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1649s) starting.

[27:33](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1653s) Now over to you Hershel

[27:40](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1660s) three.

[27:41](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1661s) Thank you P. Uh so this is the module

[27:44](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1664s) three where we will try to learn about

[27:46](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1666s) the different strategies uh that have

[27:48](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1668s) been rolled out for the admission

[27:50](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1670s) control. So far we have seen that no

[27:52](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1672s) layer in the today's stack asks where uh

[27:55](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1675s) whether there is a room and that layer

[27:58](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1678s) is ours to add and now we give it a

[28:01](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1681s) brain. So we will look at why counting

[28:03](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1683s) requests isn't enough. Breaking a policy

[28:06](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1686s) down into four different choices. See

[28:08](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1688s) what four different research paper chose

[28:12](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1692s) and finish with a decision tree that you

[28:15](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1695s) can use onto your own ecosystem.

[28:20](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1700s) So let's start with what most team

[28:22](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1702s) already have which is counting. Yes.

[28:26](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1706s) Counting the number of the requests.

[28:30](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1710s) A simple idea is to rate limit at the

[28:32](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1712s) gateway which gives API key a fixed

[28:35](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1715s) allowance set in advance. It's great for

[28:38](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1718s) keeping the clients fair, keeping your

[28:40](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1720s) tenants fair with each other,

[28:43](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1723s) but it does nothing to protect your GPUs

[28:47](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1727s) because it doesn't change with the load

[28:50](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1730s) that is happening onto your GPUs or your

[28:52](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1732s) cluster. A fixed cap at the proxy gets

[28:57](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1737s) closer. It counts how many requests are

[29:00](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1740s) in flight on each replica. The problem

[29:03](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1743s) is that the quick crep and a long

[29:05](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1745s) planning to call

[29:08](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1748s) count exactly the same while the real

[29:10](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1750s) capacity of the GPU changes with every

[29:12](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1752s) batch. So in practice it mostly cues and

[29:17](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1757s) clients timeout ends up doing the

[29:20](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1760s) refusing.

[29:22](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1762s) Neither one knows whether our 40 calls

[29:25](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1765s) are cheap or expensive.

[29:27](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1767s) The cost of air policy does though and

[29:31](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1771s) it estimates each request will actually

[29:35](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1775s) cost what each request will cost in the

[29:39](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1779s) KV memory and the tokens using the live

[29:42](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1782s) state from your engine. And it says no

[29:45](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1785s) before the request actually takes up the

[29:47](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1787s) memory. So the short version is count is

[29:50](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1790s) not a cost.

[29:55](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1795s) How do you build your own policy? So

[29:57](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1797s) there are four choices here. The first

[30:00](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1800s) is the unit

[30:02](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1802s) which tells you what you are actually

[30:04](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1804s) admitting. Are you admitting a single

[30:06](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1806s) request? Are you admitting an agent

[30:09](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1809s) session or are you admitting a tenant?

[30:13](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1813s) The two second is the signal

[30:17](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1817s) which means what is that the policy is

[30:20](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1820s) watching for taking an admission

[30:22](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1822s) decision. This could be as simple as

[30:24](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1824s) your memory pressure which is your KB

[30:27](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1827s) pressure. It could be your Q depth. It

[30:29](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1829s) could be your deadline slack. That means

[30:32](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1832s) whether your request is able to meet

[30:34](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1834s) your SLOs's or not.

[30:37](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1837s) And whatever it is, it has to actually

[30:39](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1839s) move when the system gets busy. So three

[30:43](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1843s) is the rule which means how does the

[30:46](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1846s) signal gets turned into a decision.

[30:50](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1850s) So rule is used for that and that can be

[30:54](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1854s) anything from a simple threshold that

[30:56](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1856s) takes microscond to a full optimizer

[30:58](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1858s) that takes seconds.

[31:01](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1861s) Fourth is the action. Once you have the

[31:04](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1864s) rule that gives you a decision what do

[31:06](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1866s) you do with it?

[31:09](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1869s) What does your how does your client gets

[31:11](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1871s) impacted through those actions?

[31:14](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1874s) That's the admit defer and the reject

[31:17](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1877s) that per just talked about in the module

[31:20](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1880s) two. Each choice limits the next one.

[31:24](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1884s) For example, a deadline signal only

[31:26](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1886s) works if your request actually carry the

[31:28](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1888s) deadlines. So let's go through the first

[31:31](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1891s) three.

[31:37](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1897s) Choice one. Do you admit a single

[31:39](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1899s) request or do you admit a poll sessions?

[31:44](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1904s) Say you have two agent task 20 calls

[31:46](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1906s) each room for almost half of them.

[31:50](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1910s) You can uh point you can see the top

[31:54](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1914s) row. So if you admit per request about

[31:57](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1917s) half of each task gets in but neither

[32:00](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1920s) task can finish because each one is

[32:03](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1923s) stuck waiting on the calls that didn't

[32:05](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1925s) make it. You have used all the capacity

[32:08](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1928s) and completed nothing. If you admit per

[32:12](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1932s) session, session A gets all of its call

[32:15](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1935s) and finishes. Session B gets one clear

[32:17](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1937s) answer. Come back later. And this

[32:20](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1940s) picture is illustrative, but the logic

[32:22](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1942s) holds.

[32:24](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1944s) Our 40 calls are one session. Coding

[32:27](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1947s) agents send sessions and yet most

[32:30](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1950s) published policies admit only the single

[32:33](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1953s) requests.

[32:38](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1958s) So which metric do you watch for what

[32:40](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1960s) particular problem?

[32:44](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1964s) This is the

[32:46](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1966s) question that we are trying to answer.

[32:49](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1969s) It depends on the problem that you are

[32:51](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1971s) trying to prevent. If you are if you

[32:54](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1974s) want to keep the cash from overflowing,

[32:57](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1977s) you definitely have to watch your KV

[32:59](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1979s) pressure.

[33:02](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1982s) If you want to avoid your wasted

[33:04](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1984s) recomputee, you have to watch out

[33:07](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1987s) the KV used together with the cash hit

[33:10](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1990s) rate. A full cache is fine if it's being

[33:12](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1992s) reused.

[33:15](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1995s) If you want to keep your weights to be

[33:18](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1998s) shorter, you have to watch for your

[33:20](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2000s) queue depth and you have to take actions

[33:22](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2002s) based on that. If you want to meet your

[33:25](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2005s) latency targets, you need to watch your

[33:27](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2007s) deadline slack. Take actions based on

[33:30](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2010s) that. If you need to plan for your long

[33:32](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2012s) answers, you need to predict for the

[33:34](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2014s) output length.

[33:37](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2017s) And look at the last column

[33:40](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2020s) fails when it tells you when each signal

[33:44](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2024s) stops working. Look at the first row. KV

[33:48](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2028s) pressure fails when the cache never

[33:50](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2030s) fills up.

[33:52](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2032s) Remember that row? It comes back to bite

[33:54](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2034s) someone in the part four.

[34:01](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2041s) So what are the four ways to take that

[34:03](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2043s) decision?

[34:07](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2047s) That is how do you convert your signal

[34:09](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2049s) into a decision from the cheapest to the

[34:12](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2052s) most expensive. Here are the ways a

[34:16](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2056s) threshold just compares one number to

[34:18](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2058s) the one limit. Example is a policy named

[34:23](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2063s) clued weight that was uh created

[34:27](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2067s) somewhere in 2026.

[34:29](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2069s) It takes tens of microsconds per

[34:31](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2071s) decision.

[34:34](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2074s) Second is a dual gate which needs two

[34:37](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2077s) signals to agree before it acts. Example

[34:40](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2080s) is a paper called concour

[34:43](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2083s) uh which checks like KVUs and the cash

[34:45](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2085s) hits under 100 microsconds.

[34:49](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2089s) Third is an optimizer which solves like

[34:51](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2091s) one big planning problem across all the

[34:54](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2094s) cues at once. Example is a QLM.

[34:57](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2097s) It's thorough but it takes about 5

[34:59](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2099s) seconds per decision.

[35:02](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2102s) And the fourth is the formal bound uh

[35:05](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2105s) which does the math on the worst case.

[35:08](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2108s) Can this request still meets its

[35:10](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2110s) deadline?

[35:12](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2112s) The example is the Kronos on which uses

[35:16](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2116s) the like time to first token and it uh

[35:19](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2119s) provides you like a microcond level

[35:21](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2121s) latency.

[35:23](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2123s) And now hold that cost column up against

[35:26](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2126s) our agent. It sends 40 calls in 2

[35:28](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2128s) seconds. A 5-second decision shows up

[35:31](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2131s) after every party is over.

[35:38](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2138s) So what the four papers did

[35:42](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2142s) as the complete policy fluid weight

[35:44](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2144s) admits single request uses the threshold

[35:46](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2146s) on your KV use and reports 10 to 25%

[35:50](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2150s) improvement. Conqueror is the one that

[35:53](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2153s) admits your VO agents. It uses your dual

[35:56](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2156s) gate and reports four to 4x

[36:00](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2160s) uh the throughput of the SG lang but it

[36:02](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2162s) was tested offline.

[36:05](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2165s) Uh QLM admits the request plans across

[36:09](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2169s) the W feed and reports 40 to 90% gains

[36:13](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2173s) at about 5 seconds per decision. Kronos

[36:16](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2176s) admits requests against their deadline

[36:18](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2178s) slack with a formal bound and takes

[36:21](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2181s) deadline misses from 99% down to zero.

[36:25](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2185s) Definitely very amazing accomplishments.

[36:29](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2189s) So each purper used a different model

[36:32](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2192s) engine and a workload. So please don't

[36:34](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2194s) read this as a leaderboard.

[36:37](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2197s) The question that matters for this room

[36:39](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2199s) is does any of them fit a coding agent

[36:45](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2205s) and the answer is no. We placed every

[36:49](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2209s) paper on the two axises. One axis is

[36:52](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2212s) what it optimizes deadlines or the

[36:55](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2215s) throughputs. Other is what it admits

[36:57](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2217s) single request or the VO agent sessions.

[37:01](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2221s) A coding agent needs a policy that

[37:03](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2223s) admits bold sessions, works online with

[37:06](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2226s) live traffic and runs engine you haven't

[37:08](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2228s) fought

[37:11](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2231s) and nothing sits there cleanly. Conc

[37:13](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2233s) doesn't need a for but it was only

[37:16](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2236s) tested offline.

[37:18](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2238s) So in practice you will pick the closest

[37:20](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2240s) match and build just build your own. And

[37:23](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2243s) here is how you can choose which polic

[37:27](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2247s) like which kind of a policy you want to

[37:29](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2249s) try it out.

[37:31](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2251s) So here is that decision tree.

[37:34](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2254s) First

[37:36](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2256s) the question that you have to answer is

[37:38](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2258s) does your request carry any deadlines?

[37:40](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2260s) If yes, set up a some sort of a formal

[37:44](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2264s) bound on that deadline. This will take

[37:46](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2266s) some microsconds level of latency.

[37:50](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2270s) If there is no deadline requirement, the

[37:53](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2273s) second question that you can ask is does

[37:56](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2276s) the KV cache fill at the peak load? If

[38:00](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2280s) your KV cache fills at the peak load,

[38:05](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2285s) then

[38:07](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2287s) you would want something like a way to

[38:11](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2291s) detect the cache prefixes used. So you

[38:14](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2294s) can use some sort of a dual gate

[38:15](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2295s) mechanism or a KV threshold mechanism.

[38:19](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2299s) Um and then if your KV cache gets filled

[38:24](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2304s) at the peak load, it does not gets

[38:27](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2307s) filled at the peak load, then you what

[38:28](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2308s) you could do is you can set up a Q level

[38:30](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2310s) of threshold on it.

[38:35](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2315s) And that gets us to the benchmarking

[38:38](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2318s) like once you have a policy, how do you

[38:41](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2321s) benchmark?

[38:43](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2323s) So I will be handing it over to pop.

[38:50](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2330s) Thank you. So now moving on to part four

[38:53](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2333s) which is benchmarking. As we just saw no

[38:57](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2337s) publish policy fit a coding agent

[39:00](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2340s) cleanly. So you are going to adopt one

[39:02](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2342s) or you you will have to build your own

[39:05](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2345s) and then you have to prove if it's

[39:07](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2347s) actually working or not. So to now

[39:11](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2351s) moving on to the next slide

[39:14](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2354s) you would have to test your policies

[39:16](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2356s) right so for that you need four things

[39:19](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2359s) first is your cluster your policy and

[39:23](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2363s) trace and score. So let's dig deeper

[39:26](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2366s) into each. So your cluster of the first

[39:29](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2369s) measure it it is capacity with no

[39:33](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2373s) admission control at all. The load your

[39:35](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2375s) replica can sustain on their own. Every

[39:38](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2378s) load level at after this is described as

[39:42](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2382s) multiple of your number like multiple of

[39:45](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2385s) that number and uh about the policy u

[39:49](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2389s) which signals it reads what limit it

[39:52](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2392s) uses and what it does when it says no.

[39:56](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2396s) The third one is trace which is a record

[39:58](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2398s) of real traffic that you replay using

[40:02](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2402s) code coding agent traffic ramping from

[40:04](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2404s) half your capacity up to twice your

[40:08](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2408s) capacity.

[40:10](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2410s) Four is a score. Then they run the same

[40:13](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2413s) trace twice. Once with the policy and

[40:16](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2416s) once without and then you can compare

[40:19](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2419s) your result. The score is where the most

[40:22](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2422s) people go wrong. So let's look look at

[40:25](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2425s) that. Now moving on to the next slide

[40:28](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2428s) which is uh how how you score. The score

[40:32](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2432s) we recommend is called goodput under

[40:35](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2435s) admission. So you can see the formula on

[40:38](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2438s) the screen. The top of the fraction is

[40:41](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2441s) the request that met their SLO their

[40:44](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2444s) latency target. That means the answer

[40:47](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2447s) started on time and it never stalled for

[40:50](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2450s) too long between tokens.

[40:53](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2453s) The bottom of the fraction is every

[40:56](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2456s) request that arrived including the one

[40:59](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2459s) which got rejected. Why does the bottom

[41:02](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2462s) matter so much?

[41:05](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2465s) So try this. Uh refuse 95% of your

[41:09](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2469s) traffic and serve the remaining 5%

[41:11](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2471s) perfectly. If you only count what you

[41:15](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2475s) admitted, your SLO attainment is 100%.

[41:19](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2479s) Looks amazing, right? Just ship it.

[41:22](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2482s) But no like good put under admission

[41:25](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2485s) says 5%. Shedding looks three when you

[41:29](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2489s) only account the request you kept. It

[41:33](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2493s) isn't.

[41:35](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2495s) So moving on to slide 42.

[41:39](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2499s) The second thing to get right is the

[41:42](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2502s) workload. A benchmark only tell you

[41:44](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2504s) something if it looks like your real

[41:47](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2507s) traffic.

[41:49](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2509s) So replay agent traces with their fan

[41:52](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2512s) out retries session not uniform random

[41:56](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2516s) arrival.

[41:57](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2517s) Describe load as multiple of the

[42:00](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2520s) capacity you measured and sweep from

[42:03](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2523s) half of it to double it.

[42:08](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2528s) Set your guideline as multiple of your

[42:11](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2531s) latency with no load. And remember a

[42:14](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2534s) benchmark that never goes past capacity

[42:17](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2537s) never actually tests the policy because

[42:19](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2539s) the policy never has to say no. So you

[42:22](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2542s) have to make sure that you reach at that

[42:25](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2545s) point.

[42:27](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2547s) Now

[42:29](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2549s) move moving on to the seven uh seven

[42:32](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2552s) things every admission result should

[42:34](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2554s) disclose.

[42:36](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2556s) Signal liess is one of the seven things

[42:39](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2559s) every admission result should disclose.

[42:42](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2562s) Here is the full full list that you can

[42:45](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2565s) see on your screen.

[42:47](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2567s) The load as a multiple of measured

[42:50](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2570s) capacity

[42:51](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2571s) deadline sets relative to latency with

[42:54](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2574s) no load.

[42:56](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2576s) Signal liveless which is no paper

[42:59](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2579s) report.

[43:01](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2581s) How many times you repeated the run and

[43:04](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2584s) how much the result varied not one lucky

[43:07](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2587s) run. Where each parameters comes from,

[43:10](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2590s) turned on this setup, borrowed from

[43:12](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2592s) somewhere else or left at the default.

[43:16](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2596s) Whether the rejected request count in a

[43:19](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2599s) good put denominator and the full

[43:22](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2602s) configuration

[43:25](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2605s) and the full configuration is model,

[43:28](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2608s) engine, version, hardware, workload,

[43:31](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2611s) SLOs's and baseline. None of this ask

[43:35](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2615s) anyone to change how they evaluate. It

[43:38](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2618s) just ask them to say what they did. Use

[43:41](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2621s) it as a checklist when you read on on a

[43:44](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2624s) paper and when your benchmark and when

[43:47](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2627s) you benchmark your own policy. Doing all

[43:51](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2631s) seven by hand is painful. So we built a

[43:54](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2634s) tool that records all of this uh as as a

[43:58](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2638s) default.

[44:00](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2640s) So uh the tool is admin perf and it's it

[44:05](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2645s) it's two things in one repository. So in

[44:09](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2649s) production if you see it's a library you

[44:13](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2653s) put in front of VLLM or SG lang it

[44:16](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2656s) decides admit defer or reject for every

[44:20](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2660s) request that's ingress layer from part

[44:23](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2663s) two.

[44:26](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2666s) So if you also see we on the right side

[44:31](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2671s) uh it's uh in in a benchmark it's a

[44:34](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2674s) harness that runs any policy against a

[44:36](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2676s) real engine on a real GPU and reports

[44:40](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2680s) what the policy cost

[44:43](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2683s) both sides calls the same function

[44:45](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2685s) policy decide. So the policy you

[44:49](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2689s) benchmark is exactly the policy you

[44:51](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2691s) deploy.

[44:53](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2693s) We build it because across those 16

[44:56](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2696s) paper no two share the same baseline

[45:00](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2700s) engine workload or SLO admitf makes a

[45:04](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2704s) head-to-head comparison possible.

[45:07](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2707s) Now moving on to the next slide.

[45:11](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2711s) So using its pive command. So you can

[45:15](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2715s) see on the screen we have five commands.

[45:18](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2718s) So infra up starts a real VLM on a real

[45:22](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2722s) GPU. Infra smoke checks that it's

[45:25](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2725s) actually serving. Then we have bench

[45:27](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2727s) runs every policy several times each.

[45:31](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2731s) Then we have bench computer tell you

[45:34](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2734s) which policy won and how by how much.

[45:37](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2737s) And then we have infraown which which

[45:40](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2740s) basically means it should shuts all all

[45:43](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2743s) off so you stop paying. And that's my

[45:46](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2746s) favorite one.

[45:48](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2748s) Uh and so now provisioning and

[45:50](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2750s) benchmarking are separate commands on

[45:52](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2752s) purpose. Loading a model takes minutes

[45:55](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2755s) and every policy has to face exactly

[45:58](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2758s) same deployment.

[46:00](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2760s) If you are reprovisioned between policy,

[46:02](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2762s) you would change the very thing you are

[46:05](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2765s) trying to hold a constant.

[46:08](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2768s) The deployment is constant. The policy

[46:10](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2770s) is variable.

[46:15](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2775s) So now uh here a policy must declare the

[46:19](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2779s) signal it reads. This is how admit perf

[46:22](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2782s) deals with the trap I just showed you.

[46:25](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2785s) Every policy has to declare the signal

[46:27](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2787s) it needs. This KV cache threshold policy

[46:31](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2791s) says I require KV used correction which

[46:35](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2795s) is the share of the KV cache in use.

[46:40](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2800s) If the engine cannot report that signal,

[46:42](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2802s) the run stops at the startup with an

[46:45](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2805s) error. You can actually read without the

[46:48](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2808s) check a missing signal reads as a zero

[46:51](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2811s) and KV cache threshold that always sees

[46:54](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2814s) zero admits everything. It's quietly

[46:57](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2817s) turned into the no admit baseline while

[47:01](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2821s) your results looks completely normal. So

[47:04](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2824s) policy that cannot see its signal should

[47:07](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2827s) fail loudly and not just fail silently.

[47:14](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2834s) Um so here

[47:17](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2837s) what we did is every decision is

[47:20](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2840s) recorded with the state it saw. Every

[47:22](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2842s) run leaves behind a folder you can open

[47:25](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2845s) inspect. There would be four files as

[47:27](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2847s) you can see on the screen. Those are

[47:30](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2850s) manifest JSON summary JSON decision JSON

[47:33](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2853s) and uh out outcome JSON. So

[47:39](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2859s) what each does

[47:41](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2861s) is

[47:42](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2862s) uh you can take a look deeply once you

[47:46](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2866s) run the results. Now we we will be

[47:48](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2868s) looking at the shading cut. So shade

[47:51](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2871s) shading 37% cut tail latency. So here

[47:55](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2875s) are the four results how they will look

[47:57](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2877s) like on a real GPU. The setup small

[48:00](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2880s) model we used quen 2.5 uh and it's a

[48:04](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2884s) half billion

[48:06](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2886s) model on a A10 GPU. The engine was kept

[48:10](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2890s) out at four sequ sequences at a time.

[48:14](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2894s) Eight request at 14 per second two

[48:17](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2897s) repeats.

[48:20](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2900s) If you look at the first row with no

[48:23](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2903s) admission control, the P95 time to first

[48:27](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2907s) token was about two two and two and a

[48:29](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2909s) half around 2 and a half seconds and P

[48:32](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2912s) 45 sorry P95 means 95% of the requests

[48:37](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2917s) were faster than that and it swung by

[48:40](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2920s) about 700 milliseconds either way.

[48:44](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2924s) Now the last row if you see with a queue

[48:47](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2927s) up policy that allows at most two

[48:50](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2930s) request waiting 60 63% of requests were

[48:54](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2934s) admitted and the P95 dropped to about

[48:58](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2938s) around 1 second that's 2.5 times better

[49:03](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2943s) with the exact the good same put

[49:07](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2947s) now if we look at the signal matter more

[49:10](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2950s) than the threshold

[49:12](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2952s) but this This is the result I love most.

[49:15](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2955s) Same run. KV cache usage peaked at half

[49:18](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2958s) a percent. The queue got 24 requests

[49:21](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2961s) deep on a small model. KV cache is huge

[49:25](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2965s) compared to the four short sequences. So

[49:28](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2968s) it never comes close to filling. All of

[49:31](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2971s) the pressure showed up in the queue.

[49:33](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2973s) Instead, a KV cache pressure policy

[49:35](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2975s) would have botched a flat line and

[49:38](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2978s) admitted everything. that that's the row

[49:42](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2982s) of choice from two. So

[49:48](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2988s) now here and you you can try it today.

[49:50](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2990s) You don't need a GPU. There is a small

[49:53](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2993s) fake engine that speaks the three

[49:55](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2995s) endpoints admit puff uses and it gets

[49:58](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2998s) busy under load just like a real one. So

[50:02](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3002s) you can test the whole setup on your

[50:04](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3004s) laptop. If your right policy is your own

[50:06](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3006s) package,

[50:08](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3008s) admit perf discovers it automatically.

[50:11](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3011s) No change to the repo and contribute

[50:13](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3013s) contributions are very welcome. New

[50:16](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3016s) policies, workload, engine adapter or

[50:19](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3019s) matrices.

[50:25](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3025s) Okay.

[50:29](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3029s) So if you want to remember like one

[50:32](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3032s) slide just make it this one that

[50:37](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3037s) when you are accepting the request in

[50:40](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3040s) your inference clusters

[50:42](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3042s) you need to have a mechanism to say no

[50:45](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3045s) early something that works at the

[50:47](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3047s) ingress something before saying yes to

[50:51](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3051s) the request before that request goes

[50:54](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3054s) into the GPU before that request takes

[50:57](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3057s) up your memory.

[51:00](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3060s) and per added clearly. There are a

[51:03](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3063s) couple of error codes that your

[51:06](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3066s) admission policies return. You need to

[51:08](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3068s) handle those requests and those errors.

[51:15](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3075s) And with that we would like to come to

[51:18](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3078s) the closure. We would like to thank you

[51:20](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3080s) all for your patience, for your time.

[51:24](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3084s) And here are like two of our projects.

[51:28](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3088s) It all started as part of the LLM

[51:30](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3090s) inference at scale handbook. We and when

[51:33](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3093s) we were doing the survey, we found the

[51:35](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3095s) gaps and we prepared admit puff. So here

[51:39](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3099s) is the source code for it. Feel free to

[51:42](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3102s) go to it and like experiment with it.

[51:46](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3106s) Raise the issues, contribute to it. idea

[51:49](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3109s) is to

[51:52](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3112s) build your admission policies, benchmark

[51:54](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3114s) them and be confident that it uh helps

[51:58](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3118s) you keep your cluster stable. Yeah.

[52:01](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3121s) Thank you everyone.

[52:02](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3122s) Thank you everyone.
