OpenAI

The inference load balancer runs in front-end CPU clusters, which prepare incoming requests for inference engines usually hosted in GPU clusters. A model can have engines distributed across regions, providing resilience against localized failures. Selecting among them requires considering time to first token (TTFT), time between output tokens (TBOT), health, utilization, and network distance. Serving architecture and signals

Cache locality adds a consequence beyond distributing current work: a follow-up conversation turn can reuse context already held in an engine’s KV cache, avoiding recomputation. Moving related requests between engines can sacrifice that benefit, so an apparently balanced traffic allocation can still undermine serving efficiency. KV-cache reuse

Zhang describes an earlier design that filtered engines by eligibility—including capabilities, compute restrictions, and data residency—then selected among eligible destinations using weighted consistent hashing. A periodic controller smoothed engine signals into performance scores, compared those scores with the fleet average, and adjusted routing weights upward or downward. It borrowed the proportional component of PID control, rather than presenting a full PID controller. Eligibility, hashing, and weight generation

This approach combined multiple signals and helped redistribute less constrained requests when broadly eligible engines became busy. Its tradeoff was coupled behavior: explaining a particular weight or tuning one property without affecting others became difficult, especially across different GPU types. Removing traffic cooled an engine; the resulting improvement encouraged the controller to send traffic back. Repeated reversals produced oscillation that disrupted KV-cache utilization. Controller benefits and limitations Oscillation mechanism

Lao explains why neither round robin nor independent local choices provide sufficient coordination. Round robin ignores differences in capacity, hardware, health, and distance while scattering cache-related requests. Independently choosing a preferred engine from each CPU cluster can make several clusters converge on one destination, overloading it while other engines remain underused. Why local choices need coordination

The newer architecture gives a control plane a global view and lets each cluster’s data plane select engines from locally cached candidate lists and routing weights. Those snapshots refresh asynchronously, so request-time selection does not require a synchronous control-plane call. Live engine health and ready-replica signals also supply fast local guardrails. The control plane combines engine signals, network overhead, and offline regressions of capacity, TTFT, and TBOT to compute and publish subsequent weights. Control-plane and data-plane responsibilities

The distinction is between three paths: synchronous request-time selection and forwarding, asynchronous engine-signal collection, and asynchronous publication and pulling of routing weights. Global planning influences each request through installed local state, while immediate guardrails respond to changing engine conditions. Three system paths

Nearest-engine routing can fail when regional demand exceeds nearby GPU capacity. Sending some traffic farther adds network latency but can avoid a larger engine-side wait, provided the receiving engine has sufficient spare capacity. The recording’s illustrative example supports this comparison, without establishing a measured latency saving; its transcript and description disagree on the farther engine’s numerical capacity. Geographic demand and capacity

The optimizer takes demand from each CPU cluster, network latency to each engine, available engine capacity and health, and load-dependent TTFT and TBOT profiles. It outputs weights representing the fraction of each cluster’s traffic allocated to each engine. Its stated objective minimizes expected end-to-end latency across routed traffic, including both network and engine-side latency, subject to routing all demand, respecting effective capacity, and keeping weights non-negative. Optimizer inputs, objective, and constraints

The presentation makes these policy choices explicit but does not specify the solver or how cache affinity enters the newer optimization. Cache locality remains a stated routing concern; the described objective does not demonstrate a separate cache-affinity term. Scope of the described optimization

Zhang closes with controls addressing separate sources of production pressure. Outlier penalties reduce an anomalous engine’s routing weight, allowing transient recovery or operator intervention. Retry budgets cap additional attempts because failures under heavy utilization can trigger retries that increase load and cause still more failures. Those budgets must become more restrictive as utilization rises. Outlier penalties and retry storms

Load shedding is the last resort when serving capacity cannot meet demand: proactively rejecting a portion of traffic allows graceful degradation. This also clarifies the practical boundary of the optimizer’s requirement to route all demand within capacity. Routing redistributes available work, penalties reduce exposure to unhealthy engines, retry budgets constrain amplification, and shedding reduces the work the fleet attempts when total capacity is insufficient. Dynamic budgets and load shedding