← All speakers

Lu Zhang

Conference affiliation: Member of Technical Staff · OpenAI · 2026

Lu Zhang is an infrastructure engineer whose work spans GPU clusters and inference serving. He contributed to inference compute for GPT-4o, also credited in the model’s 2024 system card. Earlier in his career, he built and operated GPU clusters for machine-learning workloads before focusing on serving models: deciding where requests should run and how to use available hardware without wasting computation already performed.

On this page

In his joint presentation with Qianru Lao, Lu introduced both speakers as members of OpenAI’s inference team. He explained why routing inference requests involves more than finding an engine with spare capacity. KV-cache reuse lets an engine retain computation from earlier conversation turns, making follow-up requests cheaper and faster to process. Sending the next turn to a less busy engine can sacrifice that advantage. Routing also has to respect engine capabilities and data-residency restrictions, while accounting for performance and health across different hardware and regions.

When adaptive routing becomes unstable

Lu described an earlier routing design that used weighted consistent hashing to select an engine after filtering out ineligible destinations. A periodic controller combined engine performance signals into scores, compared them with the fleet average, and adjusted routing weights. Better-performing engines received more traffic; worse-performing engines received less. This feedback could accommodate changing conditions and uneven restrictions on which engines could serve particular requests, reducing the need for manual intervention.

The same adaptation could undermine cache reuse. Moving traffic away from an engine improved its performance, prompting the controller to send traffic back. Repeated adjustments could create routing oscillations, shifting requests between engines and disrupting the cached context that made subsequent turns efficient. Combining many signals into one weight also made routing decisions difficult to explain or tune: improving one aspect could change another, and different GPU types complicated comparisons.

Lu and Qianru described their team’s move toward more explicit routing policies. Qianru explained the newer architecture, which computes routing weights globally and refreshes them asynchronously so individual requests can use locally stored routing state. Lu’s portions examined the earlier design’s tradeoffs and the protections needed when production conditions deteriorate.

Preventing failures from creating more work

Lu explained three complementary safeguards:

  • Engine penalties. Reducing an unhealthy engine’s routing weight gives it room to recover from a transient problem or allows operators to replace faulty hardware.
  • Dynamic retry budgets. Retrying failed requests adds load. Near capacity, that extra work can cause further failures and more retries. The system therefore constrains retries according to utilization, allowing more under normal conditions and fewer when heavily loaded.
  • Load shedding. When demand exceeds available capacity, rejecting a portion of traffic helps preserve service for the remaining requests instead of allowing overload to spread through the system.

Read the topics behind these talks

1 conference talk

Loading more talks and highlights…

References