▶ Watch ↗18:12
Conference affiliation: Member of Technical Staff · OpenAI · 2026
Lu Zhang is an infrastructure engineer whose work spans GPU clusters and inference serving. He contributed to inference compute for GPT-4o, also credited in the model’s 2024 system card. Earlier in his career, he built and operated GPU clusters for machine-learning workloads before focusing on serving models: deciding where requests should run and how to use available hardware without wasting computation already performed.
In his joint presentation with Qianru Lao, Lu introduced both speakers as members of OpenAI’s inference team. He explained why routing inference requests involves more than finding an engine with spare capacity. KV-cache reuse lets an engine retain computation from earlier conversation turns, making follow-up requests cheaper and faster to process. Sending the next turn to a less busy engine can sacrifice that advantage. Routing also has to respect engine capabilities and data-residency restrictions, while accounting for performance and health across different hardware and regions.
Lu described an earlier routing design that used weighted consistent hashing to select an engine after filtering out ineligible destinations. A periodic controller combined engine performance signals into scores, compared them with the fleet average, and adjusted routing weights. Better-performing engines received more traffic; worse-performing engines received less. This feedback could accommodate changing conditions and uneven restrictions on which engines could serve particular requests, reducing the need for manual intervention.
The same adaptation could undermine cache reuse. Moving traffic away from an engine improved its performance, prompting the controller to send traffic back. Repeated adjustments could create routing oscillations, shifting requests between engines and disrupting the cached context that made subsequent turns efficient. Combining many signals into one weight also made routing decisions difficult to explain or tune: improving one aspect could change another, and different GPU types complicated comparisons.
Lu and Qianru described their team’s move toward more explicit routing policies. Qianru explained the newer architecture, which computes routing weights globally and refreshes them asynchronously so individual requests can use locally stored routing state. Lu’s portions examined the earlier design’s tradeoffs and the protections needed when production conditions deteriorate.
Lu explained three complementary safeguards:
▶ Watch ↗18:12
Loading more talks and highlights…