AI Engineer
← All speakers

Bio, Work & Ideas

Lu Zhang

Conference affiliation: Member of Technical Staff · OpenAI · 2026

Lu is an engineer working on large-scale inference platforms, focused on making AI model serving reliable, efficient, and scalable. His work includes distributed systems, workload scheduling, performance optimization, and production reliability. Previously, Lu built and operated GPU clusters supporting large machine learning workloads.

1 conference talk

▶ Watch ↗

18:12

Routing LLM Inference in Production: From Engine Signals to Policy — Qianru Lao & Lu Zhang, OpenAI

Read the full talk →

Key ideas

Scroll to read ↓

Qianru Lao and Lu Zhang explain how OpenAI moved from feedback-driven routing weights to a global latency optimizer, while keeping engine selection local and protecting the fleet against failures, retry storms, and excess demand.

  • Inference routing must account for reusable conversation context as well as performance, health, and geography. Bouncing traffic between engines can undermine KV cache reuse.
    2:41 ↗
  • Global coordination can stay outside request-time selection: the control plane computes weights asynchronously, and data planes select engines from locally cached snapshots with live guardrails.
    8:39 ↗
  • Compare network latency plus engine-side delay when choosing a destination. A farther engine can serve a request sooner, provided the allocation respects its effective capacity.
    12:10 ↗
  • Production needs separate controls for unhealthy engines, retry amplification, and insufficient total capacity: penalties, dynamic retry budgets, and last-resort load shedding.
    15:33 ↗

References