AI Engineer
← All speakers

Bio, Work & Ideas

Qianru Lao

Conference affiliation: Member of Technical Staff · OpenAI · 2026

Qianru Lao is a Member of Technical Staff on the Inference team at OpenAI, where she works on infrastructure for large-scale model serving. Previously, she contributed to the open-source Delta Lake project at Databricks and worked on distributed storage systems at Alibaba Cloud and infrastructure tooling at Google. She holds degrees in Computational Science and Engineering from Harvard and Computer Science from Sun Yat-sen University.

1 conference talk

▶ Watch ↗

18:12

Routing LLM Inference in Production: From Engine Signals to Policy — Qianru Lao & Lu Zhang, OpenAI

Read the full talk →

Key ideas

Scroll to read ↓

Qianru Lao and Lu Zhang explain how OpenAI moved from feedback-driven routing weights to a global latency optimizer, while keeping engine selection local and protecting the fleet against failures, retry storms, and excess demand.

  • Inference routing must account for reusable conversation context as well as performance, health, and geography. Bouncing traffic between engines can undermine KV cache reuse.
    2:41 ↗
  • Global coordination can stay outside request-time selection: the control plane computes weights asynchronously, and data planes select engines from locally cached snapshots with live guardrails.
    8:39 ↗
  • Compare network latency plus engine-side delay when choosing a destination. A farther engine can serve a request sooner, provided the allocation respects its effective capacity.
    12:10 ↗
  • Production needs separate controls for unhealthy engines, retry amplification, and insufficient total capacity: penalties, dynamic retry budgets, and last-resort load shedding.
    15:33 ↗

References