Yuchen Fama works in AI product leadership and inference infrastructure, connecting model capabilities with the systems needed to deploy them. Her career spans statistical modeling, insurance analytics, AI platforms, and GPU infrastructure; at AI Engineer World’s Fair 2026, she was a senior principal product manager for inference at Red Hat and a contributor to vLLM and GuideLLM.
Fama completed a PhD in statistics at the University of Connecticut in 2010. Her dissertation on self-exciting switching processes examined how earlier events influence subsequent behavior and transitions between regimes. It developed methods for detecting dependent clusters and applied them to event sequences and economic cycles.
She brought statistical and machine-learning methods into insurance operations as an analytics manager at Travelers and then an analytics director at Hartford Steam Boiler, part of Munich Re. There, she led a data-science team combining insurance-risk modeling with Hadoop, Spark, and DevOps processes for model workflows. Her work also included customer-lifetime-value models and digital-advertising targeting, pairing predictive methods with the infrastructure needed to use them.
At Clarifai, Fama served as director of AI products and later vice president of product, working on an end-to-end deep-learning platform. With Alfrick Opidi and Ian Kelk, she co-authored a guide to model evaluation covering precision, recall, confusion matrices, and the inspection of problematic inputs. These tools help practitioners move from an overall performance score to a specific diagnosis of what a model gets wrong.
By 2024, she was chief product officer at Normal Computing. Her career also includes CTO responsibilities at AI startups and leadership of AI and machine-learning research teams at larger companies. At Red Hat, her work turned toward open-source inference. She co-authored distributed-inference guidance with Fatih E. Nar, Yuan Tang, and Greg Pereira, explaining how model architecture, hardware, parallelism, and latency requirements shape a deployment.
Making inference work for agents
Agentic inference makes those choices more demanding. Fama and Ashish Kamra’s joint work on agentic serving examines workloads in which agents repeatedly reuse system prompts and tool definitions while context sizes and concurrent activity fluctuate. A steady-state throughput benchmark can miss the delays users experience between turns. Their approach emphasizes interactive latency, cache reuse, and context-size distributions, including the long tail that affects capacity planning.
KV cache-aware routing addresses one source of that delay. The KV cache retains intermediate attention state from previously processed context. A follow-up request can reuse that work if it reaches a worker holding the relevant prefix. Fama and Kamra describe how llm-d’s endpoint picker weighs prefix-cache availability and cache utilization alongside running and waiting requests, balancing reuse against worker load. Their demonstration showed an initial request taking roughly three seconds and a subsequent request with the same system prompt taking roughly one second. Changing the system prompt required fresh processing; another request reusing that new prefix was faster again. These were demonstration timings, rather than a general performance guarantee.
Keeping useful state available also requires cache-management decisions. Their joint work identifies offloading tiers and session-aware eviction policies, including prioritization and session pinning, as areas of ongoing development. Routing can exploit a cached prefix only while the system retains it in an accessible place.
Prefill and decode disaggregation tackles a different problem: interference between processing a prompt and streaming an answer. In Fama and Kamra’s joint treatment, and in Fama’s collaborative distributed-inference guidance, prefill is compute-intensive and benefits from processing prompts in parallel; decode generates tokens one at a time and places greater pressure on memory bandwidth. When both share a GPU, a burst of long prompts can interrupt ongoing token generation. Separate worker pools allow the phases to scale independently, with the prefill worker constructing cached state that the decode worker transfers across the network and uses to continue generation.
That separation has costs. Long contexts, heavy input workloads, and strict inter-token latency requirements can favor disaggregation, provided the network can move cached state efficiently. Shorter contexts, low concurrency, or insufficient network capacity can favor aggregated serving. The choice depends on the workload and latency goal; improving time to the first token and keeping subsequent tokens flowing smoothly are distinct objectives.
Fama also co-authored a GLM-5.2 agentic-serving study using production Claude Code traces to examine deployment on H200 GPUs. The team combined prefix-aware routing, CPU cache offloading, expert parallelism, separate prefill and decode pools, and multi-token prediction. Those techniques addressed different constraints: reusing context, increasing cache capacity, distributing model computation, scaling prompt processing, and accelerating output. The study’s sweep measured scaling within the prefill/decode layout. It included no matched aggregated-serving control, so those results do not establish a performance advantage over aggregated serving.
Her work connects product decisions with deployment behavior: understanding the tasks users run, identifying the resource or latency constraint, and choosing infrastructure that addresses it. The inference work makes that connection concrete through cache reuse, workload-sensitive scheduling, and measured tradeoffs between prompt processing and token generation.
▶ Watch ↗