Ashok Chandrasekar

Ashok Chandrasekar is a Staff Software Engineer at Google working on AI inference performance evaluation and optimization for Google Kubernetes Engine. He leads and maintains Inference Perf and co-leads llm-d’s SIG Benchmarking. His work addresses a practical problem in model serving: engineers need to measure how systems respond under load, while accounting for delays introduced by the tools doing the measuring.

Previously a Staff Engineer at VMware, Chandrasekar holds a master’s degree from Carnegie Mellon University. His research interests span distributed systems, with a current focus on systems for AI and machine-learning applications. At Google and through his open-source work, that focus includes comparing serving stacks, designing benchmark workloads, and improving the measurements used to evaluate inference performance.

Comparing model-serving systems

Inference Perf originated in Kubernetes wg-serving’s effort to standardize inference benchmarking tools and metrics. It is model-server agnostic, allowing engineers to compare different servers and serving stacks using a common tool. Chandrasekar co-authored its 2026 Journal of Open Source Software publication with Sachin Varghese, Jason Kramberger, Brendan Slabe, Chen Wang, and Yuan Tang.

The tool supports configurable input and output distributions, shared-prefix and multi-turn workloads, and several ways to generate load: constant request rates, Poisson arrivals, concurrent-user simulation, and trace replay. Multi-stage runs vary request rates and durations to help identify saturation points. These controls let engineers examine how performance changes with the workload and traffic pattern rather than relying on a single fixed test.

Its measurements also distinguish raw capacity from service quality. Alongside latency and token throughput, Inference Perf reports goodput: the rate of requests that meet specified service-level objectives. A system may process more traffic while serving fewer requests within the required latency limits; goodput makes that distinction visible.

Accounting for the benchmark’s own delays

In Identifying and Mitigating Systemic Measurement Bias in Production LLM Inference Benchmarks, Chandrasekar and Jason Kramberger examine how benchmark clients can distort the performance they report. Their analysis connects client architecture, queuing, and latency metrics:

• Check whether the client becomes the bottleneck. Single-process, asyncio-driven benchmarking clients can accumulate their own queues under high concurrency. The authors model the client as an M/G/1 queue and analyze how Python’s Global Interpreter Lock can inflate measured Time to First Token and Time Per Output Token as request rates rise. Those measurements can therefore include delays in the benchmarking client as well as the serving system.

• Distribute load generation across processes. The paper proposes a multi-process evaluation framework to reduce client-side queuing overhead. The authors report that their evaluation isolates serving-engine performance at production scales exceeding thousands of queries per second, supporting more accurate and reproducible comparisons.

• Include the wait before tokens arrive. The paper formalizes Normalized Time Per Output Token, which spreads end-to-end latency across output sequence length while including prefill and scheduling delays. This incorporates costs that a measure focused only on token-generation speed can leave out.

Coordinating distributed-inference benchmarking

Chandrasekar co-leads llm-d’s SIG Benchmarking with Marcio A L Silva. The group coordinates benchmarking frameworks and methodologies, performance regression testing, workload simulation and synthetic data generation, hardware-specific optimization, and performance analysis and profiling tools.

That remit extends beyond an individual benchmark run. It brings workload design, regression checks, and profiling into a shared effort to evaluate distributed inference across different workloads and hardware configurations.