Jason Kramberger

Jason Kramberger works at Google and is a maintainer of Inference Perf, an open-source tool for measuring language-model serving performance. His software contributions and co-authored research address a practical problem: a benchmark can introduce delays of its own, leading engineers to mistake an overloaded testing client for a slow model server.

Building tools that can keep up with the server

Kramberger authored a GKE AI Labs benchmarking guide for a model-server-agnostic tool that benchmarks language models running on GPUs and TPUs. The guide explains how to run the tool directly or in a container on Google Kubernetes Engine, then generate latency and throughput profiles across request rates. The older project is no longer actively developed and now directs users to Inference Perf.

Within Inference Perf, Kramberger has worked on both load generation and the machinery that makes experiments reproducible. The project’s release history credits him with adding multiprocess, multithreaded load generation for its initial release. Subsequent contributions moved request-data generation into worker processes, reduced CPU use by waiting workers, improved request-rate accuracy, and added automatic load sweeps and saturation detection. These changes help the client deliver the intended workload without becoming the experiment’s limiting factor.

He also contributed a Helm chart for Kubernetes deployment, configuration saved alongside reports, support for the CNN/DailyMail dataset, and prompt-generation fixes. Later changes corrected generators that ignored their configured distribution types and added CI controls to hold merges until checks pass. These contributions connect measurement accuracy to everyday engineering practice: the workload must match its configuration, the experiment must be practical to run, and changes need checks before they reach users.

Kramberger is one of six co-authors of Inference Perf: A Benchmarking Tool for GenAI Inference, published in the Journal of Open Source Software on August 27, 2026.

What makes an inference benchmark trustworthy

Kramberger and Google colleague Ashok Chandrasekar examined the measurement problem in Identifying and Mitigating Systemic Measurement Bias in Production LLM Inference Benchmarks. Their paper connects Inference Perf’s implementation to several evaluation principles:

• Separate client delays from server delays. Under heavy load, a single-process Python benchmark can struggle to dispatch requests and process streamed tokens. If it records a token’s arrival only when the event loop processes it, client-side waiting becomes part of the reported latency. The authors distribute work across isolated processes and use simulator experiments to examine benchmark overhead separately from model computation.

• Measure the progression toward saturation. A result at one load level gives limited guidance about how much traffic a deployment can handle. Sweeping request rates produces a latency profile that shows how throughput and latency change together, helping teams identify where additional traffic produces sharply increasing wait times.

• Include the wait before decoding. The paper proposes Normalized Time Per Output Token, calculated by dividing a request’s total elapsed time by its output-token count. This incorporates initial queuing and prompt processing, which decode-only timing can obscure. The distinction matters for retrieval-heavy requests: substantial input context can delay the response even when output tokens arrive quickly once generation begins.

• Check the workload actually delivered. Prompt lengths, sampling settings, request order, and prefix-cache behavior can change results. The authors advocate retaining configuration, monitoring actual dispatch rates, and reporting scheduling delays. A requested load is only useful if the client can deliver it; otherwise, the benchmark may understate the pressure on the serving system.

The paper’s empirical results concern specific simulator workloads and client hardware. Its future-work discussion identifies multi-node load generation, controlled prefix-cache datasets, and richer models of bursty traffic as areas still to develop. That scope matters when applying the findings: Kramberger’s work gives engineers tools to inspect both serving capacity and the reliability of the experiment used to measure it.