Sitanshu Gupta

Sitanshu Gupta is Director of Engineering, Inference Services at CoreWeave, where he leads engineering work on serving AI models under production load. His work addresses the pressures that emerge when a model moves beyond a demonstration: simultaneous requests, long contexts, and repeated agent interactions competing for memory and compute.

Coordinating distributed computation

Gupta is one of seven co-inventors of a SambaNova-assigned patent on flow control for reconfigurable processors, whose patent family traces to a June 2, 2020 filing. The invention describes a compiler that inserts buffers and control connections to coordinate asynchronous processing stages. Credit counters track downstream buffer space and writes still in flight. When either allowance runs out, a producer pauses until the downstream component signals that it can accept more data. This lets distributed stages proceed at different speeds without overflowing buffers or losing coordination.

At CoreWeave, Gupta’s writing examines coordination across model-serving infrastructure. His June 2026 article on GLM 5.2 explains the engineering between evaluating a managed endpoint and sustaining its performance in production: matching hardware to the model, tuning networking and the serving runtime, loading models and containers, and managing cached context. He also argues that open weights preserve deployment choices. A team can begin with a managed endpoint and retain the option to fine-tune or self-host as its needs change.

His August 2026 account of serving Kimi K3 develops that systems perspective through long-context, multi-turn workloads. CoreWeave uses NVIDIA Dynamo at the infrastructure layer and vLLM at the runtime layer; Gupta credits runtime optimizations to the vLLM team, Moonshot AI, and NVIDIA. He explains how deployment topology, request routing, and runtime features combine to determine the performance customers experience.

Making inference predictable

• Benchmark the workload people will run. Gupta’s Kimi K3 article examines concurrent, multi-turn, tool-calling interactions rather than isolated requests. CoreWeave tested NVIDIA GB200 and GB300 NVL72 systems across concurrency levels from 1 to 32 users, measuring output throughput per GPU, request latency, and decode speed per user. These measures reveal the tradeoff between infrastructure efficiency and interactive responsiveness. In the reported workload, GB200 throughput dropped sharply beyond 16 concurrent users, while GB300 continued scaling through 32. The comparison shows why a high aggregate throughput number needs context: it does not, by itself, establish that each user receives a timely response.

• Route repeated context to a warm replica. Agent interactions and long conversations repeatedly send context the model has already processed. CoreWeave’s KV-cache-aware routing directs repeat queries to replicas that retain the relevant prefix, allowing earlier computation to be reused. Gupta connects this decision to lower token cost and higher throughput, particularly for coding agents and multi-turn workloads.

• Keep communication local and scale serving phases independently. Gupta describes pinning each Kimi K3 replica to one NVLink domain so tensor- and expert-parallel traffic stays on the local high-bandwidth interconnect rather than crossing racks. The serving stack also supports speculative decoding to improve generation speed and prefill/decode disaggregation so processing the input and generating the response can scale independently. Together, these mechanisms explain the work required after the model’s weights fit onto GPUs.

Building and explaining the inference stack

Gupta also recruits engineers and managers for this work. In his public Kimi K3 post, retained in his public LinkedIn profile, he paired the engineering article with openings for inference engineering managers and staff and senior software engineers.

His Inside Inference session with Inferact’s Zachary Xi addresses the same production problem from two layers. The session covers vLLM scheduling, batching, memory management, and quantization alongside CoreWeave’s networking, topology, and orchestration. Its practical question is whether serving-software improvements reach users as measured gains—or are absorbed by limits elsewhere in the system.