Harshul Jain builds data and AI infrastructure and is the creator of AgentShip, a project that supplies the production machinery surrounding AI agents. His work connects ML and data platforms with a practical concern: helping engineers spend more time on application behavior and less time rebuilding the systems needed to run it.
His professional background includes work as a Software Development Engineer II in Audible’s Fraud Engineering and Content Data Science Engineering teams. He helped build systems protecting content from abuse and AWS data pipelines supporting recommendations through relationships among topics, genres, and series. His broader infrastructure work includes streaming pipelines, feature stores, and generative-AI platforms. By the time of his recorded LLM inference workshop with Tanmay Sah, he was a senior software engineer at Audible and described five years of experience building ML and data platforms.
From production plumbing to AgentShip
Jain created AgentShip after repeatedly finding that production plumbing took far longer than writing the agent logic. API serving, streaming, execution, and observability required substantial work around the agent itself. AgentShip grew out of that imbalance: shared infrastructure could make those recurring responsibilities easier to handle.
The project separates agent definitions from execution machinery through engine adapters around a vendor-independent core. Developers can describe agents and teams in YAML or Python while using shared infrastructure built with LangGraph, LiteLLM, MCP adapters, and OpenTelemetry. This separation lets developers define what an agent should do while the surrounding runtime handles how that work executes. Keeping those concerns distinct aims to make runtime changes manageable without rewriting the application.
Jain also develops LLM Inference at Scale, a public practitioner’s handbook connecting GPU memory, capacity planning, model optimization, cache management, serving engines, and production operations. He began assembling it while learning inference because useful resources were scattered. With Tanmay Sah, he developed workshop material that pairs explanations with slides, notebooks, and benchmarks engineers can inspect.
Jain approaches inference through the constraints an application must meet. His work on inference capacity and serving starts with model weights and the GPU memory left for request state. The KV cache stores attention keys and values so generation can reuse earlier computation. Longer contexts and more concurrent requests both increase its memory requirements; latency targets impose another constraint on how much work a GPU can serve at once.
Several practical distinctions shape that approach:
Capacity depends on the application. An interactive chat application may prioritize context and responsiveness, while asynchronous tasks can tolerate more delay to gain throughput. Hourly GPU price is therefore an incomplete measure of cost. Jain emphasizes calculating how many useful tokens the hardware can serve under the application’s context, concurrency, and latency requirements.
Prefill and decode expose different bottlenecks. Prefill processes the incoming prompt and builds its attention state, involving substantial parallel computation. Decode generates successive tokens and often depends heavily on moving model weights and cached state through GPU memory. Time to first token and inter-token latency reveal different kinds of work. Understanding the limiting resource helps engineers choose an optimization that addresses the bottleneck they actually face.
Cache and scheduling techniques solve different problems. Paged attention allocates cache memory in blocks as requests grow, reducing waste from contiguous reservations. Continuous batching admits new work as requests finish, improving use of the GPU. Prefix caching reuses computation for shared prompt beginnings across requests, while KV cache quantization reduces the memory occupied by cached state. Saving memory can enable longer contexts or more concurrent requests; it does not by itself establish faster generation or better answers.
Serving engines need workload-specific evaluation. Jain recommends vLLM as a starting point for standard serving workloads and considering SGLang when repeated prefixes and branching agent workflows make cache reuse especially valuable. The comparisons he and Sah developed remain specific to their tested setup. Their practical lesson is to measure the workload the service will actually run rather than treat one benchmark as a universal ranking.
Across AgentShip and his inference handbook, Jain works on the infrastructure that makes AI applications usable beyond a prototype. He gives engineers concrete ways to separate application logic from runtime responsibilities, calculate resource needs, and distinguish memory savings from latency and throughput gains.
Harshul Jain and Tanmay Sah build inference capacity from attention and GPU memory, then explain how quantization, caching, batching, and serving engines change the costs of producing tokens.
KV storage multiplies across tokens, layers, KV heads, precision, and concurrent requests. The workshop’s roughly 4,000-token context across 80 users requires about 42 GB of KV storage alone.
Prefill performs prompt-wide computation before output begins. Decode repeatedly moves weights and preceding context for each new token, making bandwidth and inter-token latency central.
Quantization frees storage, but task quality and execution speed need separate evaluation. The workshop corrects its MLA comparison to 14× savings after restoring the missing layer multiplier.
Standard traffic and repeated agent prefixes can produce different engine comparisons. Benchmark the context reuse your application creates, then study eviction, compression, and distributed serving as its bottlenecks demand.
▶ Watch ↗