Audible

Start with Deep dive on LLM Inference at Scale for a beginner-to-intermediate path from attention and KV storage to GPU constraints. The Mistral 7B notebooks connect context length and concurrency to memory, time to first token, inter-token latency and throughput. Follow the distinction between compute-heavy prefill and memory-bandwidth-limited decode before exploring quantization and attention variants. In the recorded quantization demo, reported weight memory drops from roughly 15 GB at FP16 to 7.5 GB at int8 and 4.5 GB at int4; throughput is not measured. Jain corrects the earlier MLA savings calculation to 14×. The ostrich and World Cup algorithms are fictional teaching analogies.

Return to the same workshop for KV reuse, paged attention, continuous batching, prefix caching, KV quantization and speculative decoding. Treat the benchmarks as recorded results from particular setups: the presenters report nearly 15× throughput for default vLLM over a plain Hugging Face baseline on H100, alongside higher first-token latency and lower inter-token latency. They report no statistical difference between vLLM and SGLang on their ShareGPT workload, but three to four times better performance for SGLang on their repeated two-turn agentic branching workload, explicitly noting setup dependence. Their recommendation to default to vLLM and evaluate SGLang for agentic workloads reflects the workshop’s conclusions, not a verified comparison of today’s engines. Sah’s skepticism about speculative decoding is presented as personal testing experience.