Philip Kiely is Head of AI Education at Baseten, an early employee of the model-inference company, and the author of Inference Engineering. He helps engineers understand how model architecture, GPU hardware, serving software, networking, and application design determine the speed and cost of production AI.
Kiely began publishing technical tutorials in 2019 while studying computer science at Grinnell College. He wrote for Smashing Magazine, Twilio, and other technology publications, published Writing for Software Developers shortly before graduating with an honors degree in 2020, and created Who Pays Technical Writers to connect writers with paid opportunities.
He joined Baseten in January 2022 as its tenth employee, working across developer relations, technical content, documentation, and education. As generative AI moved into production, he concentrated on inference: choosing hardware, deploying models, measuring bottlenecks, and balancing latency, throughput, reliability, and cost. His 2026 book Inference Engineering maps that discipline across model architecture, GPUs, serving runtimes, optimization, multimodal applications, and production infrastructure.
Inference performance depends on the workload. GPU type, batch size, sequence length, concurrency, and latency targets determine which configuration actually performs well. Kiely emphasizes representative benchmarks, tail latency, and the difference between throughput experienced by one user and total GPU output. His TensorRT-LLM workshop also addresses hardware-specific engines, quantization, deployment, and autoscaling constraints.
Open serving frameworks expand practical choices. Kiely works across TensorRT-LLM, SGLang, and vLLM, choosing among hardware optimization, extensibility, and model compatibility. His SGLang workshop connects runtime selection to speculative decoding, realistic benchmarking, inspectable implementations, and open-source contributions.
Real-time voice inference changes the optimization target. For Orpheus text-to-speech, roughly 83 tokens per second satisfies real-time playback; first-byte latency, simultaneous streams, and GPU utilization then matter more than additional token throughput. Kiely combines FP8 quantization, TensorRT-LLM, GPU-based audio decoding, and dynamic batching while accounting for connection reuse, network placement, and coordination across listening, reasoning, and speech systems.
Technical books can become reusable audio pipelines. Kiely built an AI-narrated audiobook of Inference Engineering using Rime’s Coda model and a custom clone of his voice. The project involved rewriting diagrams and tables for audio, segmenting the manuscript, correcting pronunciation, generating speech, and assembling audiobook files.
Build a GPU-specific inference engine, repeat the build with FP8, deploy it behind an API, and measure where throughput gains meet latency and serving constraints.
Deploy an OpenAI-compatible model server, diagnose CUDA graph coverage from its logs, and tune speculative decoding against the workload you actually expect to serve.
Orpheus TTS shows how LLM runtimes, audio decoding, batching and client connections work together to deliver more concurrent speech streams with less startup delay.
Philip Kiely of Baseten examines the tradeoffs behind four-bit caches, learned memory compaction, and diffusion drafting—and the growing role of training in making inference faster.
Local inference and data center serving can favor different tradeoffs. A smaller cache may justify slower decode when memory capacity prevents a workload from fitting.
DFlash proposes eight or 16 tokens together. Complete drafting time and target acceptance determine its benefit, rather than the duration of one drafting invocation.
Continuous retraining can improve acceptance on live traffic, but requires data-use permission, storage, compute, data movement, and changes when the target model changes.
▶ Watch ↗
