Charles Frye
Conference affiliation: Member of Technical Staff · Modal · 2026
Related work and projects
- Modal is a computer ↗
An architectural explanation of Modal’s distributed compute, container runtime, and I/O planes.
modal.com - Modal GPU Glossary ↗
The GPU architecture and performance guide that Charles primarily authored.
modal.com · Source ↗ - Build an AI coding platform that scales ↗
Coauthored practical guide to isolated coding-agent execution and snapshot-capable sandboxes.
modal.com - Critical Point-Finding Methods Reveal Gradient-Flat Regions of Deep Network Losses ↗
First-author research on deep-network optimization and gradient-flat loss landscapes.
arxiv.org - askFSDL ↗
A hands-on retrieval-based question-answering project for Full Stack Deep Learning.
fullstackdeeplearning.com - Running an LLM on one machine: GLM-130B ↗
A self-authored deep dive into running and deploying a large open language model.
charlesfrye.github.io - Achieve state-of-the-art inference latencies with speculative decoding ↗
Coauthored guide to speculative decoding, inference-engine overhead, GPU kernels, and low-latency model serving.
modal.com - Boost your throughput with dynamic batching ↗
Coauthored demonstration of dynamic batching for Whisper transcription and efficient GPU inference.
modal.com
Charles Frye is a member of technical staff at Modal, where he helps teams deploy and optimize neural-network inference. He primarily authored Modal’s GPU Glossary, explaining how accelerator hardware and model execution shape application design. His work connects two practical questions: how to make better use of expensive computing resources, and how to tell whether the outputs are worth producing.
On this page
From neuroscience research to complete applications
Charles studied biology and computational neuroscience at the University of Chicago before pursuing doctoral research at UC Berkeley’s Redwood Center for Theoretical Neuroscience, supported by an NSF Graduate Research Fellowship. He completed his neuroscience doctorate in 2020, studying the geometry of neural-network optimization.
His first-author paper on gradient-flat regions, written with five collaborators, identified a problem with methods used to investigate that geometry. Algorithms searching for critical points—where a loss function’s gradient vanishes—could instead converge to regions where the gradient norm was stationary while the gradient itself remained nonzero. In these gradient-flat regions, the gradient lies approximately in the Hessian’s null space, making the loss locally approximately linear along the gradient direction. Mistaking these regions for critical points could distort conclusions about neural-network loss landscapes and the optimization methods operating on them.
Teaching developed alongside his research. Charles created browser-accessible mathematics and statistics notebooks and designed a course in data science for research psychology in which students constructed probabilistic models using Python and PyMC3. Interactive notes, labs, and automated feedback made the material available beyond the classroom.
At Weights & Biases, he worked first as a deep-learning instructor and then as an educator, producing material on machine learning, Python, and applied mathematics. From February 2022 to February 2024, he was a deep-learning educator at The Full Stack. Through Full Stack Deep Learning, he taught engineers to build complete machine-learning applications, from GPUs and training through deployment, user experience, and monitoring.
His askFSDL assistant put that approach into practice with a bot that answered questions using retrieved course materials. Initially, scraping and chopping up the material produced poor results; preserving source-data structure improved the system. In his 2023 account of building with LLMs, Charles described another adjustment prompted by actual use: people asked what the bot could do and how it operated, questions he had not anticipated. He added prompt instructions for that class of question. His approach to retrieval extends beyond vector similarity to keyword search, metadata filtering, and combinations suited to the information being sought.
When self-hosting became worthwhile
In January 2023, Charles ran GLM-130B on an eight-GPU machine. He wanted experience operating a language model himself and was concerned about concentrating a new computing capability in privately controlled services. Earlier openly available models had not delivered the free-form generation he needed.
The experiment worked, but it exposed difficult software dependencies, weaker instruction following than the OpenAI models he was accustomed to, and serving costs substantially above the API after a day of work. He also found an error in his own German prompt that complicated judging a translation. His interest in self-hosting survived those results without turning them into a claim that it was already the better option.
In his 2023 inference workshop, Charles argued that open and proprietary systems could coexist: accessible weights permit deployment and inference customization, while hosted services offer convenience, capability, and enterprise support. His conditional prediction was that open models would become especially competitive when applications needed a sufficient level of capability rather than the strongest available model. By 2025, he saw improvements in both model quality and inference engines making open-weight model self-hosting practical for more applications.
Charles joined Modal as a developer advocate in February 2024 and became a member of technical staff in February 2026. His account of Modal as a distributed computer explains its architecture through operating-system concepts: container images become running programs, shared files are reused, storage forms a hierarchy of caches, and routing carries information to and from containers. The analogy explains how the platform combines and shares cloud resources without assigning its construction to Charles alone.
Making expensive hardware do useful work
- GPU literacy for application decisions. Charles’s GPU teaching emphasizes arithmetic intensity—computation performed per memory load—and low-precision matrix-matrix multiplication. Sequential token generation can repeatedly move model weights while leaving arithmetic capacity underused. Batching requests, generating multiple candidates, or checking proposed tokens together can expose more parallel work. In their Whisper batching article, Charles and Cathy Zhou explain the serving tradeoff: process requests together, but impose a waiting-time limit so sporadic traffic does not leave users waiting indefinitely for a full batch.
- Benchmarks that answer deployment questions. Charles’s LLM Engineer’s Almanac benchmarking work grew from repeated customer questions about particular models and workloads. It compared vLLM, SGLang, and TensorRT-LLM across model configurations and input/output lengths, relating throughput to first-token latency. The initial results measured out-of-the-box performance rather than exhaustively tuned engines. They made the cost difference between parallel prompt processing and sequential output generation visible. Charles recommends considering better context before automatically requesting longer generated reasoning; the performance results do not establish which choice improves every task’s answers.
- Keeping the CPU out of the GPU’s way. In his explanation of inference engines, Charles identifies scheduling and host-device synchronization as potential bottlenecks. Faster GPU arithmetic does little good if CPU work leaves the accelerator idle. CUDA graphs address one part of this problem by recording a sequence of GPU operations that can then be launched together, reducing repeated host-side launch work. He treats correctness as a separate obligation: performance optimizations must also be checked for changes in model behavior.
His speculative-decoding article with Shankha Biswas develops a complementary approach. A smaller draft model proposes tokens that the target model checks in parallel, turning some sequential decoding into small parallel processing steps. Task-specific training can increase the number of accepted tokens per iteration. Their account describes collaborative work involving Modal colleagues, Decagon, Z Lab, and SGLang, including training on synthetic data when production data is restricted and checking for failures to generalize beyond the training distribution.
Charles also explains where batching intuition stops applying. His discussion of KV-cache arithmetic intensity distinguishes shared model weights from context-specific cached information. Independent requests reuse model weights, but attention generally loads different cached context for each request unless prefixes overlap. Batching therefore does not improve every component of inference equally.
When a beautiful output still fails
The QArt codes project account, coauthored with Will Shainin and Erik Dunteman, gives Charles’s evaluation work a concrete test: an attractive generated image must also transmit the information encoded in its QR pattern.
The team compared automated scanning and aesthetic evaluators with human judgments before using them for larger experiments. Will performed the laborious manual scanning that helped establish whether automated checks matched phone-based use. A configuration that produced better-looking images initially reduced scan success, so choosing parameters alone did not resolve the tradeoff.
The team then applied inference-time compute scaling: generate eight candidates per request, evaluate them, and display four ranked first by scannability and then by aesthetics. They reported reaching a 95% scan-rate objective measured per batch while keeping end-to-end latency below 20 seconds at the 95th percentile. Additional generation helped because the system could select usable results. Charles explores the same mechanism for coding tasks, where executable tests can identify successful candidates among multiple smaller-model outputs.
With Joy Liu, Charles coauthored Modal Vibe, a coding-platform demonstration built around a sandbox manager, a frontend, and isolated execution environments. Their account examines the startup and scaling demands of running many generated applications concurrently and explicitly cautions against copying the demo directly into production. His satire about ever more elaborate agent harnesses questions the expectation that additional orchestration alone will turn unreliable generation into dependable software.
Charles’s applications also include a camera-driven installation that turns whatever it sees into a museum-style descriptive placard, using a Qwen vision-language model served through SGLang on Modal. It supplies the curatorial language that makes an ordinary scene sound like an artwork—a playful use of the same inference systems he measures and explains.
Read the topics behind these talks
6 conference talks
▶ Watch ↗AI Engineer World's Fair 202516:07
How fast are LLM inference engines anyway?
Explore the talk →
▶ Watch ↗AI Engineer World's Fair 202519:52
What every AI engineer needs to know about GPUs
Explore the talk →
▶ Watch ↗AI Engineer World's Fair 202435:21
What We Learned From A Year of Building With LLMs
Explore the talk →
▶ Watch ↗AI Engineer World's Fair 202659:58
What Is an Inference Engine, Anyway? — Charles Frye, Modal
Explore the talk →Loading more talks and highlights…
References
- modal.com/blog/modal-is-a-computer ↗
- modal.com/gpu-glossary ↗
- modal.com/gpu-glossary/contributors ↗
- modal.com/blog/modal-vibe ↗
- arxiv.org/abs/2003.10397 ↗
- fullstackdeeplearning.com/llm-bootcamp/spring-2023/askfsdl-walkthrough/ ↗
- charlesfrye.github.io/programming/2023/01/12/running-glm-130b.html ↗
- modal.com/blog/achieve-sota-specdec ↗
- modal.com/blog/batching-whisper ↗
- ai.engineer/worldsfair/2026 ↗

