AI Engineer Summit 2023

AI Engineering 201: Inference

Charles Frye1:43:16

Loading slides…

Read the talk

AI Engineering 201: Inference

Moving beyond an API wrapper means understanding where inference runs, what its hardware spends time doing, and which operational responsibilities your application should own.

From a talk by Charles Frye

Before you start: Familiarity with calling an LLM API and basic ideas about arrays, memory, and application deployment will help; no model-training background is required.

What comes after the API wrapper?

You have built a thin wrapper around the OpenAI API. In Charles Frye’s opening joke, you have also raised $100 million at a $4 billion valuation. What are you supposed to engineer next? The API call works, but it tells you little about the workload behind it—or what will happen when cost, latency, reliability, and control become product requirements.

Slide asks what comes next after making “ChatGPT for your dog’s documents,” with a small presenter inset.Open full source frame
You can call the OpenAI API—what next?

Inference is where those requirements become concrete engineering constraints. You can define service-level objectives, compare service-level agreements, decide between open and proprietary models, choose device or cloud execution, and determine whether to serve a model yourself. Frye places this before application architectures, monitoring, evaluation, and observability in the workshop: first understand the machinery on which those applications depend.

Frye approaches the subject through teaching complete ML applications. He studied neural networks at Berkeley in the 2010s, worked in developer relations and education at Weights & Biases, and taught with Full Stack Deep Learning—from GPUs through the user experience. That breadth matters here because an inference decision can begin with model quality and end with power delivery, deployment policy, or interface responsiveness.

0:000:25
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:00 · section reference included

From human-readable data to tensors

From outside, generative inference looks like a data-to-data function. Supply text, an image, or sound; receive something a person can interpret. In the PaLM-E example, an image of a restaurant accompanies a question about how a robot could help. The generated answer proposes cleaning tables, picking up trash, and moving or wiping chairs. That plan could become input to a robot controller; producing the plan and physically executing it are separate operations.

Inside the API, this becomes three stages:

  1. Preprocess the input. A tokenizer converts text into numerical token identifiers, represented in arrays called tensors.
  2. Run the network. Neural computation maps tensors to tensors.
  3. Produce the output. Output conversion and sampling turn numerical results into text or other human-readable data.

Tokens, log probabilities, and temperature belong to this more detailed view. Tokenization and sampling have their own complications, but the network’s tensor-to-tensor computation dominates the compute, memory, and optimization burden. This is the expensive work you either engineer or delegate.

Diagram connects text, image and sound to tensors through .preprocess, tensors to tensors through .forward, and tensors to outputs through .predict.Open full source frame
Generative inference decomposed into three functions.

A neural network is a composition of tensor-to-tensor functions. Their compatible interfaces give them a LEGO-like quality: an output from one block becomes an input to the next. Inception v3, the computer-vision example, starts with an RGB image and passes through intermediate representations such as an 8 × 8 × 2048 tensor. That large block of numbers is an internal representation, not the final classification.

Each block also depends on another set of tensors: its weights and biases. The first-layer weights of AlexNet can be visualized as small RGB patches that detect edges, textures, and color differences. Most later weights are less directly interpretable, but the principle is the same: learned numbers determine the transformation. Training produces those numbers through optimization and data collection—a process Frye also criticizes for collecting internet material without consent. Foundation-model teams now do that work and either release weights or expose a proprietary service, letting application developers begin with inference rather than training.

3:524:05
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:52 · section reference included

Buying capability before optimizing cost

The first infrastructure choice is familiar: build from available components or buy a service. In the workshop’s historical snapshot, Frye describes the top five LMSYS leaderboard entries as proprietary models from OpenAI and Anthropic. His practical sequence is to establish that an application works with a highly capable model, then investigate a cheaper or easier-to-run substitute. Otherwise, a model limitation can look like a failure of the product idea. The prices, rankings, and API availability discussed here belong to that snapshot, rather than a current procurement guide.

Self-hosting does not automatically make an equivalent application cheaper. Capability differences complicate the comparison, and providers can spread optimization and hardware costs over many customers. Frye reports that a one-day attempt to run GLM-130B brought him within an order of magnitude of OpenAI’s cost, on the more expensive side. He also cites Honeycomb’s natural-language query transformation feature as a case where the team found OpenAI inexpensive. These are operating experiences, not a controlled comparison of identical models and service levels.

For scale, Frye quotes about $1 per million tokens for Claude Instant and $10 per million for Claude 2, without separating input and output prices. He contrasts those amounts with the cost of human-produced text. Anthropic’s longer context windows offered a reason to choose it, while his contemporary preference for OpenAI rested on price and capability together. The important comparison is the cost of completing your task at the required quality, not just the price attached to a model name.

Slide titled “Good: proprietary models are priced affordably” shows Honeycomb and FSDL quotations above OpenAI and Anthropic pricing tables.Open full source frame
Proprietary model pricing alongside accounts of operating costs.
10:1610:28
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:16 · section reference included

What a managed API lets you control

A proprietary model’s capability comes with an access boundary. Frye uses Alex Graveley’s reaction to gpt-3.5-turbo-instruct as an example: access to token log probabilities made it possible to inspect more than the generated text. Probabilities reveal which alternatives a model considered likely and provide richer evidence during prompt development. They can also support applications that inspect or manipulate the output distribution.

Frye recounts log-probability access returning in mid-September and disappearing again shortly before the workshop. He interprets that restriction as protection against reverse engineering, comparing the situation to clones of IBM’s personal computer. That motive is his explanation, rather than an established provider statement. The durable issue is that a service controls which parts of a probabilistic model you can observe, and that boundary can change independently of your application.

Enterprise offerings address a different set of constraints: GDPR concerns, SOC 2, privacy, security reviews, and contractual assurances. For a company that cannot simply send business data to an external endpoint, those services can determine whether an application ships at all. Asked about Azure versus direct-API prices, Frye says he does not know. He instead identifies the enterprise benefits he considers consequential: an SLA, more generous rate limits, and potentially different customization options. An SLA creates a contractual reliability commitment; it does not eliminate failures.

Fine-tuning complicates those economics. An attendee notes that Azure’s offering then supported only three models. Frye points to a Gradient example where fine-tuning a style could replace repeatedly supplying style instructions in the prompt. He is less persuaded by fine-tuning as a way to inject fresh knowledge or eliminate retrieved context. A customized model also gives the provider fewer users over whom to amortize its serving costs, helping explain the higher inference prices he encountered. The exchange closes with audience corrections that Vertex already included Google’s managed model offering and AWS Bedrock was available.

14:2914:47
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

14:29 · section reference included

Open weights enable a different development model

Asked whether to bet on open or proprietary models, Frye first separates raw capability from hackability. He describes four of the next five leaderboard positions as Llama 2 models and derivatives. Their attraction includes the surrounding development process: many independent contributors can implement small features, support unusual hardware, or solve specialized last-mile problems, then share those improvements. The resulting ecosystem can compete with a centrally funded team without matching its organization.

Available weights are not the same thing as an unrestricted open-source license. Frye contrasts familiar licenses such as MIT, Apache, and LGPL with Meta’s custom terms. Llama 1 had research-only distribution before its weights spread through torrents. The Llama 2 Community License permits much broader use but restricts using the materials or their outputs to improve other LLMs, while exempting Llama 2 and its derivatives. Its additional commercial-permission provision applies above 700 million monthly active users in the calendar month preceding release—not at exactly that number, and not as a permanent unconditional ban.

Those restrictions matter to the meaning of open source, not merely to whether a small team can download a checkpoint. Frye connects them to the Open Source Initiative’s principles against restricting who may use software and to the then-ongoing effort to define open-source AI. A model can be highly useful to an open development community while still carrying legal constraints that conventional open-source software does not.

Slide asks “but is it FOSS?” beside a Jim Fan post about Llama 2 license restrictions and an Open Source Initiative definition announcement.Open full source frame
Llama 2 licensing and the definition of open-source AI.
20:5621:05
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

20:56 · section reference included

A convincing answer can still be wrong

An early ecosystem lead is not permanent. Frye points to Mistral 7B as a recent challenge to larger models. Its release-specific claim was more precise than a general victory over larger LLMs: the 7.3B-parameter model outperformed Llama 2 13B across the reported benchmarks and Llama 1 34B on many of them. Model size alone does not establish a capability ranking.

Enthusiasm can also outrun evaluation. Frye describes the popular imitation recipe: begin with Llama weights, collect outputs from a proprietary chatbot, and fine-tune on those conversations. The widely repeated Vicuna announcement illustrates why the evaluation method matters. Its original result concerned Vicuna-13B trained on about 70,000 conversations; the reported 92% was a ratio of aggregate GPT-4 judging scores across 80 questions. It was not general task accuracy, and the announcement itself treated the evaluation as preliminary.

The workshop makes the failure mode tangible with a modified illustrative screenshot, rather than a live model test. Two anonymized answers address the same question: how does actor–critic improve on REINFORCE?

AnswerWhat the reader seesSubstantive issue
APolicy/value methods, a critic, advantage estimationA coherent explanation
BSimilar terminology and an authoritative listClaims REINFORCE has a separate critic for each action

Answer B also claims that actor–critic learns a policy and actor simultaneously while REINFORCE learns them separately. Its vocabulary sounds appropriate, but the relationships are wrong. Frye asks the audience to choose between the answers; the polished presentation makes that harder for readers without reinforcement-learning expertise.

After correcting an initial verbal mix-up, Frye identifies A as GPT-4 and B as a fine-tuned imitation model. The point is not that formatting is bad; it is that preference judgments can reward formatting without testing correctness. The False Promise of Imitating Proprietary LLMs investigates this separation between convincing style and substantive capability. Frye also credits the Guanaco work with recognizing evaluation problems. An evaluator must be able to detect the errors that matter to the application. Otherwise, a friendly, well-organized answer can pass while failing its actual task.

25:2325:42
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

25:23 · section reference included

Capability requirements shape the market

The history of infrastructure offers a better starting point than asking which licensing model must win. Frye walks from IBM mainframes through Xerox, DOS, Mac OS, Unix, and GNU/Linux: open systems gained ground, but proprietary systems did not disappear. Mobile platforms likewise support both models. Databases show a similar pattern, from Oracle and DB2 through open SQL implementations, NoSQL systems, and streaming databases. His talk-era database ranking was roughly split between open and proprietary products.

Foundation models occupy a low layer of the software stack, much like an operating system or database. Some organizations value the ability to inspect, modify, and deploy that layer themselves. Others value a supported enterprise service. Those needs can sustain coexistence. Frye’s conditional forecast is that open models gain the advantage if users eventually want a model that is simply good enough for a defined task, rather than the most capable model available.

Chasing the next capability tier requires expensive research teams and infrastructure; Frye invokes investments on the order of hundreds of millions of dollars. If requirements saturate, that frontier expense becomes less valuable to ordinary buyers, leaving room for open implementations to catch up. He compares this with processors becoming fast enough for many users and growing interest in RISC-V. If demand for cognition instead remains effectively unbounded—as he suggests demand for RAM often feels—proprietary frontier investment may retain its advantage. The outcome depends on what applications continue to need.

Slide reads: “If capabilities requirements saturate, open models will catch up, then dominate.”Open full source frame
Capability saturation could allow open models to catch up.
30:4030:51
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

30:40 · section reference included

Language coverage, team knowledge, and specialization

Multilingual capability introduces another dimension to model selection. The amount of available training text in a language matters more directly than the number of people who speak it. A language-focused model may use a fixed capacity more effectively for that language. English-heavy instruction tuning and human feedback also mean that behavior and safety policies may transfer unevenly; Frye cites failures in low-resource languages and notes the cultural consequences of reinforcing English’s existing advantage.

Could an application translate into English, prompt for reasoning there, and translate the answer back? Frye thinks it might help but explicitly has no benchmark to establish that. Any reasoning benefit has to exceed translation loss. His example of asymmetric recall—asking for Tom Cruise’s mother versus asking for that woman’s son—illustrates why language-model behavior need not resemble traversing a knowledge graph in either direction. Changing the tokens and direction of a question can change whether the model retrieves the relationship.

The next audience question asks how much deep learning an application engineer still needs. Frye distinguishes individuals from organizations. An individual can use a database without implementing B-trees or understanding every page-layout decision, and can similarly choose how deeply to study models according to interest. A team with no database expertise, however, will struggle when its managed service fails, needs reconfiguration, or should be replaced. AI teams need the same diagnostic capacity somewhere: application engineering, modeling engineering, and LLM reliability engineering are different specializations, not a requirement that everyone master all three.

That distinction leads to a business question about broad vertical models for finance, law, or healthcare. Frye worries about their position between general foundation models and very narrow specialists. His bootcamp once culminated in building and deploying OCR; he reports that GPT-4V subsequently handled it better as an incidental capability. From the other direction, a cardiac-ultrasound model could be much more efficient for one specific clinical task. His hypothetical tenfold efficiency advantage illustrates the competitive risk, rather than a measured result: a broad healthcare model must defend itself against both the general API and the focused specialist.

36:0536:12
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

36:05 · section reference included

Moving inference onto the user’s device

Inference does not have to cross a network. Frye reports running Llama 2 13B on his laptop, and local execution opens possibilities that a remote API cannot always support. He uses the familiar Norvig/Dean latency example of an approximately 150 ms California–Europe round trip alongside the rough 400 ms Doherty interaction threshold. Network travel alone consumes a substantial part of that illustrative response budget. Asynchronous work can hide some delay, but a system that must react immediately—such as a vehicle deciding to brake—has less room to compensate.

Local computation also moves the inference bill to the user’s hardware. It can remove the application provider’s per-request compute expense, but it reduces operational control. Telemetry is more constrained, hardware and software environments vary, and installed clients may remain on old versions. A centrally operated service can change its environment once; a client application has to survive the environments its users actually have.

Phones, wearables, and robots make the appeal concrete. Frye describes the Pixel 8 Pro announcement in terms of summarization and light image editing, and a robotics hack involving household point clouds and voice control of a Roomba. These are constrained features, rather than evidence that unrestricted chatbot capability fits everywhere. Storage, RAM, and accelerator memory all impose limits; his single-board computer example had only 2 GB shared between CPU and GPU.

Power and heat sit beneath those memory limits. Frye compares a 300 W A100 configuration with a 10 W Jetson Nano, a 30-fold power difference. Those are the configurations in his illustration, not universal device ratings. A small enclosure cannot simply accept data-center-scale power or memory movement: the resulting heat has to leave the device. This is a physical engineering constraint, not something a cleaner API removes.

Desktops relax some limits while increasing hardware diversity. Native assistants, OS features, and intelligent game characters cannot assume that every user has a recent NVIDIA GPU or that all its memory is available. Frye uses llama.cpp’s build complexity as a reminder of that support burden. Consoles look more promising in his forecast: a controlled platform, a substantial hardware budget, and developer influence over custom silicon could make them attractive inference targets. That is a proposed direction for future game systems, not a capability demonstrated in the workshop.

44:1144:26
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

44:11 · section reference included

Quantization saves bytes; kernels realize the benefit

When memory is the constraint, reducing the representation size of each parameter helps. Quantization moves from two-byte parameters toward one byte, half a byte, or even experimental one-bit representations. Frye describes practical progress at the time as clustering around four bits. Saving storage is only part of the task: realizing fast execution with packed low-precision values can require specialized kernels or assembly.

Sparsity attacks a different source of waste. If many weights are zero or near zero, perhaps their multiplications can be skipped. But zeros scattered arbitrarily through a matrix create unstructured sparsity, while convenient hardware and software support often expects a regular structure. A sparse checkpoint therefore does not automatically run faster. The execution system must avoid the work without spending more time locating and handling the irregular values. Frye points to hand-tuned CUDA kernels as one way to realize that benefit and mentions distillation alongside quantization for smaller-device models.

The basic memory arithmetic is straightforward: the B in a model size means billions of parameters, not bytes. At two bytes per parameter, a 7B model requires approximately 14 GB for weights alone. Paging weights from disk can make an otherwise oversized model executable, but repeatedly moving them into memory can severely slow it down.

The audience’s question about four-byte parameters exposes the importance of stating the representation. float32 uses four bytes; float16 and bfloat16 use two; float64 uses eight. Lower-bit quantized formats reduce the nominal weight payload further. The following Python calculation makes the storage assumption explicit:

python

parameters = 7_000_000_000

for bits in (32, 16, 8, 4):
    weight_gb = parameters * bits / 8 / 1_000_000_000
    print(f"{bits:2d}-bit weights: {weight_gb:.1f} GB")

This counts decimal gigabytes of parameter values. Runtime buffers, caches, and quantization metadata require additional space. Frye closes the exchange by noting Georgi Gerganov’s work converting models to four- and three-bit representations: the checkpoint’s advertised parameter count is only the beginning of a memory estimate.

53:0853:13
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

53:08 · section reference included

The browser standardizes execution, not delivery

Browser inference combines a user-owned computer with a more uniform software runtime. The model runs inside the browser; it is not merely a web interface calling a remote model. Frye discusses WebAssembly as a compilation target and emerging WebGPU support as a cleaner path to GPU execution. But this introduces a distinct bottleneck: the model’s weights must reach the browser. A capable desktop can still face a mobile-like model-size budget during first load.

Progressive delivery or a model bundled into the browser could reduce the application’s burden. Frye imagines a successor to V8 with a foundation model built in. That is a speculative deployment idea, and the audience immediately tests it against download size. In his rough taxonomy, conversational models already occupy tens of billions of parameters, while PaLM 540B would require roughly a terabyte of two-byte weights. An attendee corrects his assumption about browser installers toward hundreds of megabytes. Under that download budget, bundling a large model is a nonstarter; standardizing the runtime does not make the bytes disappear.

Would a new language or compiler such as Mojo change this arithmetic? Frye’s answer is that he has already assumed reasonably efficient execution. Python at the application layer is not the fundamental problem when PyTorch dispatches expensive operations to optimized native kernels. The hardware still has to hold and move the same model values.

For a CPU-versus-GPU cost comparison, measure delivered throughput and combine it with hardware spend:

cost per token=hardware cost per seconddelivered tokens per second\text{cost per token} = \frac{\text{hardware cost per second}} {\text{delivered tokens per second}}

Use an observation interval that reflects the traffic you actually serve, including paid idle time. Frye offers no universal CPU/GPU price ratio: CPU competitiveness depends on the workload, and matching a provider’s unit economics generally requires substantial optimization.

58:0158:23
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

58:01 · section reference included

Centralized serving trades latency for simplicity

A data center gives you control over the execution environment, simplifying deployment and support. The audience challenges Frye’s emphasis on network latency: if a local machine generates tokens more slowly, could local execution actually take longer? Yes—the relative importance of network travel depends on the full request. Sequential calls with intervening application logic accumulate round trips, while long generations may be dominated by decoding. Frye’s thirty-token example does not supply absolute generation rates that establish where local execution wins.

That distinction separates throughput from response latency without making them independent. Generation speed contributes to completion time, while network travel and waiting add other costs. Frye acknowledges that ChatGPT-like applications may not feel network delay as their main problem. Motor-control applications are different: cognition inside a moving system may need responses in tens of milliseconds, leaving little room for a distant service.

Inference-as-a-service makes centralized execution easy to consume. It can serve proprietary models or open models through providers such as Replicate. Even when the model weights are open, the service may limit customization because its economics depend on sharing infrastructure across many similar requests. Frye compares this with cloud providers reusing cached container components: infrastructure sharing is what lets them amortize costs, and arbitrary model customization can make that harder.

The resulting management problem differs by ownership:

ArrangementMain economic taskConstraints you inherit
Metered inference APIMinimize usage while meeting quality needsRate limits and service controls
Rented cloud GPUsMatch capacity to demandInstance availability and rental costs
Committed capacityKeep purchased capacity usefulLock-in and reduced flexibility

Frye identifies specialist GPU clouds such as Lambda Labs as alternatives to the large providers and highlights GPU attachment flexibility in Google Cloud. Renting gives more control without owning the physical facility, but a large prepaid commitment starts to resemble hardware ownership financially: the money is committed even if the workload changes.

1:04:451:04:57
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:04:45 · section reference included

Scale to zero, then load the model again

Serverless GPU services aim to combine usage-based pricing with more control over the computation. Frye’s definition includes scale-to-zero resources and billing, no direct server or OS management, and high availability. He names Modal, Replicate, Hugging Face inference endpoints, and Banana.dev. Low-traffic features and occasional investor demos fit the economics especially well: an unused feature need not keep a GPU running. Autoscaling down to one instance is materially different from scaling to zero.

The trade-off appears when traffic arrives or expands. A dedicated operations team can tune autoscaling around particular throughput and tail-latency objectives; a shared serverless platform owns some of those decisions for you. For a cold replica, model loading follows a costly path:

  1. Read the weights from storage.
  2. Place them in host RAM.
  3. Transfer them into accelerator memory.
  4. Begin serving requests.

Frye reports cold boots ranging from 30 seconds to three minutes in cases involving roughly half a terabyte of model data. Such starts can damage tail latency even when warm inference is fast. A memory problem has reappeared as a latency problem. Caching and sharing weights offer ways to reduce it.

Slide lists scale-to-zero semantics, server management, availability, low-traffic benefits, autoscaling and latency drawbacks, and cost management beside provider logos.Open full source frame
Serverless inference: scale to zero and operational trade-offs.

An attendee who identifies themselves as a Cloudflare employee challenges the generalization using Cloudflare’s new GPU offering. Frye has not tried it and asks how the loading problem is handled. The exchange does not establish a technical resolution, so it leaves a useful provider-evaluation question: what remains resident or shared when your application scales down, and what must be loaded on its next request? Frye points readers to Full Stack Deep Learning’s Cloud GPUs resource for provider and compute comparisons.

1:11:401:12:00
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:11:40 · section reference included

Owning hardware means owning its constraints

Physical ownership need not mean constructing a data center; co-location can supply the facility. GPU workloads nevertheless put unusual demands on it. Frye describes older facilities as optimized for disk and network workloads rather than high power density. Even obtaining a huge supply of accelerators would not make a database-oriented facility ready to power and cool them. This mismatch helps explain the room for specialist infrastructure providers.

Ownership can deliver control and favorable long-run economics when utilization justifies the capital expense. It also requires a scarce combination of ML, hardware, and operational expertise. The financial constraint is illiquidity: buying GPUs commits the business to an architecture while inference might shift toward CPUs or different custom silicon. Frye recommends Nitesh Agrawal’s Lambda Labs presentation for the operational details, while acknowledging the provider’s commercial interest in making that work look difficult.

1:17:511:18:07
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:17:51 · section reference included

Why loading a weight once should serve many inputs

The tensor-to-tensor workload reduces to two tasks: load numbers from memory and perform arithmetic on them. The imbalance between those tasks explains the appeal of GPUs. For a small input batch, a weight may be fetched only to perform very little arithmetic before moving on. Reading the value is expensive relative to the operation. To use the hardware economically, load a weight and reuse it across multiple inputs.

This reuse happens as data moves from GPU memory toward the places where computation occurs. Applying the same operations to many inputs becomes parallel linear algebra, closely related to the workload graphics hardware was designed to accelerate. Optimized CUDA kernels can overlap memory transfers and computation, but overlap does not eliminate a mismatch between how fast values arrive and how fast arithmetic units can consume them.

Frye uses an A100 example with 312 TFLOP/s of two-byte tensor-core arithmetic and approximately 1.5 TB/s of memory bandwidth. In the idealized weight-multiplication model, loading one two-byte weight and using it for one multiply-add per input gives approximately one FLOP per byte per batch element. The balance point is therefore:

machine balance=312×1012 FLOP/s1.5×1012 byte/s≈208 FLOP/byte,arithmetic intensity≈B FLOP/byte,Bbalance≈208.\begin{aligned} \text{machine balance} &= \frac{312 \times 10^{12}\ \mathrm{FLOP/s}} {1.5 \times 10^{12}\ \mathrm{byte/s}} \\ &\approx 208\ \mathrm{FLOP/byte}, \\ \text{arithmetic intensity} &\approx B\ \mathrm{FLOP/byte}, \\ B_{\mathrm{balance}} &\approx 208. \end{aligned}

This explains the initially near-flat runtime curve: adding inputs can fill otherwise underused arithmetic capacity while roughly the same weights are being loaded. Frye’s roughly 200-element crossover comes from this analytical model, not a universally optimal serving batch size. Carol Chen’s Transformer Inference Arithmetic develops the calculation under 16-bit, 40 GB A100 assumptions and compares a fuller account with measured timings. H100 increases both capabilities while making the compute-to-bandwidth imbalance still more pronounced.

Batching amortizes weight movement across useful work. A busy service can collect inputs from many users and process them together. A personal device may have only one user issuing requests too slowly to fill such a batch. That changes the desirable hardware balance: a CPU or another accelerator can become more attractive when there is insufficient independent work to exploit a GPU’s arithmetic throughput.

1:20:191:20:33
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:20:19 · section reference included

A latency deadline limits how long you can batch

A low-traffic API cannot wait indefinitely for enough requests to reach an efficient batch size. Its latency objective may force it to run small batches, which changes the balance between computation, memory bandwidth, and memory capacity. Frye uses cached attention keys and values as an example of that trade-off: caching saves recomputation but occupies memory and creates memory traffic. His discussion follows a conditional analysis of when that exchange is favorable; being memory-bound alone is not a rule to disable the KV cache.

WorkloadPrimary constraintBatching consequence
Single-user or low-load serviceFew arrivals and a response deadlineSmall batches may be unavoidable
Interactive shared APIThroughput plus per-request latencyBalance reuse against waiting time
Offline batch jobTotal work completedFavor larger batches that fit

An overnight job can prioritize throughput without making a user wait for each individual answer. Frye uses training as a familiar throughput-oriented analogy. For an interactive API, however, the batch that maximizes device utilization may violate the product’s response-time requirement.

Annotated quotation discusses low-load APIs, per-request latency and mass inference, with colored labels for End-User, Over a Network and Batch Jobs.Open full source frame
Batch size and caching choices depend on latency requirements.
1:25:431:26:00
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:25:43 · section reference included

Specialized chips need a sufficiently stable workload

Would TPUs offer a better balance? Frye recalls seeing improvements around 30%, rather than an order-of-magnitude change, but has not used TPUs seriously and supplies no specific benchmark or metric for that figure. An attendee adds that operational and power efficiency may matter more than raw throughput. Frye accepts the distinction: FLOP utilization and total arithmetic do not account for every economically important property of an accelerator. Switching hardware also means adopting a different software stack.

Custom chips work especially well when the workload stays fixed in precise detail. Blockchain mining provides the comparison: an ASIC can target a computation whose rules change only through consensus. Neural-network workloads vary with architecture, model size, and tensor shape. A design tailored to one arrangement may fit the next model less well. The GPU’s flexibility is valuable while those details keep changing.

The Cerebras discussion separates memory capacity from memory bandwidth. An attendee contrasts a 40 GB memory figure with an 80 GB A100 and mentions Jais, an Arabic model trained with Cerebras. Frye emphasizes very high memory bandwidth, while accepting that capacity can still prevent a model from fitting. Networking chips can increase aggregate storage, and a better memory-to-arithmetic balance may help small batches, but a wafer-scale system is not a solution that fits inside a phone. He suggests that convergence on transformer architectures could make specialization more attractive over time.

Adding GPUs increases aggregate memory capacity, memory bandwidth, and arithmetic throughput. Ideally, bandwidth and FLOPs scale together, leaving their ratio similar while the extra memory makes larger batches possible. Achieving that scaling requires an implementation that handles communication effectively. For a service, the resulting GPU group helps determine the size of a serving replica—the rough deployment-unit idea Frye connects to Kubernetes pod sizing. The correct unit depends on the actual model’s memory footprint and efficient batch size.

1:27:001:27:07
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:27:00 · section reference included

Check the hardware model against a trace

First-principles arithmetic tells you where to look; a profiler tells you what actually happened. Frye describes a vLLM implementation that launched a CUDA kernel for every batch element. Replacing those launches with a single launch reportedly increased throughput by approximately 30–33%; the workshop does not identify the workload or exact change. The mechanism is still instructive: a supposedly batched operation can retain per-element overhead that prevents the intended amortization.

Execution traces expose that problem through repeated kernel launches and gaps where the GPU is idle. Frye recommends inspecting the timeline directly, alongside statistical profiles, because a long sequence of tiny operations or a large idle span is often easier to recognize visually. Memory traces deserve equal attention when memory is the central constraint. His examples use PyTorch; other inference runtimes may require generic GPU, CPU, and memory-profiling tools.

Library selection must retain the workload assumptions too. Frye points to Hamel Husain’s Optimizing latency comparison specifically for batch size one. It evaluates Llama 2 7B implementations, mostly on A6000 hardware, using nine prompts, warmup, and a 200-token cap; model variants and generated lengths differ. vLLM’s community momentum does not make it the automatic winner for that case. MLC and CTranslate2 are alternatives to investigate when single-request latency, rather than shared-service throughput, is the target.

1:33:461:34:00
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:33:46 · section reference included

A container cannot hide every GPU dependency

Optimizing an individual workload is only part of building an inference platform. GPU containerization has improved, but model weights create unusually large artifacts when packaged in the application layer. Moving them into remote storage reduces image size at the cost of making the deployment less self-contained. The startup path now depends on both the container and the availability and transfer of its model data.

Hardware choices also leak upward into application decisions. A GPU’s compute-to-memory-bandwidth ratio affects the batch size at which execution changes from memory-bound to compute-bound. Change the GPU and an otherwise identical container may need a different batching policy. Frye’s driver anecdote makes the isolation boundary visible: a CUDA 11.2 container did not make the host’s reported CUDA 12.0 driver capability disappear. Host drivers and container libraries are different layers, and their interaction affects the features available to PyTorch and Hugging Face Transformers.

When inference performance is expensive enough to optimize aggressively, these abstraction boundaries matter. Containers remain useful, but they do not erase accelerator compatibility or the consequences of mixing GPU, Arm, and x86 machines. A heterogeneous cluster therefore demands more operational expertise than a uniform fleet whose hardware can be treated as interchangeable.

1:36:111:36:19
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:36:11 · section reference included

Choose the operational responsibility you want to own

Kubernetes can orchestrate inference services, but it introduces another layer between model behavior and NVIDIA hardware dependencies. Frye relays a useful piece of MLOps advice: if serving inference yourself is important enough to choose over API calls, consider making MLOps a core company competency. Building from KubeRay or Seldon Core can provide control, but that control comes with responsibility for the system’s behavior.

Managed platforms offer several places to draw the boundary:

ApproachExamplesWhat the platform supplies
Build around KubernetesKubeRay, Seldon CoreServing components you operate
End-to-end cloud ML platformAmazon SageMaker, Google Vertex AIA broader integrated service
Managed serving toolkitBentoCloud, Seldon, AnyscaleManaged BentoML, Seldon Core, or Ray

Existing cloud agreements can make one option more practical than another. Frye is careful about his own experience: he has explored Ray Serve and Anyscale but does not present an exhaustive platform comparison. Ray’s use in foundation-model training and cluster management gives it relevant expertise and ecosystem support; it motivates considering Ray for inference rather than proving it is the best choice. The final decision is how much of this hardware, runtime, and operational stack your team needs to understand, tune, and keep running itself.

1:39:311:39:44
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:39:31 · section reference included

Resources

From the talk

Transcript download
Full transcript

Loading transcript…