AI Engineer World's Fair 2024
A Practical Guide to Efficient AI
Loading slides…
Read the talk
A Practical Guide to Efficient AI
Getting an AI demo into customers’ hands means fitting real resource budgets. Small models and quantization offer two practical starting points, with evaluation deciding what is safe to deploy.
From a talk by Shelby Heinecke
Before you start: Basic familiarity with language models, model weights, fine-tuning and inference will help; no prior quantization experience is required.
From a working demo to a service customers can use
A new model demonstrates an impressive capability. What does it take to turn that prototype into something customers can use at scale? The demonstration establishes that a task is possible; production also demands that the system fit its operating budget. Efficiency helps close the gap between capability and deployment. That is the practical problem behind Shelby Heinecke’s guide.
Heinecke leads an AI research team delivering LLM solutions to Salesforce’s data platform, the foundation beneath its products. Alongside that production work, the team investigates agents, language models and on-device AI. At the time of the talk, she reports more than 15 research papers and more than six open-source repositories from her team. Her own research background includes learning algorithms designed to use samples and communication efficiently.
That work sits inside a longer enterprise deployment history. Heinecke reports ten years of Salesforce AI deployment, with its research team founded in 2014, more than 300 AI patents and more than 227 research papers. Trust is part of the deployment requirement: she describes participation in six ethical AI councils and the White House commitment to trusted AI. The timeline then moves into the 2022–2023 wave of code-generation products, Service GPT, Einstein GPT and Tableau GPT. At Fortune 500 scale, the resources needed to serve these products become a central engineering concern.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Every deployment has a resource budget
The constraint changes with the deployment environment, but it never disappears.
| Environment | Main constraint |
|---|---|
| Public or private cloud | Ongoing GPU, CPU and disk costs |
| On-premises cluster | Operating costs and a fixed supply of GPUs |
| Phones and other small devices | Tighter hardware and resource limits |
In the cloud, resource consumption becomes cost to serve. On premises, paying for infrastructure does not remove the capacity ceiling: the model must work with the GPUs available.
Apple’s then-recent announcement of an LLM for newer devices gives Heinecke a concrete reason to look beyond servers. Phones suggest a wider set of possibilities for language and multimodal models: tablets, personal laptops and edge devices. Those options also impose tighter budgets. A model that is affordable in the cloud may still be too demanding for a small device, so the deployment target needs to influence model design and selection.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Five places to improve efficiency
Efficiency is not a single optimization applied just before release. Heinecke separates it into five dimensions, each offering a different place to intervene:
- Architecture. Start with a small model where possible. Other choices include mixture-of-experts architectures and, when building a model, efficient attention mechanisms.
- Pre-training. If training from scratch is necessary, mixed precision and scaling methods help address the substantial GPU expense.
- Fine-tuning. Parameter-efficient methods such as LoRA and QLoRA avoid updating every base-model weight. They concentrate trainable changes in a much smaller set of parameters.
- Inference. Once the model is trained and adapted, post-training quantization and speculative decoding offer ways to reduce serving costs.
- Prompting. Prompt tokens consume memory and affect latency. Concise templates and prompt compression reduce the amount of input the model must process.
These are separate opportunities, so choosing an efficient architecture does not eliminate the need to examine training, serving or prompts.
The two practical directions developed here are small-model selection and post-training quantization. The first changes the model you begin with; the second changes how an already-trained model’s weights are represented for inference.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Fewer weights expand the deployment options
The resource implications of model size become clear when looking at the large models that helped define the preceding wave of releases. Heinecke’s examples are PaLM at 540 billion parameters, BLOOM at 176 billion and YaLM at 100 billion. Parameters are the network’s weights: they occupy storage and participate in computation. Large weight sets therefore affect pre-training, fine-tuning and serving, not just the size of a downloaded checkpoint.
For this discussion, Heinecke defines a small LLM as one with 13 billion parameters or fewer. The sizing slide puts that category beside the much larger models, making the difference in scale visible.
Fewer weights generally mean lower RAM and disk requirements, less GPU or CPU work, and faster fine-tuning. A forward pass also has less work to do, which can lower latency. Together, these benefits widen the set of places a model can run: cloud and on-premises systems remain options, while sufficiently small models can also fit phones, personal laptops and edge devices. The next question is whether that smaller model can perform the task well enough.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Small models become compelling on specific tasks
The first example is Phi-3-mini, the 3.8-billion-parameter member of the Phi-3 family. Heinecke presents a technical-report comparison in which it outperforms a 7B model on the displayed evaluation. The useful implication is that parameter count alone does not determine capability: a much smaller model can be a serious candidate. The comparison is specific to that evaluation, rather than a claim that it exceeds every larger model.
The next example goes below a billion parameters. MobileLLM is presented in its 350-million-parameter configuration, small enough to make mobile and edge deployment especially interesting. Heinecke reports that, after fine-tuning, MobileLLM-350M performs on par with a 7B model on the referenced tasks. Here, the relevant setting is specialized API calling, not general language-model parity. Fine-tuning is part of the result: the question is what the small model can do after adaptation to its intended job.
Function calling provides an even more explicit specialization example. Octopus v2 adapts Gemma 2B for Android tasks. Heinecke reports that the fine-tuned Octopus model outperforms GPT-4 and a Llama 7B model on the referenced Android function-calling tasks. The scope matters: this is a model trained for a particular interface and workload. It shows why a deployment decision should compare candidates on the job they will actually perform, including the benefit of task-specific fine-tuning.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Reduce the bits used to represent each weight
After model selection comes an inference optimization: post-training quantization. Quantization is an established technique; the application here is to language and multimodal models. Instead of retaining every weight as a 32- or 16-bit floating-point value, use a lower-precision representation such as 8 or 4 bits. The operation reduces representational precision, rather than reducing the number of model parameters.
The storage arithmetic explains the appeal. Moving from 32 to 4 bits reduces the nominal bits per weight eightfold. The following Python calculation lets you inspect that weight-storage budget for a model with seven billion parameters:
python
parameter_count = 7_000_000_000
for bits_per_weight in (32, 16, 8, 4):
weight_bytes = parameter_count * bits_per_weight // 8
weight_gb = weight_bytes / 1_000_000_000
print(f"{bits_per_weight:2d}-bit: {weight_gb:.1f} GB of weight data")
This calculates weight data only. A deployed model also needs runtime memory, and a quantized representation can carry additional metadata, so the bit ratio is not an identical reduction in total system memory.
Heinecke’s displayed Llama examples—7B, 13B and 70B—show reduced disk usage after 4-bit quantization. She then turns to a separate multimodal-model study: the displayed comparison reports lower time to first token at 4 bits than at the original 16 bits. Time to first token measures how long a request waits before generation begins, rather than the rate of generating the entire response. Lower precision can help latency, but the gain depends on the model, hardware and runtime. Once storage and latency improve, the remaining question is whether the model still performs its task well.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Accuracy determines how far quantization can go
The vision-language evaluation compares multimodal models at their original 16-bit precision with 8- and 4-bit quantization. Heinecke reports essentially unchanged task performance through 4 bits in the displayed vision-language evaluation. In that example, the storage and latency benefits come without a meaningful movement in the task score.
The next part of the graph establishes the limit: performance drops at 3 bits in the displayed comparison. This is why a favorable 4-bit result is a reason to test quantization, not a reason to skip testing. Evaluate the quantized model itself. The appropriate precision is the one that preserves the behavior needed for the deployment, and quantization can be considered for conventional ML models as well as language and multimodal models.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Start with existing quantization tools and releases
You may not need to produce the quantized weights yourself. Heinecke points first to llama.cpp, describing a range from 16-bit representations down to approximately 1.5 bits. She recommends checking Hugging Face for an existing compatible quantized release before doing the conversion, and mentions Python and Java wrappers for integration.
The second route is ONNX Runtime, a runtime with a history beyond LLMs and support across many programming languages. Heinecke highlights its 8-bit quantization. These are the capabilities discussed in the recording; the linked project documentation describes current tooling, whose formats and APIs have continued to evolve. The choice of route should fit both the model and the environment where it will run.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Test quality, safety and behavior on the device
Obtaining a quantized checkpoint is not the final deployment step. Heinecke introduces Mobile AI Bench, a recently released open-source framework from her team, to make evaluation more systematic before a model reaches customers. Its purpose is to confirm that the quantized model still behaves as expected.
The evaluation covers text tasks, trust and safety, and vision-language tasks. Trust and safety deserve explicit attention: preserving a task score is insufficient if quantization degrades safety behavior. For on-device deployment, the described iOS app adds latency and hardware-usage measurements, including battery drainage. Quality evaluation and device profiling answer different questions—whether the model remains suitable for its job, and whether it can do that job within the device’s operating budget.
That brings the deployment problem back to all five dimensions: architecture, pre-training, fine-tuning, inference and prompting. Small models and quantization are practical entry points, while the final choice depends on measured quality and resource use in the environment customers will actually use.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
The original report describes the 3.8B-parameter Phi-3-mini, its training approach and benchmark comparisons.
Sub-billion-parameter architectures and fine-tuned API-calling evaluations, including comparisons with LLaMA-v2 7B.
A specialized small model for function calling, evaluated on Android APIs against larger model baselines.
Evaluation tools and mobile app setup instructions for testing model quality and resource consumption.
C/C++ model inference project with tools and documentation for running quantized models.
Further reading
Evaluation methodology and results for quantized language and multimodal models, including quality, safety, latency and device resource usage.
Updates since the talk
Current guidance for quantizing ONNX models and assessing the resulting accuracy and performance.
Full transcript
Loading transcript…