AI Engineer World's Fair 2024

A Practical Guide to Efficient AI

Loading slides…

Read the talk

A Practical Guide to Efficient AI

Getting an AI demo into customers’ hands means fitting real resource budgets. Small models and quantization offer two practical starting points, with evaluation deciding what is safe to deploy.

From a talk by Shelby Heinecke

Before you start: Basic familiarity with language models, model weights, fine-tuning and inference will help; no prior quantization experience is required.

From a working demo to a service customers can use

A new model demonstrates an impressive capability. What does it take to turn that prototype into something customers can use at scale? The demonstration establishes that a task is possible; production also demands that the system fit its operating budget. Efficiency helps close the gap between capability and deployment. That is the practical problem behind Shelby Heinecke’s guide.

Heinecke leads an AI research team delivering LLM solutions to Salesforce’s data platform, the foundation beneath its products. Alongside that production work, the team investigates agents, language models and on-device AI. At the time of the talk, she reports more than 15 research papers and more than six open-source repositories from her team. Her own research background includes learning algorithms designed to use samples and communication efficiently.

That work sits inside a longer enterprise deployment history. Heinecke reports ten years of Salesforce AI deployment, with its research team founded in 2014, more than 300 AI patents and more than 227 research papers. Trust is part of the deployment requirement: she describes participation in six ethical AI councils and the White House commitment to trusted AI. The timeline then moves into the 2022–2023 wave of code-generation products, Service GPT, Einstein GPT and Tableau GPT. At Fortune 500 scale, the resources needed to serve these products become a central engineering concern.

Slide titled “10 years of Trusted AI Deployment” with a rising timeline, product milestones, a rocket, and counts of ethical AI councils, research papers, and patents.Open full source frame
Salesforce’s timeline of AI deployment and trust commitments.
0:290:40
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:29 · section reference included

Every deployment has a resource budget

The constraint changes with the deployment environment, but it never disappears.

EnvironmentMain constraint
Public or private cloudOngoing GPU, CPU and disk costs
On-premises clusterOperating costs and a fixed supply of GPUs
Phones and other small devicesTighter hardware and resource limits

In the cloud, resource consumption becomes cost to serve. On premises, paying for infrastructure does not remove the capacity ceiling: the model must work with the GPUs available.

Apple’s then-recent announcement of an LLM for newer devices gives Heinecke a concrete reason to look beyond servers. Phones suggest a wider set of possibilities for language and multimodal models: tablets, personal laptops and edge devices. Those options also impose tighter budgets. A model that is affordable in the cloud may still be too demanding for a small device, so the deployment target needs to influence model design and selection.

4:134:21
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:13 · section reference included

Five places to improve efficiency

Efficiency is not a single optimization applied just before release. Heinecke separates it into five dimensions, each offering a different place to intervene:

  1. Architecture. Start with a small model where possible. Other choices include mixture-of-experts architectures and, when building a model, efficient attention mechanisms.
  2. Pre-training. If training from scratch is necessary, mixed precision and scaling methods help address the substantial GPU expense.
  3. Fine-tuning. Parameter-efficient methods such as LoRA and QLoRA avoid updating every base-model weight. They concentrate trainable changes in a much smaller set of parameters.
  4. Inference. Once the model is trained and adapted, post-training quantization and speculative decoding offer ways to reduce serving costs.
  5. Prompting. Prompt tokens consume memory and affect latency. Concise templates and prompt compression reduce the amount of input the model must process.

These are separate opportunities, so choosing an efficient architecture does not eliminate the need to examine training, serving or prompts.

The two practical directions developed here are small-model selection and post-training quantization. The first changes the model you begin with; the second changes how an already-trained model’s weights are represented for inference.

5:385:50
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:38 · section reference included

Fewer weights expand the deployment options

The resource implications of model size become clear when looking at the large models that helped define the preceding wave of releases. Heinecke’s examples are PaLM at 540 billion parameters, BLOOM at 176 billion and YaLM at 100 billion. Parameters are the network’s weights: they occupy storage and participate in computation. Large weight sets therefore affect pre-training, fine-tuning and serving, not just the size of a downloaded checkpoint.

For this discussion, Heinecke defines a small LLM as one with 13 billion parameters or fewer. The sizing slide puts that category beside the much larger models, making the difference in scale visible.

Bubble diagram labeled PaLM 540B, BLOOM 176B, YaLM 100B, and Small LLMs below 13B, under “LLM Sizing by Parameters.”Open full source frame
Parameter counts for PaLM, BLOOM, YaLM, and small LLMs.

Fewer weights generally mean lower RAM and disk requirements, less GPU or CPU work, and faster fine-tuning. A forward pass also has less work to do, which can lower latency. Together, these benefits widen the set of places a model can run: cloud and on-premises systems remain options, while sufficiently small models can also fit phones, personal laptops and edge devices. The next question is whether that smaller model can perform the task well enough.

8:058:19
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:05 · section reference included

Small models become compelling on specific tasks

The first example is Phi-3-mini, the 3.8-billion-parameter member of the Phi-3 family. Heinecke presents a technical-report comparison in which it outperforms a 7B model on the displayed evaluation. The useful implication is that parameter count alone does not determine capability: a much smaller model can be a serious candidate. The comparison is specific to that evaluation, rather than a claim that it exceeds every larger model.

The next example goes below a billion parameters. MobileLLM is presented in its 350-million-parameter configuration, small enough to make mobile and edge deployment especially interesting. Heinecke reports that, after fine-tuning, MobileLLM-350M performs on par with a 7B model on the referenced tasks. Here, the relevant setting is specialized API calling, not general language-model parity. Fine-tuning is part of the result: the question is what the small model can do after adaptation to its intended job.

Function calling provides an even more explicit specialization example. Octopus v2 adapts Gemma 2B for Android tasks. Heinecke reports that the fine-tuned Octopus model outperforms GPT-4 and a Llama 7B model on the referenced Android function-calling tasks. The scope matters: this is a model trained for a particular interface and workload. It shows why a deployment decision should compare candidates on the job they will actually perform, including the benefit of task-specific fine-tuning.

10:1810:25
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:18 · section reference included

Reduce the bits used to represent each weight

After model selection comes an inference optimization: post-training quantization. Quantization is an established technique; the application here is to language and multimodal models. Instead of retaining every weight as a 32- or 16-bit floating-point value, use a lower-precision representation such as 8 or 4 bits. The operation reduces representational precision, rather than reducing the number of model parameters.

The storage arithmetic explains the appeal. Moving from 32 to 4 bits reduces the nominal bits per weight eightfold. The following Python calculation lets you inspect that weight-storage budget for a model with seven billion parameters:

python

parameter_count = 7_000_000_000

for bits_per_weight in (32, 16, 8, 4):
    weight_bytes = parameter_count * bits_per_weight // 8
    weight_gb = weight_bytes / 1_000_000_000
    print(f"{bits_per_weight:2d}-bit: {weight_gb:.1f} GB of weight data")

This calculates weight data only. A deployed model also needs runtime memory, and a quantized representation can carry additional metadata, so the bit ratio is not an identical reduction in total system memory.

Heinecke’s displayed Llama examples—7B, 13B and 70B—show reduced disk usage after 4-bit quantization. She then turns to a separate multimodal-model study: the displayed comparison reports lower time to first token at 4 bits than at the original 16 bits. Time to first token measures how long a request waits before generation begins, rather than the rate of generating the entire response. Lower precision can help latency, but the gain depends on the model, hardware and runtime. Once storage and latency improve, the remaining question is whether the model still performs its task well.

12:1112:17
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

12:11 · section reference included

Accuracy determines how far quantization can go

The vision-language evaluation compares multimodal models at their original 16-bit precision with 8- and 4-bit quantization. Heinecke reports essentially unchanged task performance through 4 bits in the displayed vision-language evaluation. In that example, the storage and latency benefits come without a meaningful movement in the task score.

The next part of the graph establishes the limit: performance drops at 3 bits in the displayed comparison. This is why a favorable 4-bit result is a reason to test quantization, not a reason to skip testing. Evaluate the quantized model itself. The appropriate precision is the one that preserves the behavior needed for the deployment, and quantization can be considered for conventional ML models as well as language and multimodal models.

Slide titled “Efficient AI with Quantization” pairs resource and latency tables with a graph comparing model scores at 16-bit, 8-bit, 4-bit, and 3-bit quantization.Open full source frame
Quantization results show relatively stable scores through 4-bit and declines at 3-bit.
14:0314:16
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

14:03 · section reference included

Start with existing quantization tools and releases

You may not need to produce the quantized weights yourself. Heinecke points first to llama.cpp, describing a range from 16-bit representations down to approximately 1.5 bits. She recommends checking Hugging Face for an existing compatible quantized release before doing the conversion, and mentions Python and Java wrappers for integration.

The second route is ONNX Runtime, a runtime with a history beyond LLMs and support across many programming languages. Heinecke highlights its 8-bit quantization. These are the capabilities discussed in the recording; the linked project documentation describes current tooling, whose formats and APIs have continued to evolve. The choice of route should fit both the model and the environment where it will run.

15:1615:25
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

15:16 · section reference included

Test quality, safety and behavior on the device

Obtaining a quantized checkpoint is not the final deployment step. Heinecke introduces Mobile AI Bench, a recently released open-source framework from her team, to make evaluation more systematic before a model reaches customers. Its purpose is to confirm that the quantized model still behaves as expected.

The evaluation covers text tasks, trust and safety, and vision-language tasks. Trust and safety deserve explicit attention: preserving a task score is insufficient if quantization degrades safety behavior. For on-device deployment, the described iOS app adds latency and hardware-usage measurements, including battery drainage. Quality evaluation and device profiling answer different questions—whether the model remains suitable for its job, and whether it can do that job within the device’s operating budget.

That brings the deployment problem back to all five dimensions: architecture, pre-training, fine-tuning, inference and prompting. Small models and quantization are practical entry points, while the final choice depends on measured quality and resource use in the environment customers will actually use.

16:0916:18
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

16:09 · section reference included

Resources

From the talk

  • The original report describes the 3.8B-parameter Phi-3-mini, its training approach and benchmark comparisons.

  • MobileLLMPaper11:11

    Sub-billion-parameter architectures and fine-tuned API-calling evaluations, including comparisons with LLaMA-v2 7B.

  • Octopus v2Paper11:38

    A specialized small model for function calling, evaluated on Android APIs against larger model baselines.

  • Evaluation tools and mobile app setup instructions for testing model quality and resource consumption.

  • C/C++ model inference project with tools and documentation for running quantized models.

  • Evaluation methodology and results for quantized language and multimodal models, including quality, safety, latency and device resource usage.

Updates since the talk

Transcript download
Full transcript

Loading transcript…