AI Engineer World's Fair 2024

Insights from Snorkel AI running Azure AI Infrastructure

Loading slides…

Read the talk

Insights from Snorkel AI running Azure AI Infrastructure

Enterprise model quality depends on the right domain data—and on infrastructure that can turn that data into repeated training, evaluation and synthesis cycles.

From a talk by Lachlan Ainley and Humza Iqbal

Before you start: Familiarity with model fine-tuning, GPU training and the distinction between training and inference will help you follow the infrastructure discussion.

What gets a bank’s model ready to deploy?

What does it take to make a foundation model reliable enough for a bank or insurer? Humza Iqbal works on that problem as an applied research scientist on Snorkel’s computer vision team, fine-tuning foundation models for enterprise use cases. The company’s name invites a detour: after joining, he discovered that snorkel.com belonged to a hot-tub company. Snorkel’s actual business is developing data for enterprise AI.

For its Fortune 500 customers, quality, latency and cost all constrain deployment. A model must also be reliable and accurate on the work the organization actually needs it to do. Off-the-shelf Claude, GPT-4 and Gemini can get partway there, Iqbal explains, but task-specific fine-tuning data supplies the missing final mile.

Slide charts domain complexity against quality requirements, contrasting out-of-the-box AI with development required, alongside a 77.5% fine-tuning statistic.Open full source frame
Out-of-the-box LLMs face enterprise quality, latency and cost requirements.

Access to enterprise knowledge is only the beginning. Retrieval-augmented generation, or RAG, can connect a model to information absent from its pretraining. But finding and maintaining the right data remains essential. A large instruction-tuning dataset may contain little about the particular bank policy or industry rule the model must learn. Dataset size does not substitute for that required slice of information.

Iqbal places this work in Snorkel’s longer history: origins in the Stanford AI Lab, roughly a decade of data-development experience, and research spanning prompting, RAG and model architectures. He describes the company as a pioneer in data development for LLMs, extending an emphasis on data that predates today’s foundation models.

0:160:43
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:16 · section reference included

Scale expert knowledge, then evaluate the actual task

The research question is how enterprises can best develop data for custom models. Snorkel approaches it through three connected requirements:

  • Keep subject-matter experts involved. Industry specialists know the details that model feedback must capture; the workflow should maximize the value of their limited time.
  • Make development programmatic, scalable and auditable. Expert input needs a way to influence more data than experts could manually review themselves.
  • Evaluate continuously on domain-specific, dynamic benchmarks. The evaluation must remain relevant to the organization’s task as its models and data change.

Broad rankings such as LMSYS provide a general view of model capability. They do not establish whether a model handles a bank’s specific requirements. Strong grade-school math performance answers a different question.

3:534:06
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:53 · section reference included

Evaluate long context and align models to the domain

Fine-grained evaluation starts by asking what a test actually measures. In a needle-in-a-haystack test, an evaluator inserts a sentence into a collection of Paul Graham essays and asks the model to find it. Success demonstrates retrieval under those conditions; it need not describe how the same model handles long context in another domain. Iqbal’s research emphasis is to break performance down domain by domain, then use those findings to improve long-context behavior.

Enterprise alignment adds another requirement: outputs must comply with company goals and regulations. A fluent answer can still create an organizational problem. Multimodal alignment faces a related gap because models trained on public data can underperform in specialized domains. Snorkel is exploring how large vision-language models, or LVLMs, can generate synthetic data without manual annotation to train downstream models. Domain-specific generation feeds model training, creating a repeated cycle of data development and learning.

5:175:32
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:17 · section reference included

Optimize the whole infrastructure path

Those research cycles depend on more than accelerator performance. Microsoft’s Lachlan Ainley describes Azure’s approach as optimization across the entire system. At the time of the talk, he cites more than 300 datacenters worldwide. Inside that footprint, the choices extend through VM host CPUs and throughput, accelerators from AMD and NVIDIA alongside Microsoft’s Maia, I/O topology, and networking. Each layer must support the workload rather than constrain the next one.

Microsoft Azure slide with an end-to-end systems optimization arrow above Datacenter, CPU, AI accelerator, IO and Network icons.Open full source frame
Azure presents end-to-end systems optimization across five infrastructure layers.

Ainley describes a feedback process from large training deployments to customer infrastructure: experience supporting organizations such as OpenAI and Mistral at supercomputing scale informs the systems Azure makes available more broadly. Snorkel’s deployment is one concrete use of that infrastructure.

6:537:25
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:53 · section reference included

PyTorch, Horovod and a shared filesystem

Snorkel’s distributed training stack separates model computation, communication and storage:

LayerComponentResponsibility
Model frameworkPyTorchTraining computation
Multi-node communicationHorovodExchange gradients across workers
ComputeAzure VMs with A100 or H100 GPUsRun model workloads
Shared storageNetwork file system, or NFSSupply data and store checkpoints

All the machines access the same NFS as though it were a local filesystem. That gives workers a common location for reading training data and writing model checkpoints, while Horovod handles communication between the machines.

The projects range from a single node to dozens of nodes. They include models aligned using Direct Preference Optimization, or DPO, with instruction-response preference datasets. The most compute-intensive work is large-scale multimodal training and inference, using dozens of GPUs. These different workloads make both distributed throughput and flexible capacity important.

8:569:10
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:56 · section reference included

Find out why the GPUs are waiting

More nodes are useful only if they fit the training configuration. While training CLIP-based models, Iqbal needed enough nodes to support the desired batch size. Too many nodes could force an excessively large batch or leave some of the allocated compute underused. Choose capacity around the intended training workload, not simply the largest available cluster.

Next comes locality. Iqbal’s deliberately obvious example puts data in US West and compute in Asia: the distance introduces a communication problem before model computation even begins. Within distributed training, exchanging gradients between model replicas can also become a bottleneck. Data placement and communication between workers both matter.

Storage can produce the same visible symptom through a different mechanism. Data loading runs asynchronously alongside the model’s forward and backward passes. If the NFS cannot read data quickly enough, that overlap is insufficient: the GPU finishes its work and waits for the next input instead of continuing to compute.

Start the investigation with nvidia-smi. A simple command keeps its status display updating:

bash

nvidia-smi -l 1

Its GPU-utilization counter measures sampled time with kernels executing, not achieved FLOPS or overall training efficiency. Use it as an initial signal, then investigate:

  1. Look for low GPU utilization while the job should be computing.
  2. For multi-node jobs, test networking and communication between workers.
  3. For single-node jobs, inspect data loading first. Iqbal considers it a likely cause in that setting, rather than a diagnosis the utilization counter can establish on its own.
11:0511:22
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:05 · section reference included

Research needs capacity that can change

Reliability, flexibility and manageability shape whether a research team can use its infrastructure effectively. Snorkel’s experiments sometimes need every node and sometimes very few. That pattern continues over a long period. Some providers offered substantial compute within a fixed window—perhaps one or two months—but that did not fit a team doing ongoing, intermittent experimentation rather than one bounded training run.

Iqbal reports that dedicated Azure VMs let the team adjust capacity on demand, with dependable operation and no significant issues in its experience. He also reports good NFS throughput and values being able to resize storage as requirements change. Unexpected data growth requires more capacity; overprovisioned storage should shrink so the team does not keep paying for space it does not need.

Ease of use matters after the initial setup, too. Iqbal configured Azure for the team and wanted colleagues to work without repeatedly asking him for help. He credits clear documentation and a straightforward process with making the environment usable once configured, reducing support requests and avoiding paging.

13:0013:10
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:00 · section reference included

Compare hardware at approximately equal cost

When moving from A100s to H100s, Snorkel asked an economic question: what training and inference performance could each configuration deliver for the same spend? Iqbal reports that two H100s cost approximately the same as four A100s in Snorkel’s comparison and performed better on both training and inference.

The appeal was the simplicity of the change. Inference optimization can involve KV caches, partial KV caches and speculative decoding; changing hardware offered Snorkel a straightforward improvement. Faster inference also mattered upstream of the final model: it let the team process more synthetic data and iterate faster, supporting the path toward higher downstream model accuracy.

The completed slide shows the measurements below. Training is expressed as time per step, where lower is better; inference is expressed as iterations per second, where higher is better.

ConfigurationTraining time per stepInference throughput
2× H1006.80 s22.37 it/s
4× A10010.88 s14.32 it/s

The result supports Snorkel’s reported improvement for this experiment. The model, precision, exact batch sizes and pricing basis are not specified, so these figures are not enough to reproduce its cost comparison.

Table lists 2x H100 at 6.80s training time per step and 22.37 inference it/s, versus 4x A100 at 10.88s and 14.32 it/s. Below, arrows connect faster inference, more synthetic data and higher end-model accuracy.Open full source frame
H100 and A100 comparison links faster inference to more synthetic data and higher end-model accuracy.
15:1015:34
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

15:10 · section reference included

Separate batch-size effects from hardware effects

An audience member asks why inference appears to take longer than training. Iqbal tentatively attributes that to a larger inference batch: training typically needs a smaller batch because backpropagation consumes additional memory. The slide’s columns, however, use different units—seconds per training step and inference iterations per second—so their numerical magnitudes do not directly compare elapsed time. His answer should be read as an explanation of the workload setup, not a conversion between those columns.

A follow-up raises batch-size comparability across the two hardware configurations. Iqbal says the settings were held fixed across hardware, explicitly confirming that inference used the same batch size on H100s and A100s. That distinguishes two questions: training and inference can use different batches, while the hardware comparison for a given task keeps its batch setting constant.

17:0217:13
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

17:02 · section reference included

Match evolving accelerators to the workload

Ainley closes Azure’s side of the discussion by returning to optimization at every layer. In the recording’s 2024 context, he describes Maia as serving internal Microsoft 365 workloads, AMD’s MI300X with high-bandwidth memory as newly announced, A100s and H100s as existing options, and Blackwell adoption as planned. This is the roadmap presented at the time, rather than a current availability guide.

Five labeled chip images show AMD MI300 X, Azure Maia, NVIDIA A100, NVIDIA H100 and NVIDIA Blackwell beneath a headline about the pace of hardware innovation.Open full source frame
Hardware innovation spans AMD MI300 X, Azure Maia and NVIDIA accelerators.

He characterizes NVIDIA’s pace as approaching two releases a year, compared with his tentative recollection of one every two years previously. The practical direction is to absorb improvements in silicon while continuing to choose the right GPU for each workload. Snorkel’s ability to do the compared work with fewer GPUs illustrates why those hardware changes can matter to users of the infrastructure.

18:1818:36
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

18:18 · section reference included

Turn preference signals and expertise into better data

Snorkel’s next research directions return to the original deployment problem: improving data so generative AI prototypes can reach production.

  • Preference-driven synthesis and curation: programmatically use preference signals to generate and select data.
  • Scalable expert participation: let subject-matter experts contribute through rationales and custom taxonomies.
  • Domain-specific multimodal retrieval: develop better retrieval algorithms and evaluate them on domain-specific datasets to scale the retrieval models.

Iqbal plans to pursue these projects on Azure AI infrastructure, connecting continued data development with the compute needed to train and evaluate the resulting models.

19:4619:58
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

19:46 · section reference included

Resources

From the talk

Transcript download
Full transcript

Loading transcript…