AI Engineer World's Fair 2024
Insights from Snorkel AI running Azure AI Infrastructure
Loading slides…
Read the talk
Insights from Snorkel AI running Azure AI Infrastructure
Enterprise model quality depends on the right domain data—and on infrastructure that can turn that data into repeated training, evaluation and synthesis cycles.
From a talk by Lachlan Ainley and Humza Iqbal
Before you start: Familiarity with model fine-tuning, GPU training and the distinction between training and inference will help you follow the infrastructure discussion.
What gets a bank’s model ready to deploy?
What does it take to make a foundation model reliable enough for a bank or insurer? Humza Iqbal works on that problem as an applied research scientist on Snorkel’s computer vision team, fine-tuning foundation models for enterprise use cases. The company’s name invites a detour: after joining, he discovered that snorkel.com belonged to a hot-tub company. Snorkel’s actual business is developing data for enterprise AI.
For its Fortune 500 customers, quality, latency and cost all constrain deployment. A model must also be reliable and accurate on the work the organization actually needs it to do. Off-the-shelf Claude, GPT-4 and Gemini can get partway there, Iqbal explains, but task-specific fine-tuning data supplies the missing final mile.
Access to enterprise knowledge is only the beginning. Retrieval-augmented generation, or RAG, can connect a model to information absent from its pretraining. But finding and maintaining the right data remains essential. A large instruction-tuning dataset may contain little about the particular bank policy or industry rule the model must learn. Dataset size does not substitute for that required slice of information.
Iqbal places this work in Snorkel’s longer history: origins in the Stanford AI Lab, roughly a decade of data-development experience, and research spanning prompting, RAG and model architectures. He describes the company as a pioneer in data development for LLMs, extending an emphasis on data that predates today’s foundation models.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Scale expert knowledge, then evaluate the actual task
The research question is how enterprises can best develop data for custom models. Snorkel approaches it through three connected requirements:
- Keep subject-matter experts involved. Industry specialists know the details that model feedback must capture; the workflow should maximize the value of their limited time.
- Make development programmatic, scalable and auditable. Expert input needs a way to influence more data than experts could manually review themselves.
- Evaluate continuously on domain-specific, dynamic benchmarks. The evaluation must remain relevant to the organization’s task as its models and data change.
Broad rankings such as LMSYS provide a general view of model capability. They do not establish whether a model handles a bank’s specific requirements. Strong grade-school math performance answers a different question.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Evaluate long context and align models to the domain
Fine-grained evaluation starts by asking what a test actually measures. In a needle-in-a-haystack test, an evaluator inserts a sentence into a collection of Paul Graham essays and asks the model to find it. Success demonstrates retrieval under those conditions; it need not describe how the same model handles long context in another domain. Iqbal’s research emphasis is to break performance down domain by domain, then use those findings to improve long-context behavior.
Enterprise alignment adds another requirement: outputs must comply with company goals and regulations. A fluent answer can still create an organizational problem. Multimodal alignment faces a related gap because models trained on public data can underperform in specialized domains. Snorkel is exploring how large vision-language models, or LVLMs, can generate synthetic data without manual annotation to train downstream models. Domain-specific generation feeds model training, creating a repeated cycle of data development and learning.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Optimize the whole infrastructure path
Those research cycles depend on more than accelerator performance. Microsoft’s Lachlan Ainley describes Azure’s approach as optimization across the entire system. At the time of the talk, he cites more than 300 datacenters worldwide. Inside that footprint, the choices extend through VM host CPUs and throughput, accelerators from AMD and NVIDIA alongside Microsoft’s Maia, I/O topology, and networking. Each layer must support the workload rather than constrain the next one.
Ainley describes a feedback process from large training deployments to customer infrastructure: experience supporting organizations such as OpenAI and Mistral at supercomputing scale informs the systems Azure makes available more broadly. Snorkel’s deployment is one concrete use of that infrastructure.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
PyTorch, Horovod and a shared filesystem
Snorkel’s distributed training stack separates model computation, communication and storage:
| Layer | Component | Responsibility |
|---|---|---|
| Model framework | PyTorch | Training computation |
| Multi-node communication | Horovod | Exchange gradients across workers |
| Compute | Azure VMs with A100 or H100 GPUs | Run model workloads |
| Shared storage | Network file system, or NFS | Supply data and store checkpoints |
All the machines access the same NFS as though it were a local filesystem. That gives workers a common location for reading training data and writing model checkpoints, while Horovod handles communication between the machines.
The projects range from a single node to dozens of nodes. They include models aligned using Direct Preference Optimization, or DPO, with instruction-response preference datasets. The most compute-intensive work is large-scale multimodal training and inference, using dozens of GPUs. These different workloads make both distributed throughput and flexible capacity important.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Find out why the GPUs are waiting
More nodes are useful only if they fit the training configuration. While training CLIP-based models, Iqbal needed enough nodes to support the desired batch size. Too many nodes could force an excessively large batch or leave some of the allocated compute underused. Choose capacity around the intended training workload, not simply the largest available cluster.
Next comes locality. Iqbal’s deliberately obvious example puts data in US West and compute in Asia: the distance introduces a communication problem before model computation even begins. Within distributed training, exchanging gradients between model replicas can also become a bottleneck. Data placement and communication between workers both matter.
Storage can produce the same visible symptom through a different mechanism. Data loading runs asynchronously alongside the model’s forward and backward passes. If the NFS cannot read data quickly enough, that overlap is insufficient: the GPU finishes its work and waits for the next input instead of continuing to compute.
Start the investigation with nvidia-smi. A simple command keeps its status display updating:
bash
nvidia-smi -l 1
Its GPU-utilization counter measures sampled time with kernels executing, not achieved FLOPS or overall training efficiency. Use it as an initial signal, then investigate:
- Look for low GPU utilization while the job should be computing.
- For multi-node jobs, test networking and communication between workers.
- For single-node jobs, inspect data loading first. Iqbal considers it a likely cause in that setting, rather than a diagnosis the utilization counter can establish on its own.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Research needs capacity that can change
Reliability, flexibility and manageability shape whether a research team can use its infrastructure effectively. Snorkel’s experiments sometimes need every node and sometimes very few. That pattern continues over a long period. Some providers offered substantial compute within a fixed window—perhaps one or two months—but that did not fit a team doing ongoing, intermittent experimentation rather than one bounded training run.
Iqbal reports that dedicated Azure VMs let the team adjust capacity on demand, with dependable operation and no significant issues in its experience. He also reports good NFS throughput and values being able to resize storage as requirements change. Unexpected data growth requires more capacity; overprovisioned storage should shrink so the team does not keep paying for space it does not need.
Ease of use matters after the initial setup, too. Iqbal configured Azure for the team and wanted colleagues to work without repeatedly asking him for help. He credits clear documentation and a straightforward process with making the environment usable once configured, reducing support requests and avoiding paging.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Compare hardware at approximately equal cost
When moving from A100s to H100s, Snorkel asked an economic question: what training and inference performance could each configuration deliver for the same spend? Iqbal reports that two H100s cost approximately the same as four A100s in Snorkel’s comparison and performed better on both training and inference.
The appeal was the simplicity of the change. Inference optimization can involve KV caches, partial KV caches and speculative decoding; changing hardware offered Snorkel a straightforward improvement. Faster inference also mattered upstream of the final model: it let the team process more synthetic data and iterate faster, supporting the path toward higher downstream model accuracy.
The completed slide shows the measurements below. Training is expressed as time per step, where lower is better; inference is expressed as iterations per second, where higher is better.
| Configuration | Training time per step | Inference throughput |
|---|---|---|
| 2× H100 | 6.80 s | 22.37 it/s |
| 4× A100 | 10.88 s | 14.32 it/s |
The result supports Snorkel’s reported improvement for this experiment. The model, precision, exact batch sizes and pricing basis are not specified, so these figures are not enough to reproduce its cost comparison.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Separate batch-size effects from hardware effects
An audience member asks why inference appears to take longer than training. Iqbal tentatively attributes that to a larger inference batch: training typically needs a smaller batch because backpropagation consumes additional memory. The slide’s columns, however, use different units—seconds per training step and inference iterations per second—so their numerical magnitudes do not directly compare elapsed time. His answer should be read as an explanation of the workload setup, not a conversion between those columns.
A follow-up raises batch-size comparability across the two hardware configurations. Iqbal says the settings were held fixed across hardware, explicitly confirming that inference used the same batch size on H100s and A100s. That distinguishes two questions: training and inference can use different batches, while the hardware comparison for a given task keeps its batch setting constant.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Match evolving accelerators to the workload
Ainley closes Azure’s side of the discussion by returning to optimization at every layer. In the recording’s 2024 context, he describes Maia as serving internal Microsoft 365 workloads, AMD’s MI300X with high-bandwidth memory as newly announced, A100s and H100s as existing options, and Blackwell adoption as planned. This is the roadmap presented at the time, rather than a current availability guide.
He characterizes NVIDIA’s pace as approaching two releases a year, compared with his tentative recollection of one every two years previously. The practical direction is to absorb improvements in silicon while continuing to choose the right GPU for each workload. Snorkel’s ability to do the compared work with fewer GPUs illustrates why those hardware changes can matter to users of the infrastructure.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn preference signals and expertise into better data
Snorkel’s next research directions return to the original deployment problem: improving data so generative AI prototypes can reach production.
- Preference-driven synthesis and curation: programmatically use preference signals to generate and select data.
- Scalable expert participation: let subject-matter experts contribute through rationales and custom taxonomies.
- Domain-specific multimodal retrieval: develop better retrieval algorithms and evaluate them on domain-specific datasets to scale the retrieval models.
Iqbal plans to pursue these projects on Azure AI infrastructure, connecting continued data development with the compute needed to train and evaluate the resulting models.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Distributed training guidance covering gradient averaging, worker initialization, data sampling and batch-size scaling.
Command-line GPU monitoring documentation, including utilization, memory, clocks and power readings.
The original paper introducing DPO for learning language-model preferences.
The foundational CLIP paper on learning transferable visual representations from image-text pairs.
Further reading
A March 2024 account of Snorkel's Azure GPU infrastructure for DPO research and distributed experiments.
Snorkel's SWiM evaluation framework tests long-context models on document-specific tasks and explores positional bias.
Public code accompanying Snorkel's long-context-window evaluation research.
Microsoft's contemporaneous announcement of ND MI300X availability and its broader compute infrastructure strategy.
Full transcript
Loading transcript…