AI Engineer Summit 2025
How to Build Your Own AI Data Center in 2025
Loading slides…
Read the talk
How to Build Your Own AI Data Center in 2025
Training infrastructure must handle synchronized bursts, fragile jobs and unfamiliar power demands. Paul Gilbert explains how those constraints shape the network from GPU ports to operational telemetry.
From a talk by Paul Gilbert
Before you start: Familiarity with Ethernet links, switches and basic distributed training will help; the article explains the GPU-specific networking terms.
How much infrastructure does a model need?
How different should the infrastructure for training a model be from the infrastructure for running it? For Paul Gilbert, a New York–based technical lead at Arista Networks, that question starts with the network underneath the application. Enterprise networking teams must now understand job completion time, synchronization barriers and inference demand—not just whether servers can reach one another. Reasoning models complicate the sizing exercise because inference can involve substantially more computation than an ordinary response.
Gilbert gives an unnamed model example: training used 2,048 GPUs for one to two months, while inference after fine-tuning and alignment used four H100s. The example conveys the possible difference in infrastructure scale, but it does not specify model size, precision, latency, throughput or concurrency. Gilbert also cautions that reasoning changes the inference side of that comparison; he does not offer a settled replacement sizing rule. Training capacity does not directly determine inference capacity.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Separate GPU communication from storage
The first architectural boundary is between the backend network, which connects GPUs, and the frontend network, which supplies storage data. In Gilbert’s enterprise designs, the backend is dedicated: nothing unrelated attaches to it. GPUs are expensive, power hungry and difficult to acquire, so their communication fabric should not compete with ordinary enterprise traffic.
Each illustrated server has eight GPU-facing network ports connected through high-speed leaf and spine switches. The servers might come from NVIDIA, Supermicro or another supplier; the separation of traffic remains the same. During training, GPUs synchronize and compute, then request more data through the frontend. Depending on the model, Gilbert says the backend can run at 400 Gb/s per GPU connection, while the storage-facing network is less intense. The operational aim is continuous GPU use, supported by a simple routed fabric using iBGP or eBGP.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Read the server before designing its network
The rear of the H100 server Gilbert describes makes the separation physical. Four central connectors break out into eight GPU network ports, with separate Ethernet ports to the left. A single machine can therefore present far more network demand than an enterprise engineer accustomed to conventional servers expects.
That hardware also explains Gilbert’s practical interpretation of expansion. His customers purchase fixed eight-GPU systems: NVIDIA DGX systems or third-party systems based on HGX. They do not gradually fill those servers with additional GPUs. His uncertainty about the term scale up reflects that purchasing experience, rather than a general definition of GPU interconnects. Scale out is the part he designs for: add servers and GPU connections to a network that can grow from a small cluster to very large installations. He reserves the hundreds-of-thousands-of-GPUs scale for cloud operators, not his enterprise examples.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Collectives change the bandwidth calculation
Learning the GPU hardware is only half the transition. CUDA provides the GPU programming environment; NCCL is the communication library whose collective operations shape traffic across participating GPUs. Network engineers need to understand those operations because the application determines who sends data, who receives it and when. Gilbert’s hardware example has eight 400 Gb/s backend ports and four 400 Gb/s frontend ports.
A conventional web, application and database deployment spreads communication among services and can often redirect requests when a component fails. A distributed GPU job has a different dependency structure: participants exchange data with one another, and one failed participant can interrupt the job. The traffic is synchronized, too. Gilbert’s illustrative burst has 1,000 GPUs each transmitting at 400 Gb/s at the same time, if the network permits it.
That is why he builds the backend without oversubscription. Enterprise designs that relied on many servers not transmitting at once could provision less uplink capacity than their combined downlink capacity. Gilbert contrasts historical 10:1 or 3:1 downlink-to-uplink capacity with a 1:1 AI backend. Here, simultaneous demand is part of the workload rather than an exceptional coincidence.
The port totals show the size of the change:
| Server configuration in Gilbert’s example | Backend | Frontend | Combined nominal capacity |
|---|---|---|---|
| Twelve 400 Gb/s ports | 3.2 Tb/s | 1.6 Tb/s | 4.8 Tb/s |
| Twelve 800 Gb/s ports | 6.4 Tb/s | 3.2 Tb/s | 9.6 Tb/s |
These are sums of interface capacities across two separate fabrics, not measured application throughput. Gilbert describes the backend as capable of sustained wire-rate demand, with the storage side generally below its nominal capacity. At the time of the talk, he says the network already supports 800 Gb/s and tentatively places the next “B” server generation around March, with some systems possibly already available. For an enterprise team accustomed to one to four 100 Gb/s connections per server, either configuration changes the capacity-planning problem.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Enough bandwidth can still land on the wrong uplink
Aggregate capacity is not enough if traffic concentrates on one path. Conventional leaf-to-spine load balancing hashes flow fields to choose an uplink. The standard five-tuple comprises source and destination IP addresses, source and destination transport ports, and the transport protocol. With many independent flows, hashing can distribute traffic effectively. Gilbert’s GPU example has much less flow diversity: heavy traffic associated with a GPU can map to one uplink, overload it and cause packet loss while other paths remain available.
The alternative he describes uses the percentage of uplink bandwidth already in use to distribute traffic. Gilbert reports about 93% utilization across uplinks relative to downlinks with this approach. He does not specify the workload, collective, topology or measurement method behind that figure. The useful mechanism is the shift from relying solely on flow identity to considering actual path utilization.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Cables, power and cooling become job dependencies
A failed GPU or group of GPUs can stop training, even when checkpointing makes recovery possible. At thousands of GPUs, physical troubleshooting becomes routine: optics, transceivers, digital optical monitoring readings, link loss and cables all matter alongside the GPUs themselves. An application failure may originate in a component that looks mundane from the software layer.
Power creates a more immediate installation problem. Gilbert invokes the interest in nuclear power as a sign of the industry’s appetite, then brings the issue back to a familiar customer conversation: the GPUs have finally arrived, but nobody has checked whether the racks can support them. He cites conventional rack budgets of roughly 7–15 kW. Against that, 10.2 kW is the power-planning allowance for a DGX H100 system, not a measured draw for every eight-GPU workload. One such system can consume most of an existing rack’s allowance—or exceed a lower-capacity rack’s budget.
Gilbert describes enterprises moving toward 100–200 kW water-cooled racks and says air cooling is insufficient at that density. The infrastructure decision therefore extends beyond ordering switches and servers: rack power and heat removal can determine whether the equipment can be installed at all.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Two traffic directions, two congestion responses
The training cycle produces two traffic directions. East-west traffic is GPU-to-GPU communication across the backend. North-south traffic supplies data from storage through the frontend. Gilbert associates the synchronized, wire-rate bursts with the east-west side; he describes storage connections as nearer 100–200 Gb/s at the time, with less intense demand and room to grow. Both directions matter, but they do not put identical pressure on the switches.
A queue forms when traffic cannot move onward as quickly as it arrives. Buffers absorb some of that mismatch, but persistent buffering signals congestion. In the RoCE v2 deployment Gilbert describes, ECN and PFC provide complementary responses:
- A congested switch marks packets using Explicit Congestion Notification, or ECN.
- The receiver observes the indication and sends feedback to the sender.
- The sender’s congestion-control algorithm reduces its transmission rate, then increases it again when congestion indications cease.
- If buffers approach exhaustion, Priority Flow Control, or PFC, pauses the affected traffic priority on the link.
ECN supplies a feedback loop for reducing demand; PFC acts as the emergency stop. PFC is not an instruction to stop the whole application, but a link-level pause can still delay the work that depends on that traffic.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keep the fabric simple and involve workload owners
The dedicated backend leaves out the service chain of a conventional enterprise data center: DMZs, firewalls, load balancers, internet connectivity and other layer 4–7 services. The frontend may connect to other systems, but Gilbert advises caution there too. This isolation keeps the GPU fabric focused on its particular job; it does not make the surrounding organization’s security requirements disappear.
The operational consequence of failure is also different. A conventional application may restart or fail over with a brief interruption. A failed training job can trigger a much more consequential call to the operations center. Before designing the fabric, Gilbert therefore tells network engineers to speak with data scientists and developers about the models they are building and the collectives they use. NCCL discovers participants and organizes communication, but workload choices still affect which network design will serve them well.
Port speeds continue to rise: Gilbert describes 800 Gb/s as already supported and anticipates 1.6 Tb/s next. That is a roadmap expectation, not a deployment date to plan against. More immediately, visibility must precede the incident call. The network needs telemetry that reveals emerging trouble before developers report that the model has failed.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Control queues, then preserve evidence of failure
Gilbert uses features in Arista’s EOS switch operating system to make the operational requirements concrete. He does not treat lossless Ethernet as a claim that training can never tolerate a dropped packet: in his experience, some loss and consistent latency can be acceptable. Excessive loss is the problem, and ECN and PFC are tools for controlling it. Because GPUs synchronize, slowing one port or participant can slow the entire job.
Flow control therefore cannot substitute for finding the queue. Engineers still need to identify oversubscription, determine where traffic is accumulating and select appropriate switches for each part of the fabric. Gilbert describes adjusting buffer allocation to the packet sizes a model sends and receives. Switch buffer memory is scarce and expensive; matching its allocation to the traffic makes better use of the available capacity.
Remote Direct Memory Access, or RDMA, enables direct memory transfers without the usual CPU-mediated data-copy path. Its errors add another diagnostic layer beyond ordinary interface counters. Gilbert mentions a range of RDMA error codes rather than an exact inventory, then describes preserving evidence when a packet must be dropped: copy the packet into a buffer or send it elsewhere, or retain just its headers. The resulting snapshot includes RDMA information and the reason for the drop. Instead of learning only that a counter increased, an operator can inspect what failed and why.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Connect host state to switch state
Switch telemetry alone cannot show everything happening at the GPU endpoint. Gilbert describes a host-network coordination agent, implemented through an API and code on the NVIDIA server side, that communicates with the attached switch. This is host or NIC-side coordination, not code executing on GPU cores.
The exchange serves two purposes:
- Configuration agreement: The endpoint and switch compare their configuration so that PFC and ECN operate consistently across the connection.
- Correlated diagnostics: Packet-send, packet-receive and RDMA error statistics expose endpoint behavior alongside switch observations.
Together, these observations help distinguish a GPU-side problem from a network-side problem. A link that is up is not sufficient evidence that both ends agree on congestion handling.
Maintenance must also preserve expensive GPU work. Gilbert describes smart system upgrade as a way to update switch software for new features or security fixes without taking the switch offline. He illustrates the claim with a network of 1,024 GPUs and 64 switches whose GPUs keep working during upgrades. The talk does not establish the supported upgrade paths or prerequisites for that example, so uninterrupted maintenance remains a capability to validate for the chosen equipment and software versions.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn the requirements into a deployable fabric
Gilbert’s deployment guidance starts with the same capacity constraint: GPUs will consume the backend bandwidth available to them, so do not oversubscribe it. The remaining choices serve specific needs:
- Addressing: Use point-to-point IPv4 subnets such as
/30or/31; consider IPv6 when IPv4 address space is constrained. - Routing: Gilbert recommends BGP for a simple, fast routed fabric.
- Tenancy: Add EVPN VXLAN when separate business units or tenants need logical separation.
- Traffic distribution: Use advanced balancing; his cluster load balancing example considers the collective being run.
- Congestion and operations: Deploy RoCE with its associated congestion controls and maintain continuous visibility so the operations center gets an early warning.
These are connected decisions: addressing and routing establish reachability, balancing distributes demand, congestion control limits overload, and telemetry shows whether the design is behaving as intended.
The cluster examples use 800 Gb/s links between leaves and spines and 400 Gb/s links down to GPUs, while preserving a non-oversubscribed design. Link speed alone does not establish that property; the aggregate capacities on both sides must match the intended demand. For larger installations, Gilbert points to a 16-slot Arista 7800-series chassis. The matching 7800R4-AI platform specification lists 576 800G ports or 1,152 400G ports. Those are interface counts; translating them into attached GPUs requires the topology and port-per-GPU assumptions.
The complete deployment separates three roles: the backend where GPUs communicate during training, the frontend where storage supplies their data, and the inference environment where the trained model is deployed. That last destination brings the design back to the opening sizing question: serving the model is a distinct infrastructure task, even when training produced it on a much larger cluster.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Move more transport work into the NIC
The Ultra Ethernet Consortium discussion extends the design beyond faster links. Gilbert identifies congestion control, packet spraying and NIC-to-NIC communication as areas where Ethernet can better serve AI workloads. He forecasts version 1.0 ratification in Q1 2025 and systems appearing in Q3 or Q4. The consortium subsequently announced the specification’s public release on June 11, 2025; that announcement does not establish the internal ratification date or deployment availability.
The architectural attraction is to put more transport intelligence into NICs and less into the switching fabric. Gilbert attributes cloud operators’ interest to that division of responsibility: endpoints handle more of the communication behavior, while the network concentrates on forwarding packets.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The final measure is job completion time
The frontend supplies data; the backend carries bursts from GPUs that send and receive together. At a synchronization barrier, one slow participant holds up the others. The network’s success is therefore measured in job completion time, not merely in ports that are up or packets successfully forwarded.
Gilbert closes with an illustrative operations complaint: a job that finished in one hour yesterday now takes four days. His first suspicion would be the network. The numbers describe the kind of incident that matters, rather than a measured regression: distributed work can remain technically alive while making disastrously slow progress. Checkpointing offers recovery, but it is expensive too. The infrastructure has to preserve useful forward progress across the whole synchronized job.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Archived guide to communicators, GPU ranks, collective operations, and integrating NCCL into applications.
Arista's platform announcement describes the 7800R4-AI switch and its 800G and 400G interface capacities.
Further reading
- DGX H100 data center deployment planningDocumentation
NVIDIA guidance for rack density, power budgets, and physical deployment of DGX H100 systems.
A 2024 announcement explaining the EOS-based remote agent for coordinating network and host configuration and monitoring.
Updates since the talk
- Arista AI networking white paperDocumentation
Later vendor documentation covering AI fabrics, congestion management, load balancing, upgrades, and observability.
The consortium's June 2025 announcement of its Ethernet communication specification for AI and HPC.
Full transcript
Loading transcript…