AI Engineer Summit 2025

How to Build Your Own AI Data Center in 2025

Paul Gilbert· Technical Lead, Arista Networks23:00

Loading slides…

Read the talk

How to Build Your Own AI Data Center in 2025

Training infrastructure must handle synchronized bursts, fragile jobs and unfamiliar power demands. Paul Gilbert explains how those constraints shape the network from GPU ports to operational telemetry.

From a talk by Paul Gilbert

Before you start: Familiarity with Ethernet links, switches and basic distributed training will help; the article explains the GPU-specific networking terms.

How much infrastructure does a model need?

How different should the infrastructure for training a model be from the infrastructure for running it? For Paul Gilbert, a New York–based technical lead at Arista Networks, that question starts with the network underneath the application. Enterprise networking teams must now understand job completion time, synchronization barriers and inference demand—not just whether servers can reach one another. Reasoning models complicate the sizing exercise because inference can involve substantially more computation than an ordinary response.

Model Training and Inference slide contrasting job-completion time, time to first token and test-time compute.Open full source frame
Training, inference and test-time compute put different demands on the system.

Gilbert gives an unnamed model example: training used 2,048 GPUs for one to two months, while inference after fine-tuning and alignment used four H100s. The example conveys the possible difference in infrastructure scale, but it does not specify model size, precision, latency, throughput or concurrency. Gilbert also cautions that reasoning changes the inference side of that comparison; he does not offer a settled replacement sizing rule. Training capacity does not directly determine inference capacity.

0:170:29
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:17 · section reference included

Separate GPU communication from storage

The first architectural boundary is between the backend network, which connects GPUs, and the frontend network, which supplies storage data. In Gilbert’s enterprise designs, the backend is dedicated: nothing unrelated attaches to it. GPUs are expensive, power hungry and difficult to acquire, so their communication fabric should not compete with ordinary enterprise traffic.

Each illustrated server has eight GPU-facing network ports connected through high-speed leaf and spine switches. The servers might come from NVIDIA, Supermicro or another supplier; the separation of traffic remains the same. During training, GPUs synchronize and compute, then request more data through the frontend. Depending on the model, Gilbert says the backend can run at 400 Gb/s per GPU connection, while the storage-facing network is less intense. The operational aim is continuous GPU use, supported by a simple routed fabric using iBGP or eBGP.

AI network diagram with GPU servers between blue storage/data-center connections and green GPU connections.Open full source frame
GPU servers sit between a storage-facing front-end network and a separate GPU-facing fabric.
2:372:51
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:37 · section reference included

Read the server before designing its network

The rear of the H100 server Gilbert describes makes the separation physical. Four central connectors break out into eight GPU network ports, with separate Ethernet ports to the left. A single machine can therefore present far more network demand than an enterprise engineer accustomed to conventional servers expects.

That hardware also explains Gilbert’s practical interpretation of expansion. His customers purchase fixed eight-GPU systems: NVIDIA DGX systems or third-party systems based on HGX. They do not gradually fill those servers with additional GPUs. His uncertainty about the term scale up reflects that purchasing experience, rather than a general definition of GPU interconnects. Scale out is the part he designs for: add servers and GPU connections to a network that can grow from a small cluster to very large installations. He reserves the hundreds-of-thousands-of-GPUs scale for cloud operators, not his enterprise examples.

4:144:25
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:14 · section reference included

Collectives change the bandwidth calculation

Learning the GPU hardware is only half the transition. CUDA provides the GPU programming environment; NCCL is the communication library whose collective operations shape traffic across participating GPUs. Network engineers need to understand those operations because the application determines who sends data, who receives it and when. Gilbert’s hardware example has eight 400 Gb/s backend ports and four 400 Gb/s frontend ports.

A conventional web, application and database deployment spreads communication among services and can often redirect requests when a component fails. A distributed GPU job has a different dependency structure: participants exchange data with one another, and one failed participant can interrupt the job. The traffic is synchronized, too. Gilbert’s illustrative burst has 1,000 GPUs each transmitting at 400 Gb/s at the same time, if the network permits it.

That is why he builds the backend without oversubscription. Enterprise designs that relied on many servers not transmitting at once could provision less uplink capacity than their combined downlink capacity. Gilbert contrasts historical 10:1 or 3:1 downlink-to-uplink capacity with a 1:1 AI backend. Here, simultaneous demand is part of the workload rather than an exceptional coincidence.

The port totals show the size of the change:

Server configuration in Gilbert’s exampleBackendFrontendCombined nominal capacity
Twelve 400 Gb/s ports3.2 Tb/s1.6 Tb/s4.8 Tb/s
Twelve 800 Gb/s ports6.4 Tb/s3.2 Tb/s9.6 Tb/s

These are sums of interface capacities across two separate fabrics, not measured application throughput. Gilbert describes the backend as capable of sustained wire-rate demand, with the storage side generally below its nominal capacity. At the time of the talk, he says the network already supports 800 Gb/s and tentatively places the next “B” server generation around March, with some systems possibly already available. For an enterprise team accustomed to one to four 100 Gb/s connections per server, either configuration changes the capacity-planning problem.

5:445:59
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:44 · section reference included

Enough bandwidth can still land on the wrong uplink

Aggregate capacity is not enough if traffic concentrates on one path. Conventional leaf-to-spine load balancing hashes flow fields to choose an uplink. The standard five-tuple comprises source and destination IP addresses, source and destination transport ports, and the transport protocol. With many independent flows, hashing can distribute traffic effectively. Gilbert’s GPU example has much less flow diversity: heavy traffic associated with a GPU can map to one uplink, overload it and cause packet loss while other paths remain available.

The alternative he describes uses the percentage of uplink bandwidth already in use to distribute traffic. Gilbert reports about 93% utilization across uplinks relative to downlinks with this approach. He does not specify the workload, collective, topology or measurement method behind that figure. The useful mechanism is the shift from relying solely on flow identity to considering actual path utilization.

8:358:49
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:35 · section reference included

Cables, power and cooling become job dependencies

A failed GPU or group of GPUs can stop training, even when checkpointing makes recovery possible. At thousands of GPUs, physical troubleshooting becomes routine: optics, transceivers, digital optical monitoring readings, link loss and cables all matter alongside the GPUs themselves. An application failure may originate in a component that looks mundane from the software layer.

Power creates a more immediate installation problem. Gilbert invokes the interest in nuclear power as a sign of the industry’s appetite, then brings the issue back to a familiar customer conversation: the GPUs have finally arrived, but nobody has checked whether the racks can support them. He cites conventional rack budgets of roughly 7–15 kW. Against that, 10.2 kW is the power-planning allowance for a DGX H100 system, not a measured draw for every eight-GPU workload. One such system can consume most of an existing rack’s allowance—or exceed a lower-capacity rack’s budget.

Gilbert describes enterprises moving toward 100–200 kW water-cooled racks and says air cooling is insufficient at that density. The infrastructure decision therefore extends beyond ordering switches and servers: rack power and heat removal can determine whether the equipment can be installed at all.

9:299:38
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:29 · section reference included

Two traffic directions, two congestion responses

The training cycle produces two traffic directions. East-west traffic is GPU-to-GPU communication across the backend. North-south traffic supplies data from storage through the frontend. Gilbert associates the synchronized, wire-rate bursts with the east-west side; he describes storage connections as nearer 100–200 Gb/s at the time, with less intense demand and room to grow. Both directions matter, but they do not put identical pressure on the switches.

A queue forms when traffic cannot move onward as quickly as it arrives. Buffers absorb some of that mismatch, but persistent buffering signals congestion. In the RoCE v2 deployment Gilbert describes, ECN and PFC provide complementary responses:

  1. A congested switch marks packets using Explicit Congestion Notification, or ECN.
  2. The receiver observes the indication and sends feedback to the sender.
  3. The sender’s congestion-control algorithm reduces its transmission rate, then increases it again when congestion indications cease.
  4. If buffers approach exhaustion, Priority Flow Control, or PFC, pauses the affected traffic priority on the link.

ECN supplies a feedback loop for reducing demand; PFC acts as the emergency stop. PFC is not an instruction to stop the whole application, but a link-level pause can still delay the work that depends on that traffic.

11:0411:15
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:04 · section reference included

Keep the fabric simple and involve workload owners

The dedicated backend leaves out the service chain of a conventional enterprise data center: DMZs, firewalls, load balancers, internet connectivity and other layer 4–7 services. The frontend may connect to other systems, but Gilbert advises caution there too. This isolation keeps the GPU fabric focused on its particular job; it does not make the surrounding organization’s security requirements disappear.

The operational consequence of failure is also different. A conventional application may restart or fail over with a brief interruption. A failed training job can trigger a much more consequential call to the operations center. Before designing the fabric, Gilbert therefore tells network engineers to speak with data scientists and developers about the models they are building and the collectives they use. NCCL discovers participants and organizes communication, but workload choices still affect which network design will serve them well.

Port speeds continue to rise: Gilbert describes 800 Gb/s as already supported and anticipates 1.6 Tb/s next. That is a roadmap expectation, not a deployment date to plan against. More immediately, visibility must precede the incident call. The network needs telemetry that reveals emerging trouble before developers report that the model has failed.

12:5813:08
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

12:58 · section reference included

Control queues, then preserve evidence of failure

Gilbert uses features in Arista’s EOS switch operating system to make the operational requirements concrete. He does not treat lossless Ethernet as a claim that training can never tolerate a dropped packet: in his experience, some loss and consistent latency can be acceptable. Excessive loss is the problem, and ECN and PFC are tools for controlling it. Because GPUs synchronize, slowing one port or participant can slow the entire job.

Flow control therefore cannot substitute for finding the queue. Engineers still need to identify oversubscription, determine where traffic is accumulating and select appropriate switches for each part of the fabric. Gilbert describes adjusting buffer allocation to the packet sizes a model sends and receives. Switch buffer memory is scarce and expensive; matching its allocation to the traffic makes better use of the available capacity.

Remote Direct Memory Access, or RDMA, enables direct memory transfers without the usual CPU-mediated data-copy path. Its errors add another diagnostic layer beyond ordinary interface counters. Gilbert mentions a range of RDMA error codes rather than an exact inventory, then describes preserving evidence when a packet must be dropped: copy the packet into a buffer or send it elsewhere, or retain just its headers. The resulting snapshot includes RDMA information and the reason for the drop. Instead of learning only that a counter increased, an operator can inspect what failed and why.

15:1215:26
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

15:12 · section reference included

Connect host state to switch state

Switch telemetry alone cannot show everything happening at the GPU endpoint. Gilbert describes a host-network coordination agent, implemented through an API and code on the NVIDIA server side, that communicates with the attached switch. This is host or NIC-side coordination, not code executing on GPU cores.

The exchange serves two purposes:

  • Configuration agreement: The endpoint and switch compare their configuration so that PFC and ECN operate consistently across the connection.
  • Correlated diagnostics: Packet-send, packet-receive and RDMA error statistics expose endpoint behavior alongside switch observations.

Together, these observations help distinguish a GPU-side problem from a network-side problem. A link that is up is not sufficient evidence that both ends agree on congestion handling.

Maintenance must also preserve expensive GPU work. Gilbert describes smart system upgrade as a way to update switch software for new features or security fixes without taking the switch offline. He illustrates the claim with a network of 1,024 GPUs and 64 switches whose GPUs keep working during upgrades. The talk does not establish the supported upgrade paths or prerequisites for that example, so uninterrupted maintenance remains a capability to validate for the chosen equipment and software versions.

17:3517:45
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

17:35 · section reference included

Turn the requirements into a deployable fabric

Gilbert’s deployment guidance starts with the same capacity constraint: GPUs will consume the backend bandwidth available to them, so do not oversubscribe it. The remaining choices serve specific needs:

  • Addressing: Use point-to-point IPv4 subnets such as /30 or /31; consider IPv6 when IPv4 address space is constrained.
  • Routing: Gilbert recommends BGP for a simple, fast routed fabric.
  • Tenancy: Add EVPN VXLAN when separate business units or tenants need logical separation.
  • Traffic distribution: Use advanced balancing; his cluster load balancing example considers the collective being run.
  • Congestion and operations: Deploy RoCE with its associated congestion controls and maintain continuous visibility so the operations center gets an early warning.

These are connected decisions: addressing and routing establish reachability, balancing distributes demand, congestion control limits overload, and telemetry shows whether the design is behaving as intended.

The cluster examples use 800 Gb/s links between leaves and spines and 400 Gb/s links down to GPUs, while preserving a non-oversubscribed design. Link speed alone does not establish that property; the aggregate capacities on both sides must match the intended demand. For larger installations, Gilbert points to a 16-slot Arista 7800-series chassis. The matching 7800R4-AI platform specification lists 576 800G ports or 1,152 400G ports. Those are interface counts; translating them into attached GPUs requires the topology and port-per-GPU assumptions.

The complete deployment separates three roles: the backend where GPUs communicate during training, the frontend where storage supplies their data, and the inference environment where the trained model is deployed. That last destination brings the design back to the opening sizing question: serving the model is a distinct infrastructure task, even when training produced it on a much larger cluster.

19:0719:18
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

19:07 · section reference included

Move more transport work into the NIC

The Ultra Ethernet Consortium discussion extends the design beyond faster links. Gilbert identifies congestion control, packet spraying and NIC-to-NIC communication as areas where Ethernet can better serve AI workloads. He forecasts version 1.0 ratification in Q1 2025 and systems appearing in Q3 or Q4. The consortium subsequently announced the specification’s public release on June 11, 2025; that announcement does not establish the internal ratification date or deployment availability.

The architectural attraction is to put more transport intelligence into NICs and less into the switching fabric. Gilbert attributes cloud operators’ interest to that division of responsibility: endpoints handle more of the communication behavior, while the network concentrates on forwarding packets.

21:2221:31
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

21:22 · section reference included

The final measure is job completion time

The frontend supplies data; the backend carries bursts from GPUs that send and receive together. At a synchronization barrier, one slow participant holds up the others. The network’s success is therefore measured in job completion time, not merely in ports that are up or packets successfully forwarded.

Gilbert closes with an illustrative operations complaint: a job that finished in one hour yesterday now takes four days. His first suspicion would be the network. The numbers describe the kind of incident that matters, rather than a measured regression: distributed work can remain technically alive while making disastrously slow progress. Checkpointing offers recovery, but it is expensive too. The infrastructure has to preserve useful forward progress across the whole synchronized job.

22:0522:15
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

22:05 · section reference included

Resources

From the talk

Updates since the talk

Transcript download
Full transcript

Loading transcript…