is part of a series about distributed AI on multiple GPUs:
Introduction
Before getting into advanced parallelization techniques, we need to understand the key technologies that allow GPUs to communicate with each other.
But why do GPUs need communication in the first place? When training AI models across multiple GPUs, each GPU processes different data sets but they all need to stay synchronized by sharing gradients during backpropagation or exchanging model weights. The specifics of what to talk about and when depends on your matching strategy, which we will explore in depth in the next blog post. For now, just know that modern AI training requires communication, making efficient GPU-to-GPU data transfer critical to performance.
Communication Stack
PCIe
PCIe (Peripheral Component Interconnect Express) connects expansion cards such as GPUs to the motherboard using independent point-to-point serial lanes. Here's what the different generations of PCIe have to offer with a GPU using 16 lanes:
- by 4 x16: ~32 GB/s bidirectional
- eb5 x16: ~64 GB/s bidirectional
- by 6 x16: ~128 GB/s bidirectional (FYI 16 lanes × 8 GB/s/lane = 128 GB/s)
High-end CPUs typically provide 128 PCIe lanes, while modern GPUs require 16 lanes for full bandwidth. This is why you typically see 8 GPUs per server (128 = 16 x 8). The power consumption and physical space in the server chassis also make it impossible to exceed 8 GPUs in one place.
NVLink
NVLink enables direct GPU-to-GPU communication within the same server (node), bypassing the CPU entirely. This NVIDIA-proprietary interface creates a direct way to share memory between high-bandwidth GPUs:
- NVLink 3 (A100): ~600 GB/s per GPU
- NVLink 4 (H100): ~900 GB/s per GPU
- NVLink 5 (Blackwell): Up to 1.8 TB/s per GPU
Note: on NVLink for CPU-GPU communication
Certain CPU architectures support NVLink instead of PCIe, dramatically speeding up CPU-GPU communication by overcoming the bottleneck of PCIe in data transfer, such as moving training sets from CPU to GPU. This CPU-GPU NVLink capability makes CPU-offloading (a technique that saves VRAM by storing data in RAM instead) useful in real-world AI applications. Since sizing RAM is generally more expensive than sizing VRAM, this approach offers significant economic benefits.
CPUs with NVLink support include IBM POWER8, POWER9, and NVIDIA Grace.
However, there is a catch. On a server with 8x H100s, each GPU needs to communicate with 7 others, splitting that 900 GB/s into seven point-to-point connections of about 128 GB/s each. This is where NVSwitch comes in.
NVSwitch
NVSwitch acts as a central hub for GPU communication, routing (switching if you like) data between GPUs as needed. With NVSwitch, every Hopper GPU can communicate at 900 GB/s with all other Hopper GPUs at the same time, i.e. the maximum bandwidth does not depend on how many GPUs are communicating. This is what makes NVSwitch “non-blocking”. Each GPU connects to several NVSwitch chips via multiple NVLink connections, ensuring maximum bandwidth.
While NVSwitch started as an intra-node solution, it has expanded to interconnect multiple nodes, creating GPU clusters that support up to 256 GPUs with all-to-all connectivity at near local NVLink speeds.
The generations of NVSwitch are:
- First Generation: Supports up to 16 GPUs per server (compatible with Tesla V100)
- Second Generation: Also supports up to 16 GPUs with improved bandwidth and low latency
- The Third Generation: Designed for H100 GPUs, supports up to 256 GPUs
InfiniBand
InfiniBand handles communication between nodes. Although it is slower (and cheaper) than NVSwitch, it is often used in datacenters to scale thousands of GPUs. Modern InfiniBand supports NVIDIA GPUDirect® RDMA (Remote Direct Memory Access), allowing network adapters to access GPU memory directly without CPU involvement (no expensive copying of host RAM).
Current InfiniBand speeds include:
- HDR: ~25 GB/s per port
- NDR: ~50 GB/s per port
- NDR200: ~100 GB/s per port
These speeds are much slower than intra-node NVLink due to the network protocol and the need for two PCIe interfaces (one for the sender and one for the receiver).
Key Design Principles
Understanding Linear Scaling
Linear scaling is the holy grail of distributed computing. In simple words, it means that doubling your GPUs should double your output and reduce your training time. This happens when the overhead of connections is small compared to the computation time, allowing each GPU to be fully functional. However, perfect linear scaling is rare in AI tasks because communication requirements increase with the number of devices, and it is often not possible to achieve complete computer-communication overlap (explained next).
The Importance of Communication Stacking
If the GPU is sitting idle waiting for data to be transferred before it can be processed, you are wasting resources. Communication activities should be as computationally relevant as possible. If that doesn't happen, we call that an “obvious job”.
Intra-Node vs. Inter-Node: The Performance Cliff
Today's server-grade motherboards support up to 8 GPUs. Within this range, you can achieve near linear scaling due to high bandwidth connectivity, low intra-node connectivity.
Once you have more than 8 GPUs and start using multiple nodes connected via InfiniBand, you will see a significant drop in performance. Inter-node communication is much slower than intra-node NVLink, introducing network protocol overhead, higher latency, and bandwidth limitations. As you add more GPUs, each GPU must communicate with more peers, spending more time idle waiting for data transfers to complete.
The conclusion
Follow me on X for more free AI content @l_cesconetto
Congratulations on reaching the end! In this post you will learn about:
- Basics of CPU-GPU and GPU-GPU communication:
- PCIe, NVLink, NVSwitch, and InfiniBand
- Main design principles of distributed GPU computing
- Now you can make more informed decisions when designing your AI workloads
In the next blog post, we'll dive into our first approach to parallelism, Distributed Data Parallelism.
- NVIDIA Blog
- GPU Direct
