Large model training and serving often exceed the memory capacity, compute throughput, or availability requirements of a single accelerator. This topic covers the engineering mechanisms used to distribute model execution across multiple GPUs and nodes.
Core Ideas to Master
1. Interconnect Topologies (NVLink, InfiniBand)
- Concept: The physical and logical layout of how GPUs talk to each other (e.g., Ring, Hypercube, Fat Tree).
- Why it matters: Distributed execution can be limited by communication rather than arithmetic throughput. Bandwidth, latency, contention, and topology determine whether a placement strategy can keep accelerators utilized.
- Implementation Link: Helps decide between Data Parallelism (DP) and Model Parallelism (MP).
- Deep dive: Interconnect Topologies
2. Collective Communication Primitives (NCCL/RCCL)
- Concept: Standardized algorithms for data synchronization:
All-Reduce,All-Gather,Reduce-Scatter. - Why it matters: These operations determine the cost of gradient synchronization, activation exchange, parameter reconstruction, and expert routing. Their performance depends on message size, rank mapping, and network topology.
- Implementation Link: Forms the communication layer used by systems such as DeepSpeed, Horovod, and PyTorch Distributed.
- Deep dive: Collective Communication Primitives
3. Model Parallelism: 3D Parallelism (TP, PP, DP)
- Concept: Breaking models apart across Tensor Parallelism (intra-layer), Pipeline Parallelism (inter-layer), and Data Parallelism (sharding data).
- Why it matters: These strategies reduce per-device memory requirements and distribute computation, but they introduce communication, scheduling, and optimizer-state management costs.
- Implementation Link: Provides the basis for designing parameter, activation, gradient, and optimizer-state sharding strategies.
- Deep dive: Three-Dimensional Parallelism
4. Fault Tolerance & Checkpointing
- Concept: Mechanisms for limiting lost work when a node, process, network path, storage system, or scheduler event interrupts a distributed job.
- Why it matters: Failure probability increases with cluster size and job duration. Checkpointing and recovery design determine the amount of recomputation and the risk of inconsistent restart state.
- Implementation Link: Informs the design of training loops and persistent state management.
- Deep dive: Fault Tolerance And Checkpointing
Recommended Resources
- "Distributed Systems: Concepts and Design" (Coulouris et al.): For foundation in distributed consensus and failure modes.
- NVIDIA NCCL (NVIDIA Collective Communications Library) Documentation: Understanding optimized primitives.
- "Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM" (NVIDIA Research): For an implementation-oriented discussion of tensor, pipeline, and data parallelism.
- Microsoft DeepSpeed Blog: For practical implementations of ZeRO (Zero Redundancy Optimizer).