This topic explains how GPUs execute programs, expose parallelism, and deliver performance for ML workloads. The focus is on the execution model, memory hierarchy, synchronization, the programming stack, and the measurement techniques needed to reason about GPU behavior in practice.
Core Ideas to Master
1. SIMT Execution And Occupancy
- Concept: GPUs execute many lightweight threads in lockstep groups (warps) under the Single Instruction Multiple Threads (SIMT) model.
- Why it's important: Divergence, register pressure, and scheduling policy determine whether the GPU can hide latency and reach high utilization.
- Implementation Link: Guides kernel structure, control-flow design, and thread mapping for ML primitives.
- Deep dive: SIMT Execution And Occupancy
2. GPU Memory Hierarchy And Coalescing
- Concept: Registers, shared memory, L2 cache, and global memory have different latency and bandwidth characteristics.
- Why it's important: Memory coalescing and access patterns determine whether kernels are bandwidth-bound or compute-bound.
- Implementation Link: Informs layout decisions for tensors, KV caches, and intermediate buffers.
- Deep dive: GPU Memory Hierarchy And Coalescing
3. Thread Blocks, Synchronization, And Atomics
- Concept: Threads cooperate within blocks using barriers and atomics while blocks execute independently.
- Why it's important: Synchronization and contention determine scalability for reductions, softmax, and sparse updates.
- Implementation Link: Helps engineers choose block sizes, reduction strategies, and safe shared data access.
- Deep dive: Thread Blocks, Synchronization, And Atomics
4. Shared Memory And Tiled Kernels
- Concept: Shared memory enables reuse of data within a block through tiling and staging.
- Why it's important: Tiled kernels reduce global memory traffic and improve arithmetic intensity in GEMM and attention.
- Implementation Link: Central to custom kernels, Triton-style programming, and fused operator design.
- Deep dive: Shared Memory And Tiled Kernels
5. GPU Runtime And Programming Stack
- Concept: GPU execution passes through drivers, runtimes, compilation pipelines, and library kernels.
- Why it's important: The runtime stack determines kernel launch cost, compilation strategy, and which kernels are available.
- Implementation Link: Necessary for integrating custom kernels, JIT compilers, and vendor libraries.
- Deep dive: GPU Runtime And Programming Stack
6. Profiling And Performance Models
- Concept: Profilers and roofline-style models connect measured bottlenecks to architectural limits.
- Why it's important: Without measurement, optimizations target the wrong bottlenecks and produce regressions.
- Implementation Link: Guides iteration on kernel performance and verification of optimization impact.
- Deep dive: Profiling And Performance Models
Recommended Resources
- "Programming Massively Parallel Processors" (Kirk & Hwu): Execution model and kernel design foundations.
- NVIDIA CUDA Programming Guide: Runtime model, memory hierarchy, and kernel launch semantics.
- AMD ROCm Documentation: Open GPU stack and profiling tools.
- NVIDIA Tuning Guides (per architecture): Occupancy limits, memory behavior, and performance tips.