This topic examines the boundary between application runtimes, operating-system services, device drivers, and accelerator execution. The focus is on how queues, memory allocation, I/O paths, and launch mechanisms affect latency, bandwidth, utilization, and isolation.
Core Ideas to Master
1. GPU Scheduling & Context Switching
- Concept: How the GPU driver, runtime, and hardware schedule kernels, memory copies, command queues, and work from multiple processes.
- Why it's important: Queueing, synchronization, and GPU context switching can add latency and reduce utilization. Understanding scheduling helps engineers choose batching, stream assignment, time slicing, process isolation, or Multi-Instance GPU (MIG) where available.
- Implementation Link: Supports the design of multi-user inference services with measured tail-latency and isolation behavior.
- Deep dive: GPU Scheduling And Context Switching
2. Zero-Copy Data Paths (io_uring, GPUDirect)
- Concept: Reducing intermediate copies among kernel buffers, user-space buffers, pinned host memory, device memory, storage devices, and network adapters.
- Why it's important: In large model serving and data-intensive training, weight loading, input ingestion, or distributed communication can become limited by memory movement and CPU overhead.
- Implementation Link: Relevant to high-throughput data loaders, model loading paths, GPUDirect-style storage or networking, and distributed communication.
- Deep dive: Zero-Copy Data Paths
3. Custom Memory Allocators (e.g., Stream-Ordered Allocation)
- Concept: Using memory pools, caching allocators, and stream-ordered APIs such as
cudaMallocAsyncfor accelerator-visible memory. - Why it's important: General allocation paths and unmanaged device allocation can introduce synchronization, fragmentation, and unpredictable latency when they are not aligned with stream-ordered execution.
- Implementation Link: Helps reduce allocator-induced synchronization and memory-fragmentation effects in complex model pipelines.
- Deep dive: Custom Memory Allocators
4. Kernel Fusion & Graph Capture
- Concept: Merging compatible operations into a single device kernel or capturing a repeated execution sequence for replay.
- Why it's important: Kernel launches have host and runtime overhead, and unfused operations often materialize intermediate tensors. Fusion can reduce both launch count and memory traffic when the operations are compatible.
- Implementation Link: A common optimization in libraries and compilers such as TensorRT, framework JIT compilers, and ahead-of-time compiler pipelines.
- Deep dive: Kernel Fusion And Graph Capture
Recommended Resources
- "Operating Systems: Three Easy Pieces" (Arpaci-Dusseau): For general OS background (Virtualization, Concurrency).
- NVIDIA CUDA Programming Guide: Specifically the sections on Asynchronous Concurrent Execution and CUDA Graphs.
- Linux
io_uringDocumentation: For high-performance I/O. - GPUDirect Storage (GDS) Whitepapers: For understanding direct storage-to-GPU paths.