Kernal and OS Optimisations

Subpage of AI Engineering

Accelerating AI: across all layers

This topic examines the boundary between application runtimes, operating-system services, device drivers, and accelerator execution. The focus is on how queues, memory allocation, I/O paths, and launch mechanisms affect latency, bandwidth, utilization, and isolation.

Core Ideas to Master

1. GPU Scheduling & Context Switching

  • Concept: How the GPU driver, runtime, and hardware schedule kernels, memory copies, command queues, and work from multiple processes.
  • Why it's important: Queueing, synchronization, and GPU context switching can add latency and reduce utilization. Understanding scheduling helps engineers choose batching, stream assignment, time slicing, process isolation, or Multi-Instance GPU (MIG) where available.
  • Implementation Link: Supports the design of multi-user inference services with measured tail-latency and isolation behavior.
  • Deep dive: GPU Scheduling And Context Switching

2. Zero-Copy Data Paths (io_uring, GPUDirect)

  • Concept: Reducing intermediate copies among kernel buffers, user-space buffers, pinned host memory, device memory, storage devices, and network adapters.
  • Why it's important: In large model serving and data-intensive training, weight loading, input ingestion, or distributed communication can become limited by memory movement and CPU overhead.
  • Implementation Link: Relevant to high-throughput data loaders, model loading paths, GPUDirect-style storage or networking, and distributed communication.
  • Deep dive: Zero-Copy Data Paths

3. Custom Memory Allocators (e.g., Stream-Ordered Allocation)

  • Concept: Using memory pools, caching allocators, and stream-ordered APIs such as cudaMallocAsync for accelerator-visible memory.
  • Why it's important: General allocation paths and unmanaged device allocation can introduce synchronization, fragmentation, and unpredictable latency when they are not aligned with stream-ordered execution.
  • Implementation Link: Helps reduce allocator-induced synchronization and memory-fragmentation effects in complex model pipelines.
  • Deep dive: Custom Memory Allocators

4. Kernel Fusion & Graph Capture

  • Concept: Merging compatible operations into a single device kernel or capturing a repeated execution sequence for replay.
  • Why it's important: Kernel launches have host and runtime overhead, and unfused operations often materialize intermediate tensors. Fusion can reduce both launch count and memory traffic when the operations are compatible.
  • Implementation Link: A common optimization in libraries and compilers such as TensorRT, framework JIT compilers, and ahead-of-time compiler pipelines.
  • Deep dive: Kernel Fusion And Graph Capture

Recommended Resources

  • "Operating Systems: Three Easy Pieces" (Arpaci-Dusseau): For general OS background (Virtualization, Concurrency).
  • NVIDIA CUDA Programming Guide: Specifically the sections on Asynchronous Concurrent Execution and CUDA Graphs.
  • Linux io_uring Documentation: For high-performance I/O.
  • GPUDirect Storage (GDS) Whitepapers: For understanding direct storage-to-GPU paths.

/ Continue

Follow the technical trail.

Use the dense notes as the source material, then move through the guided route, writing, or project proof when you want a cleaner entry point.