This topic explains how high-level tensor operations are mapped onto hardware execution units and instruction set extensions. The focus is on the engineering consequences of using vector units, matrix units, domain-specific accelerators, and reduced-precision formats.
Core Ideas to Master
1. Systolic Arrays & Matrix Units (Tensor Cores / AMX)
- Concept: How matrix units implement tiled multiply-accumulate operations such as $C = A \times B + C$ using specialized datapaths and local accumulators.
- Why it's important: Dense layers, attention projections, and many convolutions spend much of their time in tensor contractions. Matrix units improve utilization by reusing operands near the arithmetic units and reducing instruction overhead.
- Implementation Link: Understanding dataflow choices such as weight-stationary and output-stationary execution helps engineers choose tile shapes, memory layouts, and custom-kernel schedules.
- Deep dive: Systolic Arrays And Matrix Units
2. SIMD vs. MIMD in AI (AVX-512, Neon)
- Concept: Single Instruction Multiple Data (SIMD) execution for vector operations, contrasted with Multiple Instruction Multiple Data (MIMD) execution for independent workers.
- Why it's important: Activation functions, softmax, layer normalization, sampling, masking, and token processing often execute outside the main GEMM path. Their runtime contribution can be significant even when their floating-point operation count is small.
- Implementation Link: SIMD and MIMD reasoning helps identify non-GEMM bottlenecks, vectorization failures, divergence, and host-side scheduling limits.
- Deep dive: SIMD And MIMD In AI
3. Precision Engineering (FP16, BF16, INT8, FP4)
- Concept: The trade-off between numerical range, precision, storage footprint, bandwidth demand, and hardware throughput.
- Why it's important: Reduced-precision formats can lower memory traffic and increase accelerator throughput, but they also change overflow, underflow, rounding, and accumulation behavior.
- Implementation Link: Precision choices determine quantization granularity, accumulator types, kernel availability, validation strategy, and mixed-precision training behavior.
- Deep dive: Precision Engineering
4. Accelerator Selection For ML Workloads
- Concept: Selecting GPUs based on workload constraints such as memory capacity, precision support, kernel availability, and interconnect bandwidth.
- Why it's important: Peak FLOPS alone does not predict real throughput; selection must align with model size, latency targets, and software support.
- Implementation Link: Defines a measurable framework for comparing devices and avoiding software or memory bottlenecks.
- Deep dive: Accelerator Selection For ML Workloads
Recommended Resources
- "Computer Architecture: A Quantitative Approach" (Hennessy & Patterson): Specifically the chapter on Domain-Specific Architectures.
- NVIDIA Tensor Core Documentation: Deep dive into the
mma(matrix multiply-accumulate) PTX instructions. - Intel AMX (Advanced Matrix Extensions) Reference Manual: Understanding how tiles are managed in the CPU register state.
- Google TPU Whitepapers: Architectural case studies of systolic-array design for AI workloads.