A general-purpose core can execute multiply and add instructions, but it usually fetches operands through registers and caches under the control of scalar instruction streams. For large matrix products, the arithmetic pattern is regular enough that specialized hardware can replace much of that instruction overhead with fixed data movement. Matrix units, including GPU tensor cores, Intel AMX tiles, and TPU-style systolic arrays, exploit this regularity.

Systolic Arrays

A systolic array is a grid of simple processing elements. Each element performs a small operation, commonly a multiply-accumulate, and passes data to neighboring elements according to a fixed schedule. Values propagate through the array over successive cycles.

Systolic arrays are subsystems in a datapath, and are created with combinational circuits.

They are homogeneous networks of data processing units, such as simple 1-bit adders in this case.

They matter for VLSI deep learning, since deep learning architectures have lots of repetitive steps of computation to be performed. This can be implemented with systolic arrays.

In an output-stationary design, each processing element holds a partial sum for one output position. Values from \(A\) and \(B\) stream through the grid, and each processing element updates its local accumulator. The output value remains stationary until the required reductions over \(k\) are complete.

In a weight-stationary design, weights remain in processing elements while activations stream across them. This can be appropriate when the same weights are reused over many input examples or sequence positions.

The central engineering point is that the hardware avoids sending every intermediate value back through the memory hierarchy. Partial sums remain close to the arithmetic units, and input values are reused as they move through the array.