Structured sparsity

Structured sparsity is a pruning method that zeroes neural network weights in a regular pattern that GPU tensor hardware can skip. Skipping them lets the hardware finish the same matrix math in up to half the time. NVIDIA's 2:4 form, introduced with the Ampere architecture and the A100, requires two zeros in every four consecutive values. Sparse Tensor Cores store only the nonzero values plus small metadata, then fetch the matching values from the other operand.

AMD lists the Instinct MI355X at 2.5 petaFLOPS (PFLOPS) of dense FP16 matrix math and 5 PFLOPS with structured sparsity. NVIDIA's eight-GPU HGX B200 table prints FP8 as 72 PFLOPS, with a footnote that dense is half that. Its FP4 row reads 144 PFLOPS sparse and 72 PFLOPS dense. The HGX B300's FP4 row reads 144 PFLOPS and 108 PFLOPS, so sparse is not always double dense. Measured gains run smaller. A sparse ResNeXt-101 in TensorRT 8.0 on an A100 ran up to about 20% faster than dense at large batch sizes.

Compare GPUs on dense figures, and read the footnotes to find them, since some rows print only the sparse rate. Count a sparse figure only for a model pruned to the hardware's pattern and retrained to recover its accuracy.

Sources

Last verified October 7, 2026.