M:N sparsity on NVIDIA tensor cores

Look at the first row of the structured-sparse matrix on the slide. Read it four cells at a time. The first four hold a value, a zero, a zero, a value. The next four hold a zero, a value, a value, a zero. Every group of four consecutive weights along the row has exactly two survivors, and the survivors can sit anywhere inside the group. Mishra et al. write the metadata for that row as [[0, 3], [1, 2]]: the positions of the survivors, group by group.

That is the whole definition of 2:4 sparsity: for each group of four values along a row, at least two must be zero. The general form keeps the same shape. Write a pattern as a:b, where the first number is how many weights survive and the second is the block size. Sparsity is then fixed by the pattern, not by the data.

2:4 is the instance that ships. Ampere sparse tensor cores accept it for FP16, BF16 and INT8 weights, and for TF32 the block halves to 1:2, still 50 percent (Mishra et al., section 3.2). NVIDIA describes the pattern as "two non-zero values in every four-entry vector" and calls it fine-grained structured sparsity (NVIDIA Ampere in-depth blog). The A100 whitepaper is explicit that the pattern is enforced along rows.

Fine-grained in what it removes, structured in how many

Lecture 04-1 laid out pruning granularities on a line from irregular fine-grained pruning, which may zero any single weight, to channel pruning, which removes whole filters. 2:4 sits between them and borrows the best of each. It is fine-grained because the unit removed is a single weight, so the pruning criterion still gets to pick within every block, which is why accuracy survives. It is structured because the count per block is constant, so the hardware knows in advance that every four weights hide exactly two multiplies.

That constancy is the entire engineering payoff. Unstructured sparsity, the kind EIE handles, needs a data-dependent index for every nonzero and a pointer to the start of every column, and the paper notes it leads to poor utilization of cache lines because the survivors land anywhere. With 2:4, the paper points out, the sparsity is constant across the matrix, so no indirection is required: a nonzero's position in memory can be computed directly from the compression rate. Two bits per survivor say which of the four slots it came from, and nothing else needs to be stored.

One more fact will matter in the next two concepts: the groups of four run along the row of the weight matrix W. In the GEMM that a layer becomes, that row is the reduction dimension, the axis along which products are summed. The pattern has to live there, because that is the only axis on which skipping a weight also skips a multiply.

Where this sits among the three case studies

The divider slide that opens this section lists three sparse-hardware case studies. EIE (Efficient Inference Engine) came first: a custom accelerator exploiting Weight sparsity and Activation sparsity at once. This part is the middle one, and it takes the opposite bet. It exploits weight sparsity only, at a fixed and modest 50 percent, in exchange for running on a GPU that already sits in every data center. TorchSparse and PointAcc, which follow, go back to activation sparsity in point clouds. Keeping the three apart by which sparsity they exploit and what hardware they need is the fastest way to remember them.

Take one group of four FP16 weights. Dense, it costs 4 x 16 = 64 bits. Under 2:4, only two values are stored, 2 x 16 = 32 bits, plus a 2-bit index for each, 2 x 2 = 4 bits. The group now costs 36 bits, a saving of about 44 percent (Mishra et al., section 3.1). Not 50 percent: the indices are small but they are not free.

Scale that to a whole matrix and you get the format on the slide. A structured-sparse W of size R x C is stored as an R x C/2 block of nonzero values and an R x C/2 block of two-bit indices. The slide's instruction is to push all the nonzero elements to the left in memory. The survivors of each row are packed contiguously at half the original width, and the index block, drawn beside them, records where each came from.

The index cost is a fixed 2 bits per stored value, so its weight relative to the values depends on the value width: 2 / 16 = 12.5% overhead for FP16 and 2 / 8 = 25% for INT8, which is why the INT8 saving is the smaller 37.5 percent (Mishra et al., section 3.1). Compare that with the format EIE relied on. CSC stores a 4-bit relative index and a 4-bit weight-sharing code per nonzero, a 16-bit pointer per column, and a padding entry whenever a run of zeros exceeds fifteen (Han et al., EIE, section III). Mishra et al. note that a plain CSR with 8-bit weights and 16-bit column indices can spend up to 200 percent of the value bits on metadata. 2:4 spends 12.5 or 25 percent, with no pointers and no padding, because the pattern itself carries most of the position information.

Why insist on pushing everything to the left rather than leaving zeros in place and skipping them? Because memory is read in wide lines. With the survivors contiguous at half width, every byte a memory read brings in is a value the tensor core will use, which the paper describes as letting hardware fully utilize large memory reads. The A100 whitepaper puts the same point as a reduction of memory storage and bandwidth by almost 2x. Zeros left in place would fill half of every line with nothing.

Follow one output element through a small GEMM with K = 8. Dense, the tensor core takes a row of A with eight values, a column of B with eight values, multiplies them pairwise and accumulates eight products. Now store that row of A in 2:4 form: four values and four 2-bit indices. The core reads the indices, pulls exactly the four elements of the B column that sit at those positions, multiplies four pairs, and accumulates. The slide's caption says it in one line: the indices are used to mask out the inputs, and only two multiplications will be done out of four.

The rule behind the example is worth stating precisely, because exam questions turn on it. A Sparse tensor core performs sparse matrix times dense matrix equals dense matrix. Only the first operand, A of size M x K, is compressed, to M x K/2 values plus indices. The second operand B (K x N) and the output C (M x N) stay dense (Mishra et al., section 3.2). NVIDIA's TensorRT blog describes the mechanism as using the metadata to pull only the necessary values from the other, uncompressed operand. The metadata belongs to the weights, but it is applied to the activations.

Halving the pairs halves the work, and the hardware is built so that the saving shows up as time. The A100 whitepaper states that a standard MMA on a 16 x 8 x 16 tile takes some number of cycles N, and the sparse MMA on the same tile takes N/2, a 2x speedup. The paper's Table 1 reports the resulting peak rates.

Why must the pattern run along K? Because K is the axis the selector walks. Each output element is a dot product over K, and a zero weight at position k means the product with B[k, n] can be dropped for every n. Zeros arranged along M or N would zero entire rows or columns of the output, which is a smaller network, not a faster GEMM. This is also why the library has shape rules: the reduction dimension must be a multiple of 16 for 16-bit formats and 32 for INT8, and layers that fail the rule, such as the first convolution of an image network with K = 3 x 7 x 7 = 147, are simply left dense (Mishra et al., sections 3.2 and 5.1). cuSPARSELt exposes the compression and the sparse GEMM to programmers, and TensorRT 8 applies them automatically to a network whose weights already follow the pattern.

What changed since EIE

Set the two designs beside each other and the trade becomes visible. EIE (Efficient Inference Engine) reaches for every zero it can find: about 90 percent in the weights and about 70 percent in the activations, both exploited at once. To do that it needs Leading non-zero detection to find the next live activation, a FIFO in front of each Processing element (PE) to keep Load balance across uneven columns, and a custom chip to hold all of it. The sparse tensor core asks for far less sparsity, exactly 50 percent in the weights alone, which is static and known before the run. That is what lets the whole mechanism collapse into a multiplexer driven by 2-bit tags, small enough to add to a tensor core that every GPU already has.

ResNet-50 on ImageNet scores 76.1 top-1 dense in FP16. After 2:4 pruning and retraining it scores 76.2 in FP16 and 76.2 in INT8 (Mishra et al., Table 2). Half the weights are gone from almost every layer, the GEMMs can run on the sparse path, and the accuracy did not move. The slide's takeaway is the plain version of this: pruning CNNs with 2:4 sparsity brings a large speedup for GEMM workloads and does not incur a performance drop for the models. The rest of this concept explains how that is achieved, how large the speedup really is, and where both claims stop.

The recipe: one prune, one retrain, no search

The deck does not show the procedure itself, so take it from the paper (Mishra et al., section 4) and from the A100 whitepaper, which calls it a simple and universal recipe. It has three steps.

1. Train the model without sparsity, exactly as you normally would.
2. Prune it to 2:4 by magnitude: in every group of four weights along the row, zero the two with the smallest absolute value.
3. Retrain from those weights using the same optimizer, learning-rate schedule and number of epochs as step 1, with the zeros held fixed so the pattern survives. Optimizer state such as momentum is reset.

Two things distinguish this from the Fine-tuning and Iterative pruning you met earlier in this lecture. It is one-shot: a single prune straight to the target, then a full second training run rather than a short low-learning-rate touch-up. And it needs no hyper-parameter search, because step 3 reuses step 1 wholesale. The paper is candid about the cost, a second full training, and argues it is worth paying once for a workflow with nothing to tune, since deployment amortizes it. NVIDIA packages the procedure as the ASP library for PyTorch.

How much faster: a staircase toward 2x

The chart on the slide compares INT8 GEMMs from cuSPARSELt, with one operand in 2:4 form, against dense cuBLAS GEMMs on an A100, at M = N = 10240 and K sweeping from 1280 to 20480. The bars start near 1.2x and climb toward 2x without reaching it. Mishra et al. read it the same way: larger GEMMs achieve nearly a 2x speedup with sparse tensor cores.

Why does the speedup depend on K at all, when the pattern halves the multiplies at every size? Because the sparse tensor core halves only the arithmetic. A GEMM also has to move its operands in and its result out, and it pays fixed launch and scheduling costs. A small GEMM has low arithmetic intensity: few operations per byte moved, so its time is dominated by memory traffic and overheads that 2:4 does not touch. As K grows, the operations per output element grow while the output traffic does not, the GEMM becomes math bound, and halving the math approaches halving the time. The paper says exactly this: larger GEMMs tend to have higher arithmetic intensity, so they get closer to the 2x speedup.

Accuracy: the table that justifies the claim

The right half of the slide is the paper's Table 2: twenty ImageNet classifiers, each in dense FP16, sparse FP16 and sparse INT8. Read down any row and the three numbers agree to within a few tenths of a point; the widest spread is SUNet-128 at 75.6, 76.0 and 75.4. The paper states that the differences are within run-to-run variation, which is the honest way to say zero loss.

Two details in that table repay attention. Sparse INT8 matches sparse FP16, so the pruning and INT8 quantization, which this course turns to after pruning, compose without compounding their losses, at least on these networks. And the largest model, ResNeXt-101-32x16 trained with weak supervision, loses 0.2 in FP16 and nothing in INT8, so the pattern is not exploiting a weakness of small or old architectures. VGG, DenseNet, Inception, Xception and the dilated residual networks all behave the same way.