Fine-grained, N:M and channel pruning in practice

Go back to the very first picture of this lecture: a network before and after pruning, with individual synapses cut out while every neuron stays. That picture is fine-grained pruning, the synapse pruning of slide 4 seen through the lens of granularity. Slides 19 and 20 bring it back only to attach numbers to it, so treat them as a short callback rather than new material.

The numbers are the ones from part 02. Han et al. pruned AlexNet from 61 M to 6.7 M parameters and VGG-16 from 138 M to 10.3 M without losing accuracy, a pruning ratio of roughly 89% and 93% (Han, Pool, Tran and Dally, NIPS 2015). Those results are only possible because the method may zero any single weight, wherever it sits in the tensor. The slide calls this "flexible pruning indices": the set of surviving positions is unconstrained, so the algorithm can find redundant weights anywhere and usually reaches the largest compression ratio of all granularities.

Why flexibility is expensive at run time

Now look at what the flexibility leaves behind. After fine-grained pruning the weight tensor still has its original shape; it is simply riddled with zeros at irregular positions. A dense matrix multiply on a GPU does not know the zeros are there and multiplies them anyway, so the work is unchanged. To skip them you must store the weights in a sparse format such as CSR, CSC or COO, where every surviving value carries an index saying where it belongs. Mishra et al. note that these formats make memory accesses data dependent and that the index metadata can cost up to 200% of the weight storage when the weights are 8-bit (Mishra et al., 2021). Zhou et al. summarize the situation in one line: fine-grained sparsity "can achieve a high compression ratio but is not hardware friendly and hence receives limited speed gains" (Zhou et al., ICLR 2021).

That is why the red line on slide 20 matters more than the table. Speedup from fine-grained sparsity needs hardware built around indices. Han's own EIE accelerator is the example the MIT source deck names: by reading compressed weights and skipping zero activations, it ran fully-connected layers 189x faster than a CPU and 13x faster than a GPU running the uncompressed network (Han et al., ISCA 2016). With sparse kernels the picture depends on batch size: in EIE's own measurements cuSPARSE ran pruned FC layers 4x to 9x faster than dense at batch 1 on a Titan X, but slower at batch 64, which is why the slide says "not GPU (easily)". On an ordinary GPU with dense kernels, the same pruned network runs at roughly dense speed.

Take one row of eight FP16 weights and split it into two groups of four. In each group keep the two largest magnitudes and zero the other two. Every group now has exactly two zeros and two nonzeros, so the row is 50% sparse, and so is every other row treated the same way. That is 2:4 sparsity, the classic case of N:M sparsity and the reason the lecture lists pattern-based pruning as its own step on the granularity spectrum.

The general rule needs one careful sentence, because the slide's sentence is ambiguous. In NVIDIA's definition, which is the one hardware implements, an N:M pattern allows at most N nonzero values in every contiguous group of M. The Ampere whitepaper describes 2:4 as a matrix "that allows two non-zero values in every four-entry vector", and Mishra et al. write of "the 2 nonzero values in each group of 4" (NVIDIA, 2020; Mishra et al., 2021). Slide 21 instead says that "N of them is pruned". For 2:4 the two readings agree, since two kept and two pruned are the same thing. For any other pattern they do not.

Why a pattern is the middle of the spectrum

Inside each group of four the pattern is still fine-grained: any two of the four positions may survive, six possible masks, chosen by magnitude. Across the matrix it is rigidly structured: every group holds exactly two values, no more and no fewer. Mishra et al. point out what this buys: because the sparsity is constant across the matrix, "there is no indirection required; a nonzero value's position in memory can be determined from the compression rate directly" (Mishra et al., 2021). The hardware always knows that group g starts at value slot 2g. That is the difference between a lookup and an address calculation, and it is what makes fine-grained-in-the-small, structured-in-the-large worth a place of its own.

The compressed format

The rule also fixes the storage layout, which is the right-hand panel of slide 21. An R by C dense matrix becomes an R by C/2 block of nonzero values plus an R by C/2 block of 2-bit indices, where each index records the position, 0 to 3, of a kept value inside its group of four. Two bits are enough because a group has only four positions. One corner case keeps the format regular: if a group happens to have three or four zeros after training, two values are stored anyway, padded with zeros as needed (Mishra et al., 2021).

The two tables carry an insight that matters when you combine pruning with the quantization lectures: the index costs two bits whatever the value width, so the narrower the value type the more the index costs relatively. Mishra et al. give the same figures, about 44% saved for 16-bit and about 38% for 8-bit weights, which is why NVIDIA says the format cuts storage and bandwidth by "almost" 2x rather than exactly 2x (Mishra et al., 2021; NVIDIA, 2020).

Where the 2x speedup comes from

Storage is only half of the payoff. The Ampere A100 added Sparse Tensor Core instructions that take a 2:4 compressed operand and, in the whitepaper's words, "skip the compute on entries that have zero values, resulting in a doubling of the Tensor Core compute throughput": a 16 x 8 x 16 matrix multiply instruction completes in half the cycles of its dense counterpart (NVIDIA, 2020). Software reaches it through TensorRT 8 and cuSPARSELt; the slide's link covers the TensorRT 8.0 path. Zhou et al. restate the same about 2x for the A100, citing NVIDIA rather than measuring it themselves (Zhou et al., ICLR 2021).

Try the pattern yourself below. Switch the convention from "N nonzero" to "N pruned" at 2:4 and nothing changes; switch it at 1:4 and the sparsity flips from 75% to 25%, which is exactly the ambiguity flagged above. Then toggle FP16 to INT8 and watch the index overhead double.

A format that halves storage and doubles matmul throughput is worth nothing if the network gets worse. So look at one row first: ResNet-50 on ImageNet scores 76.1% top-1 dense in FP16 and 76.2% after 2:4 pruning. Half the weights are gone and the accuracy is unchanged; the extra 0.1 is run-to-run noise, not a gain.

The rule that produced that row is NVIDIA's three-step recipe. Train the dense network as usual. Prune it to 2:4 by keeping the two largest magnitudes in every group of four, which fixes the mask. Then retrain with the mask held fixed, using the original schedule and hyperparameters, so the survivors absorb the job of the pruned weights. That last step is the fine-tuning loop of part 02 applied under a structural constraint, and the whitepaper reports "virtually no loss in inferencing accuracy" across dozens of networks with it (NVIDIA, 2020). The choice of mask by magnitude inside each group previews the magnitude criterion of part 05.

Read the table across, never down. The seven rows use four different metrics (top-1 accuracy, box AP, BLEU and F1), so the only meaningful comparison is dense versus sparse within one row. Read that way the evidence is unusually broad: image classification, object detection, instance segmentation, translation and question answering, spanning convolutional networks and Transformers, and not one row lost measurable accuracy. The whitepaper's explanation is that after training "only a subset of weights have acquired a meaningful purpose", and the retraining step lets the network re-adapt around the fixed mask (NVIDIA, 2020).

Now the coarse end of the spectrum. A convolution layer with 64 output channels is pruned at sparsity 0.5: thirty-two whole filters are deleted and the layer becomes a 32-channel layer. Nothing about the result is sparse. Its weight tensor, in the [c_o, c_i, k_h, k_w] layout of part 03, is simply smaller in the c_o dimension, and the next layer, which consumed those channels, is smaller in its c_i dimension.

That is the whole argument for channel pruning, and both sides of the slide's trade-off follow from it. Pro: direct speedup on any hardware, because a network with fewer channels does less work in every dense library that exists. Li et al. put it precisely: removing whole filters "does not result in sparse connectivity patterns", so it "does not need the support of sparse convolution libraries and can work with existing efficient BLAS libraries" (Li et al., ICLR 2017). He, Zhang and Sun report a 5x speedup on VGG-16 for a 0.3% increase in error and 2x on ResNet for 1.4% (He, Zhang and Sun, ICCV 2017). Con: a smaller compression ratio, because a channel is a much coarser unit than a weight. Keeping a channel means keeping all of its weights, including the weak ones that fine-grained pruning would have dropped, so you cannot reach the 9x to 13x of slide 20.

Pruning one layer shrinks two

The example above hid a compounding effect that the slide's bar chart depends on. Pruning the output channels of layer l removes rows from its own weight tensor and columns from layer l+1, because the input channels of l+1 are the output channels of l. The per-layer sparsity values on slide 24 therefore do not add up the way they look; they multiply through the chain.

Uniform shrink versus channel prune

The inequality on slide 24, "Uniform Shrink < Channel Prune", compares accuracy at a comparable budget. Uniform shrink applies one ratio to every layer, which is what a MobileNet width multiplier does. It ignores the fact that layers differ in how much redundancy they carry: some tolerate 70% removal and others only 20%. Channel pruning proper starts from the trained wide network, chooses a ratio per layer, deletes the least useful channels in each, and fine-tunes. Under the same latency or MAC budget, spending the cuts where they hurt least keeps more accuracy. Slide 24 defers the obvious question, how to find those ratios; the sensitivity analysis of lecture 04-2 (Pruning ratios, fine-tuning and sparse hardware) and the AMC search of the next concept are the answers.

On a Google Pixel 1, the full-width MobileNet takes 123.3 ms per image and scores 70.6% ImageNet top-1. Shrinking it uniformly with the 0.75 width multiplier brings the latency down to 72.3 ms but the accuracy down to 68.4%. The channel-pruned model found by AMC runs in 68.3 ms at 70.5%: as fast as the shrunken model, 2.1 points more accurate, and only 0.1 below the full model at 1.81x its speed (He et al., ECCV 2018).

AMC (AutoML for Model Compression) is the method behind the red curve on slide 25. It treats the per-layer sparsity ratios as actions for a reinforcement-learning agent, which walks the network layer by layer, proposes a ratio for each, and is rewarded by the accuracy the pruned model reaches under a FLOPs or latency constraint. The pruned model is then fine-tuned. The baseline is the easiest thing one could do instead, which the paper states plainly: "the easiest way to reduce the channels of a model is to use uniform channel shrinkage, i.e. use a width multiplier", and the result is that AMC "consistently outperforms the uniform baselines" (He et al., ECCV 2018).

The chart on the slide is AMC's Figure 5b. The four table rows above are exact; the remaining points in the table below are read off the figure and are approximate. Either way the shape is the same. At equal latency the searched ratios are about 1.5 to 2 points better. At similar accuracy they are much faster: the 50% latency-budget model keeps 70.2% at 63.3 ms, 1.95x faster than the 123.3 ms full model at 70.6%, only 0.4 points higher.

Choosing a granularity for a real board

This is where the part comes together for your project. The granularity you pick is a hardware decision, not a compression decision:

• Target is an Ampere-class or newer NVIDIA GPU (a Jetson Orin, an A100): prune to 2:4. Half the weights, about 2x on the matmuls, accuracy held by retraining.
• Target is a phone CPU, a microcontroller or an NPU with dense kernels only: channel pruning with searched per-layer ratios, AMC style. A smaller dense network is the safest choice, since it needs no special kernels. Some mobile runtimes (XNNPACK) do accelerate high unstructured sparsity in 1x1 convolutions, so check what your runtime supports.
• Target has a sparse accelerator such as EIE, or the constraint is storage and not latency: fine-grained pruning, for the largest compression ratio.