Pruning criteria: magnitude and scaling factors

Start with one neuron and three inputs. Its output is y = ReLU(10 x_0 - 8 x_1 + 0.1 x_2), and the budget says one of the three connections has to go. Removing the 10 changes the pre-activation by up to 10 |x_0|. Removing the -8 changes it by up to 8 |x_1|. Removing the 0.1 changes it by at most 0.1 |x_2|. If the three inputs are of similar size, the third cut is a hundred times gentler than the first, and that is the one to make.

That small decision contains the whole idea of a Pruning criterion. Pruning asks the network to give up parameters, and the principle the lecture states is simple: the less important the parameters being removed are, the better the performance of the pruned network is. Every criterion in this lecture is a different way of estimating importance, and importance always means the same thing underneath: how much the loss would change if this parameter, or this group, were set to zero. Magnitude, the subject of this part, is the cheapest such estimate. Second-order methods in the next part spend far more compute to estimate the same quantity more carefully (Second-order pruning).

Notice what the example needed to make the decision: nothing but the weights. No data, no gradients, no second run through the training set. That is the appeal of magnitude criteria and also their limit. They answer the Pruning formulation only approximately, because the formulation asks for the smallest increase in loss under a budget on nonzeros, and magnitude is a proxy for loss, not the loss itself. The next three sections make the proxy precise for single weights, for rows, and for whole channels.

Now scale the neuron up to a weight matrix. Take W = [[3, -2], [1, -5]] and a budget of 50 percent Sparsity, which on four weights means exactly two zeros. Write the absolute value of every entry, keep the two largest, zero the rest. The importances are [[3, 2], [1, 5]], the survivors are 3 and -5, and the pruned matrix is [[3, 0], [0, -5]].

This is Magnitude-based pruning at its finest granularity, the fine-grained case from part 03, and it is the criterion of the paper the slides cite. Han, Pool, Tran and Dally train a network, prune every connection whose weight falls below a threshold, and retrain the survivors. Done once, that took AlexNet to 5x fewer parameters; done iteratively, prune then retrain then prune again, it reached 9x on AlexNet (61M to 6.7M) and 13x on VGG-16 (138M to 10.3M) with no loss of accuracy. They also tried the obvious alternative, removing weights at random with probability tied to their magnitude, and report that it gave worse results. Hard thresholding on |w| won.

In code, this is one call. PyTorch's prune.l1_unstructured(module, name, amount) prunes a tensor by removing the units with the lowest L1 norm, and amount can be a fraction such as 0.5 or an absolute count. The name is exact: on a single element the L1 norm is just its absolute value, so element-wise L1 and |W| are the same criterion, and the slide's label "L1-norm, element-wise" is the same statement as Importance = |W|. Any other p would give the same ranking on single elements, which is why the choice of norm only starts to matter in the next section.

Keep the same matrix, [[3, -2], [1, -5]], but change the unit of removal. Instead of two individual entries, remove one whole row. A row of a linear layer's weight matrix is one output neuron's incoming connections, so deleting it is Neuron pruning, and the result is a smaller dense matrix that any library runs faster (Coarse-grained (structured) pruning). The question is how to score a row when it holds several weights of different sizes and signs.

The lecture's answer is to treat the row as a Structural set, written W^(S) for the set S of parameters it contains, and to collapse its absolute values into one number. Two collapses are standard. Add them: the L1-norm importance. Or square them, add, and take the root: the L2-norm importance. Row 0 scores |3| + |-2| = 5 under L1 and sqrt(9 + 4) = sqrt(13) = 3.61 under L2. Row 1 scores |1| + |-5| = 6 and sqrt(1 + 25) = sqrt(26) = 5.10. Both norms rank row 1 higher, so both prune row 0, and the pruned matrix is [[0, 0], [1, -5]].

One formula for every p

L1 and L2 are two settings of one dial. Goodfellow, Bengio and Courville define the Lp norm of a vector for any p ≥ 1 as the p-th root of the sum of p-th powers of the absolute values. Set p = 1 and the root and the power vanish, leaving slide 29. Set p = 2 and you get the Euclidean length of slide 30. Slide 31 is the same example under the general formula, not a new method.

When the norm changes the answer

On the slide's matrix L1 and L2 agree, and it is tempting to file the norm as a cosmetic choice. It is not. Compare a row [3, 3] with a row [0, 5] competing for the last surviving slot. L1 scores them 6 against 5 and keeps [3, 3]. L2 scores them sqrt(18) = 4.24 against 5 and keeps [0, 5]. Squaring amplifies the single largest entry, so L2 rewards a row with one dominant weight. Summing treats every unit of magnitude the same, so L1 rewards a row with many moderate weights. Goodfellow et al. describe the same contrast in general terms: the L1 norm grows at the same rate in all locations, while the squared L2 norm increases very slowly near the origin.

Try it yourself. The explorer below scores an editable matrix element-wise or row-wise, always computes both row norms so a disagreement is visible without toggling, and ships with the slide's example and the [3, 3] against [0, 5] case as presets. Change a single entry and watch which row the two norms fight over.

Where the row-wise criteria come from

The L1 row score is the criterion of Li et al. Their filter importance is the sum of the absolute kernel weights of a filter, which they describe as an expectation of the magnitude of the output feature map. They remove the m filters with the smallest sums, delete the matching input channels from the next layer, and show that pruning the smallest works better than pruning at random or pruning the largest. On CIFAR-10 this cut inference cost by up to 34 percent for VGG-16 and 38 percent for ResNet-110 with retraining recovering the accuracy. Because a filter is a structural set, this filter pruning is Channel pruning from part 04 with an explicit score attached.

The L2 row score is what Wen et al. put inside training. Their structured sparsity learning adds a group Lasso regularizer to the loss, a sum over groups of sqrt(sum of squared weights in the group), which is exactly the L2 norm of each structural set. The optimizer is then rewarded for driving whole filters, channels, filter shapes or even layers to zero, and they report 5.1x CPU and 3.1x GPU speedups on AlexNet's convolutional layers, and a ResNet on CIFAR-10 shrunk from 20 to 18 layers with accuracy moving from 91.25 to 92.60 percent, still above the original 32-layer ResNet. The difference between the two papers is when the norm is used: Li et al. score a trained network after the fact, Wen et al. shape the network during training so the sets to remove are already near zero.

PyTorch exposes the post-hoc version directly. prune.ln_structured(module, name, amount, n, dim) removes the channels with the lowest Ln norm along the specified dimension; n = 1 or 2 picks the norm, and dim = 0 on a weight of shape [c_o, c_i, k_h, k_w] scores whole filters. Finer structural sets such as a row inside a kernel (Vector-level pruning) use the same formula on a smaller S.

A convolution layer has N filters, one per output channel (Convolution weight dimensions). Give each filter a single trainable number, its scaling factor, and multiply that channel's entire output by it. On the slide the factors come out as 1.17, 0.10, 0.29, 0.82, ..., 0.56. Suppose the threshold is 0.3 (the slide only marks the two smallest factors as pruned) and two channels fall below it: filter 1 at 0.10 and filter 2 at 0.29. Delete those filters, delete the input channels that consumed their outputs in the next layer, and what remains is filter 0, filter 3, on to filter N-1: a physically narrower layer.

This is Scaling-based pruning, and it differs from the norms in one important way. A norm scores a trained network after the fact, from the weights alone. A scaling factor is a parameter that training itself sets, so the network is asked, during training, how much it wants each channel. Liu et al. call the method network slimming and make the question sharp by adding an L1 penalty on the factors to the loss.

The L1 penalty is the same L1 from the previous section, now used as a regularizer rather than a score: it charges every unit of |gamma| equally, so factors that the loss does not defend slide all the way to zero. Liu et al. use lambda = 1e-4 for VGGNet and 1e-5 for ResNet and DenseNet on CIFAR. After training they sort every factor in the whole network and set one global threshold at a percentile: pruning 70 percent of channels means the threshold is the 70th percentile of all factors, so layers with many weak channels lose more than layers with strong ones, which is the per-layer Pruning ratio falling out automatically. Then they fine-tune the slimmed network, and can repeat the whole loop.

Why the factor is already there: batch norm gamma

Where does the scaling factor live? Liu et al. observe that almost every modern convolution is followed by batch normalization, and batch normalization already contains one. Ioffe and Szegedy normalize each channel of a mini-batch to zero mean and unit variance, then let the network undo that with two learned parameters per channel: a scale gamma and a shift beta. Goodfellow et al. explain why they exist: the normalized activation is replaced by gamma H' + beta so the new variable can have any mean and standard deviation the network wants. That gamma is the Batch normalization scaling factor, one per output channel, and it is the scaling factor the slide reuses.

Two things make gamma the right choice, and both are exam material. First, cost: reusing gamma adds no new parameters and no new layers. Liu et al. say it introduces no overhead, and in PyTorch the vector is already sitting in BatchNorm2d.weight, a learnable parameter of size C initialized to one, next to beta in BatchNorm2d.bias initialized to zero, with epsilon = 1e-5 by default.

Second, and more subtle, meaning. Suppose you skipped batch normalization and simply inserted a scaling layer after the convolution. Convolution is linear and scaling is linear, so the network could halve every factor and double the corresponding filter weights without changing a single output. The factor would then say nothing about importance, because the network can move magnitude freely between the factor and the weights, and an L1 penalty on the factor would be defeated the same way. Liu et al. call such a factor meaningless. Batch normalization breaks the symmetry: the normalized activation has a fixed unit scale regardless of what the filter weights do, so gamma alone sets the size of the channel's output and its magnitude is a genuine measure of how much the channel contributes.