Why every layer needs its own pruning ratio

Start with a measurement, not a principle. Take MobileNet on ImageNet and run it on a Google Pixel 1 CPU with TensorFlow Lite. The dense network scores 70.6% top-1 at 123.3 ms per image. Now shrink it two different ways to roughly half the work. The first way multiplies every layer's width by 0.75, the built-in MobileNet knob. The second way, AMC, lets every layer keep a different fraction of its channels (He et al., ECCV 2018, Table 4). MMACs counts millions of multiply-accumulate operations per image, a hardware-neutral measure of compute; a FLOPs target asks AMC to halve that compute, while a latency target asks it to halve measured time on the phone.

Read the middle two rows against each other. At nearly the same latency, 72.3 ms versus 68.3 ms, the per-layer version keeps 2.1 points more accuracy. Read the first and third rows together and the gap is even more striking: AMC's 68.3 ms model loses only 0.1 points against the full network at 123.3 ms. Slide 9 plots the same story as a chart (AMC Figure 5b), and the values read off it land in one table.

Every AMC point sits above every uniform point. The uniform line on that chart mixes MobileNet's width multiplier with its input-resolution multiplier, which is why it has four points while Table 4 lists two; either knob shrinks every layer by the same factor.

The two stacks of bars on the slide are the whole idea in one picture. Uniform shrinking keeps every layer the same shape, just thinner: MobileNet's width multiplier alpha "thins a network uniformly at each layer" (Howard et al. 2017, section 3.3). Per-layer channel pruning ends with a ragged stack, some layers barely touched and others cut hard, because each layer got its own pruning ratio.

Why the ragged stack wins

Uniform shrinking rests on a hidden assumption: that every layer has the same slack. It does not. Han et al. (NeurIPS 2015, section 5) found that "the first convolutional layer, which interacts with the input image directly, is most sensitive to pruning", and suspected this sensitivity is "due to the input layer having only 3 channels and thus less redundancy than the other convolutional layers". The AMC paper records the standard hand-crafted rule of thumb that grew from such findings, one it then argues is non-optimal because layers are not independent: "prune less parameters in the first layer which extracts low level features and have the least amount of parameters". To see how little the first layer has to give, count the weights of the first six convolution layers of VGG-11 configuration A (Simonyan and Zisserman 2015, Table 1), each a 3 x 3 kernel over its input channels.

Prune 90% of L0 and 173 weights are left to describe every edge and colour detector that the rest of the network depends on. Prune 90% of L5 and 235,930 weights remain. A uniform rule takes the same 90% from a 1,728-weight layer and a 2.36-million-weight layer as if both had the same room to spare. This difference in slack is what the lecture calls sensitivity, and the next concept defines it precisely.

Here is the same experiment on a network you can count by hand. Train VGG-11 on CIFAR-10 until it reaches about 93% test accuracy. Prune 90% of the weights in L0, the first convolution, and touch nothing else: accuracy falls to about 32%, a drop of about 61 points. RestoreL0, prune 90% of L1 instead: accuracy falls only to about 85%, a drop of about 8. Same ratio, roughly eight times the damage.

That difference is what layer sensitivity measures: how much accuracy the network loses when one layer alone is pruned at a given ratio, with every other layer left dense. Write the dense accuracy as Acc_dense and the accuracy after pruning only layer L_i at ratio r as Acc_r^i. The degradation is their difference.

Plot Acc_r^i against r and a sensitive layer gives a steep curve while a redundant layer gives a flat one. Slide 10 states the two ends of the spectrum: some layers are more sensitive, the first layer being the usual example, and some are more redundant. The word "usual" matters, because sensitivity is measured, not assumed, and the reasons the first layer tends to top the list are worth spelling out rather than memorizing.

Three reasons the first layer is usually most sensitive

1. Fewest parameters. VGG-11's L0 has 1,728 weights against 73,728 in L1 and millions further in (VGG Table 1 arithmetic). Every weight removed from L0 is a larger share of what the layer knows.
2. Only three input channels. The layer sees raw RGB, so there is little redundancy to absorb a loss. This is the explanation Han et al. (2015) give for AlexNet, where the first convolution was the most sensitive layer in their per-layer sweep (Figure 6).
3. Everything downstream depends on it. Every feature in L1 through the classifier is a function of L0's output, so an error introduced there is inherited by every later layer rather than corrected.

If sensitivity is a curve per layer, the procedure to obtain it is fixed by the definition: isolate one layer, vary its ratio, measure, repeat. Slides 12 to 17 build that chart one curve at a time for VGG-11 on CIFAR-10. Here is the procedure as a whole, carried out on L0, then generalized.

The completed chart on slide 17 is the payoff. Every layer stays close to 93% until about 50% pruning, which tells you that any one layer can lose half its weights while the rest stay dense; whether every layer can lose half at once is a different question, taken up below. Past that point the layers separate. Values below are read off the slide to the nearest point.

Try the sweep yourself. The simulator holds the chart values, lets you pick the layer being pruned and slide its ratio, and reports the accuracy, the drop and how many of that layer's weights are gone. Leave the ratio at 0.9 and switch between L0 and L1 to feel the difference in one click.

Why each layer is swept alone

Sweeping one layer while the rest stay dense is what makes the curve interpretable. If two layers were pruned together and accuracy fell, the drop could not be attributed to either one. Isolation buys attribution. It also has a price, which the next part turns into a whole discussion: it assumes the damage from pruning several layers adds up. AMC's authors put it bluntly, the single-layer approach "assumes that errors of different pruned layers can be summed up linearly, which does not stand according to our experiments" (He et al. 2018, section 4). Keep that interaction in mind: the sweep tells you which layers are fragile, not what happens when you prune them all at once.

What the sweep costs

No gradient step runs during a sweep, but every point on every curve is one full pass over the evaluation set. The number of evaluations is the number of layers times the number of ratios, and each evaluation is a forward pass over every test image.

The CIFAR-10 VGG-11 behind this chart has 9 weight layers (8 convolutions plus a 1-layer classifier), so ten ratios mean 90 evaluations over the 10,000-image CIFAR-10 test set, about 900,000 forward passes. One small saving: r = 0 is the same dense model for every layer, so it only needs to be evaluated once. A 50-layer ImageNet network evaluated on the 50,000 ILSVRC validation images (Russakovsky et al. 2015) needs 25 million. The cost is linear in depth, which is tolerable for VGG-11 and painful for deep networks, and it is one of the reasons the automated methods later in this lecture, AMC and NetAdapt, search the ratios instead of sweeping them.