From sensitivity curves to per-layer ratios

Put your finger on the 80% column of the VGG-11 chart and read straight down. The green L1 curve is still at about 92% accuracy. The blue L0 curve has already dropped to about 71%. Same network, same dataset, same pruning ratio, and the two layers are twenty points apart. That gap is the entire reason per-layer pruning ratios exist, and it is what the six curves are there to expose.

Recall what each curve is. Sensitivity analysis prunes one layer L_i alone at ratios r in {0, 0.1, ..., 0.9}, measures the accuracy at each ratio with every other layer left dense, then moves on to the next layer and repeats. The result is one accuracy versus ratio line per layer. Every line starts at the same place, the dense accuracy of about 93% on CIFAR-10, so the starting point carries no information at all. What differs is the slope: how soon and how steeply each line bends down as r grows. That slope is the layer's sensitivity.

The rule that falls out of the table is simple. A curve that stays flat as r grows belongs to a redundant layer: most of its weights can go with no visible cost, so it should be pruned hard. A curve that bends early belongs to a sensitive layer: its weights are doing work that nothing else in the network can absorb, so it should be pruned gently. The slide summarizes it as "some layers are less sensitive to pruning" (L1, the flattest) and "some layers are more sensitive to pruning" (L0, the steepest), and the whole point of drawing all six on one axis is that you can rank them by eye.

Why the first layer is the fragile one

L0 being the most sensitive is not a quirk of this run. Han et al. (2015), the paper the chart's method comes from, report the same pattern on AlexNet: "The first convolutional layer, which interacts with the input image directly, is most sensitive to pruning. We suspect this sensitivity is due to the input layer having only 3 channels and thus less redundancy than the other convolutional layers" (section 5, Figure 6). Their Figure 6 also shows convolutional layers as a group being more sensitive than fully connected layers, which is the same story at coarser resolution: the layers with the fewest weights per unit of work have the least slack. A 3 x 3 kernel over three input channels has 27 weights per filter; removing 80% of them leaves five, and five numbers cannot describe an edge detector.

Now draw one dashed horizontal line across the chart just under 80% accuracy, where the slide puts it (measured against the gridlines it sits at about 79%). Walk along the blue L0 curve from the left. It sits above the line through 60%, is still above it at 70% (about 83.5%), and has dropped to 71% by 80%. Somewhere between those two grid points it crossed the line, at roughly 73% pruning. That crossing is L0's pruning rate. Do the same for the orange L3 (about 82%) and the steep red L4, whose 80% point sits just under the line: strict interpolation puts the crossing at about 78.5%, and the slide rounds its drawn guide up to 80%. L1, the flat red L2 and L5 never reach the line inside the sweep, so they get the largest tested rate, 90%, which is where the slide draws its last vertical guide.

That is the whole mechanism, and it deserves to be stated as an algorithm rather than a picture, because the exam version comes as a table, not a chart. Choose an accuracy threshold T. For each layer, the pruning ratio is the largest swept ratio whose accuracy is still at or above T. Layers whose curves never dip below T inside the sweep get the maximum swept ratio.

The read-off table above and the worked example below follow the slide's drawn guides, which use the chart form and interpolate between grid points. By the strict table form L0 and L4 would snap to 70% and L3 to 80%, and exact interpolation puts L4 at about 78.5%, so the slide's 80% guide for L4 is a rounding of its own chart. The formula is the exam rule; the guides are how the slide applies it by eye.

From six rates to one overall rate

The slide bullet reads "pick a degradation threshold T such that the overall pruning rate is desired", and the word overall hides a calculation students routinely get wrong. The overall rate is not the average of the six numbers. It is the fraction of all weights removed, so each layer's rate is weighted by how many weights that layer has. A 90% rate on a layer with 2.36 million weights removes far more than a 73% rate on a layer with 1,728.

T is the one knob. If R comes out below the target, lower T: every curve now runs further right before it crosses, so every r_i grows or stays pinned at the maximum swept ratio, and R never falls. If the pruned network turns out too inaccurate, raise T and no rate can grow. The loop is: pick T, read off six rates, compute R, compare with the target, move T, repeat. It converges in a handful of iterations because R is monotone in T.

The simulator below runs this loop on curves digitized from the slide, so at its default T of 79% the interpolated crossings land within about a point and a half of the slide's guides (L4 reads 78.5% against the drawn 80%) and R reads 86.0% against the worked example's 86.3%. Drag T and watch all six vertical guides move at once; switch between snapping to the 10% grid and interpolating between grid points; and read R, the unweighted mean and the compression factor as they update. Try T at 90 first, then 70, and note which direction R moves.

The exam version: a table instead of a chart

On paper you will get an accuracy table for two or three layers, a threshold and a weight count per layer. The procedure is identical. Read each row until the accuracy falls below T, take the last ratio that was still at or above T, weight by the counts, and add.

Take two of the rates you just read off. L0 pruned to 73% on its own costs about 14 points (93 down to the line at 79). L4 pruned to 80% on its own costs about 16 points. Now prune both at once. L4 was trained on features that flow through a dense L0. Its tolerance to losing 80% of its weights was measured with those features intact, and that condition no longer holds. The network now sees both perturbations at the same time, and the combined drop is usually larger than either single-layer drop, and the chart contains no measurement of it.

The slide asks "is this optimal?" and answers itself: "maybe not. We do not consider the interaction between layers." The next slide of the source deck puts it even more directly, "sensitivity analysis ignores the interaction between layers, sub-optimal". The reason is built into the procedure. Each curve is a one-dimensional slice of accuracy along one layer's axis, taken with every other layer frozen dense. The read-off then treats those axes as independent, as if you could add up six separate tolerances and get the tolerance of the whole. Layers are not independent. That is the Layer interaction the heuristic leaves out, and it is why T is only a proxy for the accuracy you will actually get, not a guarantee.

Why the heuristic ignores interactions on purpose

The omission is a cost decision, not an oversight. The sweep needs (layers) x (ratios) evaluations, 6 x 9 = 54 here, each a forward pass over the validation set. Measuring every joint setting would need 9^6 = 531,441 evaluations for six layers, and real networks have dozens of layers. The AMC paper says exactly this about hand-crafted rules: "as the layers in deep neural networks are not independent, these rule-based pruning policies are non-optimal, and doesn't transfer from one model to another model", and "the design space has exponential complexity, which is infeasible to be solved by greedy, rule-based methods" (He et al., 2018, section 1). The heuristic trades optimality for a search that finishes in an afternoon.

Two things keep it usable in practice. First, Fine-tuning after pruning (part 06) recovers a large share of the compounded loss, so the read-off does not have to be exact, only in the right region. Second, it is the starting point that the automated methods improve on rather than throw away. NetAdapt removes filters from one layer per iteration, re-measures accuracy after a short fine-tune, and keeps the best proposal, so interactions get folded in one step at a time (Yang et al., 2018). AMC goes further and learns the whole per-layer assignment as one policy, rewarded on the accuracy of the jointly pruned network (He et al., 2018). Both are Automated pruning, and both exist because the single-layer sweep leaves accuracy on the table.