Second-order, activation and regression criteria

Start with a trained network and its loss L(x; W). Setting one weight to zero moves the parameter vector from W to W_P = W + δW, where δW is zero everywhere except at the pruned position, where δw_i = -w_i. The question every criterion in this part is trying to answer is simple: how much does the loss move? You could answer it exactly by re-evaluating the loss after each candidate deletion, but LeCun, Denker and Solla call that approach prohibitively laborious, and for a network with thousands of weights it is. Second-order pruning builds a local model of the loss instead.

One weight before the general case. A weight sits at w = 0.5 in a valley of the loss surface given by L(w) = 1 + 2(w - 0.5)². Setting it to zero walks back to w = 0 and raises the loss from 1 to 1.5, and that rise is exactly ½ h w² with curvature h = 4: the gradient at the bottom of the valley is zero, so the first-order term contributes nothing and the curvature term contributes everything. Now scale the picture up to a whole parameter vector. The local model is a Taylor expansion of the loss around the trained point. Perturbing the weights by δW changes the loss by a first-order slope term, a second-order curvature term, and a remainder that shrinks as the cube of the perturbation.

Read the four pieces one at a time. The first sum is the slope: how fast the loss rises if you nudge each weight on its own. The second sum is the self-curvature: how quickly that slope itself changes along each weight's own axis, weighted by the square of the nudge. The third sum is the cross-curvature: whether moving weight i also tilts the loss along weight j. The last piece is everything the quadratic picture misses. The Hessian is the matrix that holds the second and third pieces together: its diagonal h_ii feeds the self-curvature sum and its off-diagonal h_ij feeds the cross terms.

Goodfellow, Bengio and Courville define the Hessian exactly this way, as the matrix with entries ∂²f/(∂x_i ∂x_j), and note that it is symmetric wherever the second partials are continuous. The cross sum runs over ordered pairs i ≠ j, so each unordered pair appears twice, once as (i, j) and once as (j, i); the one half in front is the second-order Taylor coefficient from ½ δWᵀ H δW, not a symmetry correction. Because H is symmetric the same term can be written without the one half as Σ_{i < j} h_ij δw_i δw_j. They also use this same second-order expansion to derive Newton's method, so the expression on the slide is not a pruning trick: it is the standard local model of any smooth function.

The table is the reason the rest of this part exists. Even a 1989 network with about 2600parameters has millions of Hessian entries, and a modern network with 10⁸ weights would have 10¹⁶. Nobody computes that matrix. The whole art of second-order pruning is deciding which of its entries you can afford to ignore.

Four terms is three too many. Optimal Brain Damage keeps the expansion usable by making three assumptions, and each one erases exactly one piece of the sum. The paper gives them names, the slide gives them plain-language bullets, and you should be able to match the two lists and say which term each one kills.

The extremal assumption does double duty. Because training stopped at a local minimum, the gradient is zero and the first sum disappears, but LeCun, Denker and Solla also point out that at a local minimum all the h_ii are non-negative, so any perturbation of the parameters will cause the loss to increase or stay the same. That single sentence is the source of the footnote on slide 36 that h_ii is non-negative, and it is what lets the surviving term serve as a cost rather than a signed change.

What survives is the diagonal curvature term alone. Deleting weight i means setting it to zero and, in the paper's words, freezing it there, so the step along that axis has the size of the weight itself, |δw_i| = |w_i|. The squared terms of the expansion do not care about the sign of δw_i, so substitute and you have the OBD estimate of the loss increase.

The paper calls this quantity the saliency of a parameter, s_k = h_kk w_k² / 2 (the paper writes its parameter as u_k, the same variable as the w_k used on the slide and here), and the recipe is to delete the parameters with the lowest saliency. Notice what the formula says about a weight's importance: it is the product of two things, how large the weight is and how sharply the loss curves along that weight's axis. Magnitude pruning from part 05 kept only the first factor.

Magnitude pruning is OBD with the curvature assumed constant

Suppose every h_ii were the same number h. Then importance would be (h/2) w_i², and ranking weights by that value is exactly ranking them by |w_i|, since squaring and scaling by a positive constant preserve the order. So Magnitude-based pruning is OBD under a fourth, hidden assumption: that the loss curves equally steeply along every weight. The 1989 paper states its goal as moving beyond the approximation that magnitude equals saliency, and the point of a second-order criterion is to drop that assumption. The table below is a small case where doing so changes the answer.

Set every h_ii in the table to the same value and watch the two decision columns fall into line. Then give a small weight a large curvature and watch them split again. That is the entire relationship between the two criteria in one gesture.

Why the slide says the Hessian is difficult

The full Hessian has n² entries, and the previous concept showed that this is out of reach even for a small network. OBD never computes it. The diagonal approximation means only the n entries h_ii are needed, and the paper derives a second back-propagation pass that computes them with, in its words, the same order of complexity as computing the gradient. Along the way it drops the terms involving the second derivative of the activation function, a Levenberg-Marquardt style approximation that gives guaranteed positive estimates of the second derivative. So the difficulty the slide flags is real for the full matrix and for networks with 10⁸ parameters, where even a diagonal pass over the whole training set is expensive, but it is not a reason OBD itself was impractical. Optimal Brain Surgeon (Hassibi and Stork, 1993) kept the cross terms by using the inverse of the full Hessian. SparseGPT (2023) makes second-order pruning practical at scale by solving layer-wise reconstruction problems with block-wise Hessian updates, which the final concept points to.

The recipe and what it achieved

The abstract on the slide states the motivation the formula serves. Deleting unimportant weights is a trade-off between network complexity and training-set error, and the paper expects better generalization, fewer training examples required and faster learning and classification from it. That is why the recipe below is framed as model selection, not only compression.

1. Choose a reasonable network architecture.
2. Train the network until reasonable convergence.
3. Compute the second derivatives h_kk for each parameter.
4. Compute the saliencies s_k = h_kk w_k² / 2.
5. Sort the parameters by saliency and delete some low-saliency parameters.
6. Return to step 2.

The loop back to training is the Iterative pruning and Fine-tuning pattern from part 02, thirty years earlier. The 1989 experiment applied it to a handwritten zip-code recognizer.

Everything so far scored individual synapses. Take a linear layer with 5 inputs and 4 outputs, so its weight matrix has shape [4, 5]. Deleting the second output neuron does not zero one entry; it removes the entire row W[1, :], all 5 incoming weights at once, along with the edges drawn into that neuron on the slide, which is why the lower layer shows three circles with a gap where the fourth stood. The matrix becomes [3, 5], a smaller dense matrix, which is the reason Coarse-grained (structured) pruning gives real speedup without any sparse format.

That is the whole content of the sentence on the slide: Neuron pruning is coarse-grained weight pruning. In a convolution layer with weights of shape [c_o, c_i, k_h, k_w] (recall Convolution weight dimensions), deleting output channel o deletes W[o, :, :, :], one full row of c_i kernels in the slide's grid, where rows 3 and 5 of six filters are blanked out. It also deletes the corresponding map from the layer's output, and therefore the input channel o of every filter in the next layer, a point He et al. make explicitly in their figure. Channel pruning cuts twice.

The rule on the slide is then almost a tautology: the less useful the removed neurons are, the better the pruned network performs. The work is in defining useful. The Pruning criterion question has simply moved up one level of granularity, and part 05 already gave one answer, the L1 or L2 norm of the row or filter. The next two concepts give two answers that need no Hessian and no norm: look at the activations the neuron produces, or look at how well the layer output can be rebuilt without it.

ReLU outputs zero for every negative pre-activation. A channel whose output map is mostly zero, image after image, contributes almost nothing to the layer that follows, whatever its weights look like. Hu, Peng, Tai and Tang turned that observation into a criterion. On VGG-16 they found 631 neurons whose activations were zero more than 90% of the time on ImageNet validation images, and they found that the mean fraction of zeros is far higher in the deeper convolutional layers and the fully connected layers than in the early layers, so most of the redundancy sits at the top of the network.

The score is the APoZ. For channel c of a layer, run N validation images through the network, look at every position of the H × W output map for every image, and count how often the value is exactly zero. Divide by the number of positions you looked at.

The slide writes the same thing as zeros divided by batch times height times width. Its example uses a batch of 2, three channels and 4 × 4 maps, so every channel is judged over 2 × 4 × 4 = 32 positions. Count the bold zeros in each grid and the numbers on the slide come out exactly.

Direction is the only thing people get wrong here, so state it in words: importance falls as APoZ rises. The paper's operating rule, the procedure it names Network Trimming, is to trim the neurons whose APoZ is more than one standard deviation above the layer's mean, which under a Gaussian assumption on APoZ values rejects about 16% of the neurons in a trimmed layer on average. Trimming is iterative: prune a few high-APoZ layers, retrain with the surviving weights initialized from the pre-trim network rather than from scratch, and repeat. The authors show that training the trimmed architecture from scratch leaves more zero-activation neurons than initializing from the trimmed weights does, so the initialization matters.

The comparison shows a split that matters for your project. Magnitude and OBD are read off the weights and their derivatives. APoZ and the regression criterion of the next concept are data-driven: they need a forward pass over real inputs, so they capture what the network does on your data rather than what its parameters look like, at the price of needing that data available at pruning time.

OBD asked what pruning does to the loss. APoZ looked at what a channel emits. He, Zhang and Sun ask a third question that is easier to answer than the first and more precise than the second: after you remove some input channels of a layer, how well can the survivors reproduce the layer's original output? Take one conv layer and sample b positions from real images (the paper used 5000 images with 10 samples each). Unroll each receptive field into a row of a matrix X of shape b × c_i in the slide's simplified picture. The layer's output at those positions is a matrix multiply.

The second equality is the whole idea. A matrix product can be split by the shared dimension: column c of X times row c of W^T is one b × c_o outer product, and Z is the sum of c_i of them, one per input channel. Removing input channel c means removing exactly one term of that sum. Regression-based pruning attaches a switch β_c to each term and asks which switches can be turned off while keeping the sum close to the original.

Name every symbol. β is the Channel selection coefficient, a vector of length c_i; β_c = 0 prunes input channel c. N_c is the number of channels allowed to survive, so the L0 norm constraint counts nonzero switches, the same kind of constraint as the pruning formulation of part 01. The objective is the Reconstruction error: the Frobenius norm is the square root of the sum of all squared entries, so its square is just the elementwise squared error summed over the whole b × c_o output. Nothing about the network loss, the labels or the layers downstream appears anywhere.

The figure on the slide shows why the input side is the one that shrinks. X_P loses a column and W_P^T loses the row that multiplied it; the two disappear as a pair, because they were only ever used together. The output Ẑ still has b rows and c_o columns. The next layer sees an input of the same shape as before, only slightly perturbed, which is what lets the paper prune a very deep network layer by layer and account for the accumulated error by always regressing toward the un-pruned model's output.

Solving it: alternate between selecting and rebuilding

The L0 constraint makes the problem NP-hard, so the paper relaxes it to an L1 penalty, adding λ ||β||₁ to the objective. That turns channel selection into LASSO regression, whose solutions are naturally sparse: as λ grows, more β_c are driven exactly to zero. With two unknowns, β and W, the paper alternates.

With W fixed, Ẑ is linear in β, so the objective is an ordinary least-squares fit in β and the added L1 penalty makes it a LASSO problem that picks the channels. With β fixed, the survivors' weights are re-fitted by ordinary least squares so that they rebuild Z as well as possible, in closed form. A constraint ||W_c||_F = 1 on each channel's weights stops the trivial solution of shrinking β while inflating W. In practice the authors run the selection step repeatedly, raising λ until ||β||₀ drops to the target, and then run the reconstruction step once.

The reference list at the end of the deck is a map of where every Pruning criterion in parts 05 and 06 came from, and of what comes next. Reading it as a family tree makes the lecture easier to hold in memory than reading it as nineteen unrelated titles.

Three of the pointers deserve a sentence each. SparseGPT is the modern descendant of Second-order pruning: it prunes GPT-scale models with 175 billion parameters in one shot to 50 to 60% unstructured sparsity with little loss in perplexity, using Hessian information of a layer-wise reconstruction problem very much like the one in the previous concept. ThiNet selects filters using statistics from the next layer rather than the current one, a cousin of Regression-based pruning. Molchanov's first-order Taylor criterion scores units by gradient times value on mini-batches. It is the same expansion as OBD with the gradient term kept and the Hessian dropped.