Where we are: the pruning problem so far

Take one layer with 16 weights and suppose the deployment budget allows you to keep only 8 of them. Which eight? That single question is the whole of Pruning. Every answer in lectures 04-1 and 04-2 is a different way of choosing those eight, and every hardware trick at the end of this lecture is a way of profiting from the eight zeros that remain.

The general rule states the question as a constrained optimization. Among all weight tensors that have at most N nonzero entries, pick the one that makes the training loss smallest.

Read the formula from the inside out. L(x; W_P) asks how badly the network with weights W_P does on the data. The arg min says we want the W_P that makes that number smallest. The constraint says we may only search among tensors whose nonzero count fits the budget. The budget is what connects this formula to the Pruning ratio of the next part: with M weights in the layer and N survivors, the fraction removed is 1 - N / M, so for M = 16 and N = 8 the layer is 50% sparse. The weight count is written M rather than |W|, because |W| below means the element-wise absolute value.

Why nobody solves this formula directly

The formula looks like something you could hand to gradient descent, and that is the trap. The nonzero count is a step function: nudge a weight from 0.30 to 0.31 and the count does not move, set it to exactly zero and the count drops by one. Its gradient is zero almost everywhere and undefined at zero, so no gradient flows through the constraint. Louizos, Welling and Kingma open their ICLR 2018 paper on this exact point: the L0 norm of the weights is non-differentiable, which is why it cannot simply be added to the training objective. Without a gradient, the exact problem is a combinatorial search, choosing which N of the M positions to keep, and for a layer with millions of weights that search is hopeless.

The heuristics are not arbitrary. Every one of them is a proxy for the same quantity: how much would L change if this weight were removed. The constraint side of the formula only fixes the budget; the loss side is what actually decides the answer, so the best criterion is the one whose ranking best tracks the change in loss. That is why the slide on selection later in this part says that the less useful the removed neurons are, the better the pruned network performs. Useful means low loss change, nothing else.

The formulation says nothing about how to find good survivors. Lecture 04-1 gave two practical answers, a shape and a score, and slides 4 to 6 of this deck replay them in the space of one minute. Start with the score on the smallest example that still teaches something: a 2 x 2 weight matrix that must lose half of its entries.

This is Magnitude-based pruning, and the slide is honest about its status: it is a heuristic criterion. The assumption is that a weight far from zero contributes more to the output, and therefore to the loss, than a weight near zero. Han et al. (NeurIPS 2015) used exactly this threshold rule to produce the 9x and 13x reductions quoted in the first concept, which is strong evidence that the assumption is usually good enough. It is not the only score. Lecture 04-1, part 05, adds scaling factors such as batch-norm gammas for whole channels, and part 06 adds second-order saliency from Optimal Brain Damage, the percentage of zero activations, and regression-based selection. All of them plug into the same three steps: score, sort, keep the top N.

The shape of what is removed

Scoring weights one at a time and zeroing the smallest gives the pattern the slide calls fine-grained: zeros scattered wherever the small weights happened to be. The pruned network figure on that slide, again from Han et al., shows synapses vanishing at arbitrary positions and whole neurons disappearing once every synapse into them, or every synapse out of them, is gone. Fine-grained removal has the most freedom, since any index may be pruned, and that freedom is what buys it the highest compression ratios. The cost is that the survivors no longer form a dense block. Their positions must be stored as indices, and the hardware needs a sparse kernel that skips the holes. Mao et al. (2017) measured the other end of the trade: coarse-grained pruning reaches a sparsity similar to unstructured pruning without losing accuracy, needs far fewer indices, and saves about 2x the memory references compared with fine-grained sparsity.

That ladder is the Pruning granularity axis, and the two ends pull against each other. Irregular patterns give the criterion freedom and therefore compression; regular patterns give the hardware dense blocks and therefore speed. Every design choice later in this lecture sits somewhere on this axis, and the 2:4 pattern of the tensor-core part is the industry's attempt to sit in both places at once.

The coarse end of the ladder deserves one concrete picture, because it explains a sentence that students often quote without understanding: Neuron pruning is coarse-grained weight pruning. Slide 6 draws both cases, the linear layer and the convolution layer, and the same picture in matrix form makes the sentence obvious.

Removing a neuron is removing a row

A linear layer computes y = W x with W of shape out x in. Output neuron i is nothing more than row i of W: its incoming weights are the entries of that row, and its output is that row's dot product with x. Deleting the neuron therefore deletes the whole row at once, which is the white stripe in the slide's weight matrix, and the next layer loses the matching column because that input no longer exists. That is neuron pruning, and it is weight pruning where the removed set is a structured block of in weights chosen together rather than one weight at a time. Lecture 04-1, part 06, first drew this equivalence next to Optimal Brain Damage.

The convolution version is the same idea one dimension up. A conv layer holds weights of shape C_out x C_in x k x k. Output channel j is produced by filter j, which is the slice of all C_in kernels that feed it. Deleting a channel deletes that entire filter, the two dashed blocks in the slide's stack of six filters and the two white rows of kernels in its weight grid (two of six output channels removed). That is Channel pruning, and its reward is that nothing sparse is left behind: the layer simply has C_out minus the pruned count of channels and runs on the same dense kernels as before.

Five questions, three answered

The outline that opens this deck on slide 2, and returns on slide 7 to close this part, lists five questions, and the same outline appeared three times in lecture 04-1. It is the map of both lectures, so it is worth knowing where each question was answered. The formulation above answers the first. Granularity, the pattern of removal, was answered in 04-1. Criterion, the score used to rank weights, was also answered in 04-1. What remains open is how many weights each layer should lose, and how to recover the accuracy once they are gone.

On slide 2 both open questions, ratio and fine-tuning, are highlighted in yellow. On slide 7, which closes this part, ratio turns green while fine-tuning stays yellow: ratio is being opened now, fine-tuning is still queued. The outline also under-sells the deck. After the five questions, parts 07 onward add a story the list never mentions, how EIE, 2:4 sparse tensor cores and sparse convolution engines turn the zeros produced by pruning into real speed and energy savings on hardware.