Pruning ratios, fine-tuning and sparse hardware

Take one layer with 16 weights and suppose the deployment budget allows you to keep only 8 of them. Which eight? That single question is the whole of Pruning. Every answer in lectures 04-1 and 04-2 is a different way of choosing those eight, and every hardware trick at the end of this lecture is a way of profiting from the eight zeros that remain.

The general rule states the question as a constrained optimization. Among all weight tensors that have at most N nonzero entries, pick the one that makes the training loss smallest.

Read the formula from the inside out. L(x; W_P) asks how badly the network with weights W_P does on the data. The arg min says we want the W_P that makes that number smallest. The constraint says we may only search among tensors whose nonzero count fits the budget. The budget is what connects this formula to the Pruning ratio of the next part: with M weights in the layer and N survivors, the fraction removed is 1 - N / M, so for M = 16 and N = 8 the layer is 50% sparse. The weight count is written M rather than |W|, because |W| below means the element-wise absolute value.

Why nobody solves this formula directly

The formula looks like something you could hand to gradient descent, and that is the trap. The nonzero count is a step function: nudge a weight from 0.30 to 0.31 and the count does not move, set it to exactly zero and the count drops by one. Its gradient is zero almost everywhere and undefined at zero, so no gradient flows through the constraint. Louizos, Welling and Kingma open their ICLR 2018 paper on this exact point: the L0 norm of the weights is non-differentiable, which is why it cannot simply be added to the training objective. Without a gradient, the exact problem is a combinatorial search, choosing which N of the M positions to keep, and for a layer with millions of weights that search is hopeless.

The heuristics are not arbitrary. Every one of them is a proxy for the same quantity: how much would L change if this weight were removed. The constraint side of the formula only fixes the budget; the loss side is what actually decides the answer, so the best criterion is the one whose ranking best tracks the change in loss. That is why the slide on selection later in this part says that the less useful the removed neurons are, the better the pruned network performs. Useful means low loss change, nothing else.

The formulation says nothing about how to find good survivors. Lecture 04-1 gave two practical answers, a shape and a score, and slides 4 to 6 of this deck replay them in the space of one minute. Start with the score on the smallest example that still teaches something: a 2 x 2 weight matrix that must lose half of its entries.

This is Magnitude-based pruning, and the slide is honest about its status: it is a heuristic criterion. The assumption is that a weight far from zero contributes more to the output, and therefore to the loss, than a weight near zero. Han et al. (NeurIPS 2015) used exactly this threshold rule to produce the 9x and 13x reductions quoted in the first concept, which is strong evidence that the assumption is usually good enough. It is not the only score. Lecture 04-1, part 05, adds scaling factors such as batch-norm gammas for whole channels, and part 06 adds second-order saliency from Optimal Brain Damage, the percentage of zero activations, and regression-based selection. All of them plug into the same three steps: score, sort, keep the top N.

The shape of what is removed

Scoring weights one at a time and zeroing the smallest gives the pattern the slide calls fine-grained: zeros scattered wherever the small weights happened to be. The pruned network figure on that slide, again from Han et al., shows synapses vanishing at arbitrary positions and whole neurons disappearing once every synapse into them, or every synapse out of them, is gone. Fine-grained removal has the most freedom, since any index may be pruned, and that freedom is what buys it the highest compression ratios. The cost is that the survivors no longer form a dense block. Their positions must be stored as indices, and the hardware needs a sparse kernel that skips the holes. Mao et al. (2017) measured the other end of the trade: coarse-grained pruning reaches a sparsity similar to unstructured pruning without losing accuracy, needs far fewer indices, and saves about 2x the memory references compared with fine-grained sparsity.

That ladder is the Pruning granularity axis, and the two ends pull against each other. Irregular patterns give the criterion freedom and therefore compression; regular patterns give the hardware dense blocks and therefore speed. Every design choice later in this lecture sits somewhere on this axis, and the 2:4 pattern of the tensor-core part is the industry's attempt to sit in both places at once.

The coarse end of the ladder deserves one concrete picture, because it explains a sentence that students often quote without understanding: Neuron pruning is coarse-grained weight pruning. Slide 6 draws both cases, the linear layer and the convolution layer, and the same picture in matrix form makes the sentence obvious.

Removing a neuron is removing a row

A linear layer computes y = W x with W of shape out x in. Output neuron i is nothing more than row i of W: its incoming weights are the entries of that row, and its output is that row's dot product with x. Deleting the neuron therefore deletes the whole row at once, which is the white stripe in the slide's weight matrix, and the next layer loses the matching column because that input no longer exists. That is neuron pruning, and it is weight pruning where the removed set is a structured block of in weights chosen together rather than one weight at a time. Lecture 04-1, part 06, first drew this equivalence next to Optimal Brain Damage.

The convolution version is the same idea one dimension up. A conv layer holds weights of shape C_out x C_in x k x k. Output channel j is produced by filter j, which is the slice of all C_in kernels that feed it. Deleting a channel deletes that entire filter, the two dashed blocks in the slide's stack of six filters and the two white rows of kernels in its weight grid (two of six output channels removed). That is Channel pruning, and its reward is that nothing sparse is left behind: the layer simply has C_out minus the pruned count of channels and runs on the same dense kernels as before.

Five questions, three answered

The outline that opens this deck on slide 2, and returns on slide 7 to close this part, lists five questions, and the same outline appeared three times in lecture 04-1. It is the map of both lectures, so it is worth knowing where each question was answered. The formulation above answers the first. Granularity, the pattern of removal, was answered in 04-1. Criterion, the score used to rank weights, was also answered in 04-1. What remains open is how many weights each layer should lose, and how to recover the accuracy once they are gone.

On slide 2 both open questions, ratio and fine-tuning, are highlighted in yellow. On slide 7, which closes this part, ratio turns green while fine-tuning stays yellow: ratio is being opened now, fine-tuning is still queued. The outline also under-sells the deck. After the five questions, parts 07 onward add a story the list never mentions, how EIE, 2:4 sparse tensor cores and sparse convolution engines turn the zeros produced by pruning into real speed and energy savings on hardware.

Start with a measurement, not a principle. Take MobileNet on ImageNet and run it on a Google Pixel 1 CPU with TensorFlow Lite. The dense network scores 70.6% top-1 at 123.3 ms per image. Now shrink it two different ways to roughly half the work. The first way multiplies every layer's width by 0.75, the built-in MobileNet knob. The second way, AMC, lets every layer keep a different fraction of its channels (He et al., ECCV 2018, Table 4). MMACs counts millions of multiply-accumulate operations per image, a hardware-neutral measure of compute; a FLOPs target asks AMC to halve that compute, while a latency target asks it to halve measured time on the phone.

Read the middle two rows against each other. At nearly the same latency, 72.3 ms versus 68.3 ms, the per-layer version keeps 2.1 points more accuracy. Read the first and third rows together and the gap is even more striking: AMC's 68.3 ms model loses only 0.1 points against the full network at 123.3 ms. Slide 9 plots the same story as a chart (AMC Figure 5b), and the values read off it land in one table.

Every AMC point sits above every uniform point. The uniform line on that chart mixes MobileNet's width multiplier with its input-resolution multiplier, which is why it has four points while Table 4 lists two; either knob shrinks every layer by the same factor.

The two stacks of bars on the slide are the whole idea in one picture. Uniform shrinking keeps every layer the same shape, just thinner: MobileNet's width multiplier alpha "thins a network uniformly at each layer" (Howard et al. 2017, section 3.3). Per-layer channel pruning ends with a ragged stack, some layers barely touched and others cut hard, because each layer got its own pruning ratio.

Why the ragged stack wins

Uniform shrinking rests on a hidden assumption: that every layer has the same slack. It does not. Han et al. (NeurIPS 2015, section 5) found that "the first convolutional layer, which interacts with the input image directly, is most sensitive to pruning", and suspected this sensitivity is "due to the input layer having only 3 channels and thus less redundancy than the other convolutional layers". The AMC paper records the standard hand-crafted rule of thumb that grew from such findings, one it then argues is non-optimal because layers are not independent: "prune less parameters in the first layer which extracts low level features and have the least amount of parameters". To see how little the first layer has to give, count the weights of the first six convolution layers of VGG-11 configuration A (Simonyan and Zisserman 2015, Table 1), each a 3 x 3 kernel over its input channels.

Prune 90% of L0 and 173 weights are left to describe every edge and colour detector that the rest of the network depends on. Prune 90% of L5 and 235,930 weights remain. A uniform rule takes the same 90% from a 1,728-weight layer and a 2.36-million-weight layer as if both had the same room to spare. This difference in slack is what the lecture calls sensitivity, and the next concept defines it precisely.

Here is the same experiment on a network you can count by hand. Train VGG-11 on CIFAR-10 until it reaches about 93% test accuracy. Prune 90% of the weights in L0, the first convolution, and touch nothing else: accuracy falls to about 32%, a drop of about 61 points. RestoreL0, prune 90% of L1 instead: accuracy falls only to about 85%, a drop of about 8. Same ratio, roughly eight times the damage.

That difference is what layer sensitivity measures: how much accuracy the network loses when one layer alone is pruned at a given ratio, with every other layer left dense. Write the dense accuracy as Acc_dense and the accuracy after pruning only layer L_i at ratio r as Acc_r^i. The degradation is their difference.

Plot Acc_r^i against r and a sensitive layer gives a steep curve while a redundant layer gives a flat one. Slide 10 states the two ends of the spectrum: some layers are more sensitive, the first layer being the usual example, and some are more redundant. The word "usual" matters, because sensitivity is measured, not assumed, and the reasons the first layer tends to top the list are worth spelling out rather than memorizing.

Three reasons the first layer is usually most sensitive

1. Fewest parameters. VGG-11's L0 has 1,728 weights against 73,728 in L1 and millions further in (VGG Table 1 arithmetic). Every weight removed from L0 is a larger share of what the layer knows.
2. Only three input channels. The layer sees raw RGB, so there is little redundancy to absorb a loss. This is the explanation Han et al. (2015) give for AlexNet, where the first convolution was the most sensitive layer in their per-layer sweep (Figure 6).
3. Everything downstream depends on it. Every feature in L1 through the classifier is a function of L0's output, so an error introduced there is inherited by every later layer rather than corrected.

If sensitivity is a curve per layer, the procedure to obtain it is fixed by the definition: isolate one layer, vary its ratio, measure, repeat. Slides 12 to 17 build that chart one curve at a time for VGG-11 on CIFAR-10. Here is the procedure as a whole, carried out on L0, then generalized.

The completed chart on slide 17 is the payoff. Every layer stays close to 93% until about 50% pruning, which tells you that any one layer can lose half its weights while the rest stay dense; whether every layer can lose half at once is a different question, taken up below. Past that point the layers separate. Values below are read off the slide to the nearest point.

Try the sweep yourself. The simulator holds the chart values, lets you pick the layer being pruned and slide its ratio, and reports the accuracy, the drop and how many of that layer's weights are gone. Leave the ratio at 0.9 and switch between L0 and L1 to feel the difference in one click.

Why each layer is swept alone

Sweeping one layer while the rest stay dense is what makes the curve interpretable. If two layers were pruned together and accuracy fell, the drop could not be attributed to either one. Isolation buys attribution. It also has a price, which the next part turns into a whole discussion: it assumes the damage from pruning several layers adds up. AMC's authors put it bluntly, the single-layer approach "assumes that errors of different pruned layers can be summed up linearly, which does not stand according to our experiments" (He et al. 2018, section 4). Keep that interaction in mind: the sweep tells you which layers are fragile, not what happens when you prune them all at once.

What the sweep costs

No gradient step runs during a sweep, but every point on every curve is one full pass over the evaluation set. The number of evaluations is the number of layers times the number of ratios, and each evaluation is a forward pass over every test image.

The CIFAR-10 VGG-11 behind this chart has 9 weight layers (8 convolutions plus a 1-layer classifier), so ten ratios mean 90 evaluations over the 10,000-image CIFAR-10 test set, about 900,000 forward passes. One small saving: r = 0 is the same dense model for every layer, so it only needs to be evaluated once. A 50-layer ImageNet network evaluated on the 50,000 ILSVRC validation images (Russakovsky et al. 2015) needs 25 million. The cost is linear in depth, which is tolerable for VGG-11 and painful for deep networks, and it is one of the reasons the automated methods later in this lecture, AMC and NetAdapt, search the ratios instead of sweeping them.

Put your finger on the 80% column of the VGG-11 chart and read straight down. The green L1 curve is still at about 92% accuracy. The blue L0 curve has already dropped to about 71%. Same network, same dataset, same pruning ratio, and the two layers are twenty points apart. That gap is the entire reason per-layer pruning ratios exist, and it is what the six curves are there to expose.

Recall what each curve is. Sensitivity analysis prunes one layer L_i alone at ratios r in {0, 0.1, ..., 0.9}, measures the accuracy at each ratio with every other layer left dense, then moves on to the next layer and repeats. The result is one accuracy versus ratio line per layer. Every line starts at the same place, the dense accuracy of about 93% on CIFAR-10, so the starting point carries no information at all. What differs is the slope: how soon and how steeply each line bends down as r grows. That slope is the layer's sensitivity.

The rule that falls out of the table is simple. A curve that stays flat as r grows belongs to a redundant layer: most of its weights can go with no visible cost, so it should be pruned hard. A curve that bends early belongs to a sensitive layer: its weights are doing work that nothing else in the network can absorb, so it should be pruned gently. The slide summarizes it as "some layers are less sensitive to pruning" (L1, the flattest) and "some layers are more sensitive to pruning" (L0, the steepest), and the whole point of drawing all six on one axis is that you can rank them by eye.

Why the first layer is the fragile one

L0 being the most sensitive is not a quirk of this run. Han et al. (2015), the paper the chart's method comes from, report the same pattern on AlexNet: "The first convolutional layer, which interacts with the input image directly, is most sensitive to pruning. We suspect this sensitivity is due to the input layer having only 3 channels and thus less redundancy than the other convolutional layers" (section 5, Figure 6). Their Figure 6 also shows convolutional layers as a group being more sensitive than fully connected layers, which is the same story at coarser resolution: the layers with the fewest weights per unit of work have the least slack. A 3 x 3 kernel over three input channels has 27 weights per filter; removing 80% of them leaves five, and five numbers cannot describe an edge detector.

Now draw one dashed horizontal line across the chart just under 80% accuracy, where the slide puts it (measured against the gridlines it sits at about 79%). Walk along the blue L0 curve from the left. It sits above the line through 60%, is still above it at 70% (about 83.5%), and has dropped to 71% by 80%. Somewhere between those two grid points it crossed the line, at roughly 73% pruning. That crossing is L0's pruning rate. Do the same for the orange L3 (about 82%) and the steep red L4, whose 80% point sits just under the line: strict interpolation puts the crossing at about 78.5%, and the slide rounds its drawn guide up to 80%. L1, the flat red L2 and L5 never reach the line inside the sweep, so they get the largest tested rate, 90%, which is where the slide draws its last vertical guide.

That is the whole mechanism, and it deserves to be stated as an algorithm rather than a picture, because the exam version comes as a table, not a chart. Choose an accuracy threshold T. For each layer, the pruning ratio is the largest swept ratio whose accuracy is still at or above T. Layers whose curves never dip below T inside the sweep get the maximum swept ratio.

The read-off table above and the worked example below follow the slide's drawn guides, which use the chart form and interpolate between grid points. By the strict table form L0 and L4 would snap to 70% and L3 to 80%, and exact interpolation puts L4 at about 78.5%, so the slide's 80% guide for L4 is a rounding of its own chart. The formula is the exam rule; the guides are how the slide applies it by eye.

From six rates to one overall rate

The slide bullet reads "pick a degradation threshold T such that the overall pruning rate is desired", and the word overall hides a calculation students routinely get wrong. The overall rate is not the average of the six numbers. It is the fraction of all weights removed, so each layer's rate is weighted by how many weights that layer has. A 90% rate on a layer with 2.36 million weights removes far more than a 73% rate on a layer with 1,728.

T is the one knob. If R comes out below the target, lower T: every curve now runs further right before it crosses, so every r_i grows or stays pinned at the maximum swept ratio, and R never falls. If the pruned network turns out too inaccurate, raise T and no rate can grow. The loop is: pick T, read off six rates, compute R, compare with the target, move T, repeat. It converges in a handful of iterations because R is monotone in T.

The simulator below runs this loop on curves digitized from the slide, so at its default T of 79% the interpolated crossings land within about a point and a half of the slide's guides (L4 reads 78.5% against the drawn 80%) and R reads 86.0% against the worked example's 86.3%. Drag T and watch all six vertical guides move at once; switch between snapping to the 10% grid and interpolating between grid points; and read R, the unweighted mean and the compression factor as they update. Try T at 90 first, then 70, and note which direction R moves.

The exam version: a table instead of a chart

On paper you will get an accuracy table for two or three layers, a threshold and a weight count per layer. The procedure is identical. Read each row until the accuracy falls below T, take the last ratio that was still at or above T, weight by the counts, and add.

Take two of the rates you just read off. L0 pruned to 73% on its own costs about 14 points (93 down to the line at 79). L4 pruned to 80% on its own costs about 16 points. Now prune both at once. L4 was trained on features that flow through a dense L0. Its tolerance to losing 80% of its weights was measured with those features intact, and that condition no longer holds. The network now sees both perturbations at the same time, and the combined drop is usually larger than either single-layer drop, and the chart contains no measurement of it.

The slide asks "is this optimal?" and answers itself: "maybe not. We do not consider the interaction between layers." The next slide of the source deck puts it even more directly, "sensitivity analysis ignores the interaction between layers, sub-optimal". The reason is built into the procedure. Each curve is a one-dimensional slice of accuracy along one layer's axis, taken with every other layer frozen dense. The read-off then treats those axes as independent, as if you could add up six separate tolerances and get the tolerance of the whole. Layers are not independent. That is the Layer interaction the heuristic leaves out, and it is why T is only a proxy for the accuracy you will actually get, not a guarantee.

Why the heuristic ignores interactions on purpose

The omission is a cost decision, not an oversight. The sweep needs (layers) x (ratios) evaluations, 6 x 9 = 54 here, each a forward pass over the validation set. Measuring every joint setting would need 9^6 = 531,441 evaluations for six layers, and real networks have dozens of layers. The AMC paper says exactly this about hand-crafted rules: "as the layers in deep neural networks are not independent, these rule-based pruning policies are non-optimal, and doesn't transfer from one model to another model", and "the design space has exponential complexity, which is infeasible to be solved by greedy, rule-based methods" (He et al., 2018, section 1). The heuristic trades optimality for a search that finishes in an afternoon.

Two things keep it usable in practice. First, Fine-tuning after pruning (part 06) recovers a large share of the compounded loss, so the read-off does not have to be exact, only in the right region. Second, it is the starting point that the automated methods improve on rather than throw away. NetAdapt removes filters from one layer per iteration, re-measures accuracy after a short fine-tune, and keeps the best proposal, so interactions get folded in one step at a time (Yang et al., 2018). AMC goes further and learns the whole per-layer assignment as one policy, rewarded on the accuracy of the jointly pruned network (He et al., 2018). Both are Automated pruning, and both exist because the single-layer sweep leaves accuracy on the table.

Take a ResNet-50 and allow each of its roughly fifty prunable layers to keep one of ten possible fractions of its weights. That is 10^50 ways to spend one overall budget. The Sensitivity analysis of part 03 escapes the explosion by never looking at combinations at all: it prunes one layer at a time, runs 50 x 10 = 500 single-layer experiments, and reads each layer's Pruning ratio off its own curve where it crosses the accuracy threshold.

The saving comes from a hidden assumption: that the accuracy lost by pruning several layers at once is the sum of what each loses alone. The AMC authors tested exactly this and report that the sensitivity approach "assumes that errors of different pruned layers can be summed up linearly, which does not stand according to our experiments" (He et al., 2018). That is the Layer interaction that slide 24 says the analysis ignores. Pruning layer 3 changes what layer 4 receives, so the curve you measured for layer 4 on a dense network is the wrong curve once layer 3 is thin. The ratios are usable, but they are sub-optimal, and they belong to the network they were measured on. Change the architecture and every curve must be measured again.

The bottleneck is people

Slide 25 shows the practical consequence. Conventionally the allocation relies on human expertise and trial and error. Three customers with three engineers is a workable shop. Many customers overwhelm even a team of engineers (the red cross and the alarmed manager in the middle column), and every new model, new target device or new budget restarts the trial and error from scratch, because rules such as prune the first layer less and the fully connected layers more are, in the paper's words, "non-optimal, and doesn't transfer from one model to another" (He et al., 2018). The right column serves the same crowd with one engineer and an engine that takes the model and the constraint and returns the compressed network.

Slide 26 names the ambition: a push-the-button solution. Today an efficient network needs someone who is both a machine learning expert and a hardware expert. The goal is that a non-expert plus a hardware-centric AutoML tool reaches the same efficient network. This is the definition of Automated pruning in this lecture: an algorithm, not a person, chooses the per-layer ratios for an overall target, and it does so with the target hardware in the loop.

Follow one pass of AMC over MobileNet. Layer t-1 has just been pruned at 30%. The agent now looks at layer t: it receives a short vector describing that layer, its index, its channel counts, its kernel size, its FLOPs, how many FLOPs earlier layers already gave up and how many remain in later layers. It answers with one number, say 50%. The environment removes half of that layer's channels by magnitude and presents layer t+1. When the last layer is done, the whole pruned network is evaluated on a held-out set, and a single number comes back: minus the error. That number is the only feedback the agent ever gets, and it arrives once per pass.

That story contains every ingredient of a reinforcement learning problem, which is what slide 27 means by "pruning as a reinforcement learning problem". The paper's own description: the agent "processes the network in a layer-wise manner", "receives a layer embedding s_t", "outputs a precise compression ratio a_t", and after every layer is compressed the "validation accuracy of the pruned model with all layers compressed is evaluated without fine-tuning, which is an efficient delegate of the fine-tuned accuracy" (He et al., 2018). Only the best network found after the search is fine-tuned.

Why the action must be continuous, and why an actor and a critic

A pruning ratio is a real number, and the compressed model's accuracy is very sensitive to it. A discrete agent would need a grid fine enough to tell 48% from 52%, which explodes the number of actions, and a grid also throws away the order between neighbouring ratios (He et al., 2018), so the agent could not exploit the fact that neighbouring ratios behave alike. DDPG (Lillicrap et al., 2015) is the standard actor-critic method for continuous control: it concurrently learns a Q-function and a policy and can only be used with continuous action spaces (OpenAI Spinning Up). Sutton and Barto give the naming: when a value function is used "to assess, or criticize, the policy's action selections, then the value function is called a critic and the policy is called an actor". In AMC the critic learns, from replayed transitions, how much reward a state and ratio pair leads to; the actor is then nudged by gradient ascent on that estimate to propose ratios the critic scores higher. The critic judges, the actor proposes, and neither ever sees a gradient of the pruned network itself.

Why the reward is minus error under a constraint

Reinforcement learning maximizes reward, so maximizing -Error is minimizing error. The budget appears as a constraint rather than a bonus on purpose: if the agent were paid for shrinking, it would keep shrinking past the target and lose accuracy for nothing. With a hard constraint there is no reward for finishing under budget, so the agent "can precisely arrive at the target compression ratio" (He et al., 2018). The minus infinity branch is the slide's compact way of saying an infeasible network earns nothing useful.

A wasted episode teaches the critic nothing, so the paper never lets one happen. Before each action it computes the duty: the share of the removal target alpha W_all that this layer must carry, given what earlier layers already removed (W_reduced) and the most the remaining layers could remove at the cap a_max (0.8 for convolutions and 0.98 for fully connected layers in fine-grained pruning; 0.8 for every layer in channel pruning). If the actor's proposal is too timid, it is raised to the duty. The budget is therefore met by construction, and the minus infinity case is a formal backstop, not a common outcome.

Slide 28 shows the other protocol. When the goal is the smallest model with no accuracy loss, the budget is unknown in advance, so the reward itself must push on size. The bubble chart on the slide (Canziani et al., 2016) shows that top-1 accuracy rises roughly with the logarithm of the operation count, so multiplying -Error by log(FLOPs) gives, in the paper's words, "a small incentive for reducing FLOPs" while staying "sensitive to Error". The ResNet-50 and MobileNet results in the next two concepts use the constrained reward, not this one.

The constraint need not be FLOPs. The paper notes that the resource can be "FLOPs or the actual inference time on mobile device", and slide 29 adds the mechanism: a pre-built Latency lookup table of measured per-layer times on the target phone, the same device-in-the-loop idea NetAdapt uses in part 05. Substituting measured time for FLOPs turns a FLOPs-constrained search into a latency-constrained one, and the MobileNet table at the end of this part shows why that matters.

Slide 30 puts the agent up against Song Han's own hand-pruned ResNet-50 from his PhD thesis. The human expert kept 29% of the weights, a 3.4x reduction. The agent kept 20%, a 5x reduction, at the same accuracy: 76.13% top-1 for the original and 76.11% for AMC (He et al., 2018). This experiment uses fine-grained weight pruning, not channel pruning, and it is run as Iterative pruning in four rounds at 50%, 35%, 25% and 20% overall density with 30 fine-tuning epochs after each round, which is the schedule part 06 explains.

What the agent noticed inside a residual block

The per-region bars hide the interesting pattern, which slide 31 shows layer by layer. ResNet-50 is built from bottleneck blocks: a 1x1 convolution that reduces the channel count, a 3x3 convolution, and a 1x1 convolution that restores it (He et al., 2016). Walk along the layer index and the kept density forms a sawtooth. The peaks, where the agent kept more, are the 1x1 layers. The crests, where it cut deep, are the 3x3 layers. The paper draws the conclusion directly: the agent "automatically learns that 3x3 convolution has more redundancy than 1x1 convolution and can be pruned more" (He et al., 2018). Nobody told it about kernel shapes; the kernel size k is just one of the eleven features in the state.

The finding makes sense once you think about where redundancy lives. A 1x1 kernel mixes channels and has no spatial extent, so every weight is a distinct channel-to-channel connection with nothing next to it to stand in for it. A 3x3 kernel has nine spatial taps per channel pair, many of which are near zero after training, and it also holds most of a block's weights: in a stage-3 bottleneck the 3x3 layer carries about 590K of the block's 1.1M parameters against 262K for each 1x1 layer (Han, 2017). Cutting the 3x3 layers hard is where the compression is, and it is also where accuracy is cheapest to spend, which is exactly the trade the reward rewards.

The final slide moves from weights to milliseconds. MobileNet is run with TF-Lite on a Samsung Galaxy S7 Edge with a Qualcomm Snapdragon SoC, single core, batch size 1, which is the latency-oriented setting of a phone serving one image at a time. Four models are measured: the full 1.0 MobileNet, two AMC results and the 0.75 MobileNet baseline, which is the same architecture with every layer's width scaled by 75%.

Two lessons in one table

First, FLOPs are a proxy. Halving the multiply-accumulates bought 1.85x, not 2x, because a phone does not spend time in proportion to arithmetic. MobileNet's depthwise layers have a low ratio of computation to memory traffic, so trimming them saves fewer milliseconds than the MAC count suggests; the paper reports about 2x speedup on the 1x1 convolutions but less on the depthwise ones (He et al., 2018). When the agent is instead constrained by measured time from the lookup table, it removes work where the phone is actually slow, and the search lands at 59.7 ms, a 1.99x speedup that all but hits the 2x target. That allocation is less accuracy-friendly, which is the 0.3 point cost. You choose the target you care about, and you pay in the other currency.

Second, non-uniform beats uniform. Uniform shrinking with the width multiplier is a single knob turned on every layer, and the thinner model must be trained from scratch (Howard et al., 2017). AMC prunes a pretrained network with a different ratio per layer, keeps 2.1 (50% FLOPs) or 1.8 (50% Time) more points of accuracy with fewer MACs than 0.75 MobileNet, and does it after a short fine-tune. In the accuracy versus latency plane the AMC points sit above and to the left, which is the definition of a better Pareto frontier.

Suppose a pretrained MobileNet runs in 10.0 ms on one core of a Pixel 1 and the product needs 7.0 ms. The Sensitivity analysis of part 03 cannot help directly, because its curves are in accuracy versus Pruning ratio, not milliseconds, and it ignores Layer interaction. AMC (AutoML for Model Compression) from part 04 could be pointed at latency, but it needs a reinforcement learning agent and a training loop of its own. NetAdapt asks a much simpler question, and asks it many times: which single layer can I thin right now so that latency falls by a fixed step while accuracy falls the least?

The slides call it a rule-based iterative, or progressive, method, and every word matters. Rule-based means no learned policy: the decision each round is a comparison of measured numbers. Iterative means the budget is reached in many small steps rather than one cut. Progressive means each step starts from the winner of the previous one. It is Automated pruning in the sense of part 04, because nobody hand-picks per-layer ratios, but the automation is a loop you could run with a spreadsheet.

The problem, and the trick of tightening it slowly

Yang et al. state the goal as a constrained optimization. Accuracy should be as high as possible while every resource of interest (latency, energy, model size) stays under its budget.

Solving that in one shot is hopeless, so the paper breaks it into a chain of easier problems. At iteration i the constraint is not the final budget but the previous network's resource use minus a step ΔR.

The paper calls ΔR the resource reduction schedule and compares it to a learning-rate schedule. In the small MobileNetV1 experiments it starts at 0.5 ms and decays by 0.96 every iteration; for larger networks the initial step is scaled with the network's latency. The slides simplify this to a manually defined constant, and the worked example below does the same. Think of it as a progressive barrier: each round the wall moves in a little, and the network only has to squeeze past the new wall, never the final one.

One iteration, step by step

Read slides 34 to 39 as a single loop body. The paper's Algorithm 1 has the same shape, with K equal to the number of convolutional and fully connected layers, so a network with K such layers produces K proposals per iteration.

1. Set the constraint for this round: Con = Res_i minus ΔR_i.
2. For each layer L_k, choose the number of filters to keep, from the lookup table, so the whole network meets Con; then choose which filters to keep by l2 magnitude.
3. Short-term fine-tune that proposal (the slides say about 10k iterations) and measure its accuracy on a holdout set.
4. Keep the proposal with the highest accuracy; it becomes the starting point of the next iteration.
5. Repeat while the latency is still above the budget.
6. Long-term fine-tune the final network until convergence.

Why a lookup table and not FLOPs

The constraint in Eq. 2 is a latency, so the loop needs to know the latency of every proposal, and there are K of them per iteration. Timing each one on the phone is slow and hard to parallelize. NetAdapt instead builds a Latency lookup table per layer before the loop starts: for every shape a layer might take (input channels, output channels, resolution), measure it once on the device and store the number. Layers that share a shape share an entry. The latency of a whole proposal is then the sum of its layers' entries. The paper's own illustration is a two-layer network where layer 1 with 4 filters on a 3-channel input costs 6 ms, and layer 2 with 6 filters on the 4 channels that layer 1 now produces costs 4 ms, so the network is estimated at 6 + 4 = 10 ms. Notice that the second entry is indexed by the first layer's filter count, which is how the table accounts for the removed input channels. A table indexed by whole networks would grow exponentially with depth; a table per layer grows linearly.

Why not skip the measurement and count multiply-accumulates? Because MACs are an indirect metric, and their relation to latency is neither linear nor the same across devices. The paper makes the point with a striking table: a network guided by MACs ended up with 19% fewer MACs than the baseline and 29% longer latency on a Pixel 1 CPU.

The lesson carries over to your own boards. Memory traffic, kernel launch overheads, cache behaviour and how well a library tiles a particular layer shape all change latency without changing the MAC count. The honest number is the one the device reports, and the lookup table is a cheap way to have that number on hand for every candidate. The paper checks the approximation on a Pixel 1 and finds the summed estimate highly correlated with the real measurement. Part 04 noted that AMC can also use a pre-built lookup table to optimize latency; NetAdapt makes it the default rather than an option.

Why fine-tune twice

There are two kinds of Fine-tuning in the loop and they serve different purposes. The short-term fine-tune happens once per proposal, so K times per iteration. Its only job is to make the K accuracies comparable. The paper reports that without it, the accuracy of a small network after pruning rapidly drops to nearly zero, and the algorithm then picks the best proposal solely based on noise. With 10k iterations the accuracy stays above 20% and the ranking becomes meaningful.

The long-term fine-tune happens once, after the loop has stopped, and runs until convergence. This is the step that repairs the damage of all the cuts on the final architecture. In the paper it adds between 1.8 and 4.5 points, 3.4 on average, on top of the last short-term number.

A run you can redo by hand

Take a four-layer network with 32, 64, 64 and 128 filters (layers A to D), a measured latency of 10.0 ms and a budget of 7.0 ms. Fix ΔR = 1.0 ms, ten percent of the starting latency. The lookup tables below are illustrative, the accuracies are synthetic, and the effect of one layer's cut on the next layer's input channels is ignored so the sums stay readable. The paper decays ΔR and accounts for the next layer.

Now run it yourself. The simulator uses the same tables and the same synthetic accuracy model, so its default run reproduces the example above. Then change the budget or the step and watch which layer gets picked: a smaller ΔR takes more iterations, produces more intermediate models, and tends to end slightly higher in accuracy, which is what the paper's schedule study found.

NetAdapt against AMC

Both methods answer the question of part 02, which Pruning ratio for which layer, without a human in the loop, and both were published at the same conference. They differ in almost every mechanism, and the NetAdapt paper even uses AMC (under its earlier name, ADC) as a baseline, beating it by 1.2x in latency on the large MobileNetV1.

Look back at the worked example. The run was aiming for one network at 7.0 ms, but on the way it produced Net_1 at 9.0 ms and Net_2 at 7.6 ms, and neither took any extra work. Each was the best of its round and each already satisfied its own, slightly looser, constraint. Slide 40 is that observation at full scale: the number of models equals the number of iterations.

The paper states it directly. Besides the final network, NetAdapt can generate a sequence of simplified networks, the highest-accuracy network from each iteration Net_1 to Net_i, which together provide the efficient frontier of accuracy and resource consumption trade-offs. The authors call it a family of simplified networks that allows dynamic network selection. With the 0.5 ms initial step and 0.96 decay, the small MobileNetV1 run in Table 2 took 28 iterations, so it delivered 28 networks. The red dots on slide 40 are a series of the same kind plotted the same way, one dot per iteration winner, from about 3.5 ms to 11 ms.

Reading the chart

The horizontal axis is latency on a single large core of a Pixel 1 CPU, measured with TensorFlow Lite as the median of eleven runs. The vertical axis is ImageNet top-1 accuracy. The red dots are the NetAdapt model series for the small MobileNetV1; the green triangles are MobileNetV1 shrunk with the width and resolution multipliers, which is Uniform shrinking from part 02; the blue diamond is MorphNet. The two arrows pick a multiplier point and the MorphNet point and slide left along a horizontal line to the NetAdapt curve, which is where the speedups on the slide come from.

From about 12.1 ms to about 7.3 ms is 1.7x faster with 0.3% higher accuracy than the multiplier baseline. From about 7.0 ms to about 4.4 ms is 1.6x faster with 0.3% higher accuracy than MorphNet. Across the whole curve the paper summarises it as up to 1.7x faster with the same or higher accuracy. On a mobile GPU (Samsung Galaxy S8 through SNPE) the gain drops to about 1.2x, because about 6.2 ms of that pipeline's latency is overhead that no amount of pruning can remove.

Why does the frontier beat the multipliers at all? A width multiplier thins every layer by the same factor, so it cannot express that some layers matter more. NetAdapt's greedy picks build a non-uniform architecture: the paper's Fig. 10 shows it keeping more filters in layer 6, where the feature map resolution drops, and in the last convolution that feeds the 1000-class classifier, while cutting layers 7 to 10 harder. That is the same lesson as part 02, now discovered by measurement rather than by a sensitivity sweep.

What the schedule does to the series

Because the series is a by-product of the loop, the step size shapes it. The paper compares three schedules for the same target latency.

Take the AlexNet that Han, Pool, Tran and Dally started from in 2015: 61 million parameters, five convolution layers and three fully connected ones, 57.2% top-1 accuracy. They kept roughly one weight in nine and ended with a network that was not measurably worse. That is the headline of this part, and the whole trick behind it is what happens after the cut.

Now look at what those hours say. Retraining took more than twice as long as training. The authors did not treat it as optional polish; section 3 of the paper calls the final retraining step critical and states that if the pruned network is used without retraining, accuracy is significantly impacted. So the answer to the outline's fifth question, how to fine-tune a pruned network, is a method, not a footnote: the Pruning mask decides which weights exist, and Fine-tuning decides whether the survivors can still do the job.

Why the survivors are wounded, and why the wound heals

A trained layer is a set of weights that were optimized together. Zeroing the smallest of them, the magnitude criterion from lecture 04-1, is a small perturbation to any one output, but a large Pruning ratio removes so many small contributions that the outputs shift and the accuracy drops, more steeply the higher the ratio (this is what the chart on the slide shows for the dashed curve). The survivors are still near a good solution, though. Nothing about them was wrong; the function they were tuned to complete simply lost some of its terms. Fine-tuning continues gradient descent on exactly those survivors, with the removed weights held at zero, so they can absorb the work of the missing ones.

This is why the learning rate matters so much. The slide states the rule: the fine-tuning learning rate is usually 1/100 or 1/10 of the original learning rate. In the paper, LeNet was retrained at 1/10 of the original rate (section 4.1) and AlexNet at 1/100 of the initial rate (section 4.2). A large step would throw the weights out of the basin in which the magnitude ranking was measured, undoing the very judgement the pruning made. Small steps walk the network back down the loss surface from where the cut left it.

Slide 41 makes the same point as an optimization statement. Pruning poses the problem as minimizing the loss over the pruned weights W_P subject to a budget N on the L0 norm, the count of nonzero weights. The mask chooses which entries of W_P may be nonzero at all (the constraint), and fine-tuning is the argmin over the survivors with that mask fixed: the best values the kept weights can take.

The paper adds a second reason to keep the survivors instead of re-initializing them. Section 3.3 observes that networks contain fragile co-adapted features: gradient descent finds a good solution when the network is first trained, but not after re-initializing some layers and retraining them. Fine-tuning starts from the co-adapted weights and preserves that structure. The paper also notes that during retraining it can help to fix the convolution parameters while the fully connected ones retrain, and vice versa, to avoid vanishing gradients through the sparse layers.

The three-box pipeline

The figure on the slide is Figure 2 of the paper. The first box, Train Connectivity, is ordinary training, but its purpose is reframed: it learns which connections are important, not their final values. The second box, Prune Connections, applies the magnitude threshold. The third box, Train Weights, retrains the sparse network. In Han et al.'s implementation the threshold was a quality parameter multiplied by the standard deviation of each layer's weights, and Caffe was modified to add a mask that disregards pruned parameters during network operation for each weight tensor.

Reading the chart on slide 43

The chart plots accuracy loss against the fraction of parameters pruned away for AlexNet. The dashed purple curve is pruning with no retraining; the green curve adds fine-tuning. Reading the points off the figure gives the following.

The paper summarizes the same curves in section 5: without retraining, accuracy begins dropping much sooner, at one third of the original connections rather than one tenth. It describes a free lunch of removing half the connections with no loss even without retraining, while with retraining the connections can be reduced by 9x. Read that 9x carefully: it is reached with the iterative scheme of the next two concepts. A single prune and retrain, the green curve, is already at about -1.7% by 90% pruned. Two green points (at about 67% and 75% pruned) sit slightly above zero, which the authors attribute to pruning finding the right capacity of the network and thereby reducing overfitting. Because sparsity and compression ratio are two ways of saying the same thing, keep the conversion at hand: ratio = 1 / (1 - s).

Once retraining is part of the recipe, a natural question follows: why cut to the final sparsity in one go? The slides tell the story as a sequence. Prune to 30%, retrain the weights. Prune to 50%, retrain. Prune to 70%, retrain. The only change to the pipeline figure is a single arrow from Train Weights back up to Prune Connections, and that arrow is the whole idea of Iterative pruning.

Define the unit first: one iteration is a prune followed by a fine-tune. Iterative pruning repeats that unit while gradually increasing the target sparsity in each iteration, instead of jumping to the final sparsity in one step. Han et al. put it plainly in section 3.4: learning the right connections is an iterative process, pruning followed by retraining is one iteration, and after many such iterations the minimum number of connections can be found. Each iteration is a greedy search for the best connections. They also tried pruning parameters probabilistically by absolute value and found it gave worse results.

Why gradual beats one-shot

The magnitude criterion ranks weights on the network as it is right now. After a 30% cut and a fine-tune, the survivors have re-settled: some grew to cover the removed terms, some shrank because their job was taken over. The ranking for the next cut is therefore measured on a network that has adapted, and the weights it marks as small really are the least needed. A one-shot cut to 90% ranks all 90% on a network that has never adapted to any loss, so it discards weights that would have become important once their neighbors were gone. Zhu and Gupta describe the same principle in the general form of a sparsity schedule: the binary mask is updated every few hundred steps to gradually increase the sparsity of the network while allowing the training steps to recover from any pruning-induced loss in accuracy. Their cubic schedule prunes quickly at first, when redundant connections are abundant, and slowly near the end.

The schedule in this part uses five hand-picked targets rather than a closed form, but the shape is the same: larger cuts early, smaller ones late. On a 6 x 6 matrix of 36 weights, rounding 36 s gives the following zero counts, which are also what the simulator below shows.

The loop also connects backwards to part 05. NetAdapt runs a short-term fine-tune after every layer it thins and a long-term fine-tune at the end, which is the same prune-then-fine-tune loop driven by a latency budget instead of a sparsity schedule. Whatever criterion or granularity you choose, the outer loop looks like this figure.

Slide 51 adds one bullet and one curve, and together they carry the number you will be asked for: Iterative pruning boosts the Pruning ratio from 5x to 9x on AlexNet compared to single-step aggressive pruning. The red curve is what that sentence looks like.

Notice where the curve begins. It does not start at 40% like the others. The paper explains why in section 5: the biggest gain comes from iterative pruning, where the pruned and retrained network of the green curve is pruned and retrained again. The leftmost red dot corresponds to the point on the green line at 80% (5x) pruned further to 8x. There is no accuracy loss at 9x, and not until 10x does the accuracy begin to drop sharply.

The same paper reaches 13x on VGG-16 with five iterations of pruning and retraining, shrinking it to 7.5% of its original size with fc6 and fc7 each pruned to less than 4% of their original size. These pruned networks are the starting point of Deep Compression, which quantizes the surviving weights to 8 bits in convolution layers and 5 bits in fully connected layers and Huffman codes the result to reach 35x on AlexNet (240 MB to 6.9 MB) and 49x on VGG-16. The roughly 90% Weight sparsity that EIE assumes in the next part is this number.

Fine-tuning repairs the network after the cut. Regularization (for pruning) prepares it before the cut. Follow one weight w = 0.05 through training with a penalty coefficient lambda = 0.01 and a learning rate eta = 0.1, ignoring the data loss for the moment so only the penalty acts.

The two regularized losses

The slide states the general rule. During training, or during the fine-tuning of a pruned network, a penalty is added to the loss to penalize nonzero parameters and to encourage smaller ones. The two common choices are the L1 and L2 penalties on the weights W, with lambda setting how strongly the penalty competes with the data loss L(x; W).