Pruning ratios, fine-tuning and sparse hardware

AOverview

Lecture 04-1 taught you to remove weights. This lecture answers the two questions it left open, how many to remove from each layer and how to get the accuracy back, and then confronts the fact that a network full of zeros runs no faster on ordinary hardware. The second half is about the engines, tensor cores and libraries that were built so that sparsity on paper becomes speed in silicon.

As a PhD student you need it in three places. Exams ask you to run a Sensitivity analysis and read ratios off a threshold, to write the NetAdapt loop or name the state, action and reward of AMC (AutoML for Model Compression), to explain why Fine-tuning uses a tenth of the learning rate, and to trace a sparse matrix-vector product through EIE (Efficient Inference Engine) or a 2:4 block through a Sparse tensor core. Your research project puts a compressed model on a board, and the reviewer will ask the question this lecture drills: you pruned 90% of the weights, so why is it not 10x faster? Beyond the course, any efficient system you design has to choose between irregular sparsity with custom hardware and structured sparsity on commodity chips, and that trade-off is the spine of parts seven to thirteen.

The path has five stops. You learn why Uniform shrinking wastes accuracy and how to sweep each layer alone to measure its Layer sensitivity. You hand the search over to machines, an RL agent in AMC and a greedy latency-driven loop in NetAdapt, because humans allocate ratios badly and layers interact. You recover the lost accuracy with fine-tuning and see Iterative pruning lift AlexNet from 5x to 9x. You open EIE, the first engine that computes directly on compressed weights, stage by stage, and weigh what held up seven years later. Finally you follow sparsity into commodity GPUs with M:N sparsity and into point clouds with Sparse convolution, TorchSparse and PointAcc.

Numbers you will own by the end

These are the anchors the parts derive and the exam returns to. Each one is a ratio, and the skill is knowing what it compares and under which conditions it holds.

Success looks like

• Run a sensitivity sweep on paper, read a per-layer ratio off a threshold for every layer, and compute the parameter-weighted overall rate.
• Explain why sensitivity analysis is sub-optimal, then contrast AMC and NetAdapt on who chooses the ratios, what cost model they use and what comes out of one run.
• Write the prune, fine-tune, repeat loop with its learning rate and quote the AlexNet 5x to 9x result, and say what L1 and L2 regularization each do to the weights.
• Trace one nonzero activation through EIE from broadcast to accumulator, encode a column in relative-index CSC, and rank the four energy factors.
• Compress a 2:4 matrix, compute its storage saving, and explain why GEMM speedup approaches 2x while end-to-end inference gains less.
• Build the (In, Out, Wgt) map for a small sparse grid, cost the gather, matmul, scatter pipeline, and say what TorchSparse and PointAcc each fix.
• Map every kind of sparsity in the lecture, weight, activation, structured and point cloud, to the hardware that exploits it and the granularity it needs.

How to study this lecture

1. Read parts one to six in order, since each ratio method exists to fix the limit of the one before. Parts seven to thirteen can be read as three shorter arcs: EIE, tensor cores, and point clouds.
2. Keep a pencil next to the worked examples. Read ratios off the VGG-11 chart, run three NetAdapt iterations, encode a CSC column and build a sparse convolution map before you reveal the answer.
3. Answer every recall prompt in your head before opening it. The numbers that stick are the ones you retrieved.
4. Take each quiz and read the explanation even when you are right. The wrong options are the exam traps, especially FLOPs versus measured latency and static versus dynamic sparsity.
5. Mark a concept understood only when you could redraw its diagram, the EIE dataflow or the gather, matmul, scatter pipeline, from a blank page.
6. Keep the glossary open for terms and the reference sheet for the result table, the CSC format and the 2:4 arithmetic when you solve problems.
7. Come back after a few days and redo the sweeps and traces cold on a network of your own, such as your project model. Spaced practice on a new example is what turns a procedure into a skill.

BThe 13 parts

1. 01Where we are: the pruning problem so farA compact recap of lecture 04-1: pruning as constrained optimization, the granularity spectrum, magnitude as the default criterion and neuron or channel pruning as coarse-grained weight pruning, setting up the two open questions this lecture answers.3 conceptsSlides 1-718 min
   1. 1.1Pruning is a loss minimization under a nonzero budget
   2. 1.2Magnitude scores and the granularity ladder
   3. 1.3Neurons, channels and the two questions still open
2. 02Why every layer needs its own pruning ratioUniform shrinking loses to non-uniform pruning, so each layer needs its own ratio. Sensitivity analysis measures how much accuracy each layer loses as it is pruned harder.3 conceptsSlides 8-1718 min
   1. 2.1Uniform shrinking wastes accuracy: every layer deserves its own ratio
   2. 2.2Sensitivity: how fast accuracy falls when one layer alone is pruned
   3. 2.3The sweep: one layer, ten ratios, repeat for all layers
3. 03From sensitivity curves to per-layer ratiosPick an accuracy threshold, read off each layer's pruning rate where its curve crosses it, and see why this heuristic ignores the interaction between layers.3 conceptsSlides 18-2318 min
   1. 3.1Reading the chart: which layers are sensitive and which are redundant
   2. 3.2One horizontal line turns six curves into six ratios
   3. 3.3Why the read-off is a heuristic: layers are swept alone but pruned together
4. 04AMC: pruning ratios as a reinforcement learning problemGiven an overall compression target, AMC lets a DDPG agent choose each layer's sparsity, rewarded by accuracy under a FLOPs or latency constraint, and beats hand-tuned and uniformly scaled MobileNets on a real phone.4 conceptsSlides 24-3224 min
   1. 4.1Per-layer ratios are a search, and humans search badly
   2. 4.2AMC: state, action, reward and a DDPG agent
   3. 4.3What the agent discovers on ResNet-50: peaks at 1x1, crests at 3x3
   4. 4.4Targeting measured time: the MobileNet table
5. 05NetAdapt: platform-aware pruning to a latency budgetA rule-based automatic method that removes latency in fixed steps, tries pruning each layer with a measured latency lookup table, keeps the layer whose short fine-tune loses the least accuracy, and produces a whole series of models.2 conceptsSlides 33-4012 min
   1. 5.1NetAdapt: shave Delta R of latency per iteration, keep the layer that hurts least
   2. 5.2One run, a whole accuracy versus latency frontier
6. 06Fine-tuning, iterative pruning and regularizationPruning costs accuracy, fine-tuning at a much smaller learning rate recovers it, repeating prune and fine-tune pushes AlexNet from 5x to 9x compression, and L1 or L2 regularization during training makes weights easier to prune.4 conceptsSlides 41-5224 min
   1. 6.1Pruning costs accuracy, and fine-tuning at a tiny learning rate buys it back
   2. 6.2One iteration is prune then fine-tune, and the loop raises sparsity gradually
   3. 6.3Reading the three curves: iterative pruning takes AlexNet from 5x to 9x
   4. 6.4Regularization: L1 parks weights at zero, L2 shrinks them, both make pruning safer
7. 07EIE: the first accelerator for sparse, compressed networksWhy sparsity needs hardware support, Han's co-design paradigm, and how EIE exploits weight sparsity, activation sparsity and weight sharing by splitting a sparse matrix-vector product across processing elements.3 conceptsSlides 53-6018 min
   1. 7.1Sparsity on paper versus sparsity in silicon: why Han built an engine
   2. 7.2Three sources of savings: sparse weights, sparse activations, shared weights
   3. 7.3Interleaving rows across PEs and broadcasting only the nonzero activations
8. 08Inside EIE: dataflow, the PE and its storage formatsHow a nonzero activation walks through the PE array, what one processing element contains, how load is balanced, and how activation sparsity, CSC weight storage and 4-bit codebooks are realised in hardware.3 conceptsSlides 61-7018 min
   1. 8.1The dataflow: one broadcast, many local multiply-adds
   2. 8.2Inside one PE: five stages from queue to ReLU
   3. 8.3Four mechanisms and the storage format that makes them cheap
9. 09EIE results and what came afterNon-zero detection after ReLU, EIE's special design choices, its throughput and energy lead over CPU, GPU, FPGA and other ASICs, and the retrospective lessons that shaped later sparse hardware.5 conceptsSlides 71-7730 min
   1. 9.1Closing the loop: ReLU, non-zero detection and why the PE never leaves the chip
   2. 9.2Nine sparse layers against a CPU, a GPU and a mobile GPU
   3. 9.3Reading the log-scale charts: order-of-magnitude evidence, not a fair race
   4. 9.4Seven years later: what held up and what did not
   5. 9.5The first principle: be lazy, and where sparsity went next
10. 10M:N sparsity on NVIDIA tensor coresFine-grained structured sparsity keeps exactly N nonzeros in every block of M weights, compresses the matrix to half plus two-bit indices, and lets Ampere sparse tensor cores double GEMM throughput with no accuracy loss.4 conceptsSlides 78-8324 min
    1. 10.12:4 sparsity: a pattern the hardware can count on
    2. 10.2Push the nonzeros left: half the values plus 2-bit indices
    3. 10.3The tensor core multiplies only the columns the indices name
    4. 10.4Train, prune, retrain: near 2x GEMM speed at zero accuracy cost
11. 11Sparse inputs and sparse convolutionPoint clouds are mostly empty space, so convolution should compute only where inputs exist: sparse convolution keeps output sparsity equal to input sparsity and is a sparse set of dense matrix multiplies driven by an input, output and weight map.3 conceptsSlides 84-9618 min
    1. 11.1Point clouds are almost empty, and ordinary convolution fills them in
    2. 11.2Maps: a list of (In, Out, Wgt) tuples is the whole computation
    3. 11.3Weight-stationary: gather, one matmul per offset, scatter-add
12. 12TorchSparse: regular computation from irregular sparsityThe gather, matmul, scatter pipeline on GPUs, why separate small matmuls waste the GPU, and how TorchSparse trades a little padding for regularity with adaptive grouping, locality-aware access and, in TorchSparse++, overlapped memory and compute.3 conceptsSlides 97-10918 min
    1. 12.1The baseline pipeline: one gather, one matmul and one scatter per weight offset
    2. 12.2Trading a little padding for regularity: separate, dense and adaptive grouping
    3. 12.3What grouping buys, and why TorchSparse++ fuses the three phases
13. 13PointAcc, the lecture summary and referencesPointAcc builds the sparse convolution map in hardware with a merge-sort based mapping unit, then the lecture closes with what was covered and what quantization brings next.4 conceptsSlides 110-11524 min
    1. 13.1Building the map in hardware: from hash tables to a merge sort
    2. 13.2What the merge sort buys: PointAcc against GPU, TPU and CPU
    3. 13.3What this lecture built and what quantization needs next
    4. 13.4The reading list, and the papers the slides cite but do not list

CGlossary and reference

• GlossaryEvery term from the lecture, defined once, with links back to the slides that use it.
• Reference sheetThe lecture on one printable page, for revision before a quiz or exam.