Glossary

Two of every four contiguous weights are zero (50 percent sparsity), stored as nonzero values plus 2-bit indices and accelerated about 2x on NVIDIA Ampere GPUs.

• Slide 13
• Slide 21
• Slide 22

Zeros in a layer's activations, mostly produced by ReLU; a MAC is skipped only when the weight or the input activation is zero, and APoZ measures this fraction per channel.

• Slide 11
• Slide 39

AutoML for Model Compression, which finds per-layer pruning ratios and achieves higher ImageNet accuracy than uniform scaling at the same latency.

• Slide 25

Average Percentage of Zeros: the fraction of zero ReLU activations of a channel over batch and spatial positions; smaller APoZ means a more important neuron.

• Slide 39
• Slide 40

The gamma in z_o = gamma (z_i - mu_B)/sqrt(sigma_B^2 + epsilon) + beta, reused as the per-channel scaling factor for pruning.

• Slide 33

Removing whole channels so the network has fewer channels; gives direct speedup but a smaller compression ratio.

• Slide 18
• Slide 23
• Slide 24
• Slide 25
• Slide 41
• Slide 42

beta, a vector of length c_i where beta_c = 0 prunes input channel c, constrained by ||beta||_0 <= N_c.

• Slide 42

Pruning whole structures such as rows or channels; less flexible, a subset of fine-grained choices, but easy to accelerate because the result is a smaller dense matrix.

• Slide 15
• Slide 38

A convolution weight tensor has shape [c_o, c_i, k_h, k_w]: output channels (filters), input channels, kernel height and kernel width.

• Slide 16
• Slide 17

Efficient Inference Engine (Han et al., ISCA 2016), a custom accelerator that holds a pruned model in on-chip SRAM and skips zero weights and activations; the example of hardware that turns fine-grained sparsity into speed.

• Slide 13
• Slide 19

Removing whole output channels (filters, W[o, :, :, :]) of a convolution layer, which also deletes the matching input channel of the next layer; Li et al. score each filter by the sum of its absolute kernel weights.

• Slide 18
• Slide 32
• Slide 38

Pruning any individual weight; most flexible pruning indices and usually the largest compression ratio, but irregular and hard to accelerate on GPUs.

• Slide 15
• Slide 18
• Slide 19
• Slide 20

Retraining the remaining weights after pruning to recover the accuracy lost by removing connections.

• Slide 9
• Slide 10
• Slide 22

||A||_F, the square root of the sum of the squared entries of a matrix; its square in the reconstruction error is the total squared error over the whole layer output.

• Slide 41
• Slide 42

The matrix of second derivatives h_ij of the loss with respect to weights; its diagonal h_ii is non-negative and costly to compute.

• Slide 34
• Slide 36

The positions a sparse format (CSR, CSC, COO or the 2:4 index strip) stores next to surviving weights; fine-grained pruning pays one index per weight, coarser units share one index per unit, and channel pruning needs none.

• Slide 18
• Slide 19
• Slide 21

Repeating the prune then retrain loop several times, reaching higher pruning ratios without accuracy loss than one-shot pruning.

• Slide 10

Removing entire k_h by k_w kernels from a convolution weight tensor.

• Slide 18

The count of nonzero entries in a weight tensor, written ||W_P||_0; used as the constraint in the pruning formulation.

• Slide 5
• Slide 42

Importance of a structural set equal to the sum of absolute values of its weights.

• Slide 28
• Slide 29

Importance of a structural set equal to the square root of the sum of squared weights.

• Slide 30
• Slide 31

Replacing the NP-hard L0 constraint on beta with an L1 penalty lambda ||beta||_1, so channel selection becomes a LASSO regression with a sparse solution, alternated with least-squares reconstruction of W.

• Slide 42

||W^(S)||_p = (sum over i in S of |w_i|^p)^(1/p), the general magnitude measure for a structural set.

• Slide 31

Multiply-accumulate operations, a measure of the compute cost of a network, reported alongside parameter reduction.

• Slide 11

The cut-off below which weights are removed in magnitude-based pruning; Han et al. set it per layer as a quality parameter times the standard deviation of that layer's weights.

• Slide 7
• Slide 8
• Slide 28

A heuristic criterion that treats weights with larger absolute values as more important; element-wise importance is |W|.

• Slide 28
• Slide 29
• Slide 30
• Slide 31

A 32-bit DRAM access costs about 640 pJ in 45 nm, roughly 200 times a 32-bit integer multiply, so data movement dominates energy.

• Slide 3

A pattern where every group of M contiguous elements has a fixed number of pruned elements; in NVIDIA's convention at most N of each M are nonzero.

• Slide 21
• Slide 22

Liu et al. (2017): train with an L1 penalty on batch norm gamma, prune the channels whose gamma falls below a global percentile threshold, then fine-tune.

• Slide 32
• Slide 33

Hu et al. (2016): measure APoZ on validation images, trim the neurons whose APoZ is more than one standard deviation above the layer mean, retrain from the pre-trim weights, and repeat a few layers at a time.

• Slide 39
• Slide 40

Removing whole neurons with all their incoming and outgoing connections; a coarse-grained form of weight pruning that deletes a row of a linear layer's weight matrix.

• Slide 4
• Slide 38

LeCun et al. (1989) method assuming a nearly quadratic loss, converged training and independent deletion errors, giving importance 1/2 h_ii w_i^2.

• Slide 34
• Slide 35
• Slide 36
• Slide 37

Pruning with fixed shapes of kept weights inside kernels or groups, a semi-regular granularity between fine-grained and vector-level.

• Slide 18
• Slide 21

Making a neural network smaller by removing synapses (weights) and neurons whose removal least affects performance.

• Slide 4
• Slide 7
• Slide 8

The rule for measuring parameter or neuron importance; removing less important ones yields better pruned performance.

• Slide 26
• Slide 27
• Slide 38

argmin over W_P of L(x; W_P) subject to ||W_P||_0 <= N, where L is the training objective and N the target number of nonzeros.

• Slide 5

The structural unit in which weights are removed, ranging from individual weights to whole channels.

• Slide 15
• Slide 18

The set of weight positions zeroed by pruning; it stays fixed during fine-tuning so pruned weights remain zero, and coarse-grained masks are a subset of fine-grained ones.

• Slide 9
• Slide 10
• Slide 15

The percentage of parameters pruned away from a network.

• Slide 7
• Slide 8
• Slide 9
• Slide 10

||Z - Z_hat||_F^2, the squared Frobenius distance between the original and pruned layer outputs.

• Slide 41
• Slide 42

How many times smaller the pruned model is, k = 1 / (1 - r) for a pruning ratio r: 80 percent pruned is 5x, 90 percent is 10x.

• Slide 8
• Slide 9
• Slide 10
• Slide 11

Channel pruning that minimizes the reconstruction error of a layer's outputs rather than the network loss.

• Slide 41
• Slide 42

OBD's name for the estimated loss increase from deleting one parameter, s_k = 1/2 h_kk w_k^2; parameters with the lowest saliency are deleted first.

• Slide 36
• Slide 37

Filter pruning that attaches a trainable scaling factor to each output channel and prunes channels whose factor magnitude is small.

• Slide 32
• Slide 33

Choosing weights to prune by approximating the loss change with a Taylor series that uses the Hessian.

• Slide 34
• Slide 35
• Slide 36

The NVIDIA Ampere (A100) unit that reads 2:4 compressed weights with their 2-bit indices and skips the zeros, giving up to 2x matrix-multiply throughput.

• Slide 13
• Slide 21
• Slide 22

The fraction of weights in a tensor or layer that are zero after pruning.

• Slide 5
• Slide 21
• Slide 23
• Slide 24

A group S of parameters, W^(S), such as a row, kernel or channel, scored together for structured pruning.

• Slide 29
• Slide 30
• Slide 31

Removing individual connections (weights) between neurons while the neurons remain.

• Slide 4
• Slide 19

Synapses per neuron grow from about 2500 at birth to 15000 at 2 to 4 years, then are pruned to about 7000 in adults.

• Slide 6

delta L = sum g_i delta w_i + 1/2 sum h_ii delta w_i^2 + 1/2 sum_{i != j} h_ij delta w_i delta w_j + O(||delta W||^3).

• Slide 34
• Slide 35

Reducing every layer's channels by the same ratio, which gives worse accuracy than channel pruning with per-layer sparsity.

• Slide 24
• Slide 25

Removing whole rows or columns of weights within a kernel.

• Slide 18