Glossary

A chosen minimum accuracy on the sensitivity chart; each layer's pruning rate is read where its curve crosses T, and T is adjusted until the overall pruning rate meets the target.

• Slide 19
• Slide 20
• Slide 21
• Slide 22

EIE's dataflow: the central control unit scans the input vector for its next nonzero a_j and sends the pair (a_j, j) to every PE at once over an H-tree; each PE multiplies it by the nonzero weights in its own slice of column j and accumulates into outputs it alone owns, so zero activations skip a whole column for all PEs and no partial sums are ever exchanged.

• Slide 60
• Slide 61
• Slide 62
• Slide 63
• Slide 64

Zeros in the activations produced at run time, typically by ReLU (dynamic sparsity, about 70 percent in EIE), letting hardware skip multiplications since W times 0 = 0.

• Slide 55
• Slide 57
• Slide 60
• Slide 67
• Slide 71

TorchSparse's strategy of padding weight-offset groups of similar size so they run as one batched matmul, sitting between separate per-offset matmuls (no waste, irregular) and fully dense batching (regular, large overhead).

• Slide 101
• Slide 104
• Slide 105

A method by He et al. (ECCV 2018) that casts per-layer pruning ratio selection as reinforcement learning: a DDPG agent receives a layer embedding, outputs a sparsity ratio as a continuous action, and is rewarded by negative error subject to a FLOPs or latency constraint.

• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 32

Operations performed per byte moved from memory; a GEMM with small K has low intensity and is bound by memory traffic and launch overhead, so halving its math with 2:4 sparsity barely helps, while a large-K GEMM is math bound and approaches the 2x ceiling.

• Slide 83

Choosing per-layer pruning ratios for an overall compression target with an algorithm instead of hand-tuned heuristics, as in AMC and NetAdapt.

• Slide 24
• Slide 25
• Slide 26
• Slide 33

The EIE retrospective's first principle of efficient AI computing: avoid redundant computation, quickly reject the work, or delay the work; applied as spatial sparsity (SIGE, point clouds), token sparsity and progressive quantization (SpAtten) and temporal sparsity (TSM).

• Slide 77

Removing entire output channels (filters) of a convolution layer, which is coarse-grained weight pruning that shrinks the layer's weight matrix and keeps dense computation.

• Slide 6
• Slide 9
• Slide 27
• Slide 35

EIE's weight storage format: for each column, the nonzero values with a small relative index (distance from the previous nonzero, padded when the gap is too large) and a pointer to where the column starts.

• Slide 65
• Slide 68

How uniform and batchable the matrix multiplies are; higher regularity uses the GPU better but, for sparse convolution, is bought with padded rows that add computation overhead.

• Slide 102
• Slide 103
• Slide 104
• Slide 105

Deep Deterministic Policy Gradient, an actor-critic reinforcement learning algorithm for continuous action spaces; AMC uses it because a pruning ratio is a continuous value.

• Slide 27

Han, Mao and Dally (ICLR 2016): a three-stage pipeline of pruning, trained quantization with a k-means codebook (weight sharing) and Huffman coding that shrinks AlexNet from 240 MB to 6.9 MB (35x) and VGG-16 by 49x with no accuracy loss, small enough to live in on-chip SRAM; EIE runs directly on its output.

• Slide 56
• Slide 57

The first DNN accelerator (Han et al., ISCA 2016) that runs directly on a sparse, compressed model, exploiting weight sparsity, activation sparsity and weight sharing with an array of processing elements and on-chip SRAM.

• Slide 55
• Slide 56
• Slide 57
• Slide 58
• Slide 73
• Slide 74
• Slide 75
• Slide 76
• Slide 77

Continuing to train the surviving weights after pruning to recover accuracy, usually with a learning rate 1/10 to 1/100 of the original training rate.

• Slide 43
• Slide 44
• Slide 51

The GPU pipeline for sparse convolution: gather the input feature rows named by the map for one weight offset, multiply by that weight matrix, then scatter-accumulate partial sums to the output rows.

• Slide 95
• Slide 96
• Slide 97
• Slide 98
• Slide 101

A fused, output-stationary sparse convolution kernel (SpConv v2, TorchSparse++) in which each thread block owns a tile of output rows, walks the kernel weights and fetches matching input rows on demand, overlapping loads with tensor-core work; lockstep execution adds redundant multiplies that row reordering and column splitting trim (12 to 10 to 8 on the slide).

• Slide 108
• Slide 109

Repeating prune-then-fine-tune iterations while gradually raising the target sparsity, which boosted AlexNet's pruning ratio from 5x to 9x compared with single-step aggressive pruning.

• Slide 44
• Slide 45
• Slide 46
• Slide 47
• Slide 48
• Slide 49
• Slide 50
• Slide 51

||W||_0, the count of nonzero entries in a weight tensor; the pruning constraint bounds it by the target number of nonzeros N.

• Slide 3
• Slide 41
• Slide 53

Pre-measured per-layer latencies on the target device for different channel counts, used by NetAdapt so that pruning decisions track real latency rather than FLOPs.

• Slide 29
• Slide 35

The state vector s_t = [N, C, H, W, i, ...] that AMC feeds its agent, summarizing the current layer's index, shape, kernel size and FLOPs budget context.

• Slide 27
• Slide 28

The compounding effect of pruning several layers at once, which sensitivity analysis ignores because it sweeps one layer at a time, making its per-layer rates sub-optimal.

• Slide 23
• Slide 24

How much accuracy a network loses when one layer alone is pruned at a given ratio; sensitive layers such as the first layer lose accuracy quickly, redundant layers tolerate high ratios.

• Slide 10
• Slide 18

The EIE unit that scans an activation vector after ReLU to find the next nonzero activation and its index so zero activations are never broadcast.

• Slide 65
• Slide 67
• Slide 71

Keeping all PEs busy despite uneven numbers of nonzeros per column, achieved in EIE with an activation FIFO queue that lets PEs run ahead.

• Slide 65
• Slide 66

The single fine-tune to convergence that NetAdapt runs once the latency budget is met, repairing the accumulated cuts of the final architecture; its result, 1.8 to 4.5 points above the last short-term number in the paper, is the accuracy that gets reported.

• Slide 39

Fine-grained structured sparsity written a:b, in which at most a of every b consecutive weights along a row are nonzero (the slides write M:N, papers usually N:M); NVIDIA Ampere tensor cores support 2:4, storing R x C/2 values plus R x C/2 two-bit indices.

• Slide 79
• Slide 80
• Slide 81

A heuristic criterion that treats weights with larger absolute value as more important, Importance = |W| for element-wise pruning, and removes the smallest.

• Slide 5

PointAcc's hardware block that shifts input coordinates by a kernel offset, merge-sorts them with the output coordinates, and emits an (In, Out, Wgt) tuple for every pair of equal neighbors.

• Slide 111
• Slide 112

The list of tuples that drives sparse convolution: input point P, output point Q and the kernel weight offset W_(dx,dy) relating them; for each entry f_out = f_out + f_in x W_wgt.

• Slide 89
• Slide 90
• Slide 91
• Slide 93

A rule-based automatic pruning method (Yang et al., ECCV 2018) that repeatedly reduces latency by a fixed step Delta R by pruning the single layer whose short-term fine-tuned accuracy is highest, using a measured latency lookup table, until a latency budget is met, then long-term fine-tunes.

• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40

Liu et al. (ICCV 2017): train with an L1 penalty lambda sum |gamma| on the per-channel batch normalization scaling factors, prune the channels whose gamma falls below a global percentile threshold, then fine-tune; a channel-level use of regularization that reports a 20x smaller VGGNet with 5x fewer operations.

• Slide 52

Removing whole neurons from a linear layer, equivalent to deleting a full row of its weight matrix.

• Slide 6

The fraction of all weights removed when every layer is pruned at its own rate, R = sum_i r_i N_i over sum_i N_i, a parameter-weighted average rather than the mean of the per-layer rates; the threshold T is tuned until R meets the target, and the compression factor is 1 / (1 - R).

• Slide 21
• Slide 22

A point-cloud accelerator (Lin et al., MICRO 2021) whose mapping unit builds sparse convolution maps in hardware with merge sort instead of hash tables, delivering large speedup and energy savings over GPUs.

• Slide 110
• Slide 111
• Slide 112
• Slide 113

One EIE compute unit that owns a round-robin subset of weight-matrix rows, stores them in compressed form in its own SRAM, and multiplies every broadcast nonzero activation by its matching nonzero weights.

• Slide 58
• Slide 59
• Slide 60
• Slide 65

Removing weights or whole neurons and channels from a trained neural network to reduce parameters and computation, formulated as minimizing the loss L(x; W_P) subject to the number of nonzeros ||W_P||_0 staying at or below a budget N.

• Slide 3
• Slide 41
• Slide 53
• Slide 114

The pattern in which weights are removed, from irregular fine-grained (individual weights) through pattern-based, vector and kernel level to regular channel-level pruning; finer is more flexible, coarser is more hardware friendly.

• Slide 4
• Slide 6

The fraction of a layer's weights (or channels) removed by pruning; sparsity is the fraction of zeros that results, and the ratio may differ per layer.

• Slide 8
• Slide 10
• Slide 18
• Slide 23

r = 1 minus theoretical FLOPs over actual FLOPs for a padded group of weight offsets; adaptive grouping extends a group while r stays at or below the tolerance epsilon, and a second threshold S decides whether a group runs as a batched matmul or, when its largest map already fills the GPU, as a plain matmul.

• Slide 104
• Slide 105

Adding a penalty to the training loss to penalize nonzero and encourage smaller parameters: L1 gives L' = L(x; W) + lambda |W| and L2 gives L' = L(x; W) + lambda ||W||^2; magnitude pruning uses L2 on weights and Network Slimming uses an L1 penalty (the slide says smooth-L1) on the batch norm channel scaling factors.

• Slide 52

The amount of latency (or another resource) that each NetAdapt iteration must remove, Con = Res minus Delta R; the slides treat it as a manually defined constant, the paper sets an initial value that decays every iteration, like a learning-rate schedule, and smaller steps take more iterations but end slightly more accurate.

• Slide 34
• Slide 38

The procedure of pruning each layer L_i in isolation at ratios r in {0, 0.1, ..., 0.9}, recording the accuracy degradation Delta Acc_r^i, and repeating for all layers to obtain a per-layer accuracy versus ratio curve.

• Slide 11
• Slide 12
• Slide 13
• Slide 14
• Slide 15
• Slide 16
• Slide 17
• Slide 18

The by-product of NetAdapt's loop: each iteration's winning network already satisfies its own, slightly looser, constraint, so Net_1 to Net_i form a deployable accuracy versus latency frontier from one run, with the number of models equal to the number of iterations (the slide writes serial of models).

• Slide 40

The brief fine-tune (about 10k iterations on the slides) that NetAdapt applies to every per-layer proposal inside an iteration so that their accuracies can be ranked; without it the pruned proposals collapse to near-zero accuracy and the pick becomes noise.

• Slide 36
• Slide 37

Convolution over sparse inputs such as point clouds that computes outputs only at positions where inputs exist, keeping output sparsity equal to input sparsity, implemented as a sparse set of dense matrix multiplies defined by maps.

• Slide 86
• Slide 87
• Slide 88
• Slide 91
• Slide 92

An Ampere tensor core mode that uses the 2:4 metadata to select only the activation operands paired with nonzero weights, roughly doubling GEMM throughput with no accuracy loss after fine-tuning.

• Slide 82
• Slide 83

Static sparsity is fixed at deployment (pruned weights); dynamic sparsity depends on the input at run time (activations after ReLU, point-cloud occupancy), so hardware must detect it on the fly.

• Slide 57
• Slide 67
• Slide 86

Graham et al.'s name for what dense convolution does to sparse input: one active site becomes 3^d active sites after one 3^d convolution and 5^d after two, so a thin surface thickens into a slab and the sparsity is gone within a few layers; on the slide grid 4 of 20 cells become 17 of 20. Submanifold sparse convolution avoids it by computing outputs only at input positions.

• Slide 86
• Slide 87
• Slide 88

An efficient point-cloud inference engine (Tang et al., MLSys 2022) that speeds sparse convolution with locality-aware gather and scatter and adaptive grouping of matmuls, trading a little padded computation for regularity.

• Slide 101
• Slide 102
• Slide 103
• Slide 104
• Slide 105
• Slide 106
• Slide 107

The successor engine that overlaps memory movement (gather and scatter) with computation by pipelining the phases, along with a sparse kernel generator and autotuner.

• Slide 108
• Slide 109

Scaling every layer's width by the same factor (for example the 0.75 MobileNet width multiplier); it is dominated by non-uniform per-layer pruning on the accuracy versus latency curve.

• Slide 9
• Slide 32
• Slide 40

Store weights in 4 bits, decode them to 16 bits and compute in 16 bits, the storage trick EIE used with a codebook that the 2023 retrospective sees reborn (with linear integer weights) in single-batch LLM decoding engines such as GPTQ, AWQ, llama.cpp and MLC LLM.

• Slide 76

Quantizing weights to a small per-layer codebook so each weight is stored as a 4-bit index and decoded to a full value at compute time, giving EIE an 8x smaller memory footprint.

• Slide 57
• Slide 69

Zeros in the weight matrix that are fixed after training (static sparsity, about 90 percent in EIE's models), letting hardware skip stored values and multiplications since 0 times A = 0.

• Slide 55
• Slide 57
• Slide 68

The dataflow of the existing GPU sparse convolution: hold one offset's C_in x C_out weight matrix still and stream every map entry for that offset through it as gather, matmul, scatter, launching a separate matmul per weight; contrasted with output-stationary implicit GEMM, where each thread block owns a tile of outputs and walks all the weights.

• Slide 96
• Slide 97