EIE results and what came after

Follow one output row to the end. The accumulator for row i has just finished summing every product of a broadcast activation and a decoded weight from that row. The sum lands in a destination activation register. ReLU clamps it: a negative sum becomes zero, a positive one is kept. Now the leading non-zero detection logic scans the register file and reports the first non-zero it finds, as a pair "value v at index j". For the next layer, the central control unit broadcasts only those pairs. The output of this layer is already the sparse input of the next one, and no dense vector was ever written out.

That is the loop the slides close. Part 08 showed how a broadcast activation fans out to the processing elements and how each PE walks its CSC column; this part shows how the results become the next broadcast. The detection is hierarchical: each group of four PEs feeds a leading non-zero detection node, those nodes feed another, and the root of the tree sits inside the central control unit and drives the H-tree broadcast. Han et al. report 21 such nodes for 64 PEs (16 + 4 + 1), each about 0.023 mW and 189 µm², well under 0.3% of one PE. The machinery that keeps both ends of the datapath sparse is almost free.

The four special aspects, and the saving each one buys

The lecturer highlights four blocks on the PE diagram as EIE's special aspects. Each of them is a concrete piece of hardware, and each maps onto one of the four factors in the paper's energy breakdown. Reading them as pairs is what turns the block diagram into an argument.

The FIFO deserves one extra sentence because it is easy to miss. Columns have different numbers of non-zeros, so PEs finish at different times. Han et al. swept the queue depth and found that at depth 1 some benchmarks lose about half their cycles to imbalance, while the gain flattens past depth 8, which is what they built. That is load balance bought with a few registers rather than with a different pruning pattern, and the retrospective will show why that trade did not scale.

Why on-chip SRAM is the whole story

The factors multiply, but they are far from equal. Table I of the EIE paper restates Horowitz's 45 nm energy table from ISSCC 2014, the same table lecture 04-1 opened with. A 32-bit integer add costs 0.1 pJ, a 32-bit integer multiply 3.1 pJ, a 32-bit SRAM cache read 5 pJ, and a 32-bit DRAM access 640 pJ. The caption says it plainly: DRAM uses three orders of magnitude more energy than simple arithmetic and 128x more than SRAM; the paper rounds this 128x table ratio to the 120x it uses in its energy breakdown. A multiplier that is twice as efficient saves picojoules. A weight that never leaves the chip saves hundreds of them, on every fetch, for every weight.

Pruning and weight sharing matter here for a reason that has nothing to do with arithmetic: they are what make the model small enough to fit. After compression, AlexNet's fully connected layers need about 131K weights per PE, and 64 PEs at 0.638 mm² each hold all of them. Memory takes 93% of a PE's area and 59% of its power, so EIE is, physically, an SRAM chip with a little arithmetic attached. The order of causes is the exam answer: compression makes the model fit on chip, fitting on chip removes DRAM, and removing DRAM is the 120x.

Take AlexNet's FC6 layer: a 4096 x 9216 matrix, 37.7 million weights. After pruning only 9% of the weights survive, and after ReLU only 35% of the input activations are non-zero. A multiply is needed only where a surviving weight meets a non-zero activation, which is 0.09 x 0.35, about 3.2% of the dense count. That is where the 33x FLOP reduction on the slide comes from.

The general rule is that the two sparsities multiply. Every layer in the benchmark set is a fully connected layer of a real model, and each one has its own pair of densities. Han et al. chose nine of them: three from AlexNet, three from VGG-16, and three from NeuralTalk, an RNN and LSTM image captioning model. The last three are the interesting outliers.

What EIE was measured against

The comparison platforms were a desktop CPU (Intel Core-i7 5930k, running Intel MKL: CSRMV for the sparse model and GEMV for the dense one), a desktop GPU (NVIDIA Titan X, cuSPARSE CSRMV and cuBLAS GEMV), and a mobile GPU (the Tegra K1, measured on a Jetson TK1 developer board, which is the same silicon the slide names differently). Each baseline was given the best library available for its case, dense or sparse, so the comparison is against optimized software rather than naive loops.

On those nine layers EIE is 189x faster than the CPU and 13x faster than the GPU, while using 24,000x and 3,400x less energy respectively. The paper adds a number that is easy to forget and worth remembering: the same compressed model, run on the CPU or GPU with sparse libraries, gives only about 3x speedup over dense. Compression alone is not the win. Compression plus hardware built for it is.

Look at the two right-most bars first. The fabricated EIE, 64 PEs at 45 nm, pushes 81,967 matrix-vector products per second through AlexNet FC7 at 0.59 W. DaDianNao, the strongest rival ASIC, reaches 147,938 per second but draws 15.97 W. Divide, and the picture flips: 138,927 frames per joule for EIE against 9,263 for DaDianNao, a 15x gap in EIE's favor at an older process node. The projected 256 PE version at 28 nm would reach 426,230 per second and 180,606 per joule.

The reason energy is where EIE separates from the pack is the one from the previous concept, now visible across platforms. A matrix-vector product at batch size 1 is completely memory bound: every weight is used once, so throughput is set by how fast weights can be delivered, and energy by where they come from. DaDianNao's rate in the table is in fact a bandwidth estimate from its eDRAM (Han et al. compute 16 x 4 x 1024 bit / 8 x 606 MHz = 4,964 GB/s), A-Eye pulls every parameter from DDR3, and the CPU and GPUs all stream from DRAM. Only TrueNorth and EIE keep weights in SRAM, and TrueNorth pays for it with 1-bit weights and an asynchronous design measured on a different task.

Consider what a large language model does when it generates one token: it multiplies a single activation vector by weight matrices holding billions of parameters, reads every weight once, and does it again for the next token. That is a matrix-vector product at batch size 1, memory bound on weights, which is exactly the workload EIE was built for in 2016. So EIE's storage trick came back: keep weights in 4 bits, decode to 16, compute in 16. Han et al. call it W4A16 and point at GPTQ, AWQ, llama.cpp and MLC LLM, with one difference: today's engines use linear integer weights rather than a K-means codebook.

In 2023, for ISCA's 50th anniversary retrospective collection, the authors wrote down what they think held up and what did not. The slide condenses it to four pros and four cons. The tables below expand each line with the evidence the retrospective gives and the design that replaced or extended the idea.

The con that matters most for the rest of this lecture

The first con is the one that shaped GPUs. A vector processor or a tensor core executes the same instruction on many lanes at once, so it needs each lane to have work at the same time. With irregular fine-grained sparsity, one lane may hold a non-zero and its neighbor a zero, and there is no way to line them up. EIE solved this with a private index stream per PE and a FIFO for load balance; ESE (FPGA 2017) went further with load-balance-aware pruning. NVIDIA instead changed the pattern itself: prune so that in every block of four consecutive weights exactly two are zero, a constraint called 2:4 sparsity. The Sparse Tensor Core then reads a 2-bit index per kept weight, selects the matching activations, and doubles math throughput with metadata overhead far below EIE's 50%. NVIDIA's recipe is to train dense, prune to 2:4 by magnitude, and retrain with the original hyperparameters. The next part takes this up in full.

The second con is the index overhead you met in the energy gap: EIE stores a 4-bit index next to every 4-bit weight, and surrounds a single MAC with pointer reads, CSC traversal and the detector. Coarser granularity, whether 2:4 or whole blocks, amortizes that overhead. The third con, FC layers only, was answered by SCNN, Cambricon-X and Eyeriss v2 for sparse convolution. The fourth is a matter of scale: everything in SRAM is right for TinyML and vision models, and Cerebras pursued it commercially, but a model of 10 to 100 billion parameters does not fit.

Edit one corner of a generated image and ask the model to redraw it. A dense pipeline recomputes every pixel. SIGE (NeurIPS 2022) computes only the edited region and reuses the rest. Nothing about this is specific to pruned weights, yet it is the same move EIE made: do not spend energy where the answer is already known to be zero or unchanged.

The retrospective states the move as a principle. The first principle of efficient AI computing is to be lazy: avoid redundant computation, quickly reject the work, or delay the work. Each of the four bullets on the slide is one of those three verbs applied to a different axis of the data, and reading them that way is more useful than memorizing four paper names.

• Avoid redundant computation: SIGE skips unedited pixels; TSM replaces a 3D convolution over time with a shift that costs zero FLOPs.
• Quickly reject the work: SpAtten prunes tokens and attention heads that contribute little, before they reach the expensive layers.
• Delay the work: SpAtten's progressive quantization fetches the most significant bits first and only fetches the least significant bits when the attention scores are not yet confident.

The point-cloud line is the bridge to the last parts of this lecture. A LiDAR scan occupies a tiny fraction of its bounding volume, so sparse convolution computes only where points exist. TorchSparse makes that fast on a GPU and PointAcc builds the map generation into hardware. Notice that this is dynamic sparsity of the input, not static sparsity of the weights, which is the static versus dynamic split again: the hardware has to discover the sparsity on the fly, exactly as EIE's leading non-zero detector did for activations.