Neural network efficiency metrics

AOverview

Accuracy tells you whether a network is right. This lecture gives you the five numbers that tell you whether it can run at all on the board in front of you: how many bytes it stores, how many it must hold while computing, how many multiply-accumulates it performs, how long that takes on a given chip, and how much charge it draws. Every result later in the course, from pruning to quantization to neural architecture search, is a ratio of two of these numbers.

As a PhD student you need them in three places. Exams ask you to count parameters, activations and MACs for a layer list, convert the counts into megabytes and milliseconds, and argue from the max latency model and the 45 nm energy table. Your research project puts a detector on an embedded board, and its evaluation section is unreadable without these metrics stated under the right conditions. And any efficient system you design or review, on a Jetson or a microcontroller, is judged first by whether its weights fit in flash, its peak activations fit in SRAM and its latency fits inside one frame period.

The path has five stops. You place every metric in the efficiency triangle, memory-related or computation-related, and see which of storage, latency and energy it predicts. You learn why latency and throughput are different questions, why a latency is meaningless without its hardware, runtime, precision and batch size, and why the milliseconds of an inference are a max of compute time and memory time rather than a sum. You count what a network stores, then what it must hold in memory, then what it must compute, with AlexNet worked layer by layer at each step. The lecture ends where it began: the count of operations divided by the rate of a chip is the compute term of that latency model.

Numbers you will own by the end

AlexNet is the running example, and by the last part you will have derived every one of these figures yourself rather than quoted them. They are the anchors the exam and the reference sheet return to.

Success looks like

• Place any efficiency metric in the memory or computation group and name which of storage, latency and energy it predicts.
• Compute latency and throughput for a batched design and explain with numbers why neither implies the other.
• Write Latency ≈ max(T_computation, T_memory) with its sub-formulas, label each term NN or hardware specification, and decide compute-bound versus memory-bound.
• Count parameters for linear, convolution, grouped and depthwise layers, total a network, and convert the count to model size at any bit width.
• Compute total and peak activations for a layer list and say whether the model fits a microcontroller whose SRAM holds activations and whose flash holds weights.
• Count MACs per layer, convert to FLOPs and OPs, and estimate T_computation on a chip from its OPS.
• Argue from the 45 nm energy table that one DRAM access costs about two hundred integer multiplies, so efficient design is above all a design for moving fewer bytes.

How to study this lecture

1. Read the parts in order. Each one feeds the next: the costs of part one are what parts two to four learn to predict, and the final part closes the loop back to the latency model.
2. Keep a pencil next to AlexNet. Every worked example hides one layer behind a question; compute it before you reveal it, then check yourself in the layer calculators.
3. Answer every recall prompt in your head before opening it. The numbers that stick are the ones you retrieved, not the ones you read.
4. Take each quiz and read the explanation even when you are right; the wrong options are the exam traps.
5. Mark a concept understood only when you could redo its formula and its AlexNet number from a blank page. Unmarked concepts show you where to return.
6. Keep the glossary open for terms and the reference sheet for the formula table, the energy table and the AlexNet totals when you solve problems.
7. Come back after a few days and redo the counts cold on a different network, such as your project detector. Spaced practice on a new example is what turns a formula into a skill.

BThe 4 parts

1. 01Latency, throughput and energyThe efficiency triangle of storage, latency and energy, what latency and throughput each measure, why one does not imply the other, the analytical latency model, and why data movement dominates energy.5 conceptsSlides 1-930 min
   1. 1.1Smaller, faster, greener: three goals, three costs, two resources
   2. 1.2Latency is the delay for one task, and the number is meaningless without its conditions
   3. 1.3Throughput is a rate, and batching raises it without lowering latency
   4. 1.4Latency as max(T_computation, T_memory): the network supplies numerators, the hardware denominators
   5. 1.5Energy is spent moving data, not multiplying it
2. 02Parameters and model sizeCounting weights for linear, convolution, grouped and depthwise layers, totaling AlexNet to 61M parameters, and converting parameter count into storage at different bit widths.4 conceptsSlides 10-1724 min
   1. 2.1#Parameters counts synapses, and a weight tensor is where they live
   2. 2.2Convolution, grouped and depthwise: one formula and a single division by g
   3. 2.3AlexNet layer by layer: 61M parameters, 96 percent of them in three linear layers
   4. 2.4Model size is parameters times bit width, and the units hide a factor of eight
3. 03Activations and peak memoryWhy activations rather than parameters are the memory bottleneck in inference and training, how memory is spread across layers of MobileNetV2 and MCUNet, and how to compute total and peak activations for AlexNet.4 conceptsSlides 18-2224 min
   1. 3.1Shrinking the weights did not shrink the peak: activations bound inference memory
   2. 3.2Training keeps every activation alive: 707 MB against 102 MB
   3. 3.3Early layers are activation heavy, late layers are weight heavy
   4. 3.4Counting AlexNet activations: total versus peak
4. 04MACs, FLOPs and operationsCounting computation as multiply-accumulate operations for matrix products and each layer type, AlexNet's 724M MACs, and converting to FLOPs, FLOPS, OPs and OPS.5 conceptsSlides 23-2930 min
   1. 4.1One MAC, one matrix product: the unit of neural network compute
   2. 4.2MACs per layer: every weight fires once per output position
   3. 4.3AlexNet's 724M MACs: convolution owns compute, linear owns parameters
   4. 4.4From MACs to FLOPs and OPs, and from counts to rates
   5. 4.5The reading list behind this lecture

CGlossary and reference

• GlossaryEvery term from the lecture, defined once, with links back to the slides that use it.
• Reference sheetThe lecture on one printable page, for revision before a quiz or exam.