COE 592Lecture 03Glossary
Glossary
Every term in Neural network efficiency metrics, defined once and used the same way in every part. Each entry links to the slides where the idea appears.
- Terms
- 42
- Letters
- 17
A
B
- Batch size (n)
The number of inputs processed together in one pass; larger batches can raise throughput while increasing per-input latency.
C
- Computation time
The part of latency spent on arithmetic, approximated as the number of operations in the model divided by the number of operations the processor can do per second.
- Compute-bound
The regime of the latency model in which computation time exceeds memory time, so latency is set by the operation count over the processor's operations per second and a faster processor lowers it.
D
- Data movement energy
The observation that memory access costs far more energy than arithmetic: in 45 nm at 0.9 V a 32-bit DRAM read costs 640 pJ, about 200 times a 32-bit integer multiply at 3.1 pJ.
- Depthwise convolution
A grouped convolution with one group per channel, so each channel has its own k_h x k_w filter; it has c_o·k_h·k_w parameters and c_o·k_h·k_w·h_o·w_o MACs.
- DRAM memory access
Reading or writing off-chip dynamic RAM, the most energy expensive operation in the Horowitz table at 640 pJ per 32-bit access.
E
- Efficiency metrics
Quantities used to judge how small, fast and energy frugal a neural network is, grouped into memory-related metrics (#parameters, model size, total and peak #activations) and computation-related metrics (MAC, FLOP, FLOPS, OP, OPS).
F
- Floating point operation (FLOP)
A single floating point multiply or add; one MAC counts as two FLOPs, so AlexNet's 724M MACs are about 1.4G FLOPs.
G
- General matrix-matrix multiplication (GEMM)
Multiplying two matrices, costing m·n·k MACs for the dimensions m, n and k shown on the slide.
- Grouped convolution
A convolution whose channels are split into g independent groups, giving c_o·c_i·k_h·k_w/g parameters and c_o·c_i·k_h·k_w·h_o·w_o/g MACs.
- Groups (g)
The number of independent channel partitions in a grouped convolution; parameters and MACs are divided by g.
I
- Inverted residual block
The MobileNetV2 building block that expands the channel count about six times with a 1x1 convolution, applies a depthwise 3x3 convolution, then projects back down; cheap in weights and MACs, but the expanded tensor at full spatial resolution gives the early blocks a large peak activation, up to 1372 kB.
K
L
M
- Matrix-vector multiplication (MV)
Multiplying an m x n matrix by an n-element vector, costing m·n MACs.
- MCU flash budget
The read-only non-volatile flash memory of a microcontroller, where the weights are stored; for a typical MCU of this class such as the STM32F746 used by MCUNet (Lin et al., NeurIPS 2020) it is about 1 MB. The slides do not give a flash size. Flash is bounded by model size, while the separate and smaller SRAM is bounded by peak activations.
- MCU memory constraint
The small runtime memory of a microcontroller, shown as 256 kB, which MobileNetV2's early blocks exceed with a peak of 1372 kB.
- MCUNet
A tiny deep learning system for IoT devices (Lin et al., NeurIPS 2020) whose per-layer memory shows high activation memory in early layers and high weight memory in late layers.
- Memory bandwidth
A hardware specification giving how fast a processor can move data to and from memory; it divides model size and activation size to estimate data movement time.
- Memory bottleneck
The resource that limits whether a model fits a device; for CNNs it is activation memory, which barely improved from ResNet to MobileNetV2 even though parameters shrank several times.
- Memory time
The part of latency spent moving data, approximated as the time to move activations plus the time to move weights.
- Memory-bound
The regime of the latency model in which memory time exceeds computation time, so latency is set by the bytes of weights and activations moved over the memory bandwidth, and only fewer bytes or more bandwidth lowers it.
- MobileNetV2
An efficient CNN built from inverted residual blocks; it has far fewer parameters than ResNet but similar or higher activation memory, with an imbalanced per-block memory profile.
- Model size
The storage needed for a network's weights, equal to #Parameters x bit width when one data type is used; AlexNet is 244 MB at 32-bit and 61 MB at 8-bit.
N
- NN specification vs hardware specification
The split in the latency model between terms set by the network (number of operations, model size, activation sizes) and terms set by the processor (operations per second, memory bandwidth).
O
- Operation (OP)
A generalized count of operations used when weights and activations are not floating point; AlexNet has about 724M x 2 = 1.4G OPs.
- Output height and width (h_o, w_o)
The spatial size of a layer's output feature map; convolution MACs scale with h_o·w_o because every filter is applied at every output position.
P
- Peak #activations
The largest number of activations that must be held at once during inference, approximated as the input plus output activations of the worst layer; 150,528 + 290,400 = 440,928 for AlexNet.
S
- SRAM cache access
Accessing on-chip static RAM, costing about 5 pJ per 32-bit access, far cheaper than DRAM but costlier than arithmetic.
T
- Throughput
The rate at which data is processed, for example images or videos per second; it can rise through batching or parallelism without lowering latency.
- TinyTL
An on-device learning method (Cai et al., NeurIPS 2020) motivated by the finding that activations, not trainable parameters, dominate training memory.
- Total #activations
The sum of activation elements over the input and all layer outputs; 932,264 for AlexNet.
W
- Weight tensor
The multidimensional array holding a layer's learned weights, for example c_o x c_i for a linear layer or c_o x c_i x k_h x k_w for a convolution.