COE 592Lecture 03Reference

Reference sheet

Neural network efficiency metrics compressed onto one page: the definitions, formulas and numbers to have in your head before a quiz or exam.

Efficiency triangle

Three goals, three costs, two resources. Memory feeds all three costs; computation feeds latency and energy. Part 01: Latency, throughput and energy

Goal	Cost	Resource	Metrics that predict it
Smaller	Storage	Memory	#parameters, model size
Faster	Latency	Computation and memory	MACs, OPs, model size, peak #activations
Greener	Energy	Computation and memory	MACs, OPs, and above all the bytes moved

Goal, cost, resource and the metrics that predict it

Group	Members	How obtained	Predicts
Memory-related	#parameters, model size, total and peak #activations	Counted on paper from layer shapes	Storage, and the memory share of latency and energy
Computation-related	MACs, FLOPs and FLOPS, OPs and OPS	Counts on paper, rates from the datasheet	The computation share of latency and energy

The two metric groups

Latency and throughput

Property	Latency	Throughput
Definition	Delay to complete one specific task	Work completed per unit time
Also called	Response time, execution time	Bandwidth
Unit	ms	image/s
Measured at	Batch size 1	Whatever batch fills the processor
Who cares	A brake camera, an on-device detector	A data center indexing video overnight
Batching	Raises it: each input waits for the whole batch	Raises it: more inputs finish per second

Definitions side by side

\text{throughput} = \frac{b}{T_{\text{batch}}}, \qquad \text{latency} = T_{\text{batch}}

b inputs processed together in one batch of duration T_batch

Design	How inputs are handled	Latency	Inputs per batch	Throughput
Design 1	One image at a time	50 ms	1	1 / 0.050 = 20 image/s
Design 2	Four images together	100 ms	4	4 / 0.100 = 40 image/s
Server, batch 1	One request	2 ms	1	500 image/s
Server, batch 32	Thirty-two requests	8 ms	32	4000 image/s

Batching raises both latency and throughput

The four conditions every reported latency must state (slide 5 footnote)

Hardware: NVIDIA Jetson AGX Orin: 43 dense FP16 TFLOPS on Tensor Cores, 204.8 GB/s LPDDR5
Runtime: TensorRT
Precision: fp16
Batch size: 1

Model	mIoU	Parameters	MACs	Orin latency	A100 throughput
SegFormer-B5	82.4	85M	1460G	638 ms	12 image/s
EfficientViT-L1	82.7	40M	282G	45.9 ms	122 image/s
Ratio	+0.3	2.1x fewer	5.2x fewer	13.9x faster	10x more

SegFormer-B5 versus EfficientViT-L1 on Cityscapes at 1024 x 2048

The latency model

Data movement and computation overlap on real hardware, so the layer finishes when the slower stream finishes. Numerators are NN specification, denominators are hardware specification. Part 01: Latency, throughput and energy

\text{Latency} \approx \max\left(T_{\text{computation}},\; T_{\text{memory}}\right)

Max, not sum: the two streams overlap

T_{\text{computation}} \approx \frac{\text{operations in the model}}{\text{operations per second}}, \qquad T_{\text{memory}} \approx T_{\text{move}}(\text{activations}) + T_{\text{move}}(\text{weights})

The memory terms add because both cross the same bus

T_{\text{move}}(\text{weights}) \approx \frac{\text{model size}}{\text{bandwidth}}, \qquad T_{\text{move}}(\text{activations}) \approx \frac{\text{input size} + \text{output size}}{\text{bandwidth}}

Bytes over bytes per second

Term	Numerator: NN specification	Denominator: hardware specification
T_computation	Number of operations (MACs, OPs)	Operations per second (OPS)
T_move(weights)	Model size	Memory bandwidth
T_move(activations)	Input activation size + output activation size	Memory bandwidth

Each term, and which part of the lecture counts its numerator

Configuration	T_computation	T_memory	Latency	Regime
100 GOPS, 32-bit weights	10 ms	5 + 2 = 7 ms	10 ms	Compute-bound
200 GOPS, 32-bit weights	5 ms	5 + 2 = 7 ms	7 ms	Memory-bound
200 GOPS, 8-bit weights	5 ms	1.25 + 2 = 3.25 ms	5 ms	Compute-bound

Toy model: 1 GOP of work, 50 MB weights, 20 MB activations, 10 GB/s

Model	T_computation	T_memory	Floor	Measured	Miss, utilisation
EfficientViT-L1	564 / 43 000 = 13.1 ms	0.39 + 0.45 = 0.84 ms	13.1 ms	45.9 ms	3.5x, about 29%
SegFormer-B5	2920 / 43 000 = 68 ms	0.83 + 0.45 = 1.3 ms	68 ms	638 ms	9.4x, about 11%

Slide 5 predicted from Orin datasheet numbers, fp16, 43 TFLOPS, 204.8 GB/s

Energy of data movement

45 nm, 0.9 V, 32-bit operations, Horowitz ISSCC 2014 as tabulated by Han et al. 2015. Part 01: Latency, throughput and energy

Rough energy per operation (slide 9)

int ADD: 0.1 pJ
float ADD: 0.9 pJ
Register file access: 1 pJ
int MULT: 3.1 pJ
float MULT: 3.7 pJ
SRAM cache access: 5 pJ
DRAM memory access: 640 pJ

Ratio	Arithmetic	Meaning
DRAM over int MULT	640 / 3.1 ≈ 206	The slide's 200x
DRAM over one int MAC	640 / 3.2 = 200	Exact reading of the slide icon
DRAM over SRAM	640 / 5 = 128	Why on-chip storage is worth two decades
float ADD over int ADD	0.9 / 0.1 = 9	The seed for quantization
DRAM over int ADD	640 / 0.1 = 6400	Three orders of magnitude (Han et al.)

Ratios worth memorizing

Item	Count	Energy each	Total
Integer MACs	724M	3.1 + 0.1 = 3.2 pJ	2.32 mJ
Weights read once from DRAM	61M	640 pJ	39.0 mJ
Same weights read from SRAM	61M	5 pJ	0.31 mJ
Float MACs instead of integer	724M	3.7 + 0.9 = 4.6 pJ	3.33 mJ

One AlexNet inference: 724M MACs, 61M weights

Counting formulas for parameters and MACs

Bias is ignored throughout. Parameters never depend on n, h or w; MACs multiply by h_o · w_o because every filter is reapplied at every output position. Part 02: Parameters and model size, Part 04: MACs, FLOPs and operations

Notation

n: Batch size
c_i, c_o: Input and output channels (features for a linear layer)
h_i, w_i, h_o, w_o: Input and output height and width
k_h, k_w: Kernel height and width
g: Number of groups in a grouped convolution

Layer	#Parameters	MACs	MACs per parameter
Linear	c_o · c_i	c_o · c_i	1
Convolution	c_o · c_i · k_h · k_w	c_o · c_i · k_h · k_w · h_o · w_o	h_o · w_o
Grouped convolution	c_o · c_i · k_h · k_w / g	c_o · c_i · k_h · k_w · h_o · w_o / g	h_o · w_o
Depthwise convolution	c_o · k_h · k_w	c_o · k_h · k_w · h_o · w_o	h_o · w_o
Pooling	0	0

Per layer type, batch size 1

\#\text{Params}_{\text{grouped}} = \frac{c_o}{g}\cdot\frac{c_i}{g}\cdot k_h k_w \cdot g = \frac{c_o \cdot c_i \cdot k_h \cdot k_w}{g}

Each group loses g squared, but there are g groups: one division by g

\text{MACs}_{MV} = m \cdot n \qquad \text{MACs}_{GEMM} = m \cdot n \cdot k \qquad a \leftarrow a + b \cdot c

Matrix-vector, matrix-matrix, and the MAC itself

Layer	Count	#Parameters	Ratio
Standard, g = 1	256 · 256 · 9	589,824	1
Grouped, g = 2	256 · 256 · 9 / 2	294,912	1/2
Grouped, g = 8	256 · 256 · 9 / 8	73,728	1/8
Depthwise, g = 256	256 · 9	2,304	1/256

One 3 x 3 layer, 256 in and 256 out, under each grouping

Layer	Count	MACs
Standard 3 × 3	64 · 32 · 9 · 112 · 112	231,211,008
Depthwise 3 × 3	32 · 9 · 112 · 112	3,612,672
Pointwise 1 × 1	64 · 32 · 112 · 112	25,690,112
Separable over standard	1 / 64 + 1 / 9	0.1267, about 7.9x fewer

MobileNet block at 112 x 112, 32 to 64 channels: MACs

AlexNet, every column

Input 3 × 224 × 224. The linear layers hold 96% of the weights and do 8% of the MACs; the convolutions are the reverse. Part 02, Part 03, Part 04

Layer	Output C × H × W	Activations	#Parameters	MACs	MACs per parameter
Image	3 × 224 × 224	150,528	0	0
conv1, 11 × 11, stride 4, pad 2	96 × 55 × 55	290,400	34,848	105,415,200	3,025
maxpool	96 × 27 × 27	69,984	0	0
conv2, 5 × 5, pad 2, g = 2	256 × 27 × 27	186,624	307,200	223,948,800	729
maxpool	256 × 13 × 13	43,264	0	0
conv3, 3 × 3, pad 1	384 × 13 × 13	64,896	884,736	149,520,384	169
conv4, 3 × 3, pad 1, g = 2	384 × 13 × 13	64,896	663,552	112,140,288	169
conv5, 3 × 3, pad 1, g = 2	256 × 13 × 13	43,264	442,368	74,760,192	169
maxpool	256 × 6 × 6	9,216	0	0
fc6, 9,216 → 4096	4096	4,096	37,748,736	37,748,736	1
fc7, 4096 → 4096	4096	4,096	16,777,216	16,777,216	1
fc8, 4096 → 1000	1000	1,000	4,096,000	4,096,000	1
Five convolutions			2,332,704 (3.8%)	665,784,864 (91.9%)
Three linear layers			58,621,952 (96.2%)	58,621,952 (8.1%)
Total		932,264	60,954,656	724,406,816

Output shape, output activations, parameters, MACs and uses per weight, per layer

Model size from bit width

\text{Model Size} = \#\text{Parameters} \cdot \text{Bit Width}, \qquad \text{Size}_{\text{MB}} = \frac{\#\text{Parameters} \cdot \text{bits}}{8 \cdot 10^{6}}

The product is in bits; divide by 8 for bytes and by 10^6 for decimal MB

\text{Model Size} = \sum_{l} \#\text{Params}_l \cdot \text{bits}_l

Mixed precision: sum per layer

Data type	Size in bits	Size (decimal MB)	Slide rounding
fp32	1,950.5 Mbit	243.8 MB	244 MB
fp16 or bf16	975.3 Mbit	121.9 MB	122 MB
int8	487.6 Mbit	61.0 MB	61 MB
int4	243.8 Mbit	30.5 MB	30.5 MB
1-bit	61.0 Mbit	7.6 MB	7.6 MB

AlexNet at each bit width, exact count 60,954,656 and the slide's rounded 61M

1 MB = 10^6 B and 1 MiB = 2^20 B (NIST). The slide's 244 MB is decimal, about 232.5 MiB. Say which you use.
The 8-bit size is simply N bytes. Forgetting the divide by 8 overstates the size eightfold and is the most common error on this question.
Halving the bit width halves the size, not the accuracy; those are separate questions.

Activations decide fit

SRAM constrains activations (read and write); flash constrains model size (read only). SRAM is the scarcer budget, so peak activation, not parameter count, decides whether a CNN runs on a microcontroller. Part 03: Activations and peak memory

Tensor	Lives in	Access pattern	Metric	Budget
Weights	Flash	Read only, written once at flashing	Model size	Larger, about 1 MB to 2 MB
Activations	SRAM	Written and read on every inference	Peak #activations	Smaller, 256 kB to 512 kB

Two budgets on a microcontroller

Model	Param memory (flash)	Peak activation (SRAM)	Fits F746?	Fits H743?
ResNet-18	11.2 MB	0.9 MB	No	No
MobileNetV2-0.75	2.5 MB	1.7 MB	No	No
MCUNet	1.9 MB	0.49 MB	No	Yes
Ratio ResNet-18 to MobileNetV2	4.6x smaller	1.8x larger

Inference, int8, about 70 percent ImageNet top-1, against STM32F746 (1 MB flash, 320 kB SRAM) and STM32H743 (2 MB flash, 512 kB SRAM)

Model	Weights	Activations	Activations over weights
ResNet-50	102 MB	707 MB	About 7x
MobileNetV2-1.4	About 24 MB	About 640 MB	About 27x
Ratio	4.3x smaller	1.1x smaller

Training, fp32, batch size 16 (TinyTL)

\frac{\partial L}{\partial W_i} = a_i^{\mathsf{T}}\,\frac{\partial L}{\partial a_{i+1}}, \qquad \frac{\partial L}{\partial b_i} = \frac{\partial L}{\partial a_{i+1}}

Backprop needs the stored input a_i for the weight gradient, but not for the bias gradient

Mode	Which activations are held	Memory rule
Inference	Input and output of the layer running now	Largest input + output pair
Training	Every layer input a_i until dL/dW_i is computed	Sum of all activations × batch size, plus weights, gradients and optimizer state

What must be alive at once

Stage	Shape	Dominant tensor	What it decides
Early layers	Large H × W, few channels	Activations	SRAM peak; patch-based inference fixes it
Middle layers	Both small	Neither	Cheapest place for on-device training
Late layers	Large C_o · C_i, tiny H × W	Weights	Flash budget; pruning and quantization pay here

Where memory sits across depth: resolution shrinks faster than channels grow

Total and peak activations

\#\text{act}_{\text{total}} = |x_0| + \sum_{i=1}^{L} |a_i|, \qquad \#\text{act}_{\text{peak}} \approx \max_{i}\big(|a_{i-1}| + |a_i|\big)

Sum every tensor for the total; scan adjacent input + output pairs for the peak

AlexNet adjacent pair sums; the first pair wins

conv1 (image + conv1 out): 150,528 + 290,400 = 440,928
pool1 (conv1 out + pool1 out): 290,400 + 69,984 = 360,384
conv2 (pool1 out + conv2 out): 69,984 + 186,624 = 256,608
pool2 (conv2 out + pool2 out): 186,624 + 43,264 = 229,888
conv3 to fc8: All under 130,000; fc8 is 4,096 + 1,000 = 5,096

Quantity	Values	8-bit	32-bit
Peak	440,928	441 kB	1.76 MB
Total	932,264	932 kB	3.73 MB
MobileNetV2 first conv	150,528 + 401,408 = 551,936	552 kB (539 KiB)	2.21 MB

Values to bytes (decimal kB and MB)

Assumptions behind the approximation: layer-by-layer execution, weights streamed from flash and not counted, no residual branches (else add all branches alive at once, sharing the input), no in-place tricks (a depthwise layer can run in N + 1 instead of 2N), plus real runtimes add scratch buffers.
The peak is a pair, never the largest single tensor. Forgetting the input undercounts AlexNet by a third.
The total (932,264) is what training holds; inference holds only the peak pair (440,928, about 47% of the total).
Per-layer input + output is also the activation traffic in T_move(activations) of the latency model.

FLOPs, OPs and computation time

Counts belong to models, rates belong to chips. Divide a count by a rate and you have a time. Part 04: MACs, FLOPs and operations

\text{FLOPs} = 2 \cdot \text{MACs}, \qquad \text{FLOPS} = \frac{\text{FLOPs}}{\text{second}}, \qquad T_{computation} = \frac{\text{FLOPs of the model}}{\text{FLOPS of the processor}}

One multiply plus one add per MAC; OPs and OPS are the same with any data type

Term	Kind	Meaning	Belongs to
MAC	Count	One multiply accumulated into a sum	A model, per inference
FLOP	Count	One floating point multiply or add; a MAC is two	A model, per inference
OP	Count	One operation of any data type; a MAC is two	A model, per inference
FLOPS	Rate	Floating point operations per second	A processor
OPS	Rate	Operations per second, any data type	A processor

Five terms, two kinds of quantity

Processor	Count ÷ rate	T_computation
1 GFLOPS	1.449 × 10^9 / 10^9	1,449 ms
100 GFLOPS	1.449 × 10^9 / 10^11	14.5 ms
1 TFLOPS	1.449 × 10^9 / 10^12	1.45 ms
67 TOPS (Orin Nano, INT8 sparse peak)	1.449 × 10^9 / 6.7 × 10^13	0.022 ms

AlexNet's 1.449G FLOPs on different processors

Procedures

Six steps that cover every numeric question in this lecture

1. Parameters: One formula instance per weighted layer, pooling is 0, write /g explicitly, sum. Add a share column if asked which layer dominates.
2. Model size: Bits first (N × bits), divide by 8 for bytes, then by 10^6, and say "decimal MB".
3. Activations: Write the C × H × W column, sum for the total, scan adjacent input + output pairs for the peak, multiply by bytes per value last. State the layer-by-layer assumption.
4. MACs: Parameter formula with h_o · w_o appended; h_o · w_o = 1 for a linear layer; c_i drops out of depthwise.
5. Time: FLOPs = 2 × MACs, then T_computation = FLOPs / FLOPS in milliseconds. Note that a peak rate gives a floor, not a prediction.
6. Regime: Compare T_computation with T_memory; the larger one names the bound and the lever that helps.

Slide errata

Answer with the corrected fact and name the slide if a question depends on it. Part 02, Part 03, Part 04

What the slides get wrong or leave out

Slide 5: 46 ms is the rounded 45.9 ms; the 82.4 mIoU is single-scale, the paper's 84.0 is multi-scale.
Slide 8: The MIT original has a Load Input, Load Weight, Compute, Store Output pipeline above the formula. That overlap is the whole reason for max rather than plus.
Slide 9: The 200x arrow is 640 / 3.1 ≈ 206 (int MULT) or 640 / 3.2 = 200 (int MAC). Memory rows are halved from Horowitz's 64-bit figures, via Han et al. 2015.
Slides 15, 16: conv1 is printed as 24,848; the product 96 · 3 · 11 · 11 is 34,848. Exact total 60,954,656; the rounded 61M stands.
Slide 18: Right chart x-axis reads 14, 16, 16, 17, should be 14, 15, 16, 17. Peak activation grew 1.8x, it did not merely "not improve". The right chart is MCUNetV2 (2021) Figure 1, not MCUNet (2020).
Slide 20: The figure is Figure 11(a) of On-Device Training Under 256KB Memory (NeurIPS 2022), a per-layer training update cost, not an inference profile; the x-axis (layers 0 to 42) was cropped.
Slide 21: The peak formula is stated without its assumptions: layer-by-layer execution, no residual branches, no in-place operations, no scratch buffers.
Slide 23: The GEMM figure is loosely labelled. Read m · n · k as rows of C times columns of C times the shared dimension.
Slides 24, 25: The grouped MAC formula appears in two factor orders; both equal c_o · c_i · k_h · k_w · h_o · w_o / g.
Slide 27: "1.4G FLOPs" is 2 × 724,406,816 = 1,448,813,632, so 1.45G exactly. Show the factor of two.