COE 592Lecture 02Reference

Reference sheet

Neural network fundamentals compressed onto one page: the definitions, formulas and numbers to have in your head before a quiz or exam.

Vocabulary and the two counts

Parameters are stored (flash), activations are computed per input (SRAM). A network fits a microcontroller only when both fit: the STM32F746 has 320 kB SRAM and 1 MB flash, and ResNet-50's 25.6 M parameters are about 100 MB in fp32. Part 01: Neurons and linear layers

y_j = f\left(\sum_i w_i x_i + b\right)

One neuron: c_i weights, one bias, one activation function

Diagram element	Names	Behaviour	Memory
Edge	Synapse, weight, parameter	Stored after training, one per connection	Flash
Node	Neuron, feature, activation	Recomputed for every input, one per computed unit	SRAM
Bias	One per neuron, never per edge	Stored, counted with the weights	Flash
Width	Size of a hidden layer	Goodfellow, Bengio and Courville
Depth	Number of layers in the chain, input row not counted	5-4-3-2 is a 3-layer network, 2 hidden

Synonym families and where each count lives

Fully connected layer and MLP

\mathbf{Y} = \mathbf{X}\,\mathbf{W}^{T} + \mathbf{b}, \qquad y_i = \sum_j w_{ij}\, x_j + b_i

W is (c_o, c_i), indexed (output, input), matching PyTorch nn.Linear

Shapes and counts

X: (n, c_i)
W: (c_o, c_i)
b: (c_o,)
Y: (n, c_o)
Parameters: c_o × c_i + c_o
Activations per sample: Sum of layer widths, plus the input if budgeting memory

Network	Weights	Biases	Parameters	Activations per sample
Slide 4, 5-4-3-2	20 + 12 + 6 = 38	4 + 3 + 2 = 9	47	9
Slide 5, 5 to 3	15	3	18	3
Slide 7, 5 to 3 to 2	15 + 6 = 21	3 + 2 = 5	26	5 (10 with inputs)
CS231n, 3-4-4-1	12 + 16 + 4 = 32	4 + 4 + 1 = 9	41	9
MNIST 784 to 128	100,352	128	100,480 (392 KiB fp32)	912 (7,296 at n = 8)

Worked parameter and activation counts

Convolution shapes and output size

Each output sees only the k inputs in its receptive field (sparse) and the same filter is reused at every position (weight sharing), so the parameter count never depends on h_i or w_i. Part 02: Convolution layers

Tensor	Fully connected	1D conv	2D conv
X	(n, c_i)	(n, c_i, w_i)	(n, c_i, h_i, w_i)
Y	(n, c_o)	(n, c_o, w_o)	(n, c_o, h_o, w_o)
W	(c_o, c_i)	(c_o, c_i, k_w)	(c_o, c_i, k_h, k_w)
b	(c_o,)	(c_o,)	(c_o,)

Tensor shapes, PyTorch order, output channels first in W

h_o = \left\lfloor \frac{h_i + 2p - k_h}{s} \right\rfloor + 1, \qquad p = \frac{k - 1}{2} \text{ keeps the size at } s = 1

General output size; p = 0 and s = 1 give h_i - k_h + 1

h_i	k	s	p	Arithmetic	h_o
4	3	1	0	(4 - 3)/1 + 1	2
4	2	2	0	(4 - 2)/2 + 1	2
5	3	1	1	(5 + 2 - 3)/1 + 1	5
32	5	1	0	(32 - 5)/1 + 1	28
32	5	1	2	(32 + 4 - 5)/1 + 1	32
32	5	2	2	floor(31/2) + 1	16
8	3	2	0	floor(5/2) + 1	3, not 3.5
224	7	2	3	floor(223/2) + 1	112

Output height for the configurations in the lecture

Y[o, i, j] = b[o] + \sum_{c=0}^{c_i - 1} \sum_{u=0}^{k_h - 1} \sum_{v=0}^{k_w - 1} W[o, c, u, v]\, X[c,\; s i + u,\; s j + v]

One output value sums every input channel into one number, then adds the bias

Slide 15 worked numbers

Setup: X (1, 2, 4, 4), W (1, 2, 2, 2), stride 2, no padding, no bias
Output size: (4 - 2)/2 + 1 = 2, so 2 × 2
Top left: channel 1 2·1 + 1·2 + 1·1 + 2·2 = 9, channel 2 1·5 + 2·6 + 2·5 + 1·6 = 33, sum 42
Top right: channel 1 21, channel 2 45, sum 66
Result: [[42, 66], [42, 66]]
Stride 1 instead: 3 × 3 output, middle column 15 + 39 = 54, one row 42, 54, 66

Parameter counts, conv versus FC

\#\text{params}_{\text{conv}} = c_o \cdot c_i \cdot k_h \cdot k_w + c_o, \qquad \#\text{params}_{\text{FC}} = c_o \cdot c_i + c_o

Multiply the four numbers in the shape of W, then add c_o biases

Layer	Arithmetic	Parameters	Depends on image size?
Conv2D 16 filters 5 × 5, 3 to 16	16 × 3 × 5 × 5 + 16	1216	No
Same layer, 3 × 3 kernels	16 × 3 × 9 + 16	448	No
Same layer, 32 filters	32 × 75 + 32	2432	No
Conv2D 128 filters 3 × 3, 64 to 128	128 × 64 × 9 + 128	73,856	No
AlexNet conv1, 96 filters 11 × 11, 3 to 96	96 × 3 × 121 + 96	34,944 (105.7 M unshared)	No
FC 32 × 32 × 3 = 3072 to 16	3072 × 16 + 16	49,168	Yes
FC 64 × 64 × 3 = 12,288 to 16	12,288 × 16 + 16	196,624	Yes

Worked counts, all on a 32 × 32 × 3 input unless stated

Padding, stride and receptive field

Padding and stride change h_o × w_o, hence activations and MACs, and never the parameter count. Part 03: Padding, stride and grouped convolution

Mode	Rule	Row 0
Zero (default)	Fill the border with 0	0 0 0 0 0 0 0
Reflection	Mirror across the edge, edge value not repeated, needs p < input size	9 8 7 8 9 8 7
Replication	Repeat the nearest edge value outward	1 1 1 2 3 3 3
Constant	Fill the border with a chosen value v	v v v v v v v

Padding modes on the slide 16 grid (3 × 3 image holding 1 to 9, p = 2, row 0 of the 7 × 7 result)

RF_L = 1 + \sum_{l=1}^{L} (k_l - 1) \prod_{i=1}^{l-1} s_i, \qquad RF_L = L\,(k - 1) + 1 \text{ when every stride is } 1

Each layer adds (k - 1) times the product of the strides before it (the jump)

L	All stride 1	First layer stride 2	Every layer stride 2
1	3	3	3
2	5	7	7
3	7	11	15
4	9	15	31

Receptive field after L layers of 3 × 3

Grouped and depthwise convolution

\mathbf{W}: \left(c_o,\; \tfrac{c_i}{g},\; k_h,\; k_w\right), \qquad \#\text{weights} = c_o \cdot \tfrac{c_i}{g} \cdot k_h k_w, \qquad \text{MACs} = \#\text{weights} \cdot h_o w_o

g must divide both c_i and c_o. Depthwise is g = c_i = c_o, one k × k filter per channel

Layer	W shape	Weights	Weights + biases	Fewer than standard
Standard, g = 1	(64, 64, 3, 3)	36,864	36,928	1×
Grouped, g = 2	(64, 32, 3, 3)	18,432	18,496	2×
Grouped, g = 4	(64, 16, 3, 3)	9,216	9,280	4×
Grouped, g = 8	(64, 8, 3, 3)	4,608	4,672	8×
Depthwise, g = 64	(64, 1, 3, 3)	576	640	64×
Depthwise + 1 × 1 pointwise	(64, 1, 3, 3) + (64, 64, 1, 1)	576 + 4,096 = 4,672	4,800	7.9×

c_i = c_o = 64, k = 3, biases included in the fourth column

Depthwise separable cost ratio (Howard et al.): 1/N + 1/D_K², so 1/64 + 1/9 ≈ 0.127, about 7.9× cheaper. On a 56 × 56 map: 115.6 M MACs standard against 14.7 M separable. A depthwise layer never mixes channels; the 1 × 1 pointwise layer (c_o · c_i weights) restores mixing. Grouped convolution does not change h_o, w_o or c_o.

Pooling

A fixed summary of a window inside each channel independently: spatial size drops, channel count unchanged, zero parameters. Part 04: Pooling and CNN features

W_2 = \left\lfloor \frac{W_1 - F}{S} \right\rfloor + 1, \qquad H_2 = \left\lfloor \frac{H_1 - F}{S} \right\rfloor + 1, \qquad C_2 = C_1, \qquad \text{parameters} = 0

The convolution formula with p = 0 and the kernel renamed F; PyTorch floors unless ceil_mode

	Max pooling	Average pooling
What it returns	Largest value in the window	Mean of the window
Shift by one cell	Usually unchanged (approximate translation invariance)	Changes slightly
Gradient	Routed to the argmax cell only	1/F² to every cell
Typical place	Inside the trunk after a conv block	Global average pool as the head
Slide 22 slice, F = 2, S = 2	[[6, 8], [3, 4]]	[[3.25, 5.25], [2, 2]]

Max against average

Input, F, S	Arithmetic	Output
224, F = 2, S = 2	(224 - 2)/2 + 1	112, activations 3,211,264 to 802,816
224, F = 3, S = 2	(224 - 3)/2 + 1 = 111.5	111 floored, 112 with ceil_mode
55, F = 3, S = 2 (AlexNet)	(55 - 3)/2 + 1	27
56 × 56 × 128, F = 3, S = 2	(56 - 3)/2 + 1 = 27.5	27 × 27 × 128
4, F = 3, S = 2	(4 - 3)/2 + 1 = 1.5	1 × 1, value [[7]]

Pooling configurations worked

Feature hierarchy and full network counts

Tier	Layers	Responds to	Receptive field
Low	First conv layers	Oriented edges, bars, blobs	3 × 3 to 11 × 11 px
Mid	Middle layers	Eyes, wheels, tusks, chair legs	tens of pixels
High	Deepest layers	Whole faces, cars, elephants, chairs	most of the image

Edges, parts, objects (Lee et al. 2009; Zeiler and Fergus 2013)

Layer	Output	Parameters
input	32 × 32 × 3	0
conv 16 × 5 × 5 × 3, pad 2	32 × 32 × 16	1216
max pool 2, stride 2	16 × 16 × 16	0
conv 20 × 5 × 5 × 16, pad 2	16 × 16 × 20	8020
max pool 2, stride 2	8 × 8 × 20	0
conv 20 × 5 × 5 × 20, pad 2	8 × 8 × 20	10,020
max pool 2, stride 2	4 × 4 × 20	0
softmax 320 to 10	10	3210
Total		22,466

ConvNetJS CIFAR-10 demo (slide 25)

Layer	Output	Parameters
input	64 × 64 × 3	0
conv_1_1, 10 × 3 × 3, valid	62 × 62 × 10	280
conv_1_2, 10 × 3 × 3, valid	60 × 60 × 10	910
max_pool_1, 2, stride 2	30 × 30 × 10	0
conv_2_1	28 × 28 × 10	910
conv_2_2	26 × 26 × 10	910
max_pool_2	13 × 13 × 10	0
flatten 1690, dense 10	10	16,910
Total		19,920

CNN Explainer Tiny VGG (slide 26)

Only the first fully connected layer's count depends on image size (320 inputs in ConvNetJS, 1690 in Tiny VGG). The SRAM high-water mark is the largest activation tensor, in Tiny VGG the 62 × 62 × 10 = 38,440 output of the first conv.

Batch normalization

Inputs that are off-centre force a large bias; inputs with different scales force W entries to span magnitudes. BN standardizes each feature over the batch with a differentiable map inside the network. Part 05: Normalization

\mu_j = \frac{1}{N}\sum_{i} x_{i,j}, \quad \sigma_j^2 = \frac{1}{N}\sum_{i} (x_{i,j} - \mu_j)^2, \quad \hat{x}_{i,j} = \frac{x_{i,j} - \mu_j}{\sqrt{\sigma_j^2 + \varepsilon}}, \quad y_{i,j} = \gamma_j\,\hat{x}_{i,j} + \beta_j

Per feature, over the batch. Biased variance during training

Shapes and defaults for a fully connected input

x, x_hat, y: N × D
mu, sigma squared: D, computed over the batch axis N
gamma, beta: D, the only learned parameters
eps, momentum (PyTorch): 1e-5, 0.1
Identity recovery: gamma = sqrt(sigma² + eps), beta = mu
BatchNorm2d over C channels: 2C parameters, running mean and variance are buffers

W' = \operatorname{diag}\!\left(\frac{\gamma}{\sqrt{\sigma^2 + \varepsilon}}\right) W, \qquad b' = \gamma \odot \frac{b - \mu}{\sqrt{\sigma^2 + \varepsilon}} + \beta

BN fusion at inference: frozen running statistics make BN affine, so it folds into the previous linear or conv layer at zero cost

Normalization axes

\hat{x}_i = \frac{x_i - \mu_i}{\sigma_i}, \qquad \mu_i = \frac{1}{m}\sum_{k \in S_i} x_k

Wu and He: one rule, the set S_i picks BN, LN, IN or GN

Normalization	Averages over	mu, sigma shape	gamma, beta shape	Same at train and test?
BN, fully connected	N	1 × D	1 × D	No
BN, convolution	N, H, W	1 × C × 1 × 1	1 × C × 1 × 1	No
Layer norm	D (or C, H, W)	N × 1	1 × D on slide 35, per channel in Wu and He	Yes
Instance norm	H, W	N × C × 1 × 1	1 × C × 1 × 1	Yes
Group norm, G groups	H, W and C/G channels	N × G × 1 × 1	1 × C × 1 × 1	Yes

The family on a conv activation of shape N × C × H × W

Norm	mu shape	Values per mean
BN	1 × 64 × 1 × 1	8 × 32 × 32 = 8192
LN	8 × 1 × 1 × 1	64 × 32 × 32 = 65,536
IN	8 × 64 × 1 × 1	32 × 32 = 1024
GN, G = 32	8 × 32 × 1 × 1	2 × 1024 = 2048

Worked on 8 × 64 × 32 × 32

Row	Feature 1	Feature 2	Feature 3
After ReLU	3, 1, 3, 2	0, 2, 3, 0	0, 5, 2, 0
Mean (divide by N = 4)	2.25	1.25	1.75
Variance (biased)	0.6875	1.6875	4.1875
Standard deviation	0.829	1.299	2.046
x_hat	0.905, -1.508, 0.905, -0.302	-0.962, 0.577, 1.347, -0.962	-0.855, 1.588, 0.122, -0.855
gamma, beta	2, 0	3, 0	-1, 1
y	1.81, -3.02, 1.81, -0.60	-2.89, 1.73, 4.04, -2.89	1.86, -0.59, 0.88, 1.86
Check: mean beta, variance gamma²	0, 4	0, 9	1, 1

Slide 38 batch norm by hand (W = [[1, 0, 1], [1, 1, 0], [0, 2, -1]], b = (0, -1, 0), then ReLU, then BN)

Activation functions

Without a nonlinearity stacked linear layers collapse into one. On an MCU the two questions are: does it need an exponential, and is the output bounded? Part 06: Activations and transformers

\text{h-swish}(x) = x \cdot \frac{\operatorname{ReLU6}(x + 3)}{6} = \begin{cases} 0 & x \le -3 \\ x & x \ge 3 \\ x(x + 3)/6 & \text{otherwise} \end{cases}

Swish traced with a ruler: one clamp, one add, one multiply

Function	Formula	Range	Derivative	Needs	Note
Sigmoid	1 / (1 + e^-x)	(0, 1)	s(1 - s), at most 0.25	exp	Saturates both sides
ReLU	max(0, x)	[0, inf)	0 or 1	max	Default hidden unit, range must be calibrated for int8
ReLU6	min(max(0, x), 6)	[0, 6]	1 on (0, 6), else 0	max, min	Fixed int8 grid, MobileNetV2
Leaky ReLU	max(alpha x, x)	(-inf, inf)	alpha or 1	max	alpha = 0.01 default
Swish (SiLU)	x / (1 + e^-x)	[-0.278, inf)	s + x s(1 - s)	exp	Beats ReLU on deep models, costly on MCUs
Hard swish	x ReLU6(x + 3) / 6	[-0.375, inf)	0, (2x + 3)/6, or 1	clamp, multiply	Within 0.142 of swish, MobileNetV3
tanh	2 s(2x) - 1	(-1, 1)	1 - tanh²	exp	Zero centred, still saturates
GELU	x Phi(x)	[-0.17, inf)	Phi(x) + x phi(x)	erf or tanh	BERT and GPT default

Formula, range, derivative, cost

Activation	Range	Step size over 255 steps
ReLU6	[0, 6], fixed at design time	6 / 255 = 0.0235
ReLU, calibrated on data	[0, 60] if one channel spikes	60 / 255 = 0.235, ten times coarser

Why bounded is what int8 wants

Transformer block and attention

The encoder layer (Vaswani et al. 2017)

Encoder layer: Multi-head attention, Add and Norm, position-wise FFN, Add and Norm, stacked N times
Add and Norm: LayerNorm(x + Sublayer(x)), LayerNorm not BatchNorm because it needs no batch statistics
Base hyperparameters: N = 6, d_model = 512, h = 8, d_k = d_v = 64, d_ff = 2048
Positional encoding: PE(pos, 2i) = sin(pos / 10000^(2i / d_model)), cosine for odd indices, added before layer 1
Decoder extras: Masked self-attention, then encoder-decoder attention (queries from decoder, keys and values from encoder)
Weights per base layer: attention 4 × 512 × 512 = 1,048,576, FFN 2 × 512 × 2048 = 2,097,152, about 67% in the FFN

\text{Attention}(Q, K, V) = \text{softmax}\!\left(\frac{Q K^{T}}{\sqrt{d_k}}\right) V, \qquad \text{FFN}(x) = \max(0,\; x W_1 + b_1)\, W_2 + b_2

Scaled dot-product attention, and the position-wise FFN applied to every token with the same weights

Shapes for N tokens

Q, K: N × d_k
V: N × d_v
Q Kᵀ and its softmax: N × N
Output: N × d_v
Learned: W_Q, W_K, W_V (and W_O); the N × N weights are recomputed per input

d_k	Unscaled	softmax unscaled	softmax scaled
4	(2, 0, -2)	(0.867, 0.117, 0.016)	(0.665, 0.245, 0.090)
64	(8, 0, -8)	(1.000, 0.000, 0.000)	(0.665, 0.245, 0.090)
512	(22.6, 0, -22.6)	(1.000, 0.000, 0.000)	(0.665, 0.245, 0.090)

Why divide by sqrt(d_k): three scores one standard deviation apart

Tokens N	Entries	Memory
64	4,096	16 KB
256	65,536	256 KB
1,024	1,048,576	4 MB
4,096	16,777,216	64 MB

The N × N bill, one head, one layer, fp32

Attention and FFN by hand

Slide 43: keys k1 = (0, 0, 1), k2 = (2, 1, 0), k3 = (1, 0, -1), k4 = (0, 0, 1); values v1 = (20, 0, 0), v2 = (0, 0, 10), v3 = (0, 10, 0), v4 = (20, 10, 0). Shortcut: halve and truncate instead of dividing by sqrt(3), use 3^x instead of e^x.

Query	Scores	Slide weights	Slide z	Exact z
q1 = (2, 0, 3)	3, 4, -1, 3	.2, .6, 0, .2	(8, 2, 6)	(10.3, 2.8, 4.6)
q2 = (1, 1, 2)	2, 3, -1, 2	.3, .3, .1, .3	(12, 4, 3)	(10.1, 3.0, 4.5)
q3 = (0, 1, 2)	2, 1, -2, 2	.4, .2, 0, .4	(16, 4, 2)	(15.0, 4.1, 2.1)
q4 = (2, 1, 1)	1, 5, 1, 1	.1, .7, .1, .1	(4, 2, 7)	(3.1, 1.5, 7.7)

All four queries, slide arithmetic and exact (q2 corrected)

Slide 44: W1 rows (1, -1, 0), (1, 1, 0), (0, 1, 1), (-1, 1, 1) with b1 = (1, 0, 1, 0); W2 rows (1, 0, 0, -1), (0, 1, 1, 0), (0, 0, 1, -1) with b2 = (0, 0, 1).

z	W1 z + b1	ReLU	W2 h + b2
z1 = (11, 2, 1)	(10, 13, 4, -8)	(10, 13, 4, 0)	(10, 17, 5)
z2 = (6, 6, 1)	(1, 12, 8, 1)	(1, 12, 8, 1)	(0, 20, 8)
z3 = (7, 4, 2)	(4, 11, 7, -1)	(4, 11, 7, 0)	(4, 18, 8)
z4 = (7, 3, 1)	(5, 10, 5, -3)	(5, 10, 5, 0)	(5, 15, 6)
z5 = (5, 3, 1)	(3, 8, 5, -1)	(3, 8, 5, 0)	(3, 13, 6)

Five tokens through the same FFN

Slide errata

Answer with the corrected fact, and name the slide if a question depends on it.

What the slides get wrong

Slide 4: "Hidden Layer 2" labels the output row. Read it as Layer 0 hidden (4), Layer 1 hidden (3), Layer 2 output (2).
Slides 5 to 7: The edge from x4 to y2 is labelled w42; by the slide's own formula it is w24. Index W as (output, input).
Slide 15: "2 x2" means 2 × 2. Stride 2 is used before slide 18 defines it. The bias is silently 0.
Slide 16: Reflection row 1 reads 6 5 4 5 6 4 5; the mirror rule gives 6 5 4 5 6 5 4.
Slide 18: Output size is written without the floor. Use floor((h_i + 2p - k)/s) + 1: 8, 3, 0, 2 gives 3, not 3.5.
Slide 20: Depthwise W is written (c, k_h, k_w); frameworks store (c, 1, k_h, k_w). Keep four axes.
Slide 21: "Max pooling performs a lot better" is an empirical heuristic (Boureau et al. 2010); modern trunks use strided convs and global average pooling. Stray capital "It".
Slide 24: "fully connecter layer" should be "fully connected". The FC bars are class scores, probabilities only after a softmax.
Slide 25: The ConvNetJS URL is printed with http://; the live page is https://.
Slide 31: Identity recovery needs gamma = sqrt(sigma² + eps), not gamma = sigma, unless eps = 0.
Slide 38: BN is placed after the ReLU; the paper, CS231n and Goodfellow put it before the nonlinearity. Only the paper order allows fusion. Values are heavily rounded; recompute with biased variance.
Slide 43: q2 is drawn as (1, 1, 1) but equals (1, 1, 2), so column 2 scores are 2, 3, -1, 2 and z2 = (12, 4, 3), not (8, 4, 4). The drawing uses column vectors, K^T Q and Z = V A.
Slide 44: Hidden row 4 shows -9 and -4; the arithmetic gives -8 and -3. ReLU zeroes both, so the outputs stand. The attention matrix columns sum to 2, not a softmax.

Reference deck: MIT 6.5940 lecture 2 (Song Han), which slides 2 to 20 reproduce, including the Layer 0 and w42 labels.
By-hand worksheets (slides 38, 43, 44) are Tom Yeh's and round heavily; recompute before quoting.