Glossary

The output value of a neuron, also called a feature. Activations are computed per input and determine runtime memory.

• Slide 4

A typically non-linear function f applied to a neuron's weighted sum, such as sigmoid, ReLU, ReLU6, leaky ReLU, swish or hard swish.

• Slide 3
• Slide 27
• Slide 39
• Slide 40

Pooling that returns the mean of the values in each window.

• Slide 21
• Slide 22

Folding an inference-time batch normalization into the preceding fully connected or convolution layer's weights and bias, giving zero test-time overhead.

• Slide 32
• Slide 33

Normalizing each feature to zero mean and unit variance using mini-batch statistics, then applying learnable scale gamma and shift beta: y = gamma (x - mu)/sqrt(sigma^2 + epsilon) + beta.

• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Slide 33
• Slide 38

The number of samples n processed together; it adds a leading dimension to input and output tensors but does not change parameter shapes.

• Slide 6
• Slide 8
• Slide 9

A learnable additive term, one per output neuron or output channel, with shape (c_o,).

• Slide 3
• Slide 5
• Slide 8

One feature dimension of a tensor; c_i and c_o denote the numbers of input and output channels of a layer.

• Slide 5
• Slide 8
• Slide 9

A layer in which each output neuron is connected only to input neurons in its receptive field, using filters of shape (c_i, k_h, k_w) shared across spatial positions; W has shape (c_o, c_i, k_h, k_w).

• Slide 8
• Slide 9
• Slide 10
• Slide 11
• Slide 12
• Slide 13
• Slide 14

The grouped convolution with g = c_i = c_o, where each channel has its own independent k_h x k_w filter and no information mixes across channels.

• Slide 20

A depthwise k x k convolution (one filter per channel, c k^2 weights) followed by a 1 x 1 pointwise convolution (c_o c_i weights) that mixes the channels again; the answer to slide 20's question, costing about 1/c_o + 1/k^2 of a standard layer and forming the MobileNet building block.

• Slide 20

Reducing the spatial size of feature maps inside a network, via stride or pooling, so outputs can see the whole image with fewer layers.

• Slide 17
• Slide 21

A node of the network seen as the quantity it represents; the third name in the family neurons = features = activations, where feature stresses what the node encodes and activation stresses that it is a computed value held in memory.

• Slide 4

The progression of what CNN filters learn: low-level features such as edges in early layers, object parts and shapes in middle layers, whole objects in deeper layers.

• Slide 23
• Slide 24

The spatial output of a convolution or pooling layer for one channel; without padding it becomes smaller, h_o = h_i - k_h + 1.

• Slide 14
• Slide 16
• Slide 24

A layer, also called linear layer, in which every output neuron is connected to all input neurons: y_i = sum_j w_ij x_j + b_i, with W of shape (c_o, c_i).

• Slide 5
• Slide 6
• Slide 7
• Slide 27

Normalization over H, W and a group of channels within each sample, between layer normalization and instance normalization.

• Slide 37

A convolution split into g narrower convolutions over disjoint channel groups, with weights of shape (c_o, c_i/g, k_h, k_w), reducing parameters by a factor g.

• Slide 19

A piecewise approximation of swish: 0 for x <= -3, x for x >= 3, and x(x + 3)/6 otherwise.

• Slide 40

A layer between the input and output of a network. The dimensionality of hidden layers determines the width of the model.

• Slide 4

Normalization over H and W for each sample and channel separately (mean of shape N x C x 1 x 1), with the same behavior at training and test time.

• Slide 36

The product of the strides of all layers before a given layer, so the distance in input pixels between two neighbouring outputs of the layer before it. Each layer adds (k - 1) times its jump to the receptive field, which is why a stride-2 first layer followed by one 3 x 3 layer already reaches 7.

• Slide 17
• Slide 18

The small weight tensor slid over the input in convolution, with height k_h, width k_w and depth equal to the number of input channels. Each filter produces one output channel.

• Slide 9
• Slide 10
• Slide 11
• Slide 15

Normalization over the feature dimension of each sample (mean of shape N x 1), with the same behavior at training and test time; usable in recurrent networks and transformers.

• Slide 35
• Slide 41

y = max(alpha x, x), which keeps a small slope alpha for negative inputs.

• Slide 40

Pooling that returns the maximum value in each window, introducing some spatial invariance.

• Slide 21
• Slide 22

The number of layers in the chain of a network, counted without the input row; the 5-4-3-2 network on slide 4 is a 3-layer network of depth 3 with 2 hidden layers.

• Slide 4

The dimensionality (number of neurons or channels) of a network's hidden layers.

• Slide 4

Running h scaled dot-product attentions in parallel on separately linearly projected Q, K and V, then concatenating the results and applying a linear layer.

• Slide 41

A network formed by stacking fully connected layers with activations between them.

• Slide 7

A unit that computes a weighted sum of its inputs plus a bias and passes it through an activation function, y = f(sum_i w_i x_i + b). Also called a feature or activation.

• Slide 3
• Slide 4

Adding p border cells around the input so the output size can match the input size: h_o = h_i + 2p - k_h + 1.

• Slide 16

Any learnable value of a model (weights and biases). The parameter count determines model storage size.

• Slide 4
• Slide 22
• Slide 25

A parameter-free layer that downsamples each activation map independently over windows of extent F with stride S, keeping the channel count: W2 = (W1 - F)/S + 1.

• Slide 21
• Slide 22

A small MLP (linear, ReLU, linear) applied with the same weights to each token's attention-weighted features independently.

• Slide 41
• Slide 44

A vector of the model width added to each input embedding before the first block so that attention, which is otherwise order-blind, can tell positions apart; the original transformer uses sinusoids of different wavelengths.

• Slide 41

The three projections of the input in attention, analogous to retrieval: the query is the search, keys are what is matched against, values are what is returned.

• Slide 42
• Slide 43

The region of the input that one output element depends on. For one convolution it is k_h x k_w; with L stride-1 layers of size k it is L(k - 1) + 1, and with strides each layer adds (k - 1) times the product of the strides before it.

• Slide 8
• Slide 17
• Slide 18

Padding that mirrors interior values across the border without repeating the edge value.

• Slide 16

The rectified linear unit y = max(0, x).

• Slide 24
• Slide 40

A ReLU clipped at 6, y = min(max(0, x), 6).

• Slide 40

Padding that repeats the nearest edge value outward.

• Slide 16

Adding a sub-layer's input to its output, x + Sublayer(x), then applying layer normalization; each attention and feed-forward sub-layer of the transformer block is wrapped this way.

• Slide 41

The mean and variance accumulated during training that batch normalization uses at inference, turning it into a fixed linear operator.

• Slide 32

Learnable per-feature parameters applied after normalization that let the model undo it, recovering the identity when gamma equals the standard deviation and beta the mean.

• Slide 31
• Slide 33

Attention(Q, K, V) = softmax(Q K^T / sqrt(d_k)) V: query-key inner products scaled by sqrt(d_k), turned into N x N weights by softmax, then used to mix the values.

• Slide 41
• Slide 42
• Slide 43

Attention in which the queries, keys and values are all projections of the same input sequence, Q = X W_Q, K = X W_K, V = X W_V, so every token reads every other token; the decoder masks it so a position cannot see later positions.

• Slide 41
• Slide 42
• Slide 43

y = 1/(1 + e^-x), which squashes any input into (0, 1); its derivative s(1 - s) is at most 0.25, so it saturates on both sides and needs an exponential per element.

• Slide 40

softmax(z)_i = e^(z_i) / sum_j e^(z_j), which turns a vector of scores into positive weights that sum to one; it converts the scaled attention scores into the N x N weight matrix and the final linear layer's outputs into probabilities.

• Slide 41
• Slide 42
• Slide 43

A representation's insensitivity to small shifts of the input. Max pooling introduces an approximate version, because when the input moves by a pixel the maximum usually stays inside its window and the pooled value does not change.

• Slide 21

Batch normalization for convolutional networks that averages over N, H and W, giving mean, variance, gamma and beta of shape 1 x C x 1 x 1.

• Slide 34

The step s by which the kernel moves; the output size becomes h_o = floor((h_i + 2p - k_h)/s) + 1 (slide 18 omits the floor), and larger strides grow the receptive field faster.

• Slide 15
• Slide 18
• Slide 22

y = x/(1 + e^-x), the input times its own sigmoid; smooth and non-monotonic, dipping to about -0.28 before rising, but it needs an exponential per element.

• Slide 40

A connection between two neurons, carrying a learnable weight that scales the signal. Used synonymously with weight and parameter.

• Slide 3
• Slide 4

An architecture built from stacked blocks of multi-head attention and position-wise feed-forward networks with residual Add and Norm, plus positional encoding of the input embeddings.

• Slide 41
• Slide 44

A learnable value that multiplies an input on a connection; the weights of a layer form a tensor such as W of shape (c_o, c_i) for a fully connected layer.

• Slide 4
• Slide 5
• Slide 6

Reusing the same filter weights at every spatial position of the input, so a convolution's parameter count does not depend on the input size.

• Slide 8
• Slide 12

Padding that fills the input boundaries with zeros; the default in PyTorch.

• Slide 16