Activation functions and the transformer

Start with the Neuron from the beginning of the lecture: three inputs, three weights, a Bias, and then y = f(w0 x0 + w1 x1 + w2 x2 + b). Now ask what happens if f is the identity and you stack two such layers. The second layer computes W2 (W1 x + b1) + b2, which multiplies out to (W2 W1) x + (W2 b1 + b2): one matrix, one bias, one linear layer. Ten layers would collapse the same way. Depth buys nothing.

That is why the slide says activation functions are typically non-linear. The Activation function is the bend that stops the collapse, and it sits after every weighted sum in every Hidden layer. Slide 39 places it on the CNN component map beside convolution, pooling, fully connected layers and the normalization block of the previous part: the only component with no parameters and no shape arithmetic, just a pointwise bend applied to every value. Which bend to use is a real design decision, and on embedded hardware it is decided by two questions the slide leaves implicit: does the function need an exponential, and is its output bounded?

The six on the slide, and what each costs a microcontroller

Sigmoid, 1 / (1 + e^-x), squashes any input into (0, 1). Its derivative is s(1 - s), at most 0.25 at zero and already 0.0066 at x = 5. A sigmoid unit therefore saturates across most of its domain, which is why Goodfellow and colleagues say its use as a hidden unit is now discouraged. It also needs an exponential per element. ReLU, max(0, x), has derivative 0 or 1, needs a single comparison, and is the same textbook's recommended default; AlexNet is the network that made it standard.

ReLU6, min(max(0, x), 6), is ReLU with a ceiling. Krizhevsky introduced the cap in 2010 to encourage sparse features; MobileNetV2 kept it for a different reason, robustness when used with low-precision computation. Leaky ReLU, max(alpha x, x), replaces the flat negative side with a small slope so a unit that is off still passes a gradient; Maas, Hannun and Ng used alpha = 0.01, which is still PyTorch's default. Swish, x / (1 + e^-x), is x times its own sigmoid (the general form is x sigmoid(beta x), PyTorch calls it SiLU). It dips to -0.278 at x = -1.28 before rising, and Ramachandran, Zoph and Le found it beats ReLU on deeper models. It costs an exponential, which is the whole problem on a Cortex-M.

Hard swish is MobileNetV3's answer: x ReLU6(x + 3) / 6. Expand it and you get the slide's three cases, 0 for x <= -3, x for x >= 3, and x(x + 3) / 6 between. Howard and colleagues give three reasons: optimized ReLU6 kernels exist on virtually every framework and chip, the piecewise form removes the precision loss that different approximate sigmoids introduce in quantized mode, and it can run with fewer memory accesses. They saw no discernible accuracy difference against real swish, and an optimized h-swish added about 1 ms over ReLU on a Pixel 1.

The largest gap on [-6, 6] is 0.142, at the two knots x = -3 and x = 3. The derivative also lines up: swish has slope 0.5 at zero and hard swish has (2x + 3) / 6 = 0.5 there too.

Why bounded is what int8 wants

Quantizing an Activation tensor to 8 bits means choosing a range and dividing it into 255 steps. With ReLU6 the range is known before you see a single input: [0, 6], so each step is 6 / 255 = 0.0235 and every layer shares the same grid. With plain ReLU the range depends on the data. If one activation channel happens to reach 60, the step becomes 60 / 255 = 0.235, ten times coarser, for every value in that tensor, including the many small ones. MobileNetV2's sentence about low-precision robustness is the citation; the arithmetic is the reason.

Feed a four-word sentence into the model of Vaswani and colleagues. Each word becomes an embedding of d_model = 512 numbers, a positional encoding of the same size is added to it, and the four vectors enter the first encoder layer. That layer does exactly two things: a Multi-head attention step that lets each word read the others, then a position-wise feed-forward network that processes each word on its own. Around each of the two sits an Add and Norm. Repeat the layer N = 6 times and you have the encoder of the original Transformer.

The Add and Norm box is the link back to the normalization part. Each sub-layer's output is added to its own input (a residual connection, the ResNet idea) and the sum goes through Layer normalization: LayerNorm(x + Sublayer(x)). LayerNorm rather than BatchNorm because it normalizes each token's 512 features on their own, so it behaves identically at training and inference and needs no batch statistics, which is what you want for variable-length sequences. Jurafsky and Martin describe the whole stack as a residual stream that each component reads from and adds back to.

Inside the two sub-layers

Multi-head attention runs h = 8 copies of Scaled dot-product attention in parallel. Each head first projects the queries, keys and values with its own small linear layers down to d_k = 64, computes attention, and the eight 64-wide results are concatenated back to 512 and passed through one more linear layer. Because each head works at one eighth of the width, the total cost is about that of a single head at full width, and each head is free to attend to a different relationship.

The feed-forward sub-layer is a two-layer MLP with a ReLU between, widening from 512 to 2048 and back. It is applied to each position separately and identically, which the paper notes is the same as two convolutions with kernel size 1. This is where the block's non-linearity lives: once the attention weights are fixed, attention itself is a linear mix of the values.

Position, the decoder, and the output

Attention has no notion of order: swap two tokens and the same weights come out swapped. So before the first layer, a positional encoding is added to each embedding. The paper uses sinusoids of different wavelengths, PE(pos, 2i) = sin(pos / 10000^(2i / d_model)) and cosine for odd indices, so that every position gets a distinct 512-vector and relative offsets are easy to express.

The decoder stack (right side of the slide) repeats the same layer with two changes. Its self-attention is masked so that position i cannot see positions after i, which together with the outputs being shifted right by one keeps training honest about what is known when. And a third sub-layer, encoder-decoder attention, takes queries from the decoder and keys and values from the encoder output. A final linear layer and softmax turn the last vector into next-token probabilities.

Type a phrase into YouTube's search bar. The phrase is your query. Every video carries a title and description, which is its key. The video itself is the value. Search scores your query against every key, ranks the matches, and hands you back videos. Attention does the same three things with vectors, with one twist: instead of returning the single best video it returns a blend of all the values, weighted by how well each key matched. That is the Query, key, value design of the slide.

In self-attention all three come from the same input matrix X of N tokens, each through its own learned projection: Q = X W_Q, K = X W_K, V = X W_V. Jurafsky and Martin give the roles precisely: the query is the current element being compared, a key is a preceding input being compared to it to produce a similarity weight, and a value is what gets weighted and summed. The comparison is a dot product, so the whole score table is one matrix multiply, and the paper names the result Scaled dot-product attention.

Read the slide's diagram bottom to top with those shapes: Q and K of shape N x d multiply into an N x N table of scores, the optional mask sets forbidden entries to minus infinity, softmax turns each row into a probability distribution, and multiplying by V (N x d) gives one output row per token. The masked version is what the decoder uses so that a token cannot attend to what comes after it.

Why divide by the square root of d_k

If the components of a query and a key have mean zero and variance one, their dot product has variance d_k: it is a sum of d_k products. So raw scores grow with the head size, and Vaswani and colleagues observe that large dot products push the softmax into regions where it has extremely small gradients. Goodfellow and colleagues say the same about softmax in general: its outputs saturate when the differences between inputs become extreme. Dividing by sqrt(d_k) brings the variance back to one, whatever the head size.

Without the division, a head of size 64 already turns a modest spread into a one-hot answer of (1.000, 0.000, 0.000), and the gradient through those zeros is essentially zero. With it, every head size sees the same healthy distribution. Softmax is also shift-invariant, softmax(z) = softmax(z + c), so only differences between scores matter, which is what implementations exploit for numerical stability.

The N x N matrix is the embedded constraint

Every query is scored against every key, so one head at one layer produces N^2 scores, whatever the feature size. Jurafsky and Martin state it plainly: attention is quadratic in the length of the input. Vaswani's complexity table writes the per-layer cost as O(n^2 d). Double the sequence and you quadruple the score matrix; multiply by heads and layers and it is the dominant activation memory of the model.

Tom Yeh's drawing makes Scaled dot-product attention concrete with numbers you can check at a desk. Four tokens, each with six features, are the columns x1 = (2, 0, 0, 0, 2, 1), x2 = (0, 1, 2, 0, 0, 0), x3 = (0, 0, 1, 1, 0, 1) and x4 = (2, 0, 0, 1, 0, 1). Three 3 x 6 matrices W_Q, W_K and W_V project them to the queries, keys and values of head size d_k = 3. Those 54 numbers are the only learned parameters; everything else is computed from the input.

Two shortcuts keep the arithmetic in your head. Dividing by sqrt(3) = 1.732 is replaced by dividing by 2 and dropping the fraction (rounding toward zero, so -1 / 2 becomes 0), and e^x is replaced by 3^x, which has the same shape and integer values for non-negative exponents. The shortcuts preserve the ranking for three of the four queries; the comparison table below shows the one query where halving squashes the gap away.

For q1, q3 and q4 the same key wins under both arithmetics, and the second-best key is the same too: softmax cares about differences between scores, and a monotone squashing of those differences preserves the order as long as the differences survive. The numbers drift (a weight of 0.6 becomes 0.46), the decision does not. The corrected q2 is the exception that shows how coarse the shortcut is: its scores 2, 3, -1, 2 halve to 1, 1, 0, 1, so the by-hand softmax hands keys 1, 2 and 4 the same 0.3 while the exact softmax still separates key 2 at 0.45. Integer halving erased a difference of one, which is exactly the kind of information a real quantizer must be careful not to lose.

The second by-hand slide picks up where the first stopped. Five tokens with three features arrive from the previous block, x1 = (5, 0, 1), x2 = (6, 2, 0), x3 = (0, 4, 1), x4 = (7, 0, 1), x5 = (0, 3, 0). The attention weight matrix A is given rather than computed, and it is deliberately simple: each column has exactly two ones, so each token attends equally to itself and to one other token (the next neighbour for tokens 1 to 4, and token 1 for token 5). The attention-weighted feature of token 1 is then a plain sum, z1 = x1 + x2 = (11, 2, 1), and likewise z2 = x2 + x3, z3 = x3 + x4, z4 = x4 + x5 and z5 = x1 + x5.

Now the position-wise feed-forward network takes each z_j on its own. The first Fully connected layer has weights W1 of shape 4 x 3 with rows (1, -1, 0), (1, 1, 0), (0, 1, 1), (-1, 1, 1) and biases (1, 0, 1, 0). A ReLU follows. The second layer has W2 of shape 3 x 4 with rows (1, 0, 0, -1), (0, 1, 1, 0), (0, 0, 1, -1) and biases (0, 0, 1). Widen from three to four, bend, narrow back to three: the same expand-and-contract shape as the paper's 512 to 2048 to 512.

Where the parameters sit

The FFN is small on the slide and large in the real model. Derive it from the base hyperparameters: the four attention projections W_Q, W_K, W_V, W_O are each 512 x 512, about 1.05 M weights per layer. The FFN's two matrices are 512 x 2048 and 2048 x 512, about 2.1 M. Two thirds of a layer's weights live in the FFN, so pruning and quantization work in later lectures will spend most of its effort there, while the N^2 activation cost of the previous concept lives in attention.

The closing slide lists the ten works the lecturer built this lecture from. Read them as a map of where each part came from: the convolution arithmetic animations and CS231n behind parts 2 to 4, the two normalization papers behind part 5, the four architecture papers whose design choices (ReLU in AlexNet, depth in VGG, residuals in ResNet, ReLU6 in MobileNetV2) run through this part, and the survey and course that frame the whole thing for embedded deployment.