Neurons, weights and the fully connected layer

Take a concrete board. The STM32F746, an ARM Cortex-M7 microcontroller used throughout the TinyML literature, has 320 kB of SRAM and 1 MB of flash (Lin et al., MCUNet, section 1). A ResNet-50 carries about 25.6 million parameters (torchvision), which is roughly 100 MB of weights in single precision. It does not fit, and no clever compiler will change that. Whether any network fits comes down to two numbers, and the whole point of this lecture is to teach you how to produce them from a diagram.

The two memories play different roles, and MCUNet states the split in one sentence: SRAM constrains the activation size, which is read and written at runtime, while flash constrains the model size, which is read only (Lin et al., MCUNet). So a model has a storage cost, the number of parameters it carries, and a runtime cost, the number of activations it must hold in memory while it computes. This part defines both words precisely and shows how to count them for the simplest layer. The parts that follow do the same for convolutions, pooling, normalization and attention.

The lecture's outline is a two-step plan. First fix the terminology: neuron, synapse, activation, feature, weight, parameter. Then learn the building blocks one at a time: fully connected, convolution, grouped and depthwise convolution, pooling, normalization, and the transformer block. This part covers the terminology and the first block. Everything later is a variation on the counting you learn here.

Start with a single unit and three numbers arriving at it: x0 = 1, x1 = 2 and x2 = 0.5. The unit does not treat them equally. It trusts the first a little, distrusts the second, and trusts the third a lot: w0 = 0.5, w1 = -1, w2 = 2. It also has a personal lean of b = 0.25. Multiply each input by its trust, add everything up, and the total is 0.5 - 2 + 1 + 0.25 = -0.25. One last step decides whether the unit speaks: a function f applied to that total.

That is the whole artificial Neuron. Jurafsky and Martin put it in one sentence: a neural unit takes a weighted sum of its inputs with one additional term called the bias, then applies a function to the result. In symbols, with i ranging over the inputs of neuron j:

The slide draws this formula on top of a biological neuron, and the mapping is worth learning because the course reuses the biological words as technical terms. Sze, Chen, Yang and Emer, whose survey carries the same figure (adapted from Stanford CS231n), describe the key property of a Synapse: it scales the signal crossing it, and that scaling factor is the Weight. Each raw input x_i arrives along an input axon, is scaled to w_i x_i at the synapse, and travels down a dendrite into the cell body, which sums the products; the output axon carries the result out. They add that the input and output signals of a neuron are what the field calls activations.

Two details of the formula matter for counting parameters later. There is one weight per incoming edge, so a neuron with three inputs owns three weights. There is exactly one Bias per neuron, not one per edge, which is why the figure shows three w symbols and a single b. The activation function f is applied once, to the finished sum. Later in this lecture you will meet several choices for f (sigmoid, ReLU, ReLU6, swish); for now it is enough to know that it is usually non-linear and that it is what lets stacked neurons compute anything more interesting than a single straight-line function.

Now wire many neurons together. The network on the slide has a row of 5 inputs, then rows of 4, 3 and 2 neurons, with every neuron connected to every neuron in the row above. Count the edges band by band: 5 × 4 = 20, 4 × 3 = 12, 3 × 2 = 6, so 38 edges. Count the biases, one per computed neuron: 4 + 3 + 2 = 9. That is 47 numbers the network stores and 9 numbers it computes per input. Everything else in this concept is the vocabulary for those two counts.

Two synonym families

The slide attaches two labels to the diagram, and each is a family of three words the course uses interchangeably. The edges are synapses, weights or parameters. The nodes are neurons, features or activations. Sze et al. state the convention directly: the outputs of the neurons are often referred to as activations, and the synapses are often referred to as weights. Which word a paper picks depends on what it is emphasising. "Synapse" stresses the biology, "weight" the arithmetic, "parameter" the fact that it is learned and stored. "Feature" stresses what a node represents, "activation" the fact that it is a computed value that must be held in memory.

CS231n gives a second example you can use to check your method: a network with layers of sizes 3, 4, 4 and 1 has 4 + 4 + 1 = 9 neurons, 3 × 4 + 4 × 4 + 4 × 1 = 32 weights and 9 biases, for 41 learnable parameters. The recipe is always the same: multiply adjacent layer sizes for weights, add each non-input layer size for biases.

Counting layers, and why the input does not count

The slide calls this a 3-layer network with 2 hidden layers, and both numbers follow one convention: the input row is not a layer. CS231n states it plainly: when we say N-layer neural network, we do not count the input layer, so a single-layer network has no hidden layers. Jurafsky and Martin say the same in a figure caption. The reason is that the inputs compute nothing. They own no weights and no biases; they are simply the numbers handed in. The three computed rows are the layers, the two blue rows in the middle are the hidden layers, and the gray bottom row is the output layer, counted but not hidden.

Width and depth

The sentence at the bottom of the slide is quoted from Goodfellow, Bengio and Courville: the dimensionality of the hidden layers determines the width of the model. On the same pages they define the other axis: a network is a chain of functions, and the overall length of the chain gives its depth. So the slide network has width 4 at its widest hidden layer and depth 3. Making a network wider adds neurons to a row, which grows both parameters and activations. Making it deeper adds rows, which also grows both, but lets features be composed out of earlier features.

Take one band of the ladder and give it names: c_i = 5 inputs x0 to x4 and c_o = 3 outputs y0 to y2, every output connected to every input. Write out the first output by hand: y0 = w00 x0 + w01 x1 + w02 x2 + w03 x3 + w04 x4 + b0. Do the same for y1 and y2. That is 15 products and 3 biases, 18 parameters. Now stack the three rows of weights on top of each other and you have a 3 × 5 table. That table is W, and the three equations collapse into one matrix product.

This is the fully connected layer, also called a linear layer. Jurafsky and Martin define fully connected exactly this way: each unit in a layer takes as input the outputs of all the units in the previous layer, with a link between every pair of units in adjacent layers. The word Channel is the lecture's general name for a feature dimension, so c_i and c_o count input and output channels; for a fully connected layer a channel is simply one neuron.

The transpose in Y = X Wᵀ is not decoration. W is stored with one row per output neuron, shape (c_o, c_i), so that row i holds the weights of y_i. To multiply a row vector x of length c_i by it, the matrix has to be flipped to (c_i, c_o). PyTorch uses precisely this convention: nn.Linear computes y = x Aᵀ + b with a weight of shape (out_features, in_features) and a bias of shape (out_features). Jurafsky and Martin write the same matrix as W ∈ R^(n1 × n0), with element W_ji the weight from input i to hidden unit j. Output index first, input index second, in every source.

Batching: more rows, same weights

Nothing in the product cares whether X has one row. Put n samples in as n rows and the same multiplication produces n rows of outputs. That is all the batch size is: a leading dimension on X and Y. PyTorch describes the input as (*, H_in), where the star means any number of leading dimensions, and the output as (*, H_out); the weight stays (H_out, H_in) no matter what the star is. That is the fact to hold on to: W and b never change with n.

What the two counts cost on a microcontroller

Scale the same arithmetic to a realistic first layer, one that takes a flattened 28 × 28 MNIST digit (784 values) to 128 hidden units. The table separates what the layer stores from what it computes, and shows which memory of the STM32F746 each column lands in.

The 401,920 bytes of fp32 weights would not fit the board's 320 kB of SRAM but sit comfortably in its 1 MB of flash, which is exactly why weights live in flash. Storing them as 8-bit integers cuts the same column by four, the first hint of why quantization matters later. Only the activation row moved when the batch grew, and it moved linearly.

Put a second fully connected layer after the first. The slide takes the 5 to 3 layer and feeds its three outputs into a 3 to 2 layer that produces z0 and z1. The new layer has W2 of shape (2, 3), six weights, and b2 with two biases: 8 more parameters, 26 in total. Per sample it computes two more activations, so 5 computed values, 10 if you include the inputs held in memory.

A chain of fully connected layers is a multilayer perceptron, or MLP. Goodfellow, Bengio and Courville write the chain as f(x) = f⁽³⁾(f⁽²⁾(f⁽¹⁾(x))), call f⁽¹⁾ the first layer and f⁽²⁾ the second, and define depth as the length of that chain. Jurafsky and Martin add a caution about the name: modern multilayer networks are called perceptrons for historical reasons only, since their units are not perceptrons in the strict sense. Use the name, but do not read anything into it.

The shapes must click together: the output channels of layer k are the input channels of layer k + 1. Here c_o = 3 of the first layer becomes c_i = 3 of the second, and the inner dimension of the second product matches. The batch dimension n rides through unchanged, just as it did in one layer.

Once you see the MLP as a chain, counting any network becomes mechanical, and this is the skill the rest of the course leans on. For each fully connected layer the parameters are c_o × c_i + c_o; sum over the layers. The activations per sample are the sum of the layer output sizes, plus the input if you are budgeting memory for it. Multiply activations by n for a batch. Never multiply parameters by n.