Neural network fundamentals

Take a concrete board. The STM32F746, an ARM Cortex-M7 microcontroller used throughout the TinyML literature, has 320 kB of SRAM and 1 MB of flash (Lin et al., MCUNet, section 1). A ResNet-50 carries about 25.6 million parameters (torchvision), which is roughly 100 MB of weights in single precision. It does not fit, and no clever compiler will change that. Whether any network fits comes down to two numbers, and the whole point of this lecture is to teach you how to produce them from a diagram.

The two memories play different roles, and MCUNet states the split in one sentence: SRAM constrains the activation size, which is read and written at runtime, while flash constrains the model size, which is read only (Lin et al., MCUNet). So a model has a storage cost, the number of parameters it carries, and a runtime cost, the number of activations it must hold in memory while it computes. This part defines both words precisely and shows how to count them for the simplest layer. The parts that follow do the same for convolutions, pooling, normalization and attention.

The lecture's outline is a two-step plan. First fix the terminology: neuron, synapse, activation, feature, weight, parameter. Then learn the building blocks one at a time: fully connected, convolution, grouped and depthwise convolution, pooling, normalization, and the transformer block. This part covers the terminology and the first block. Everything later is a variation on the counting you learn here.

Start with a single unit and three numbers arriving at it: x0 = 1, x1 = 2 and x2 = 0.5. The unit does not treat them equally. It trusts the first a little, distrusts the second, and trusts the third a lot: w0 = 0.5, w1 = -1, w2 = 2. It also has a personal lean of b = 0.25. Multiply each input by its trust, add everything up, and the total is 0.5 - 2 + 1 + 0.25 = -0.25. One last step decides whether the unit speaks: a function f applied to that total.

That is the whole artificial Neuron. Jurafsky and Martin put it in one sentence: a neural unit takes a weighted sum of its inputs with one additional term called the bias, then applies a function to the result. In symbols, with i ranging over the inputs of neuron j:

The slide draws this formula on top of a biological neuron, and the mapping is worth learning because the course reuses the biological words as technical terms. Sze, Chen, Yang and Emer, whose survey carries the same figure (adapted from Stanford CS231n), describe the key property of a Synapse: it scales the signal crossing it, and that scaling factor is the Weight. Each raw input x_i arrives along an input axon, is scaled to w_i x_i at the synapse, and travels down a dendrite into the cell body, which sums the products; the output axon carries the result out. They add that the input and output signals of a neuron are what the field calls activations.

Two details of the formula matter for counting parameters later. There is one weight per incoming edge, so a neuron with three inputs owns three weights. There is exactly one Bias per neuron, not one per edge, which is why the figure shows three w symbols and a single b. The activation function f is applied once, to the finished sum. Later in this lecture you will meet several choices for f (sigmoid, ReLU, ReLU6, swish); for now it is enough to know that it is usually non-linear and that it is what lets stacked neurons compute anything more interesting than a single straight-line function.

Now wire many neurons together. The network on the slide has a row of 5 inputs, then rows of 4, 3 and 2 neurons, with every neuron connected to every neuron in the row above. Count the edges band by band: 5 × 4 = 20, 4 × 3 = 12, 3 × 2 = 6, so 38 edges. Count the biases, one per computed neuron: 4 + 3 + 2 = 9. That is 47 numbers the network stores and 9 numbers it computes per input. Everything else in this concept is the vocabulary for those two counts.

Two synonym families

The slide attaches two labels to the diagram, and each is a family of three words the course uses interchangeably. The edges are synapses, weights or parameters. The nodes are neurons, features or activations. Sze et al. state the convention directly: the outputs of the neurons are often referred to as activations, and the synapses are often referred to as weights. Which word a paper picks depends on what it is emphasising. "Synapse" stresses the biology, "weight" the arithmetic, "parameter" the fact that it is learned and stored. "Feature" stresses what a node represents, "activation" the fact that it is a computed value that must be held in memory.

CS231n gives a second example you can use to check your method: a network with layers of sizes 3, 4, 4 and 1 has 4 + 4 + 1 = 9 neurons, 3 × 4 + 4 × 4 + 4 × 1 = 32 weights and 9 biases, for 41 learnable parameters. The recipe is always the same: multiply adjacent layer sizes for weights, add each non-input layer size for biases.

Counting layers, and why the input does not count

The slide calls this a 3-layer network with 2 hidden layers, and both numbers follow one convention: the input row is not a layer. CS231n states it plainly: when we say N-layer neural network, we do not count the input layer, so a single-layer network has no hidden layers. Jurafsky and Martin say the same in a figure caption. The reason is that the inputs compute nothing. They own no weights and no biases; they are simply the numbers handed in. The three computed rows are the layers, the two blue rows in the middle are the hidden layers, and the gray bottom row is the output layer, counted but not hidden.

Width and depth

The sentence at the bottom of the slide is quoted from Goodfellow, Bengio and Courville: the dimensionality of the hidden layers determines the width of the model. On the same pages they define the other axis: a network is a chain of functions, and the overall length of the chain gives its depth. So the slide network has width 4 at its widest hidden layer and depth 3. Making a network wider adds neurons to a row, which grows both parameters and activations. Making it deeper adds rows, which also grows both, but lets features be composed out of earlier features.

Take one band of the ladder and give it names: c_i = 5 inputs x0 to x4 and c_o = 3 outputs y0 to y2, every output connected to every input. Write out the first output by hand: y0 = w00 x0 + w01 x1 + w02 x2 + w03 x3 + w04 x4 + b0. Do the same for y1 and y2. That is 15 products and 3 biases, 18 parameters. Now stack the three rows of weights on top of each other and you have a 3 × 5 table. That table is W, and the three equations collapse into one matrix product.

This is the fully connected layer, also called a linear layer. Jurafsky and Martin define fully connected exactly this way: each unit in a layer takes as input the outputs of all the units in the previous layer, with a link between every pair of units in adjacent layers. The word Channel is the lecture's general name for a feature dimension, so c_i and c_o count input and output channels; for a fully connected layer a channel is simply one neuron.

The transpose in Y = X Wᵀ is not decoration. W is stored with one row per output neuron, shape (c_o, c_i), so that row i holds the weights of y_i. To multiply a row vector x of length c_i by it, the matrix has to be flipped to (c_i, c_o). PyTorch uses precisely this convention: nn.Linear computes y = x Aᵀ + b with a weight of shape (out_features, in_features) and a bias of shape (out_features). Jurafsky and Martin write the same matrix as W ∈ R^(n1 × n0), with element W_ji the weight from input i to hidden unit j. Output index first, input index second, in every source.

Batching: more rows, same weights

Nothing in the product cares whether X has one row. Put n samples in as n rows and the same multiplication produces n rows of outputs. That is all the batch size is: a leading dimension on X and Y. PyTorch describes the input as (*, H_in), where the star means any number of leading dimensions, and the output as (*, H_out); the weight stays (H_out, H_in) no matter what the star is. That is the fact to hold on to: W and b never change with n.

What the two counts cost on a microcontroller

Scale the same arithmetic to a realistic first layer, one that takes a flattened 28 × 28 MNIST digit (784 values) to 128 hidden units. The table separates what the layer stores from what it computes, and shows which memory of the STM32F746 each column lands in.

The 401,920 bytes of fp32 weights would not fit the board's 320 kB of SRAM but sit comfortably in its 1 MB of flash, which is exactly why weights live in flash. Storing them as 8-bit integers cuts the same column by four, the first hint of why quantization matters later. Only the activation row moved when the batch grew, and it moved linearly.

Put a second fully connected layer after the first. The slide takes the 5 to 3 layer and feeds its three outputs into a 3 to 2 layer that produces z0 and z1. The new layer has W2 of shape (2, 3), six weights, and b2 with two biases: 8 more parameters, 26 in total. Per sample it computes two more activations, so 5 computed values, 10 if you include the inputs held in memory.

A chain of fully connected layers is a multilayer perceptron, or MLP. Goodfellow, Bengio and Courville write the chain as f(x) = f⁽³⁾(f⁽²⁾(f⁽¹⁾(x))), call f⁽¹⁾ the first layer and f⁽²⁾ the second, and define depth as the length of that chain. Jurafsky and Martin add a caution about the name: modern multilayer networks are called perceptrons for historical reasons only, since their units are not perceptrons in the strict sense. Use the name, but do not read anything into it.

The shapes must click together: the output channels of layer k are the input channels of layer k + 1. Here c_o = 3 of the first layer becomes c_i = 3 of the second, and the inner dimension of the second product matches. The batch dimension n rides through unchanged, just as it did in one layer.

Once you see the MLP as a chain, counting any network becomes mechanical, and this is the skill the rest of the course leans on. For each fully connected layer the parameters are c_o × c_i + c_o; sum over the layers. The activations per sample are the sum of the layer output sizes, plus the input if you are budgeting memory for it. Multiply activations by n for a batch. Never multiply parameters by n.

Take a CIFAR-10 image: 32 x 32 pixels in three colour channels, so 3072 numbers. A single neuron of the Fully connected layer from Part 01 has one weight for each of those 3072 inputs, and CS231n uses exactly this count to show why the fully connected design does not scale to images. A neuron in a Convolution layer does something far more modest. It looks at a 5 x 5 patch of the image across all three channels, so 5 x 5 x 3 = 75 numbers, and ignores everything else. That patch is the neuron's Receptive field, and the slide states the whole idea in one sentence: the output neuron is connected to input neurons in the receptive field.

That sentence hides two separate decisions, and Goodfellow, Bengio and Courville name them in section 9.2 of Deep Learning. The first is sparse interactions. With m inputs and n outputs, a fully connected layer stores m x n weights. If each output is allowed to touch only k inputs, the layer stores k x n. On the CIFAR image that is 75 per output instead of 3072.

The second decision is Weight sharing. Sparse connectivity alone would still give every output position its own private set of 75 weights. Convolution goes further and uses the same 75 numbers at every position. Goodfellow and colleagues put it plainly: rather than learning a separate set of parameters for every location, we learn only one set, so the storage of one filter drops to k parameters. They also give the synonym you will meet in older papers, tied weights. Slide 8 labels its figure Weight Sharing, and slides 12 and 13 show it in motion: the same green, yellow and cyan filters are applied wherever the red window sits on the blue input.

What the saving looks like in a real first layer

CS231n works through AlexNet's first layer to show the size of the effect, and the numbers are worth holding in memory for the exam. The input is 227 x 227 x 3 (the AlexNet paper says 224 x 224; CS231n uses 227 so that (227 - 11)/4 + 1 = 55 comes out exact), the filters are 11 x 11 x 3, and the output is 55 x 55 x 96.

Crossing out the fully connected shapes

The slide introduces the shapes by literally striking through the fully connected ones. In Part 01 the input was X (n, c_i): n samples in the batch (the Batch size), each a vector of c_i features. A 1D convolution adds a spatial axis of width w_i, so X becomes (n, c_i, w_i) and Y becomes (n, c_o, w_o). The weight matrix (c_o, c_i) gains a kernel width k_w and becomes (c_o, c_i, k_w). The bias does not change at all: still one number per output channel, (c_o,). The table already shows the 2D column, which adds a height axis in the same way; the next concept explains each of its axes.

Start with a layer you could type into PyTorch today: nn.Conv2d(3, 16, kernel_size=5), applied to a batch of 8 RGB images of 32 x 32. Four tensors take part, and every axis of each one has a name.

The general rule is the 2D column that the previous concept left open. The input X is (n, c_i, h_i, w_i), the output Y is (n, c_o, h_o, w_o), the weights W are (c_o, c_i, k_h, k_w) and the bias b is (c_o,). The PyTorch documentation for torch.nn.Conv2d confirms this exact ordering: the weight is stored as (out_channels, in_channels / groups, kernel_size[0], kernel_size[1]) and the bias as (out_channels). The groups divisor is 1 by default; it becomes important in Part 03 when grouped and depthwise convolutions arrive.

A filter is a block, not a square

Slides 9 to 11 make one argument in three frames. The first frame draws a single filter as a green block with edges labelled k_h, k_w and c_i, sitting next to the red window on the input. The block is as deep as the input has channels. When it is applied at one position, it multiplies all c_i x k_h x k_w input values under the window by its own c_i x k_h x k_w weights, sums everything into a single number, and drops that number into one output channel. CS231n states the rule as a sentence to memorise: the extent of the connectivity along the depth axis is always equal to the depth of the input volume.

The second frame adds a yellow filter. It has the same depth c_i, sees the same window, and drops its number into a second output channel. The third frame shows the full stack of c_o filters with a bracket labelled c_o, and the output is now c_o channels deep. That bracket is literally the first axis of W. Choosing c_o is a design decision made by whoever builds the network; it is not derived from c_i.

Bias and batch, the two easy axes

The Bias is added once per output channel, after the whole c_i-deep dot product, which is why its shape is (c_o,) exactly as in the fully connected layer of Part 01. The Batch size n appears in X and Y but never in W or b: the same weights process every sample in the batch. If you are ever unsure of a shape, check that n is absent from the parameters and that c_o leads W.

Slide 14 gives the smallest example that shows the shrink. The input is 4 x 4, the Kernel (filter) is 3 x 3, there is no padding and the window moves one cell at a time. Along the top row the window can start at column 0 or column 1. Starting at column 2 would need columns 2, 3 and 4, and there is no column 4. Two starting positions per axis means the output is 2 x 2.

Slides 12 and 13 are two frames of the same motion, taken from the MIT 6.5940 lecture that this section of the deck reproduces. The red window moves one step at a time along a spatial dimension, and at every stop the same green, yellow and cyan filters are applied, which is why slide 12 carries the label Weight Sharing. Nothing about the filters changes as the window moves; this is where Weight sharing becomes visible rather than stated. Each stop adds one short stack of c_o values to the output, one value per filter, so the output has exactly one spatial cell per valid window position. The count of valid positions is the whole story of the output size.

The general formula, one part early

Slide 15 asks you to use a Stride of 2 before slide 18 has defined stride, so you need the general rule now. With padding p on every border and stride s, Dumoulin and Visin's relationship 6 gives the output size, and the PyTorch documentation states the same formula with a dilation term that is 1 throughout this course.

Two rows deserve a second look. The 32, 5, 1, 2 row is the ConvNetJS demo on slide 25, which uses pad 2 precisely so that its 5 x 5 convolution keeps the 32 x 32 size, giving the 32 x 32 x 16 output printed there. The 224, 7, 2, 3 row is the first layer of a ResNet, which halves the spatial size in one step. Both are the same formula.

Slide 15 is the exam question in miniature: two input channels of 4 x 4, one output channel, a 2 x 2 filter, Stride 2, no padding and no Bias. Input channel 1 has every row equal to 1 2 3 4; input channel 2 has every row equal to 5 6 7 8. The filter's first slab is [[2, 1], [1, 2]] and its second slab is [[1, 2], [2, 1]]. In the shapes of the previous concepts, X is (1, 2, 4, 4) and W is (1, 2, 2, 2).

The formula says what the filter-is-a-block idea implied: the sum over c folds all input channels into one number. There is no separate output per input channel. With that in mind the four values of the slide are four applications of the same recipe. Strictly, this is cross-correlation: mathematical convolution flips the kernel first, and this formula does not. Deep learning libraries, PyTorch and this deck all call the unflipped version convolution (Goodfellow et al., section 9.1), and this course follows that convention.

Now do it with your hands. The stepper below holds the slide 15 numbers, highlights the current window on both input channels, lists the products channel by channel and fills the output one cell at a time. Switch the stride to 1 to see the 3 x 3 output, then change a few cells and predict the result before you press Next.

The stride-1 column shows what the middle window adds. At columns 1 to 2, channel 1 gives 2·2 + 1·3 + 1·2 + 2·3 = 15 and channel 2 gives 1·6 + 2·7 + 2·6 + 1·7 = 39, so 54 sits between 42 and 66. Stride 2 skips that window entirely, which is why the output is smaller and the compute drops from 9 windows to 4. On a large input the compute falls to about a quarter of the stride-1 cost, since both axes are halved.

Counting parameters

The shapes of W and b give the parameter count directly, and that count is what decides whether a model fits in a microcontroller's flash. The weights number c_o x c_i x k_h x k_w, the size of W, and there is one bias per output channel.

Slide 25 later in this lecture shows the ConvNetJS CIFAR-10 demo, whose first convolution has 16 filters of 5 x 5 over the 3 colour channels. The demo prints 16 x 5 x 5 x 3 + 16 = 1216 parameters, and the formula reproduces it: 1200 weights and 16 biases. Use that number as a retrieval hook; if your formula does not give 1216 for that layer, you have forgotten the biases or swapped a factor.

Only the filter geometry and the channel counts move the number.

Take the 5 x 5 input and 3 x 3 Kernel (filter) from Part 02. The kernel can only sit where it fits entirely inside the image, so it has three horizontal and three vertical positions and the output Feature map is 3 x 3. Stack a second such layer and you get 1 x 1. A third layer cannot run at all. Every layer eats two rows and two columns, and Goodfellow, Bengio and Courville put it plainly: without padding the representation shrinks by one pixel less than the kernel width at every layer.

Padding is the fix. Add a ring of p extra cells around the input before sliding the kernel. With p = 1 the 5 x 5 image becomes 7 x 7, the kernel now has five positions in each direction, and the output is 5 x 5 again. The same 27 weights per filter (3 x 3 over three input channels) do the work; the only thing that changed is where the kernel is allowed to stand.

Read the formula as a story. The padded input is h_i + 2p rows tall. A window of height k_h can start on any row except the last k_h - 1, which leaves h_i + 2p - k_h + 1 starting rows, one per output row. Set h_o = h_i and the equation solves to p = (k - 1)/2, which is a whole number only for odd kernels. That is why 3 x 3, 5 x 5 and 7 x 7 dominate practice: Zhang et al. note that an odd kernel lets the same amount of padding go on both sides and keeps the output pixel centred on its window. Dumoulin and Visin call this half padding, and the deep learning book calls it same convolution, after the MATLAB option of that name.

What goes in the ring

The formula does not care what the new cells contain, but the values do affect what the network learns near the border. The slide pads a 3 x 3 image holding 1 to 9 by p = 2 on every side, which is a good specimen because 2 is the largest reflection padding a 3 x 3 image can take. Zero padding writes zeros, and PyTorch uses it unless told otherwise. Reflection padding mirrors the image across its edge without repeating the edge pixel, so the row above 1 2 3 reads 4 5 6 and the row above that reads 7 8 9. Replication padding stretches each edge pixel outward, so the corner 1 fills the whole top-left 3 x 3 block. Constant padding is zero padding with a value of your choice.

The index rule behind the visual is short. For a padded coordinate j in 0 to 6, subtract p to get an offset into the original image. Replication clamps that offset into 0 to 2. Reflection folds it: -1 becomes 1, -2 becomes 2, 3 becomes 1 and 4 becomes 0. Fold both the row and the column offset and you have the source pixel. The PyTorch documentation for ReflectionPad2d walks through exactly this example with the values 0 to 8. PyTorch also offers a 'circular' mode that wraps around to the opposite edge; the slide leaves it out.

One output pixel of a 3 x 3 Convolution layer is a weighted sum of a 3 x 3 patch of its input. That patch is its Receptive field. Now feed that layer's output into a second 3 x 3 layer. One pixel of the second layer reads a 3 x 3 block of first-layer outputs, and each of those was itself computed from a 3 x 3 patch. The patches of neighbouring pixels overlap and slide by one cell, so the union of all nine is a 5 x 5 region of the original image. A third layer makes it 7 x 7.

Each stride-1 layer adds k - 1 to the side of the region, because the outermost cells of the new window reach (k - 1)/2 further in each direction. Start from a single pixel and add k - 1 once per layer.

With k = 3 this gives 3, 5, 7, 9, 11 for L = 1 to 5. The deep learning book's figure 9.4 makes the same point and adds the hint that matters next: the effect increases if the network includes strided convolution or pooling.

The problem with large images

A classifier must eventually let one output depend on the whole picture. For a 224 x 224 image and stride-1 3 x 3 layers, solve 2L + 1 >= 224: L = 112 layers before a single output pixel sees everything. Each of those layers works at full resolution, so the activation memory and the multiply-accumulates are enormous. The slide states the problem and the cure in one breath: downsample inside the network.

Stride is the cheapest way to do it. Instead of moving the kernel one cell at a time, move it s cells. The number of window positions drops by about a factor of s in each direction, and so does the output size. Goodfellow, Bengio and Courville describe stride s as mathematically equivalent to a stride-1 convolution followed by keeping every s-th output, only without wasting work on the outputs you would throw away.

The floor is not decoration. The padded input has h_i + 2p - k_h rows on which a window can start after the first, and the kernel visits every s-th of them. If that span is not a multiple of s, the last window would run past the edge and is dropped. Try h_i = 8, k = 3, p = 0, s = 2: the slide's version gives 3.5, which is not a size; the true answer is floor(5/2) + 1 = 3. Dumoulin and Visin state the relationship with the floor, and the PyTorch Conv2d documentation computes the output shape the same way.

How stride changes the receptive field rule

Stride does more than shrink maps. Once a layer has stride 2, two neighbouring outputs of that layer sit two input pixels apart, so every layer after it takes steps of two in input coordinates. Its 3 x 3 window no longer adds 2 pixels to the receptive field; it adds 2 x 2 = 4. The slide 18 top row shows the payoff: a stride-2 first layer followed by one stride-1 layer already reaches 7, where stride 1 needed three layers.

Araujo, Norris and Sim give the general closed form. Call the product of the strides before layer l the jump (some authors say effective stride). Each layer adds (k_l - 1) times its jump.

Real networks live at the strided end of that table. Araujo, Norris and Sim compute receptive fields of 195 for AlexNet, 212 for VGG-16, 483 for ResNet-50 and 1,311 for Inception-v3, all with a total downsampling of 32, and remark that the receptive field usually covers the entire input image. The stride in the ResNet stem you computed in the calculator is the first of those doublings: 224 to 112 in one layer.

Padding and stride change the shape of the computation. The last two concepts change its wiring. Take a layer with c_i = c_o = 64 channels and 3 x 3 kernels. Each of the 64 filters reads all 64 input channels, so the layer holds 64 x 64 x 9 = 36,864 weights. Now split the channels into two halves and run two separate convolutions, each mapping 32 inputs to 32 outputs. Each half holds 32 x 32 x 9 = 9,216 weights, so the pair holds 18,432: the same output shape at half the cost.

That is a Grouped convolution with g = 2, or in the slide's words a group of narrower convolutions. In general, split c_i into g blocks of c_i/g channels and c_o into g blocks of c_o/g. Output block j reads only input block j. The slide writes the new Weight shape as (g · c_o/g, c_i/g, k_h, k_w), and the first factor simplifies to c_o: there are still c_o filters, each just reads c_i/g channels instead of all c_i. The Bias stays one per output channel.

The divisibility rule

The split only works if both channel counts divide evenly. PyTorch says so directly: in_channels and out_channels must both be divisible by groups. With c_i = 64 and c_o = 96, g = 4 is fine (16 inputs and 24 outputs per group) but g = 5 is not, because 64/5 is not an integer. Output channel j belongs to group floor(j / (c_o/g)) and reads only the input channels of that group. The calculator earlier in this part flags any pair that fails the rule.

The idea is older than its name. Krizhevsky, Sutskever and Hinton trained AlexNet on two GTX 580 GPUs with 3 GB each, so they put half of the kernels on each GPU and let the kernels of layer 4 take input only from the kernel maps of layer 3 that lived on the same GPU. That is g = 2, chosen for memory rather than elegance. ResNeXt (Xie et al., 2017) turned the group count into a design axis they named cardinality and found that raising it was more effective than going deeper or wider at the same parameter budget.

That last clause is the cost. Inside a grouped layer, no information crosses a group boundary; the faint grey lines in the visual are exactly the missing connections. Networks that use grouped layers alternate them with something that mixes channels again, most often a 1 x 1 convolution. The next concept pushes g to its limit and meets that need head on.

Push the group count as far as it goes: g = c_i = c_o. Every group is now one Channel in and one channel out, so each channel gets its own k x k filter and nothing else. For 64 channels and 3 x 3 kernels that is 64 x 9 = 576 weights, 64 times fewer than the 36,864 of the standard layer. The slide 20 diagram says it in one picture: eight straight arrows from input strip to output strip, and not one of them crosses.

This is a Depthwise convolution. The slide writes its Weight tensor as (c, k_h, k_w), one k_h x k_w filter per channel. Frameworks keep the four-dimensional layout: PyTorch stores (out_channels, in_channels/groups, kH, kW), which with groups = in_channels is (c, 1, k_h, k_w), the same numbers with a singleton axis. PyTorch also allows out_channels = K · in_channels with groups = in_channels, a depthwise layer with depth multiplier K; the slide's g = c_i = c_o is the case K = 1.

Is this reduction in the number of weights really good?

The slide asks the question in red and leaves it open. The honest answer has two halves. For storage and arithmetic, yes: 576 weights fit in 2.3 KB of fp32 flash, and the multiply-accumulates fall by the same factor of 64. For what the layer can learn, no. A depthwise layer filters each channel in isolation. Sandler et al. describe it as lightweight filtering that applies a single convolutional filter per input channel, and Howard et al. point out that it does not combine input channels, so it cannot build a feature that depends on two channels at once. A standard layer does that at every position; a depthwise layer never does.

The repair is a 1 x 1 convolution, called a pointwise convolution, placed right after it. With k = 1 it looks at one position and mixes all c_i channels into c_o outputs: W of shape (c_o, c_i, 1, 1), c_o · c_i weights, and in Sandler et al.'s words it is responsible for building new features through computing linear combinations of the input channels. The depthwise layer handles space, the pointwise layer handles channels, and the pair is called a depthwise separable convolution. It is the building block of MobileNet and, wrapped in an expand and project pair of 1 x 1 layers, of MobileNetV2.

The last row is the one to remember. Howard et al. derive the cost ratio of a separable block to a standard layer as 1/N + 1/D_K², where N is the output channel count and D_K the kernel size. With N = 64 and D_K = 3 that is 1/64 + 1/9 ≈ 0.127, so the block is about 7.9 times cheaper, matching the table. For wide layers the 1/N term fades and the saving approaches D_K² = 9; Howard et al. report 8 to 9 times less computation for 3 x 3 kernels at a small accuracy cost.

What this means on an embedded device

• Fewer weights means less flash and less weight traffic from memory, which on a microcontroller is often the whole budget.
• Fewer multiply-accumulates means lower latency and lower energy per inference on the same core.
• The depthwise layer does only k² multiply-accumulates per output value, against c_i · k² for a standard layer, yet still reads a k x k window per output, so it moves a lot of activation data for little arithmetic and on many accelerators is bound by memory bandwidth rather than compute. Treat the parameter saving as an upper bound on the speedup, not a promise.
• Accuracy drops a little relative to a standard layer of the same shape, which is the price of removing the cross-channel terms.

A VGG-style block hands the next layer a volume of 224 x 224 x 64, which is 3,211,264 activations. A 2 x 2 pool with stride 2 returns 112 x 112 x 64, which is 802,816. Nothing was learned, nothing was mixed across channels, and every one of the 64 maps was simply shrunk on its own. That is the whole job of a Pooling layer.

The rule behind the example: a pooling layer applies a fixed summary over a small window of each activation map, one map at a time. Two summaries are common. Max pooling keeps the largest value in the window, and Average pooling keeps the mean. Because the summary is computed inside one Channel and never across channels, the depth of the volume is preserved while the spatial size falls. Pooling is therefore the second Downsampling tool of this lecture, next to Stride, and its output is a smaller Feature map per channel.

Why max pooling is the default, and why that is a heuristic

Pooling of any kind brings an approximate translation invariance, and max pooling shows it most sharply, which is why slide 21 attaches "Introduces spatial invariance" to max pooling. Goodfellow, Bengio and Courville describe pooling in all its forms as making the representation "approximately invariant to small translations of the input": if the input shifts by a pixel, the values of most pooled outputs do not change, because the maximum is still inside its window. Their example is face detection, where the network need not know the location of the eyes with pixel-perfect accuracy, only that there is an eye on each side. They also point out that pooling regions spaced k pixels apart give the next layer roughly k times fewer inputs to process, which is the memory saving your microcontroller will feel.

Slide 21 goes further and says max pooling "performs a lot better" because it discards noisy activations. Treat that as the slide's heuristic, not a law. Boureau, Ponce and LeCun analysed the two in 2010 and found that which one wins depends on how sparse the features are and how large the pool is; they note that earlier comparisons had been "purely empirical". CS231n states more cautiously that max pooling "has been shown to work better in practice". And the field has partly moved on: Springenberg et al. showed in 2014 that max pooling "can simply be replaced by a convolutional layer with increased stride without loss in accuracy", and Lin, Chen and Yan's Network in Network replaced the fully connected head with global average pooling, which they found "less prone to overfitting". ResNet and MobileNet families follow that pattern: strided convolutions inside, a single global average pool at the end.

Take the single depth slice from slide 22, a 4 x 4 grid, and pool it with a 2 x 2 window and Stride 2. The window lands in four places and never overlaps itself, so the four coloured quadrants of the slide are the four windows. Each quadrant becomes one output cell.

Read the two results as grids. Max pooling gives [[6, 8], [3, 4]] and Average pooling gives [[3.25, 5.25], [2, 2]]. Notice what each one threw away. Max kept only the argmax cell of each window (the 6, the 8, the 3, the 4) and forgot the other three values entirely. Mean kept a trace of all four but flattened the 8 down to 5.25.

The general rule

Let the input be W1 x H1 x C. A Pooling layer needs exactly two hyperparameters, the spatial extent F of the window and the stride S, and it produces W2 x H2 x C.

The shape formula is the convolution formula from part 03 with p = 0 and the kernel size renamed to F. PyTorch writes the same rule in its general form, H_out = floor((H_in + 2p - d(F - 1) - 1)/S + 1), with the stride defaulting to the kernel size, so MaxPool2d(2) means F = 2, S = 2 (PyTorch docs). The floor matters when the division is not exact, and the second worked example shows why.

The zero comes from what pooling does in the backward pass. CS231n notes that a pooling layer "introduces zero parameters since it computes a fixed function of the input". Implementations keep the index of the maximum, the "switch", so the gradient is routed to exactly one cell per window in max pooling; in average pooling it is spread as 1/F² to each cell. Either way the layer has no Weight and no Bias, so it adds nothing to the Parameter count.

Changing F and S on the same 4 x 4 slice shows how strongly the two hyperparameters shape the result. The values below are max pooling; the simulator lets you check the means.

The grid on slide 23 comes from Lee, Grosse, Ranganath and Ng (ICML 2009), who learned a first layer of edge detectors from natural images once, then trained the second and third layers of a convolutional deep belief network separately on unlabeled Caltech-101 images of faces, cars, elephants and chairs, and visualized what each layer responds to. Their own summary: the first, second and third layers "learn edge detectors, object parts, and objects respectively". Look at the bottom row of the slide. It is the same across all four columns because that first layer was learned once from natural images: small oriented edges like Gabor patches. The middle row diverges into eyes and noses, wheels and bumpers, tusks and ears, chair legs and backs. The top row shows entire faces, cars, elephants and chairs.

This ladder is the Feature hierarchy, and it is not designed by hand. Every Convolution layer runs the same operation with a small Kernel (filter). What changes with depth is the Receptive field: a first-layer unit sees only its k x k patch of pixels, so the most it can detect is an edge. A second-layer unit sees a k x k patch of first-layer responses, which covers a larger region of the image (L(k - 1) + 1 pixels after L stride-1 layers, faster with stride or pooling), so it can combine edges into a curve or an eye. Deep units cover most of the image and can match a whole face. Zeiler and Fergus confirmed in 2013 that the same ladder appears inside a supervised ImageNet CNN, using a deconvolutional visualization of intermediate layers.

The mechanism in action: slide 24

Slide 24 shows a network of the pattern [CONV, RELU, CONV, RELU, POOL] repeated three times and then a fully connected layer, run on a photo of a car. Each column is the activation maps of one layer. Read them left to right and the hierarchy appears live: the first columns look like edge-filtered copies of the car, the middle columns are sparse and blotchy, and the last columns are tiny grids that no longer resemble the photo at all.

The ReLU after each conv keeps only positive responses, which is why the columns after it are darker. Each POOL halves the Feature map and, because the next block's k x k window now covers twice as many original pixels, pushes the receptive field outward. Two pools have already halved the maps twice, so the third block works on maps one quarter the width of the input, and the third pool leaves the fully connected layer maps one eighth the width; that is why the last units can see a whole car. The Fully connected layer at the end reads every unit of the final maps at once and produces one score per class. On the slide, car scores highest, well above truck, then airplane, ship and horse.

Why this matters on an embedded device

The hierarchy explains a pattern you will meet throughout this course. Early layers have few parameters (a 5 x 5 x 3 filter is 75 weights) but large activation maps; late layers have many parameters but tiny maps. So SRAM pressure comes from the front of the network and flash pressure from the back, and pruning or quantizing them calls for different budgets. Transfer learning on the edge uses the hierarchy directly: the edge and part detectors are generic, so you freeze the low tiers and retrain only the top.

Open the ConvNetJS CIFAR-10 demo and read its first conv block. It prints "conv (32x32x16), filter size 5x5x3, stride 1, parameters: 16x5x5x3+16 = 1216". That one line is a complete exam answer, and it is worth rebuilding by hand.

The whole demo network is small enough to count in full. Two things to watch: every conv filter is as deep as the channels feeding it, and the three pooling layers contribute nothing to the total while cutting the activations by four each time.

The total is 22,466 parameters. The softmax layer is the only one whose count depends on the image size, because it reads the flattened 4 x 4 x 20 = 320 activations; every conv layer's count depends only on its filter shape, which is Weight sharing at work.

CNN Explainer as retrieval practice

CNN Explainer (Wang et al., IEEE VIS 2020) runs a network the authors call Tiny VGG on ten classes: lifeboat, ladybug, pizza, bell pepper, school bus, koala, espresso, red panda, orange and sport car. Its training code uses 3 x 3 convolutions with 10 filters and no padding ("valid"), and 2 x 2 max pools with stride 2. The input is 64 x 64 x 3. Before you look at the table, chain the shapes yourself: a valid conv subtracts 2, a pool halves.

The pattern repeats: 19,920 parameters, of which the two pools contribute 0 while each removes three quarters of the activations. The dense layer holds 16,910 of the total because it multiplies the image-dependent 1690 by the ten classes. Click any activation map in the live page and it animates the window that produced it, which is the pooling and convolution arithmetic of this lecture rendered one cell at a time.