Pooling and what CNN filters learn

A VGG-style block hands the next layer a volume of 224 x 224 x 64, which is 3,211,264 activations. A 2 x 2 pool with stride 2 returns 112 x 112 x 64, which is 802,816. Nothing was learned, nothing was mixed across channels, and every one of the 64 maps was simply shrunk on its own. That is the whole job of a Pooling layer.

The rule behind the example: a pooling layer applies a fixed summary over a small window of each activation map, one map at a time. Two summaries are common. Max pooling keeps the largest value in the window, and Average pooling keeps the mean. Because the summary is computed inside one Channel and never across channels, the depth of the volume is preserved while the spatial size falls. Pooling is therefore the second Downsampling tool of this lecture, next to Stride, and its output is a smaller Feature map per channel.

Why max pooling is the default, and why that is a heuristic

Pooling of any kind brings an approximate translation invariance, and max pooling shows it most sharply, which is why slide 21 attaches "Introduces spatial invariance" to max pooling. Goodfellow, Bengio and Courville describe pooling in all its forms as making the representation "approximately invariant to small translations of the input": if the input shifts by a pixel, the values of most pooled outputs do not change, because the maximum is still inside its window. Their example is face detection, where the network need not know the location of the eyes with pixel-perfect accuracy, only that there is an eye on each side. They also point out that pooling regions spaced k pixels apart give the next layer roughly k times fewer inputs to process, which is the memory saving your microcontroller will feel.

Slide 21 goes further and says max pooling "performs a lot better" because it discards noisy activations. Treat that as the slide's heuristic, not a law. Boureau, Ponce and LeCun analysed the two in 2010 and found that which one wins depends on how sparse the features are and how large the pool is; they note that earlier comparisons had been "purely empirical". CS231n states more cautiously that max pooling "has been shown to work better in practice". And the field has partly moved on: Springenberg et al. showed in 2014 that max pooling "can simply be replaced by a convolutional layer with increased stride without loss in accuracy", and Lin, Chen and Yan's Network in Network replaced the fully connected head with global average pooling, which they found "less prone to overfitting". ResNet and MobileNet families follow that pattern: strided convolutions inside, a single global average pool at the end.

Take the single depth slice from slide 22, a 4 x 4 grid, and pool it with a 2 x 2 window and Stride 2. The window lands in four places and never overlaps itself, so the four coloured quadrants of the slide are the four windows. Each quadrant becomes one output cell.

Read the two results as grids. Max pooling gives [[6, 8], [3, 4]] and Average pooling gives [[3.25, 5.25], [2, 2]]. Notice what each one threw away. Max kept only the argmax cell of each window (the 6, the 8, the 3, the 4) and forgot the other three values entirely. Mean kept a trace of all four but flattened the 8 down to 5.25.

The general rule

Let the input be W1 x H1 x C. A Pooling layer needs exactly two hyperparameters, the spatial extent F of the window and the stride S, and it produces W2 x H2 x C.

The shape formula is the convolution formula from part 03 with p = 0 and the kernel size renamed to F. PyTorch writes the same rule in its general form, H_out = floor((H_in + 2p - d(F - 1) - 1)/S + 1), with the stride defaulting to the kernel size, so MaxPool2d(2) means F = 2, S = 2 (PyTorch docs). The floor matters when the division is not exact, and the second worked example shows why.

The zero comes from what pooling does in the backward pass. CS231n notes that a pooling layer "introduces zero parameters since it computes a fixed function of the input". Implementations keep the index of the maximum, the "switch", so the gradient is routed to exactly one cell per window in max pooling; in average pooling it is spread as 1/F² to each cell. Either way the layer has no Weight and no Bias, so it adds nothing to the Parameter count.

Changing F and S on the same 4 x 4 slice shows how strongly the two hyperparameters shape the result. The values below are max pooling; the simulator lets you check the means.

The grid on slide 23 comes from Lee, Grosse, Ranganath and Ng (ICML 2009), who learned a first layer of edge detectors from natural images once, then trained the second and third layers of a convolutional deep belief network separately on unlabeled Caltech-101 images of faces, cars, elephants and chairs, and visualized what each layer responds to. Their own summary: the first, second and third layers "learn edge detectors, object parts, and objects respectively". Look at the bottom row of the slide. It is the same across all four columns because that first layer was learned once from natural images: small oriented edges like Gabor patches. The middle row diverges into eyes and noses, wheels and bumpers, tusks and ears, chair legs and backs. The top row shows entire faces, cars, elephants and chairs.

This ladder is the Feature hierarchy, and it is not designed by hand. Every Convolution layer runs the same operation with a small Kernel (filter). What changes with depth is the Receptive field: a first-layer unit sees only its k x k patch of pixels, so the most it can detect is an edge. A second-layer unit sees a k x k patch of first-layer responses, which covers a larger region of the image (L(k - 1) + 1 pixels after L stride-1 layers, faster with stride or pooling), so it can combine edges into a curve or an eye. Deep units cover most of the image and can match a whole face. Zeiler and Fergus confirmed in 2013 that the same ladder appears inside a supervised ImageNet CNN, using a deconvolutional visualization of intermediate layers.

The mechanism in action: slide 24

Slide 24 shows a network of the pattern [CONV, RELU, CONV, RELU, POOL] repeated three times and then a fully connected layer, run on a photo of a car. Each column is the activation maps of one layer. Read them left to right and the hierarchy appears live: the first columns look like edge-filtered copies of the car, the middle columns are sparse and blotchy, and the last columns are tiny grids that no longer resemble the photo at all.

The ReLU after each conv keeps only positive responses, which is why the columns after it are darker. Each POOL halves the Feature map and, because the next block's k x k window now covers twice as many original pixels, pushes the receptive field outward. Two pools have already halved the maps twice, so the third block works on maps one quarter the width of the input, and the third pool leaves the fully connected layer maps one eighth the width; that is why the last units can see a whole car. The Fully connected layer at the end reads every unit of the final maps at once and produces one score per class. On the slide, car scores highest, well above truck, then airplane, ship and horse.

Why this matters on an embedded device

The hierarchy explains a pattern you will meet throughout this course. Early layers have few parameters (a 5 x 5 x 3 filter is 75 weights) but large activation maps; late layers have many parameters but tiny maps. So SRAM pressure comes from the front of the network and flash pressure from the back, and pruning or quantizing them calls for different budgets. Transfer learning on the edge uses the hierarchy directly: the edge and part detectors are generic, so you freeze the low tiers and retrain only the top.

Open the ConvNetJS CIFAR-10 demo and read its first conv block. It prints "conv (32x32x16), filter size 5x5x3, stride 1, parameters: 16x5x5x3+16 = 1216". That one line is a complete exam answer, and it is worth rebuilding by hand.

The whole demo network is small enough to count in full. Two things to watch: every conv filter is as deep as the channels feeding it, and the three pooling layers contribute nothing to the total while cutting the activations by four each time.

The total is 22,466 parameters. The softmax layer is the only one whose count depends on the image size, because it reads the flattened 4 x 4 x 20 = 320 activations; every conv layer's count depends only on its filter shape, which is Weight sharing at work.

CNN Explainer as retrieval practice

CNN Explainer (Wang et al., IEEE VIS 2020) runs a network the authors call Tiny VGG on ten classes: lifeboat, ladybug, pizza, bell pepper, school bus, koala, espresso, red panda, orange and sport car. Its training code uses 3 x 3 convolutions with 10 filters and no padding ("valid"), and 2 x 2 max pools with stride 2. The input is 64 x 64 x 3. Before you look at the table, chain the shapes yourself: a valid conv subtracts 2, a pool halves.

The pattern repeats: 19,920 parameters, of which the two pools contribute 0 while each removes three quarters of the activations. The dense layer holds 16,910 of the total because it multiplies the image-dependent 1690 by the ten classes. Click any activation map in the live page and it animates the window that produced it, which is the pooling and convolution arithmetic of this lecture rendered one cell at a time.