Convolution layers and their tensor shapes

Take a CIFAR-10 image: 32 x 32 pixels in three colour channels, so 3072 numbers. A single neuron of the Fully connected layer from Part 01 has one weight for each of those 3072 inputs, and CS231n uses exactly this count to show why the fully connected design does not scale to images. A neuron in a Convolution layer does something far more modest. It looks at a 5 x 5 patch of the image across all three channels, so 5 x 5 x 3 = 75 numbers, and ignores everything else. That patch is the neuron's Receptive field, and the slide states the whole idea in one sentence: the output neuron is connected to input neurons in the receptive field.

That sentence hides two separate decisions, and Goodfellow, Bengio and Courville name them in section 9.2 of Deep Learning. The first is sparse interactions. With m inputs and n outputs, a fully connected layer stores m x n weights. If each output is allowed to touch only k inputs, the layer stores k x n. On the CIFAR image that is 75 per output instead of 3072.

The second decision is Weight sharing. Sparse connectivity alone would still give every output position its own private set of 75 weights. Convolution goes further and uses the same 75 numbers at every position. Goodfellow and colleagues put it plainly: rather than learning a separate set of parameters for every location, we learn only one set, so the storage of one filter drops to k parameters. They also give the synonym you will meet in older papers, tied weights. Slide 8 labels its figure Weight Sharing, and slides 12 and 13 show it in motion: the same green, yellow and cyan filters are applied wherever the red window sits on the blue input.

What the saving looks like in a real first layer

CS231n works through AlexNet's first layer to show the size of the effect, and the numbers are worth holding in memory for the exam. The input is 227 x 227 x 3 (the AlexNet paper says 224 x 224; CS231n uses 227 so that (227 - 11)/4 + 1 = 55 comes out exact), the filters are 11 x 11 x 3, and the output is 55 x 55 x 96.

Crossing out the fully connected shapes

The slide introduces the shapes by literally striking through the fully connected ones. In Part 01 the input was X (n, c_i): n samples in the batch (the Batch size), each a vector of c_i features. A 1D convolution adds a spatial axis of width w_i, so X becomes (n, c_i, w_i) and Y becomes (n, c_o, w_o). The weight matrix (c_o, c_i) gains a kernel width k_w and becomes (c_o, c_i, k_w). The bias does not change at all: still one number per output channel, (c_o,). The table already shows the 2D column, which adds a height axis in the same way; the next concept explains each of its axes.

Start with a layer you could type into PyTorch today: nn.Conv2d(3, 16, kernel_size=5), applied to a batch of 8 RGB images of 32 x 32. Four tensors take part, and every axis of each one has a name.

The general rule is the 2D column that the previous concept left open. The input X is (n, c_i, h_i, w_i), the output Y is (n, c_o, h_o, w_o), the weights W are (c_o, c_i, k_h, k_w) and the bias b is (c_o,). The PyTorch documentation for torch.nn.Conv2d confirms this exact ordering: the weight is stored as (out_channels, in_channels / groups, kernel_size[0], kernel_size[1]) and the bias as (out_channels). The groups divisor is 1 by default; it becomes important in Part 03 when grouped and depthwise convolutions arrive.

A filter is a block, not a square

Slides 9 to 11 make one argument in three frames. The first frame draws a single filter as a green block with edges labelled k_h, k_w and c_i, sitting next to the red window on the input. The block is as deep as the input has channels. When it is applied at one position, it multiplies all c_i x k_h x k_w input values under the window by its own c_i x k_h x k_w weights, sums everything into a single number, and drops that number into one output channel. CS231n states the rule as a sentence to memorise: the extent of the connectivity along the depth axis is always equal to the depth of the input volume.

The second frame adds a yellow filter. It has the same depth c_i, sees the same window, and drops its number into a second output channel. The third frame shows the full stack of c_o filters with a bracket labelled c_o, and the output is now c_o channels deep. That bracket is literally the first axis of W. Choosing c_o is a design decision made by whoever builds the network; it is not derived from c_i.

Bias and batch, the two easy axes

The Bias is added once per output channel, after the whole c_i-deep dot product, which is why its shape is (c_o,) exactly as in the fully connected layer of Part 01. The Batch size n appears in X and Y but never in W or b: the same weights process every sample in the batch. If you are ever unsure of a shape, check that n is absent from the parameters and that c_o leads W.

Slide 14 gives the smallest example that shows the shrink. The input is 4 x 4, the Kernel (filter) is 3 x 3, there is no padding and the window moves one cell at a time. Along the top row the window can start at column 0 or column 1. Starting at column 2 would need columns 2, 3 and 4, and there is no column 4. Two starting positions per axis means the output is 2 x 2.

Slides 12 and 13 are two frames of the same motion, taken from the MIT 6.5940 lecture that this section of the deck reproduces. The red window moves one step at a time along a spatial dimension, and at every stop the same green, yellow and cyan filters are applied, which is why slide 12 carries the label Weight Sharing. Nothing about the filters changes as the window moves; this is where Weight sharing becomes visible rather than stated. Each stop adds one short stack of c_o values to the output, one value per filter, so the output has exactly one spatial cell per valid window position. The count of valid positions is the whole story of the output size.

The general formula, one part early

Slide 15 asks you to use a Stride of 2 before slide 18 has defined stride, so you need the general rule now. With padding p on every border and stride s, Dumoulin and Visin's relationship 6 gives the output size, and the PyTorch documentation states the same formula with a dilation term that is 1 throughout this course.

Two rows deserve a second look. The 32, 5, 1, 2 row is the ConvNetJS demo on slide 25, which uses pad 2 precisely so that its 5 x 5 convolution keeps the 32 x 32 size, giving the 32 x 32 x 16 output printed there. The 224, 7, 2, 3 row is the first layer of a ResNet, which halves the spatial size in one step. Both are the same formula.

Slide 15 is the exam question in miniature: two input channels of 4 x 4, one output channel, a 2 x 2 filter, Stride 2, no padding and no Bias. Input channel 1 has every row equal to 1 2 3 4; input channel 2 has every row equal to 5 6 7 8. The filter's first slab is [[2, 1], [1, 2]] and its second slab is [[1, 2], [2, 1]]. In the shapes of the previous concepts, X is (1, 2, 4, 4) and W is (1, 2, 2, 2).

The formula says what the filter-is-a-block idea implied: the sum over c folds all input channels into one number. There is no separate output per input channel. With that in mind the four values of the slide are four applications of the same recipe. Strictly, this is cross-correlation: mathematical convolution flips the kernel first, and this formula does not. Deep learning libraries, PyTorch and this deck all call the unflipped version convolution (Goodfellow et al., section 9.1), and this course follows that convention.

Now do it with your hands. The stepper below holds the slide 15 numbers, highlights the current window on both input channels, lists the products channel by channel and fills the output one cell at a time. Switch the stride to 1 to see the 3 x 3 output, then change a few cells and predict the result before you press Next.

The stride-1 column shows what the middle window adds. At columns 1 to 2, channel 1 gives 2·2 + 1·3 + 1·2 + 2·3 = 15 and channel 2 gives 1·6 + 2·7 + 2·6 + 1·7 = 39, so 54 sits between 42 and 66. Stride 2 skips that window entirely, which is why the output is smaller and the compute drops from 9 windows to 4. On a large input the compute falls to about a quarter of the stride-1 cost, since both axes are halved.

Counting parameters

The shapes of W and b give the parameter count directly, and that count is what decides whether a model fits in a microcontroller's flash. The weights number c_o x c_i x k_h x k_w, the size of W, and there is one bias per output channel.

Slide 25 later in this lecture shows the ConvNetJS CIFAR-10 demo, whose first convolution has 16 filters of 5 x 5 over the 3 colour channels. The demo prints 16 x 5 x 5 x 3 + 16 = 1216 parameters, and the formula reproduces it: 1200 weights and 16 biases. Use that number as a retrieval hook; if your formula does not give 1216 for that layer, you have forgotten the biases or swapped a factor.

Only the filter geometry and the channel counts move the number.