Padding, stride, receptive field and grouped convolution

Take the 5 x 5 input and 3 x 3 Kernel (filter) from Part 02. The kernel can only sit where it fits entirely inside the image, so it has three horizontal and three vertical positions and the output Feature map is 3 x 3. Stack a second such layer and you get 1 x 1. A third layer cannot run at all. Every layer eats two rows and two columns, and Goodfellow, Bengio and Courville put it plainly: without padding the representation shrinks by one pixel less than the kernel width at every layer.

Padding is the fix. Add a ring of p extra cells around the input before sliding the kernel. With p = 1 the 5 x 5 image becomes 7 x 7, the kernel now has five positions in each direction, and the output is 5 x 5 again. The same 27 weights per filter (3 x 3 over three input channels) do the work; the only thing that changed is where the kernel is allowed to stand.

Read the formula as a story. The padded input is h_i + 2p rows tall. A window of height k_h can start on any row except the last k_h - 1, which leaves h_i + 2p - k_h + 1 starting rows, one per output row. Set h_o = h_i and the equation solves to p = (k - 1)/2, which is a whole number only for odd kernels. That is why 3 x 3, 5 x 5 and 7 x 7 dominate practice: Zhang et al. note that an odd kernel lets the same amount of padding go on both sides and keeps the output pixel centred on its window. Dumoulin and Visin call this half padding, and the deep learning book calls it same convolution, after the MATLAB option of that name.

What goes in the ring

The formula does not care what the new cells contain, but the values do affect what the network learns near the border. The slide pads a 3 x 3 image holding 1 to 9 by p = 2 on every side, which is a good specimen because 2 is the largest reflection padding a 3 x 3 image can take. Zero padding writes zeros, and PyTorch uses it unless told otherwise. Reflection padding mirrors the image across its edge without repeating the edge pixel, so the row above 1 2 3 reads 4 5 6 and the row above that reads 7 8 9. Replication padding stretches each edge pixel outward, so the corner 1 fills the whole top-left 3 x 3 block. Constant padding is zero padding with a value of your choice.

The index rule behind the visual is short. For a padded coordinate j in 0 to 6, subtract p to get an offset into the original image. Replication clamps that offset into 0 to 2. Reflection folds it: -1 becomes 1, -2 becomes 2, 3 becomes 1 and 4 becomes 0. Fold both the row and the column offset and you have the source pixel. The PyTorch documentation for ReflectionPad2d walks through exactly this example with the values 0 to 8. PyTorch also offers a 'circular' mode that wraps around to the opposite edge; the slide leaves it out.

One output pixel of a 3 x 3 Convolution layer is a weighted sum of a 3 x 3 patch of its input. That patch is its Receptive field. Now feed that layer's output into a second 3 x 3 layer. One pixel of the second layer reads a 3 x 3 block of first-layer outputs, and each of those was itself computed from a 3 x 3 patch. The patches of neighbouring pixels overlap and slide by one cell, so the union of all nine is a 5 x 5 region of the original image. A third layer makes it 7 x 7.

Each stride-1 layer adds k - 1 to the side of the region, because the outermost cells of the new window reach (k - 1)/2 further in each direction. Start from a single pixel and add k - 1 once per layer.

With k = 3 this gives 3, 5, 7, 9, 11 for L = 1 to 5. The deep learning book's figure 9.4 makes the same point and adds the hint that matters next: the effect increases if the network includes strided convolution or pooling.

The problem with large images

A classifier must eventually let one output depend on the whole picture. For a 224 x 224 image and stride-1 3 x 3 layers, solve 2L + 1 >= 224: L = 112 layers before a single output pixel sees everything. Each of those layers works at full resolution, so the activation memory and the multiply-accumulates are enormous. The slide states the problem and the cure in one breath: downsample inside the network.

Stride is the cheapest way to do it. Instead of moving the kernel one cell at a time, move it s cells. The number of window positions drops by about a factor of s in each direction, and so does the output size. Goodfellow, Bengio and Courville describe stride s as mathematically equivalent to a stride-1 convolution followed by keeping every s-th output, only without wasting work on the outputs you would throw away.

The floor is not decoration. The padded input has h_i + 2p - k_h rows on which a window can start after the first, and the kernel visits every s-th of them. If that span is not a multiple of s, the last window would run past the edge and is dropped. Try h_i = 8, k = 3, p = 0, s = 2: the slide's version gives 3.5, which is not a size; the true answer is floor(5/2) + 1 = 3. Dumoulin and Visin state the relationship with the floor, and the PyTorch Conv2d documentation computes the output shape the same way.

How stride changes the receptive field rule

Stride does more than shrink maps. Once a layer has stride 2, two neighbouring outputs of that layer sit two input pixels apart, so every layer after it takes steps of two in input coordinates. Its 3 x 3 window no longer adds 2 pixels to the receptive field; it adds 2 x 2 = 4. The slide 18 top row shows the payoff: a stride-2 first layer followed by one stride-1 layer already reaches 7, where stride 1 needed three layers.

Araujo, Norris and Sim give the general closed form. Call the product of the strides before layer l the jump (some authors say effective stride). Each layer adds (k_l - 1) times its jump.

Real networks live at the strided end of that table. Araujo, Norris and Sim compute receptive fields of 195 for AlexNet, 212 for VGG-16, 483 for ResNet-50 and 1,311 for Inception-v3, all with a total downsampling of 32, and remark that the receptive field usually covers the entire input image. The stride in the ResNet stem you computed in the calculator is the first of those doublings: 224 to 112 in one layer.

Padding and stride change the shape of the computation. The last two concepts change its wiring. Take a layer with c_i = c_o = 64 channels and 3 x 3 kernels. Each of the 64 filters reads all 64 input channels, so the layer holds 64 x 64 x 9 = 36,864 weights. Now split the channels into two halves and run two separate convolutions, each mapping 32 inputs to 32 outputs. Each half holds 32 x 32 x 9 = 9,216 weights, so the pair holds 18,432: the same output shape at half the cost.

That is a Grouped convolution with g = 2, or in the slide's words a group of narrower convolutions. In general, split c_i into g blocks of c_i/g channels and c_o into g blocks of c_o/g. Output block j reads only input block j. The slide writes the new Weight shape as (g · c_o/g, c_i/g, k_h, k_w), and the first factor simplifies to c_o: there are still c_o filters, each just reads c_i/g channels instead of all c_i. The Bias stays one per output channel.

The divisibility rule

The split only works if both channel counts divide evenly. PyTorch says so directly: in_channels and out_channels must both be divisible by groups. With c_i = 64 and c_o = 96, g = 4 is fine (16 inputs and 24 outputs per group) but g = 5 is not, because 64/5 is not an integer. Output channel j belongs to group floor(j / (c_o/g)) and reads only the input channels of that group. The calculator earlier in this part flags any pair that fails the rule.

The idea is older than its name. Krizhevsky, Sutskever and Hinton trained AlexNet on two GTX 580 GPUs with 3 GB each, so they put half of the kernels on each GPU and let the kernels of layer 4 take input only from the kernel maps of layer 3 that lived on the same GPU. That is g = 2, chosen for memory rather than elegance. ResNeXt (Xie et al., 2017) turned the group count into a design axis they named cardinality and found that raising it was more effective than going deeper or wider at the same parameter budget.

That last clause is the cost. Inside a grouped layer, no information crosses a group boundary; the faint grey lines in the visual are exactly the missing connections. Networks that use grouped layers alternate them with something that mixes channels again, most often a 1 x 1 convolution. The next concept pushes g to its limit and meets that need head on.

Push the group count as far as it goes: g = c_i = c_o. Every group is now one Channel in and one channel out, so each channel gets its own k x k filter and nothing else. For 64 channels and 3 x 3 kernels that is 64 x 9 = 576 weights, 64 times fewer than the 36,864 of the standard layer. The slide 20 diagram says it in one picture: eight straight arrows from input strip to output strip, and not one of them crosses.

This is a Depthwise convolution. The slide writes its Weight tensor as (c, k_h, k_w), one k_h x k_w filter per channel. Frameworks keep the four-dimensional layout: PyTorch stores (out_channels, in_channels/groups, kH, kW), which with groups = in_channels is (c, 1, k_h, k_w), the same numbers with a singleton axis. PyTorch also allows out_channels = K · in_channels with groups = in_channels, a depthwise layer with depth multiplier K; the slide's g = c_i = c_o is the case K = 1.

Is this reduction in the number of weights really good?

The slide asks the question in red and leaves it open. The honest answer has two halves. For storage and arithmetic, yes: 576 weights fit in 2.3 KB of fp32 flash, and the multiply-accumulates fall by the same factor of 64. For what the layer can learn, no. A depthwise layer filters each channel in isolation. Sandler et al. describe it as lightweight filtering that applies a single convolutional filter per input channel, and Howard et al. point out that it does not combine input channels, so it cannot build a feature that depends on two channels at once. A standard layer does that at every position; a depthwise layer never does.

The repair is a 1 x 1 convolution, called a pointwise convolution, placed right after it. With k = 1 it looks at one position and mixes all c_i channels into c_o outputs: W of shape (c_o, c_i, 1, 1), c_o · c_i weights, and in Sandler et al.'s words it is responsible for building new features through computing linear combinations of the input channels. The depthwise layer handles space, the pointwise layer handles channels, and the pair is called a depthwise separable convolution. It is the building block of MobileNet and, wrapped in an expand and project pair of 1 x 1 layers, of MobileNetV2.

The last row is the one to remember. Howard et al. derive the cost ratio of a separable block to a standard layer as 1/N + 1/D_K², where N is the output channel count and D_K the kernel size. With N = 64 and D_K = 3 that is 1/64 + 1/9 ≈ 0.127, so the block is about 7.9 times cheaper, matching the table. For wide layers the 1/N term fades and the saving approaches D_K² = 9; Howard et al. report 8 to 9 times less computation for 3 x 3 kernels at a small accuracy cost.

What this means on an embedded device

• Fewer weights means less flash and less weight traffic from memory, which on a microcontroller is often the whole budget.
• Fewer multiply-accumulates means lower latency and lower energy per inference on the same core.
• The depthwise layer does only k² multiply-accumulates per output value, against c_i · k² for a standard layer, yet still reads a k x k window per output, so it moves a lot of activation data for little arithmetic and on many accelerators is bound by memory bandwidth rather than compute. Treat the parameter saving as an upper bound on the speedup, not a promise.
• Accuracy drops a little relative to a standard layer of the same shape, which is the price of removing the cross-channel terms.