Parameters and model size

Start with the smallest network worth drawing: five inputs x0 to x4 and three outputs y0 to y2, every input wired to every output. Count the arrows. Each input sends one arrow to each output, so there are 3 × 5 = 15 arrows, labelled from w00 up to w42. Each arrow carries exactly one learned number, its weight. That is the whole idea of the number of parameters: it is the synapse count, the number of weights, and for this layer it is 15.

Now look at the same layer the way a framework stores it. The inputs for a batch are a matrix X of shape n × c_i, the weights are a matrix W of shape c_o × c_i, and the outputs come out as Y = X · W^T of shape n × c_o. The fifteen arrows have become the fifteen elements of W. This is the second, equivalent definition the slides give: #Parameters is the number of elements in the network's weight tensors. For one linear layer that is the product of its two dimensions.

One thing in that matrix picture deserves a second look. The batch size n appears in X and in Y, but never in W. Feed the layer one image or a thousand and the weight matrix is the same c_o × c_i block. Parameter count is a property of the model, not of the data flowing through it. Hold on to this observation: it returns in the next concept for spatial size, and it is the reason parameters and MACs behave so differently in part 04.

The slides define the count for the "given" (slide 10) or "entire" (slide 11) neural network; both mean the same thing, the sum over every layer. They also say bias is ignored, and this lecture keeps that convention throughout. In PyTorch a linear layer stores a bias vector of length c_o and a convolution stores one bias per output channel, so the true count is larger by c_o per layer (PyTorch Conv2d documentation). Next to c_o · c_i that is a rounding error, which is why the lecture drops it.

Layer mechanics live in lecture 02

How a linear layer, a convolution, padding and stride, grouped and depthwise convolution and pooling produce their outputs, including the output-shape arithmetic behind numbers like 96 × 55 × 55, was covered in lecture 02: neurons and linear layers, convolution layers, padding, stride and grouped convolution and pooling.

Take AlexNet's third convolution: 384 filters, each looking at 256 input channels through a 3 × 3 window. One filter is a small brick of numbers, 256 deep and 3 × 3 across, so it holds 256 · 9 = 2,304 weights. There are 384 such bricks, so the layer holds 384 · 2,304 = 884,736 weights. That is the whole convolution formula for the #Parameters, read off the shape of the weight tensor.

Notice what is missing. The input is 256 × 13 × 13 and the output is 384 × 13 × 13, yet 13 appears nowhere in the count. A convolution slides the same brick to every output position, so the weights are shared across space. Goodfellow, Bengio and Courville make the point with an edge detector: a two-element kernel applied to a 320 × 280 image stores two parameters, where a dense matrix doing the same job would store billions (Goodfellow et al., section 9.2). The kernel size and the channel counts set the parameter count; the output height and width do not. Part 04 will show that MACs do multiply by h_o · w_o, because the brick is reapplied at every position, and that is the single most important difference between the two metrics.

Grouped convolution: derive it, do not memorize it

A grouped convolution with g groups cuts the c_i input channels into g slices of c_i/g and the c_o output channels into g slices of c_o/g, then runs g ordinary convolutions side by side, each reading only its own input slice and writing only its own output slice. So count one group with the formula you already have: (c_o/g) · (c_i/g) · k_h · k_w. There are g groups, so multiply by g. One factor of g cancels.

This matches what a framework actually stores. PyTorch gives Conv2d a weight of shape (out_channels, in_channels / groups, kH, kW) and requires that in_channels and out_channels are both divisible by groups (PyTorch documentation). The historical reason AlexNet has g = 2 in three of its layers is hardware, not elegance: the network was trained across two GTX 580 GPUs with 3 GB each, and "the kernels of the second, fourth, and fifth convolutional layers are connected only to those kernel maps in the previous layer which reside on the same GPU" (Krizhevsky et al., 2012, section 3.5). Grouping halved the weights each GPU had to hold.

Depthwise convolution: the limit g = c_i = c_o

Push the grouping to its extreme, one channel per group, and you get a depthwise convolution. Now g = c_i = c_o and every channel has its own private k_h × k_w filter. In the MobileNets paper this is stated directly: "the depthwise convolution applies a single filter to each input channel" (Howard et al., 2017, section 3.1). Plug g = c_i into the grouped formula and the c_i cancels.

The simulator collects all four formulas, adds the bit width from the final concept of this part and the MACs from part 04, and carries a preset for every parameterized AlexNet layer. Try conv3 now and confirm 884,736, then change h_o and watch which readouts move.

AlexNet is the convolutional network whose variant won the ILSVRC-2012 challenge with a top-5 test error of 15.3%, and its authors state up front that it has "60 million parameters and 650,000 neurons" (Krizhevsky et al., 2012). It is also the best possible exercise for this part: five convolutions, three of them grouped, three max-pools and three linear layers, with input 3 × 224 × 224. Walk it top to bottom and apply the right #Parameters formula at each layer. The output shapes come from lecture 02; here we only count.

Where the weights actually are

The share column is the real lesson. The five convolutions together hold 2.3M weights, under 4 percent. The three linear layers hold 58.6M, about 96 percent, and fc6 alone holds 62 percent. Krizhevsky and colleagues knew this: a footnote in the paper says that "most of the net's parameters are in the first fully-connected layer" (Krizhevsky et al., 2012). Every input of fc6 is wired to every one of its 4,096 outputs, and it has 9,216 inputs, so the dense wiring the convolutions avoided comes back in one enormous matrix.

There is a subtle exception hiding in that 9,216. The previous concept said parameter counts do not depend on input resolution, and for every convolution that is true. But fc6 takes the flattened 256 × 6 × 6 map as its c_i, and the 6 × 6 came from 224 × 224 through the strides and pools. Feed AlexNet a larger image and fc6 would need a larger weight matrix. The first linear layer after a flatten is the one place where resolution leaks into the parameter count. Global average pooling, introduced in Network in Network, removes that leak: it averages each channel to one number, and "there is no parameter to optimize" in it (Lin, Chen and Yan, 2014), so the classifier input no longer grows with the image.

This table is the starting line for two results you will study later. Han and colleagues pruned AlexNet "from 61 million to 6.7 million" parameters with no loss of accuracy, a 9x reduction (Han et al., 2015), and Deep Compression then cut its storage "from 240MB to 6.9MB" (Han, Mao and Dally, 2016). Both papers attack exactly the layers this table flags: when 96 percent of the weights sit in three linear layers, that is where pruning pays.

A parameter count is a number of boxes. To know how much storage the boxes take, you need the size of each box. Store AlexNet's 61M weights as 32-bit floating point and each weight takes 4 bytes, so the whole network takes 61M × 4 B = 244 MB. Store the same weights as 8-bitintegers and the same network takes 61 MB. Nothing about the architecture changed; only the bit width did.

That is the definition of model size: the storage needed for the weights of the network, the #Parameters times the bits each one occupies. The slide lists MB, KB and bits as the common units, and the formula as written produces bits. To reach bytes divide by 8, and to reach megabytes divide by another 10^6. The slide is explicit that its megabyte is decimal, 244 × 10^6 bytes. For a single layer, kilobytes are the natural unit: AlexNet's conv3 holds 884,736 weights, which at 8 bits is 884,736 bytes, or 884.7 KB (divide bytes by 10^3), and at 32 bits 3,538.9 KB.

The formula assumes one data type for the whole network. Mixed-precision models break that assumption, and then the size is a sum over layers, each layer's count times its own bit width. That is the form you need when, for example, the first and last layers stay at 8 bits while the middle is pushed to 4.

Why the multiplication matters for this course

The parameter count is fixed once the architecture is chosen, but the bit width is a knob. The lecture on quantization turns that knob, and this formula says exactly what it buys: every halving of the bit width halves the model size with no change to the architecture. Deep Compression combined pruning, quantization and Huffman coding to take AlexNet "from 240MB to 6.9MB" (Han, Mao and Dally, 2016); their 240 MB starting point is, up to rounding, the 244 MB computed in this concept. Part 03 shows a microcontroller SRAM budget of 256 kB (the MCU constraint, a budget for activations). The slides do not give a flash size, but a typical MCU of this class, such as the STM32F746 used by MCUNet (Lin et al., NeurIPS 2020), has about 1 MB of flash, where the weights would live; even 8-bitAlexNet at 61 MB is about 60 times too large for that flash, which is why edge deployment starts with counting.