Neural network efficiency metrics

Suppose the detector you trained in the cloud is to run on a Jetson-class board mounted on a vehicle. Before the board can use it, three separate questions must be answered. Does the model fit in the flash and RAM you have? Does one inference finish before the next camera frame lands, roughly every 33 ms at 30 frames per second? And how much charge does each inference pull from the battery? Slide 4 draws these as the three outer goals of a triangle, Smaller, Faster and Greener, and the three costs that quantify them: Storage, Latency and Energy.

The part of the triangle that matters is the middle. All three costs are driven by only two resources, Computation and Memory. Storage is purely a memory question: how many bytes the weights occupy. Latency and energy depend on both, because an inference both performs arithmetic and moves data, and each of those takes time and burns charge. That is why the metric list on the side of the slide has exactly two groups. The efficiency metrics that are memory-related are the number of parameters, the model size and the total and peak number of activations. The computation-related ones are MACs, FLOPs with their rate FLOPS, and OPs with their rate OPS.

This taxonomy is the spine of the whole lecture. Parts 02 and 03 fill the memory group (parameters, model size, activations), part 04 fills the computation group (MACs, FLOPs, OPs), and this part explains the three costs those metrics are meant to predict. Whenever a later metric appears, place it here first: which group is it in, and which cost does it feed?

Metrics you count versus costs you measure

One distinction keeps the rest of the part honest. Parameters and MACs are properties of the network alone. You can count them on paper from the layer shapes, and they do not change when you move the model to another chip. Latency, throughput and energy are properties of the network running on a specific processor with a specific runtime. They must be measured, or at least estimated from hardware numbers. The same split reappears in the latency model of concept 4 as NN specification versus hardware specification.

Slide 5 shows the same Cityscapes street segmented twice. The two masks look almost identical, and by the usual accuracy metric they are: SegFormer-B5 reaches 82.4 mIoU and EfficientViT-L1 reaches 82.7 mIoU. The difference is time. SegFormer needs 638 ms per frame and EfficientViT needs 45.9 ms (the slide rounds to 46 ms). At 30 frames per second a new frame arrives every 33 ms, so one model falls nineteen frames behind for every frame it finishes, and the other almost keeps up.

That per-frame time is latency: the delay between starting one specific task and finishing it. Patterson and Hennessy call the same quantity response time or execution time, the time between the start and completion of a task. It is a property of one task, not of a stream of tasks, and the next concept shows why that restriction matters. The 13.9x gap (638 / 45.9) is exactly the headline of the EfficientViT paper, which reports up to 13.9x GPU latency reduction over SegFormer at the same or better accuracy (Cai et al., ICCV 2023).

Read the ratio row carefully, because it plants the seed for concept 4. The MAC count falls 5.2x, but latency falls 13.9x. If latency were simply proportional to arithmetic, the two ratios would match. They do not, because SegFormer's softmax attention moves large intermediate tensors through memory, and that data movement, not its MACs, is what makes it slow on an edge GPU. MACs alone do not predict latency; they are one input to it.

Four conditions every latency must state

The slide's footnote is not fine print. Latency is measured, so it depends on everything in the measurement. The paper's setup sentence names four things: the hardware, the runtime, the numeric precision and the batch size. Change any one of them and the number changes, sometimes by an order of magnitude. The same paper measures EfficientViT-B1 at 24.3 ms on Orin and 0.82 s on a Jetson Nano, and SegFormer-B1 at 146 ms on Orin and 5.6 s on Nano. Same model, same precision, same batch size, different board: 34x to 38x apart.

Slide 6 replaces one street with two mosaics of video clips and asks a different question: not how long one clip takes, but how many clips get done per second. One system manages 6.1 videos per second, the other 77.4. Those two numbers come from Table 5 of the TSM paper (Lin, Gan and Han, ICCV 2019): I3D at 306 GFLOPs versus TSM8F at 33 GFLOPs, both on a single Tesla P100.

That rate is throughput: the amount of work completed per unit time, what Patterson and Hennessy also call bandwidth. The TSM paper is unusually helpful here because it states its protocol: latency was measured with batch size 1 and throughput with batch size 16. TSM8F's latency is 17.4 ms. If throughput were the reciprocal of latency you would expect 1000 / 17.4 = 57.5 videos per second, yet the paper reports 77.4. The extra comes from batching, and it is the whole point of slide 7.

Two designs, same processor, opposite answers

Slide 7 pits two designs against each other. Design 1 handles one image at a time and takes 50 ms per image. Design 2 takes four images together and finishes the whole group in 100 ms. Each image in Design 2 waits the full 100 ms, because nothing leaves the processor until the batch is done.

A second scenario for practice

A cloud server serves the same model at two batch sizes. At batch 1 a request takes 2 ms. At batch 32 the whole batch takes 8 ms, because the GPU can spread thirty-two images over its cores far more efficiently than one.

Batching paid 4x latency to buy 8x throughput. Whether that is a good trade depends on who is waiting. A brake camera needs one frame answered now: latency, at batch size 1, which is why slide 5 measured that way. A data center indexing videos overnight needs clips per hour: throughput, at whatever batch size fills the GPU. Your research project sits on the first side of that line.

Concept 2 left a puzzle: MACs fell 5.2x but latency fell 13.9x. To resolve it you need a model of where the milliseconds go. Watch one layer execute. The processor loads the input activations, loads the layer's weights, multiplies and accumulates, and stores the output activations. Two of those steps are arithmetic; the other two are data movement. Real hardware does not wait for one to finish before starting the other. While the multipliers chew on the current tile, the memory system is already fetching the next weights, so the two streams run side by side and the layer finishes when the slower stream finishes.

Computation time is the operation count divided by the rate at which the processor retires operations. Memory time is the bytes that must move divided by the memory bandwidth, once for the weights and once for the activations in and out. Now look at the colour coding on the slide, because it is the sentence the exam wants. Every numerator is a property of the network: operations, model size, activation sizes. Every denominator is a property of the processor: operations per second, bandwidth. That split is the NN specification versus hardware specification distinction, and it makes efficiency a co-design problem. You change numerators by redesigning the network, denominators by choosing hardware, and latency by whichever of the two you can actually afford to change.

The same model appears in the industry literature under different names. NVIDIA's GPU performance guide states that memory time is bytes accessed divided by memory bandwidth, math time is operations divided by math bandwidth, and the longer of the two shows what limits performance: math-limited if math time is longer, memory-limited if memory time is longer. The roofline model of Williams, Waterman and Patterson (CACM 2009) writes the same idea as a rate, attainable performance = min(peak compute, bandwidth x operational intensity). The vocabulary to adopt is compute-bound when T_computation wins and memory-bound when T_memory wins. One caution before using datasheet numbers: dividing by a peak OPS figure gives a floor on T_computation, not an estimate of it, because real kernels rarely keep every multiplier busy. The measured time over that floor is the inverse of the utilisation the runtime achieved.

Worked example A: a toy model and two processors

Worked example B: back to slide 5

The third cost is energy, and slide 9 settles where it goes with one table. In a 45 nm process at 0.9 V, a 32-bit integer add costs 0.1 pJ and a 32-bit integer multiply 3.1 pJ. Reading one 32-bit word from DRAM costs 640 pJ. The multiply is the most expensive arithmetic on the list, and the memory access is two hundred times dearer than it.

Three ratios from that table are worth memorizing. DRAM over integer multiply is 640 / 3.1 ≈ 206, which the slide rounds to 200x. DRAM over SRAM is 640 / 5 = 128, so keeping data on-chip is worth two orders of magnitude on its own. And float add over integer add is 0.9 / 0.1 = 9, a seed for quantization: the same arithmetic in integers is nine times cheaper per add. Han et al. summarize the whole table in one line, that memory access is three orders of magnitude more energy expensive than simple arithmetic (640 / 0.1 = 6400).

The chain the slide writes across the top follows directly: data movement means more memory references, and more memory references mean more energy. Horowitz's own framing in the ISSCC 2014 talk is that a DRAM access at 1 to 2 nJ is a couple of orders of magnitude above a cache access or a functional operation at around 10 pJ, and that even with better I/O circuits DRAM will stay expensive, around 0.6 nJ per 8 bytes, because requests and data still have to travel a long physical distance. The slide's 640 pJ per 32-bit word is about half of the 1.3 nJ per 64-bit access in his Figure 1.1.9, as the provenance note below explains.

The log axis of the slide's chart is not decoration either. On a linear axis the six small bars would be invisible next to DRAM; a 0.1 pJ bar next to a 640 pJ bar is a pixel next to a page. Only a logarithmic axis can show both, and it is the honest way to read any table whose entries span four decades.

Worked example: AlexNet's energy bill

Every efficiency technique later in the course can be read as a way of cutting that memory line. Keep weights in on-chip SRAM, which is 128x cheaper per access than DRAM. Reuse each fetched value as many times as possible before it leaves the chip; Horowitz notes that efficient designs complete on the order of a thousand operations per DRAM fetch. And make the model small enough to fit on-chip in the first place, which is what model size in part 02, peak activations in part 03, and pruning and quantization later in the course are all about.

Start with the smallest network worth drawing: five inputs x0 to x4 and three outputs y0 to y2, every input wired to every output. Count the arrows. Each input sends one arrow to each output, so there are 3 × 5 = 15 arrows, labelled from w00 up to w42. Each arrow carries exactly one learned number, its weight. That is the whole idea of the number of parameters: it is the synapse count, the number of weights, and for this layer it is 15.

Now look at the same layer the way a framework stores it. The inputs for a batch are a matrix X of shape n × c_i, the weights are a matrix W of shape c_o × c_i, and the outputs come out as Y = X · W^T of shape n × c_o. The fifteen arrows have become the fifteen elements of W. This is the second, equivalent definition the slides give: #Parameters is the number of elements in the network's weight tensors. For one linear layer that is the product of its two dimensions.

One thing in that matrix picture deserves a second look. The batch size n appears in X and in Y, but never in W. Feed the layer one image or a thousand and the weight matrix is the same c_o × c_i block. Parameter count is a property of the model, not of the data flowing through it. Hold on to this observation: it returns in the next concept for spatial size, and it is the reason parameters and MACs behave so differently in part 04.

The slides define the count for the "given" (slide 10) or "entire" (slide 11) neural network; both mean the same thing, the sum over every layer. They also say bias is ignored, and this lecture keeps that convention throughout. In PyTorch a linear layer stores a bias vector of length c_o and a convolution stores one bias per output channel, so the true count is larger by c_o per layer (PyTorch Conv2d documentation). Next to c_o · c_i that is a rounding error, which is why the lecture drops it.

Layer mechanics live in lecture 02

How a linear layer, a convolution, padding and stride, grouped and depthwise convolution and pooling produce their outputs, including the output-shape arithmetic behind numbers like 96 × 55 × 55, was covered in lecture 02: neurons and linear layers, convolution layers, padding, stride and grouped convolution and pooling.

Take AlexNet's third convolution: 384 filters, each looking at 256 input channels through a 3 × 3 window. One filter is a small brick of numbers, 256 deep and 3 × 3 across, so it holds 256 · 9 = 2,304 weights. There are 384 such bricks, so the layer holds 384 · 2,304 = 884,736 weights. That is the whole convolution formula for the #Parameters, read off the shape of the weight tensor.

Notice what is missing. The input is 256 × 13 × 13 and the output is 384 × 13 × 13, yet 13 appears nowhere in the count. A convolution slides the same brick to every output position, so the weights are shared across space. Goodfellow, Bengio and Courville make the point with an edge detector: a two-element kernel applied to a 320 × 280 image stores two parameters, where a dense matrix doing the same job would store billions (Goodfellow et al., section 9.2). The kernel size and the channel counts set the parameter count; the output height and width do not. Part 04 will show that MACs do multiply by h_o · w_o, because the brick is reapplied at every position, and that is the single most important difference between the two metrics.

Grouped convolution: derive it, do not memorize it

A grouped convolution with g groups cuts the c_i input channels into g slices of c_i/g and the c_o output channels into g slices of c_o/g, then runs g ordinary convolutions side by side, each reading only its own input slice and writing only its own output slice. So count one group with the formula you already have: (c_o/g) · (c_i/g) · k_h · k_w. There are g groups, so multiply by g. One factor of g cancels.

This matches what a framework actually stores. PyTorch gives Conv2d a weight of shape (out_channels, in_channels / groups, kH, kW) and requires that in_channels and out_channels are both divisible by groups (PyTorch documentation). The historical reason AlexNet has g = 2 in three of its layers is hardware, not elegance: the network was trained across two GTX 580 GPUs with 3 GB each, and "the kernels of the second, fourth, and fifth convolutional layers are connected only to those kernel maps in the previous layer which reside on the same GPU" (Krizhevsky et al., 2012, section 3.5). Grouping halved the weights each GPU had to hold.

Depthwise convolution: the limit g = c_i = c_o

Push the grouping to its extreme, one channel per group, and you get a depthwise convolution. Now g = c_i = c_o and every channel has its own private k_h × k_w filter. In the MobileNets paper this is stated directly: "the depthwise convolution applies a single filter to each input channel" (Howard et al., 2017, section 3.1). Plug g = c_i into the grouped formula and the c_i cancels.

The simulator collects all four formulas, adds the bit width from the final concept of this part and the MACs from part 04, and carries a preset for every parameterized AlexNet layer. Try conv3 now and confirm 884,736, then change h_o and watch which readouts move.

AlexNet is the convolutional network whose variant won the ILSVRC-2012 challenge with a top-5 test error of 15.3%, and its authors state up front that it has "60 million parameters and 650,000 neurons" (Krizhevsky et al., 2012). It is also the best possible exercise for this part: five convolutions, three of them grouped, three max-pools and three linear layers, with input 3 × 224 × 224. Walk it top to bottom and apply the right #Parameters formula at each layer. The output shapes come from lecture 02; here we only count.

Where the weights actually are

The share column is the real lesson. The five convolutions together hold 2.3M weights, under 4 percent. The three linear layers hold 58.6M, about 96 percent, and fc6 alone holds 62 percent. Krizhevsky and colleagues knew this: a footnote in the paper says that "most of the net's parameters are in the first fully-connected layer" (Krizhevsky et al., 2012). Every input of fc6 is wired to every one of its 4,096 outputs, and it has 9,216 inputs, so the dense wiring the convolutions avoided comes back in one enormous matrix.

There is a subtle exception hiding in that 9,216. The previous concept said parameter counts do not depend on input resolution, and for every convolution that is true. But fc6 takes the flattened 256 × 6 × 6 map as its c_i, and the 6 × 6 came from 224 × 224 through the strides and pools. Feed AlexNet a larger image and fc6 would need a larger weight matrix. The first linear layer after a flatten is the one place where resolution leaks into the parameter count. Global average pooling, introduced in Network in Network, removes that leak: it averages each channel to one number, and "there is no parameter to optimize" in it (Lin, Chen and Yan, 2014), so the classifier input no longer grows with the image.

This table is the starting line for two results you will study later. Han and colleagues pruned AlexNet "from 61 million to 6.7 million" parameters with no loss of accuracy, a 9x reduction (Han et al., 2015), and Deep Compression then cut its storage "from 240MB to 6.9MB" (Han, Mao and Dally, 2016). Both papers attack exactly the layers this table flags: when 96 percent of the weights sit in three linear layers, that is where pruning pays.

A parameter count is a number of boxes. To know how much storage the boxes take, you need the size of each box. Store AlexNet's 61M weights as 32-bit floating point and each weight takes 4 bytes, so the whole network takes 61M × 4 B = 244 MB. Store the same weights as 8-bitintegers and the same network takes 61 MB. Nothing about the architecture changed; only the bit width did.

That is the definition of model size: the storage needed for the weights of the network, the #Parameters times the bits each one occupies. The slide lists MB, KB and bits as the common units, and the formula as written produces bits. To reach bytes divide by 8, and to reach megabytes divide by another 10^6. The slide is explicit that its megabyte is decimal, 244 × 10^6 bytes. For a single layer, kilobytes are the natural unit: AlexNet's conv3 holds 884,736 weights, which at 8 bits is 884,736 bytes, or 884.7 KB (divide bytes by 10^3), and at 32 bits 3,538.9 KB.

The formula assumes one data type for the whole network. Mixed-precision models break that assumption, and then the size is a sum over layers, each layer's count times its own bit width. That is the form you need when, for example, the first and last layers stay at 8 bits while the middle is pushed to 4.

Why the multiplication matters for this course

The parameter count is fixed once the architecture is chosen, but the bit width is a knob. The lecture on quantization turns that knob, and this formula says exactly what it buys: every halving of the bit width halves the model size with no change to the architecture. Deep Compression combined pruning, quantization and Huffman coding to take AlexNet "from 240MB to 6.9MB" (Han, Mao and Dally, 2016); their 240 MB starting point is, up to rounding, the 244 MB computed in this concept. Part 03 shows a microcontroller SRAM budget of 256 kB (the MCU constraint, a budget for activations). The slides do not give a flash size, but a typical MCU of this class, such as the STM32F746 used by MCUNet (Lin et al., NeurIPS 2020), has about 1 MB of flash, where the weights would live; even 8-bitAlexNet at 61 MB is about 60 times too large for that flash, which is why edge deployment starts with counting.

Take a real target: an STM32F746 microcontroller has a Cortex-M7 core, 1 MB of flash and 320 kB of SRAM. It has no DRAM and no operating system; MCUNet describes such boards as bare-metal devices (Lin et al., 2020). Now take two ImageNet classifiers that both reach about 70 percent top-1 accuracy with every value stored as an 8-bit integer. ResNet-18 needs 11.2 MB for its weights. MobileNetV2 at width 0.75 needs about 2.5 MB as MCUNet's Figure 8 labels the bar, a 4.6x reduction on the slide (strictly 11.2 / 2.5 ≈ 4.5; the paper rounds to 4.6x). If parameters were the memory story, the second model would be the obvious choice.

Then look at the other pair of bars. The largest amount of activation memory that ResNet-18 needs at any moment is 0.9 MB. For MobileNetV2-0.75 it is 1.7 MB. The efficient model is 1.8x worse on peak activations, and the MCUNet paper says so plainly: MobileNetV2 "reduces the model size by 4.6x compared to ResNet-18, but the peak activation size increases by 1.8x, making it even more difficult to fit the SRAM on microcontrollers" (Lin et al., 2020). Neither model fits 320 kB. The memory bottleneck moved without anyone shrinking it.

Two budgets, and the smaller one belongs to activations

The reason the two bars behave differently is that they are paid from different memories. Weights are read-only: they are written once when the firmware is flashed and only read during inference, so they live in flash, the larger and cheaper memory. Activations are produced by one layer and consumed by the next on every single inference, so they must sit in memory that can be written and read at speed, which on a microcontroller is the on-chip SRAM. MCUNet puts it in one sentence: "SRAM constrains the activation size (read&write); Flash constrains the model size (read-only)" (Lin et al., 2020). The model size from part 02 answers the flash question. The peak activation answers the SRAM question, and SRAM is the scarcer of the two by a factor of two to four on every board in MCUNet's list (the STM32F746 has about 3x more flash than SRAM).

This also explains why bit width alone does not rescue MobileNetV2. The slide already assumes 8-bit integers for both weights and activations. Even after that 4x shrink from 32-bit floats, the peak sits at 1.7 MB. MCUNet's Table 1 reports 1.7 MB for full-width int8 MobileNetV2 (6.8 MB in fp32), 5.3x over the 320 kB of an STM32F746, not counting im2col or other runtime buffers. Quantization scales both bars by the same factor; it does not change which bar is the problem.

The MCUNet row comes from Figure 8 of the same paper and is there to show what solving the right problem looks like: its designers shrank the peak activation below 0.5 MB rather than chasing a smaller parameter count, and only then did an ImageNet model fit a 512 kB board. The 4.6x model is not an efficient model for this class of device; it is an efficient model for a phone, where DRAM is plentiful and flash is not the constraint.

Why MobileNetV2 in particular: an imbalanced profile

The right-hand chart explains where MobileNetV2's peak comes from. Plot the memory that each of its eighteen blocks needs and the profile is wildly uneven: block 2 needs 1372 kB, blocks 0, 1, 3 and 4 sit between roughly 500 kB and 600 kB, and the remaining thirteen blocks mostly need 40 kB to 170 kB. Draw the 256 kB constraint of a typical microcontroller and exactly five blocks cross it. MCUNetV2, the follow-up paper that this chart comes from, states it directly: "The first 5 blocks have large peak memory, exceeding the memory constraints of MCUs, while the remaining 13 blocks easily fit 256kB memory constraints. The third block has 8x larger memory usage than the rest of the network" (Lin et al., 2021).

The two charts on the slide come from different papers with different memory accounting: the left bars are MCUNet's (2020) figures for MobileNetV2-0.75, while the right chart is MCUNetV2's analytic input-plus-output count for full-width MobileNetV2 in int8, so the numbers are not directly comparable. Its block 2 peak is 16×112×112 + 96×112×112 = 1,404,928 B, which is 1372 kB in 1024-byte kilobytes.

The cause is the shape of MobileNetV2's inverted residual block. Each block first expands the channel count about six times with a 1×1 convolution, applies a depthwise 3×3, and projects back down. The depthwise step is what makes the block cheap in weights and MACs, exactly as part 02 computed. But the expanded tensor exists at full spatial resolution, and in the early blocks that resolution is 112×112 or 56×56. Six times the channels at a large H×W gives a huge activation in a block that owns almost no parameters. MCUNet's per-block peak memory analysis (Figure 11) measures the same imbalance on a MobileNetV2 scaled to 0.3x width to fit 320 kB: even then one block has 2.2x the peak activation of the average block (Lin et al., 2020). A model can be light in weights and heavy in activations at the same time, and this architecture is the canonical example.

Inference keeps a few activations alive; training keeps all of them. Switch to 32-bit floats and a training step of ResNet-50. Its weights take 102 MB. Its activations, for one mini-batch, take 707 MB, about 7x more. Now swap in MobileNetV2 at width 1.4: the weights fall to about 24 MB, a 4.3x reduction, but the activations fall only to about 626 MB, a 1.1x reduction. The TinyTL paper that produced these numbers concludes: "It is the activation that bottlenecks the training memory, not the parameters" (Cai et al., 2020).

The mechanism is backpropagation itself. Write one linear layer as a_(i+1) = a_i W + b, where a_i is the layer's input activation. To update the weight you need the gradient of the loss with respect to W, and that gradient is the outer product of the input activation with the gradient flowing back from the next layer.

The forward pass computes a_1, then a_2, and so on up to the loss. The backward pass returns in reverse order, and when it reaches layer i it must still have a_i. So every layer's input has to be held from the moment it is produced until the backward pass comes back to it. Goodfellow, Bengio and Courville make the same point for a multilayer perceptron: "we need to store the input to the nonlinearity of the hidden layer. This value is stored from the time it is computed until the backward pass has returned to the same point," and the cost per layer is O(m·n_h) for a mini-batch of m examples and n_h hidden units (Goodfellow et al., 2016, section 6.5.7). Depth multiplies that cost by the number of layers, and the batch size multiplies it again.

The formula also points to the cure that TinyTL proposes and that this course returns to when it reaches on-device learning. The bias gradient in equation 2 needs no stored activation at all. If you freeze the weights and update only biases (plus a small added module), the a_i tensors never have to be kept, and the 707 MB term collapses. That is a memory argument, not an accuracy argument, and it is only visible once you have separated activations from parameters as this slide does. It matters doubly on a microcontroller, because TinyTL notes that DRAM access consumes two orders of magnitude more energy than on-chip SRAM access (Cai et al., 2020), echoing the Horowitz numbers from part 01.

Plot, layer by layer, how much memory the activations and the weights of an efficient network take, and a clear shape appears. For MCUNet, a network designed for microcontrollers, the first six layers need 50 kB to 75 kB of activation memory and almost no weight memory, and activation stays the dominant term, mostly 20 kB to 55 kB, out to about layer 17. Around layers 18 to 30 both are small, under about 15 kB. From layer 32 onward the weight memory climbs to 35 kB to 75 kB while activations stay near 5 kB to 10 kB. The total traces a U: expensive at both ends, cheap in the middle, for two different reasons. The chart comes from MCUNet's on-device training work and shows the memory needed to update each layer, which is the stored input activation plus the weight and its gradient. The same geometry governs inference, as the AlexNet arithmetic below shows.

The two counting rules from part 02 predict this. An activation tensor holds C × H × W values. A convolution weight tensor holds C_o × C_i × k_h × k_w values. Now follow a CNN from input to output. At every downsampling stage the spatial size halves in each direction, so H × W drops by 4x, while the channel count typically only doubles. The activation count C × H × W therefore falls by about 2x per stage. The weight count contains C_o × C_i, which grows by about 4x when channels double and does not care about H and W at all. MCUNetV2 states the consequence: "the memory bottleneck tends to appear at the early stage of the network" because resolution shrinks faster than channels grow (Lin et al., 2021).

The same U in numbers you already computed

You do not have to trust the chart. Take the AlexNet layers from part 02, put each layer's output activation count next to its weight count, and the profile reproduces itself from arithmetic.

The first convolution has 8x more activations than weights: a 55×55 map with 96 channels is large, and an 11×11 filter over only 3 input channels is small. By conv3 the map is 13×13 and the weights are already 14x the activations. The fully connected layers are the extreme: fc6 emits 4,096 values and owns 37.7M weights. Nothing about AlexNet was designed to show this; it is the geometry of every CNN.

What each end of the U decides

• The early, activation-heavy stage sets the SRAM peak. That is why MobileNetV2's problem in the previous concept was in blocks 0 to 4, and why MCUNetV2 runs only that stage in patches.
• The late, weight-heavy stage sets the flash budget and most of the parameter count. Pruning and quantization pay off most there.
• The cheap middle is where on-device training is affordable: updating a middle layer needs neither a large stored activation nor a large weight gradient. This band is exactly what the training chart above measures, and the paper it comes from exploits exactly that band (Lin et al., 2022).

Now make the peak a number you can compute. AlexNet takes a 3×224×224 image: 150,528 values. Its first convolution emits 96×55×55 = 290,400 values. While that convolution runs, both tensors must exist, because the layer is still reading the image as it writes the output: 150,528 + 290,400 = 440,928. When the first pooling layer runs, the image is no longer needed and can be freed, but the convolution output (290,400) and the pooling output (69,984) are both alive: 360,384. Memory rises and falls one layer at a time.

Two counts fall out of this picture. The total number of activations is the sum over the input and every layer output, which for AlexNet is 932,264. The peak number of activations is the largest number alive at any one moment. For a plain chain of layers executed one at a time, that is the largest input plus output pair. MCUNetV2 gives the rule and the reason: "the memory required for a layer is the sum of input and output activation (since weights can be partially fetched from Flash)", and the runtime "allocates the input and output activation buffer in SRAM, and releases the input buffer after the whole layer computation is finished" (Lin et al., 2021).

The approximation sign carries four assumptions, and an exam answer should name at least one of them. The network runs layer by layer, with no two layers in flight together. Weights are streamed from flash and are not counted in SRAM. There are no residual branches; if there are, MCUNetV2 says to add the memory of all branches alive at the same time, counting a shared input once. And there are no in-place tricks: MCUNet shows that a depthwise layer can overwrite its input channel by channel, needing N + 1 instead of 2N values (Lin et al., 2020). Real runtimes also need scratch buffers for im2col or accumulation, which push the true peak slightly above the formula.

Sweep through AlexNet yourself before reading the worked example. Watch which two bars are highlighted, how the running peak stops moving after the first layer, and how the bit width and the SRAM preset change whether the same 440,928 values fit.

The same arithmetic explains why MobileNetV2 failed in the first concept. Its first convolution maps a 3×224×224 image to 32×112×112: 150,528 + 401,408 = 551,936 values, which MCUNetV2 quotes as "539kB even when quantized in int8" (using 1024-byte kilobytes). A 224 input through a standard stem can never run layer by layer on a 256 kB board, whatever happens to the weights afterwards.

Finally, connect the counts back to latency. Part 01 approximated memory time as the bytes of activations and weights moved divided by the memory bandwidth. The per-layer input plus output is the activation traffic of that layer, so the same column that gives the peak also feeds the latency model. Peak tells you whether the model fits; total tells you how many activation bytes cross the memory interface per inference.

Start with one neuron that has four inputs. Its output before the activation function is w₁x₁ + w₂x₂ + w₃x₃ + w₄x₄. A processor evaluates that as four identical steps: multiply a weight by an input, add the product to a running total. Each step is one multiply-accumulate operation, and the whole of neural network inference is this step repeated a few hundred million times.

The definition is standard well beyond this course. The Wikipedia entry on the multiply-accumulate operation gives exactly a ← a + (b × c), and notes that IEEE 754-2008 defines a fused multiply-add that performs the whole step with a single rounding. Sze, Chen, Yang and Emer call the MAC "the fundamental component" of DNN processing in their survey of efficient DNN hardware. Counting MACs is therefore counting what the hardware executes, not an abstraction layered over it.

From one neuron to a matrix-vector product

A layer of m neurons that each read the same n inputs is a matrix-vector multiplication: an m × n weight matrix times an n-vector. Each of the m outputs needs n MACs, one per weight in its row, so the product costs m · n MACs. Notice that this is also the number of weights. In a matrix-vector product, every weight is touched exactly once.

Feed the layer a batch of inputs at once and the vector becomes a matrix. The slide renames the dimensions here: the input count becomes k and the letter n is reused for the batch, the number of columns of B. The result is a general matrix-matrix multiplication: A of size m × k times B of size k × n gives C of size m × n. Every one of the m · n cells of C is a dot product over the shared dimension k, so it costs k MACs, and the whole product costs m · n · k.

Why a whole network reduces to this

A Linear layer is literally a matrix-vector product at batch size one and a GEMM for a batch. Convolution looks different but is executed the same way: frameworks reorder the input patches into columns (the im2col trick described in the CS231n notes) so that a convolution becomes one large GEMM, and the MobileNets paper points out that this is why convolutions can ride on "highly optimized general matrix multiply (GEMM) functions". NVIDIA's performance guide states the cost of a GEMM as 2 · M · N · K floating point operations because "each FMA is 2 operations, a multiply and an add", which previews the FLOP conversion at the end of this part. Once you can count MACs for a matrix product, you can count them for every layer.

Take the first layer of AlexNet. One filter spans all 3 input channels with an 11 × 11 window, so it holds 3 × 11 × 11 = 363 weights. Producing a single output pixel is one dot product of those 363 weights against a patch of the image: 363 MACs. But the filter does not produce one pixel. It slides to every one of the 55 × 55 = 3,025 output positions, and there are 96 filters. So the layer costs 363 × 3,025 × 96 = 105,415,200 MACs from only 34,848 weights.

That example contains the whole rule. In a convolution layer every weight is applied once per output position, so the MAC count is the parameter count from part 02 multiplied by the output height and width:

The other rows of the slide's table follow from the same idea. A Linear layer has h_o = w_o = 1, so its MACs equal its parameters, c_o · c_i. A Grouped convolution splits the channels into g groups, so each filter sees only c_i / g input channels and both parameters and MACs are divided by g. A Depthwise convolution is the extreme case g = c_i = c_o: each filter sees exactly one channel, so c_i disappears from the formula entirely and only c_o · k_h · k_w weights remain, each still applied h_o · w_o times.

How a convolution slides, how padding and stride set the output size, and how groups partition channels were covered in lecture 02, part 02 and part 03. This part only counts, and it takes the kernel size and output shape of each layer as given.

Why this mirrors part 02

Goodfellow, Bengio and Courville explain parameter sharing in one sentence that is the whole story of this part: sharing "does not affect the runtime of forward propagation, it is still O(k × n), but it does further reduce the storage requirements of the model to k parameters". Convolution saves storage by reusing weights, and reusing weights is exactly what makes the arithmetic grow. That is why parts 02 and 04 are mirror images. The layers cheapest to store are the most expensive to run.

Worked example: the MobileNet block that makes depthwise worth it

The MobileNets paper replaces one standard 3 × 3 convolution with a depthwise 3 × 3 followed by a pointwise 1 × 1. Its equations 4 and 5 give the depthwise and separable costs, and dividing by the standard cost gives the ratio 1 / N + 1 / D_K², where N is the number of output channels and D_K the kernel size. The formulas in the table let you reproduce that with real numbers.

The calculator below lets you scrub every symbol in the table. Its FLOPs, OPs and hardware rows are explained in the last concept of this part; for now watch how parameters and MACs move apart as the output size grows.

Slide 25 hands you the eight weighted layers of AlexNet with their output shapes and an empty MACs column. Slide 26 fills it in, except for one layer hidden behind a question mark. Work through the column yourself before you read the answers. Every entry is the formula from the previous concept with the layer's numbers substituted, and the pooling layers contribute nothing because they hold no weights.

Put the parameters next to the MACs

The total is not the interesting number. The interesting number appears when you place the parameter counts from part 02 beside the MACs and add a column for their ratio, which by the previous concept is just h_o · w_o.

The five convolutions hold 2,332,704 parameters, under 4% of the network, yet perform 665,784,864 MACs, 92% of the compute. The three linear layers hold 58,621,952 parameters, 96% of the network, yet perform only 58,621,952 MACs, 8% of the compute. The last column explains it: each conv1 weight is used 3,025 times, each conv3 to conv5 weight 169 times, each linear weight once. Large spatial maps early in the network make convolution compute heavy. Large c_i · c_o products late in the network make the linear layers parameter heavy.

What this means for making a model faster

The inversion sets the strategy for the rest of the course. Pruning or quantizing the linear layers shrinks Model size dramatically, because that is where the weights are, but it barely changes the MAC count and therefore barely changes computation time. To make inference faster you must attack the convolutions: fewer channels, grouped or depthwise filters as in MobileNet, or a lower input resolution so that h_o · w_o shrinks in every layer at once. The pruning lecture that follows will keep returning to this table, and when you profile a model for the research project, report both columns, not one.

A MAC hides two arithmetic operations: a multiply and an add. When each is counted as one floating point operation, one MAC is two FLOPs, and AlexNet's 724,406,816 MACs become 1,448,813,632 FLOPs. The slide rounds that to 1.4G; the exact figure is 1.449G, and showing the factor of two matters more than the rounding.

The count on its own says nothing about time. To get time you need to know how fast a processor can drain it, and that is a different quantity with an almost identical name. FLOPS, with a capital S, is floating point operations per second: a throughput that belongs to the hardware. FLOPs, with a lowercase s, is a plural count that belongs to the model. TOP500 ranks supercomputers by exactly this rate, reporting Linpack results as Rmax and Rpeak in Flop/s, and NVIDIA's GPU performance background guide writes the time a math-limited kernel takes as its number of operations divided by the processor's math bandwidth, which is FLOPs over FLOPS.

This is the computation time term of the latency model on slide 8, where the numerator is a network specification and the denominator a hardware specification. Run AlexNet's count through a few devices and the scale of the answer changes by orders of magnitude while the model does not change at all.

OPs: the same count for any data type

The F in FLOP assumes floating point arithmetic. A quantized network, the kind you deploy on a microcontroller or an INT8 accelerator, stores weights and activations at an Bit width of 8 bits and multiplies integers. Those are not floating point operations, but they are still operations, and there are still two per MAC. The lecture therefore generalizes: an operation (OP) is one multiply or add of any data type, AlexNet is 724M × 2 = 1.4G OPs, and OPS is operations per second. The layer compute calculator in the second concept of this part uses OPS in its hardware box for exactly this reason: the formula for time does not care whether the operands were floats.

Slide 29 lists the ten works the lecture draws on. Six were used directly in this lecture's counting and examples; the other four appear here as context for the architectures and normalization layers of lecture 02 and the pruning lectures ahead.

Used in this lecture

• Dumoulin and Visin, Convolution arithmetic (GitHub repository and arXiv 1603.07285): the output-shape arithmetic behind every h_o × w_o in this part.
• Stanford CS231n, lecture 5, Image Classification with CNNs: parameter sharing, the output-size formula and im2col.
• Krizhevsky, Sutskever and Hinton, ImageNet Classification with Deep Convolutional Neural Networks, NeurIPS 2012: the AlexNet layers counted on slides 25 and 26.
• Sandler et al., MobileNetV2: Inverted Residuals and Linear Bottlenecks, CVPR 2018: the depthwise design that cuts both parameters and MACs, and the "MAdd" convention.
• Deng, Li, Han, Shi and Xie, Model Compression and Hardware Acceleration for Neural Networks: A Comprehensive Survey, Proceedings of the IEEE 108(4), 2020: the wider map of compression techniques that these metrics evaluate.
• Song Han, MIT 6.5940 TinyML and Efficient Deep Learning Computing, lecture 2: the direct source of slides 23 to 28.

Context for other lectures

• Ioffe and Szegedy, Batch Normalization, ICML 2015.
• Wu and He, Group Normalization, ECCV 2018.
• Simonyan and Zisserman, Very Deep Convolutional Networks for Large-Scale Image Recognition, ICLR 2015.
• He, Zhang, Ren and Sun, Deep Residual Learning for Image Recognition, CVPR 2016.