Activations and peak memory

Take a real target: an STM32F746 microcontroller has a Cortex-M7 core, 1 MB of flash and 320 kB of SRAM. It has no DRAM and no operating system; MCUNet describes such boards as bare-metal devices (Lin et al., 2020). Now take two ImageNet classifiers that both reach about 70 percent top-1 accuracy with every value stored as an 8-bit integer. ResNet-18 needs 11.2 MB for its weights. MobileNetV2 at width 0.75 needs about 2.5 MB as MCUNet's Figure 8 labels the bar, a 4.6x reduction on the slide (strictly 11.2 / 2.5 ≈ 4.5; the paper rounds to 4.6x). If parameters were the memory story, the second model would be the obvious choice.

Then look at the other pair of bars. The largest amount of activation memory that ResNet-18 needs at any moment is 0.9 MB. For MobileNetV2-0.75 it is 1.7 MB. The efficient model is 1.8x worse on peak activations, and the MCUNet paper says so plainly: MobileNetV2 "reduces the model size by 4.6x compared to ResNet-18, but the peak activation size increases by 1.8x, making it even more difficult to fit the SRAM on microcontrollers" (Lin et al., 2020). Neither model fits 320 kB. The memory bottleneck moved without anyone shrinking it.

Two budgets, and the smaller one belongs to activations

The reason the two bars behave differently is that they are paid from different memories. Weights are read-only: they are written once when the firmware is flashed and only read during inference, so they live in flash, the larger and cheaper memory. Activations are produced by one layer and consumed by the next on every single inference, so they must sit in memory that can be written and read at speed, which on a microcontroller is the on-chip SRAM. MCUNet puts it in one sentence: "SRAM constrains the activation size (read&write); Flash constrains the model size (read-only)" (Lin et al., 2020). The model size from part 02 answers the flash question. The peak activation answers the SRAM question, and SRAM is the scarcer of the two by a factor of two to four on every board in MCUNet's list (the STM32F746 has about 3x more flash than SRAM).

This also explains why bit width alone does not rescue MobileNetV2. The slide already assumes 8-bit integers for both weights and activations. Even after that 4x shrink from 32-bit floats, the peak sits at 1.7 MB. MCUNet's Table 1 reports 1.7 MB for full-width int8 MobileNetV2 (6.8 MB in fp32), 5.3x over the 320 kB of an STM32F746, not counting im2col or other runtime buffers. Quantization scales both bars by the same factor; it does not change which bar is the problem.

The MCUNet row comes from Figure 8 of the same paper and is there to show what solving the right problem looks like: its designers shrank the peak activation below 0.5 MB rather than chasing a smaller parameter count, and only then did an ImageNet model fit a 512 kB board. The 4.6x model is not an efficient model for this class of device; it is an efficient model for a phone, where DRAM is plentiful and flash is not the constraint.

Why MobileNetV2 in particular: an imbalanced profile

The right-hand chart explains where MobileNetV2's peak comes from. Plot the memory that each of its eighteen blocks needs and the profile is wildly uneven: block 2 needs 1372 kB, blocks 0, 1, 3 and 4 sit between roughly 500 kB and 600 kB, and the remaining thirteen blocks mostly need 40 kB to 170 kB. Draw the 256 kB constraint of a typical microcontroller and exactly five blocks cross it. MCUNetV2, the follow-up paper that this chart comes from, states it directly: "The first 5 blocks have large peak memory, exceeding the memory constraints of MCUs, while the remaining 13 blocks easily fit 256kB memory constraints. The third block has 8x larger memory usage than the rest of the network" (Lin et al., 2021).

The two charts on the slide come from different papers with different memory accounting: the left bars are MCUNet's (2020) figures for MobileNetV2-0.75, while the right chart is MCUNetV2's analytic input-plus-output count for full-width MobileNetV2 in int8, so the numbers are not directly comparable. Its block 2 peak is 16×112×112 + 96×112×112 = 1,404,928 B, which is 1372 kB in 1024-byte kilobytes.

The cause is the shape of MobileNetV2's inverted residual block. Each block first expands the channel count about six times with a 1×1 convolution, applies a depthwise 3×3, and projects back down. The depthwise step is what makes the block cheap in weights and MACs, exactly as part 02 computed. But the expanded tensor exists at full spatial resolution, and in the early blocks that resolution is 112×112 or 56×56. Six times the channels at a large H×W gives a huge activation in a block that owns almost no parameters. MCUNet's per-block peak memory analysis (Figure 11) measures the same imbalance on a MobileNetV2 scaled to 0.3x width to fit 320 kB: even then one block has 2.2x the peak activation of the average block (Lin et al., 2020). A model can be light in weights and heavy in activations at the same time, and this architecture is the canonical example.

Inference keeps a few activations alive; training keeps all of them. Switch to 32-bit floats and a training step of ResNet-50. Its weights take 102 MB. Its activations, for one mini-batch, take 707 MB, about 7x more. Now swap in MobileNetV2 at width 1.4: the weights fall to about 24 MB, a 4.3x reduction, but the activations fall only to about 626 MB, a 1.1x reduction. The TinyTL paper that produced these numbers concludes: "It is the activation that bottlenecks the training memory, not the parameters" (Cai et al., 2020).

The mechanism is backpropagation itself. Write one linear layer as a_(i+1) = a_i W + b, where a_i is the layer's input activation. To update the weight you need the gradient of the loss with respect to W, and that gradient is the outer product of the input activation with the gradient flowing back from the next layer.

The forward pass computes a_1, then a_2, and so on up to the loss. The backward pass returns in reverse order, and when it reaches layer i it must still have a_i. So every layer's input has to be held from the moment it is produced until the backward pass comes back to it. Goodfellow, Bengio and Courville make the same point for a multilayer perceptron: "we need to store the input to the nonlinearity of the hidden layer. This value is stored from the time it is computed until the backward pass has returned to the same point," and the cost per layer is O(m·n_h) for a mini-batch of m examples and n_h hidden units (Goodfellow et al., 2016, section 6.5.7). Depth multiplies that cost by the number of layers, and the batch size multiplies it again.

The formula also points to the cure that TinyTL proposes and that this course returns to when it reaches on-device learning. The bias gradient in equation 2 needs no stored activation at all. If you freeze the weights and update only biases (plus a small added module), the a_i tensors never have to be kept, and the 707 MB term collapses. That is a memory argument, not an accuracy argument, and it is only visible once you have separated activations from parameters as this slide does. It matters doubly on a microcontroller, because TinyTL notes that DRAM access consumes two orders of magnitude more energy than on-chip SRAM access (Cai et al., 2020), echoing the Horowitz numbers from part 01.

Plot, layer by layer, how much memory the activations and the weights of an efficient network take, and a clear shape appears. For MCUNet, a network designed for microcontrollers, the first six layers need 50 kB to 75 kB of activation memory and almost no weight memory, and activation stays the dominant term, mostly 20 kB to 55 kB, out to about layer 17. Around layers 18 to 30 both are small, under about 15 kB. From layer 32 onward the weight memory climbs to 35 kB to 75 kB while activations stay near 5 kB to 10 kB. The total traces a U: expensive at both ends, cheap in the middle, for two different reasons. The chart comes from MCUNet's on-device training work and shows the memory needed to update each layer, which is the stored input activation plus the weight and its gradient. The same geometry governs inference, as the AlexNet arithmetic below shows.

The two counting rules from part 02 predict this. An activation tensor holds C × H × W values. A convolution weight tensor holds C_o × C_i × k_h × k_w values. Now follow a CNN from input to output. At every downsampling stage the spatial size halves in each direction, so H × W drops by 4x, while the channel count typically only doubles. The activation count C × H × W therefore falls by about 2x per stage. The weight count contains C_o × C_i, which grows by about 4x when channels double and does not care about H and W at all. MCUNetV2 states the consequence: "the memory bottleneck tends to appear at the early stage of the network" because resolution shrinks faster than channels grow (Lin et al., 2021).

The same U in numbers you already computed

You do not have to trust the chart. Take the AlexNet layers from part 02, put each layer's output activation count next to its weight count, and the profile reproduces itself from arithmetic.

The first convolution has 8x more activations than weights: a 55×55 map with 96 channels is large, and an 11×11 filter over only 3 input channels is small. By conv3 the map is 13×13 and the weights are already 14x the activations. The fully connected layers are the extreme: fc6 emits 4,096 values and owns 37.7M weights. Nothing about AlexNet was designed to show this; it is the geometry of every CNN.

What each end of the U decides

• The early, activation-heavy stage sets the SRAM peak. That is why MobileNetV2's problem in the previous concept was in blocks 0 to 4, and why MCUNetV2 runs only that stage in patches.
• The late, weight-heavy stage sets the flash budget and most of the parameter count. Pruning and quantization pay off most there.
• The cheap middle is where on-device training is affordable: updating a middle layer needs neither a large stored activation nor a large weight gradient. This band is exactly what the training chart above measures, and the paper it comes from exploits exactly that band (Lin et al., 2022).

Now make the peak a number you can compute. AlexNet takes a 3×224×224 image: 150,528 values. Its first convolution emits 96×55×55 = 290,400 values. While that convolution runs, both tensors must exist, because the layer is still reading the image as it writes the output: 150,528 + 290,400 = 440,928. When the first pooling layer runs, the image is no longer needed and can be freed, but the convolution output (290,400) and the pooling output (69,984) are both alive: 360,384. Memory rises and falls one layer at a time.

Two counts fall out of this picture. The total number of activations is the sum over the input and every layer output, which for AlexNet is 932,264. The peak number of activations is the largest number alive at any one moment. For a plain chain of layers executed one at a time, that is the largest input plus output pair. MCUNetV2 gives the rule and the reason: "the memory required for a layer is the sum of input and output activation (since weights can be partially fetched from Flash)", and the runtime "allocates the input and output activation buffer in SRAM, and releases the input buffer after the whole layer computation is finished" (Lin et al., 2021).

The approximation sign carries four assumptions, and an exam answer should name at least one of them. The network runs layer by layer, with no two layers in flight together. Weights are streamed from flash and are not counted in SRAM. There are no residual branches; if there are, MCUNetV2 says to add the memory of all branches alive at the same time, counting a shared input once. And there are no in-place tricks: MCUNet shows that a depthwise layer can overwrite its input channel by channel, needing N + 1 instead of 2N values (Lin et al., 2020). Real runtimes also need scratch buffers for im2col or accumulation, which push the true peak slightly above the formula.

Sweep through AlexNet yourself before reading the worked example. Watch which two bars are highlighted, how the running peak stops moving after the first layer, and how the bit width and the SRAM preset change whether the same 440,928 values fit.

The same arithmetic explains why MobileNetV2 failed in the first concept. Its first convolution maps a 3×224×224 image to 32×112×112: 150,528 + 401,408 = 551,936 values, which MCUNetV2 quotes as "539kB even when quantized in int8" (using 1024-byte kilobytes). A 224 input through a standard stem can never run layer by layer on a 256 kB board, whatever happens to the weights afterwards.

Finally, connect the counts back to latency. Part 01 approximated memory time as the bytes of activations and weights moved divided by the memory bandwidth. The per-layer input plus output is the activation traffic of that layer, so the same column that gives the peak also feeds the latency model. Peak tells you whether the model fits; total tells you how many activation bytes cross the memory interface per inference.