Why efficient deep learning

Open the camera on your phone and it finds faces before you press anything. Ask a speaker a question and a language model answers. Paste a screenshot into a chatbot and it reads the text and the picture together. Vision, language and multimodal tasks are all served today by deep networks, and every one of those answers costs billions of arithmetic operations. Classifying a single image with ResNet-50 takes about 3.9 billion multiply-accumulate operations (Sze et al., 2017), and that is a model from 2015 doing the easiest of the three tasks.

That is the whole tension of this course, and slide 2 states it in one line. Machine learning (ML) and deep learning are applied everywhere, they are computationally expensive, and the challenge is to make them lightweight and fast without losing accuracy. Every technique you will learn, from pruning and quantization to architecture search, hardware accelerators and tiny Inference engines, is one answer to that single question. It helps to write the question the way an optimization course would: keep accuracy as high as possible while staying under whatever budget the target device imposes.

The budgets are what change from device to device. A cloud GPU has a generous C and M; a phone has a tight Latency T and a battery; a sensor node has a memory M measured in kilobytes. The accuracy you can afford depends on which row of that constraint list binds, and part 03 puts real numbers on each row.

Three pillars, one lens for the whole course

Slide 3 gives you the lens to organize everything that follows: deep learning advances when three things improve together. Its three pillars are the algorithm (the model and how it is trained), the hardware it runs on, and the data it learns from. None of them is enough alone. The 2012 breakthrough on ImageNet, which the next concept quantifies, needed all three at once: a new deep convolutional architecture, two GPUs able to run it for a week, and a dataset of about 1.2 million labeled images (Krizhevsky et al., 2012).

The same three words tell you where efficiency work happens. Compressing, quantizing and searching for architectures are algorithm-pillar work: the model changes, the chip does not. Building accelerators, from data-center GPUs down to the Hexagon DSP and the Neural Engine, is hardware-pillar work: the chip changes so that the same model runs faster or cooler. The deck returns to this exact slide as slide 17 to open the hardware section, so treat slide 3 as the introduction of the framework and slide 17 as the pivot to its second pillar. The data pillar makes its own appearance on slide 8, where data that cannot leave a device forces learning to happen on the device.

Why efficiency matters beyond speed

Speed is the obvious reason to want small models, but the exam will ask for more, and the deck gives you at least four. Privacy: a model that fits on the device can process a face or a voice without uploading it. Cost: training and serving large models is billed in GPU hours and, increasingly, in carbon. The Once-for-All paper notes that training a separate specialized network for every deployment target emits as much CO2 as five cars over their lifetimes (Cai et al., 2020). Connectivity: sensors in fields, mines and rural clinics may have no network at all, and MCUNet argues that on-device processing extends AI to areas without Internet access (Lin et al., 2020). Energy: a battery-powered node has a milliwatt budget, and every unnecessary multiply drains it.

• Privacy: raw data stays where it was collected.
• Cost: fewer GPU hours, smaller cloud bills, less carbon.
• Connectivity: the model works where the network does not.
• Energy: milliwatt devices need models sized for milliwatts.

Look at the left panel of slide 4 before the bar chart. It shows six test images (a crop of the eight in the AlexNet paper) with the five classes AlexNet ranked highest for each. For the mushroom photo the top guess is "agaric" and the true label "mushroom" sits second. For the cherry photo the top guess is "dalmatian", followed by grape, elderberry, a bull terrier and currant; "cherry" is nowhere in the list. Under one scoring rule the mushroom counts as correct and the cherry does not; under another rule both are wrong. Which rule you use decides every number on the right of the slide.

The rule the contest used is Top-5 error rate. The ImageNet challenge (ILSVRC) let each algorithm return five labels per image and counted the image as correct if any one of them matched the single ground-truth label; the error is the fraction of test images with no match (Russakovsky et al., 2015, Eq. 1). The organizers allowed five guesses because ImageNet photos often contain several objects but carry only one label, and a system that names a visible second object should not be punished. From 2012 the contest reported only top-5 error, since all their measures ranked entries the same way.

The other rule, Top-1 accuracy, keeps only the single highest-scoring class and counts the fraction of images where it matches. It is stricter, so the same model always scores worse on top-1 than on top-5. Slides 5 and 6 use top-1 accuracy; slide 4 uses top-5 error. Keeping the two apart is the single most common slip in reading this part of the deck.

Eight years of the winning entry

With the metric fixed, the bar chart reads as a history. The winning top-5 error went from 28.2% in 2010 and 25.8% in 2011 to 16.4% in 2012, when SuperVision, the entry now called AlexNet, was the first deep convolutional network to win. The error kept falling as the networks deepened: Clarifai at 11.7% in 2013, GoogLeNet at 6.7% in 2014, ResNet at 3.57% in 2015 with an ensemble of residual networks up to 152 layers deep (He et al., 2016), Trimps-Soushen at 2.99% in 2016 and SENet at 2.25% in 2017 (Hu et al., 2018). The slide rounds those to one decimal.

What "super-human" actually means here

The human bar is not an average person. It is one expert annotator who trained on 500 images and then labeled 1500 test images at roughly one per minute, reaching 5.1% top-5 error; a second annotator with less training scored 12.0%, and GoogLeNet scored 6.8% on the same sample (Russakovsky et al., 2015, Sec. 6.4). So "DNNs achieve super-human accuracy" means that from 2015 the winning network beats a well-trained, patient human on the top-5 rule for 1000 fine-grained classes, many of them dog breeds. It says nothing about general vision, and it says nothing about top-1.

Take two models from the same family. ResNet-50 needs about 3.8 billion multiply-adds per image and ResNet-101 about 7.6 billion, twice as many (He et al., 2016, Table 1). On slide 5 they sit at about 75.3% and 76.4% top-1 accuracy. Doubling the work bought about one percentage point. That one pair is the message of the slide: high accuracy comes at the cost of high computation, and the exchange rate gets worse as you climb.

To read the chart you need its currency. The horizontal axis counts MACs, multiply-accumulate operations: one multiplication of an input by a weight, immediately added into a running sum. Sze and colleagues call the MAC "the fundamental component" of both convolutional and fully connected layers (Sze et al., 2017). Counting MACs gives a hardware-neutral cost: the number is a property of the model and the input size, not of the chip, unlike latency or energy which change with every device. A convolutional layer's count follows directly from its shape.

How to read a bubble chart

Three quantities share the chart. Horizontal position is cost in billions of MACs, so left is cheaper. Vertical position is Top-1 accuracy on ImageNet, so up is better. Bubble area is Model size, the number of parameters, from 2 M to 64 M according to the legend. Parameters measure storage (how many bytes the weights occupy) and MACs measure work (how many operations each input triggers), and the two are related but not the same. Circles are handcrafted architectures and stars are found by AutoML, which becomes important on the next slide. The goal region is the upper left: high accuracy at low cost.

Diminishing returns

Trace the chart from the cheapest model upward and the exchange rate collapses. For each step, divide the accuracy gained by the extra MACs spent. MobileNetV1 is the cheap starting point at 70.6% for 0.57 B MACs (Howard et al., 2017); every other number below is read from the chart.

The marginal value of computation falls almost fivefold across the chart, from 1.41 to 0.30 points per billion, and the worst trade is inside a single family. Notice also that Xception is itself dominated: DPN-92 is more accurate (79.3%) with fewer MACs (6.5 B).

Now put the stars back on the chart. Once-for-All (OFA) reaches 80.0% top-1 with 595 M MACs (Cai et al., 2020). Xception, the most expensive handcrafted model on the previous slide and within 0.3 points of the most accurate (DPN-92), needed about 8.4 B for 79.0%. Divide and you get 8400 / 595 = 14.1: the same accuracy, or one point more, for one fourteenth of the computation. That is the dashed arrow at the top of slide 6, and it is drawn straight from Figure 2 of the Once-for-All paper.

The MobileNetV1 row makes the point from the other side: at essentially the same budget, the searched model is 9.4 points more accurate. The stars are all products of Neural architecture search (NAS): rather than a person choosing kernel sizes, widths and depths, an algorithm searches over them under an explicit MAC or latency budget. EfficientNet, NASNet-A, MobileNetV3, ProxylessNAS, AmoebaNet, PNASNet and DARTS all sit inside the boxed region below about 2.5 B MACs, next to two efficient handcrafted models, MobileNetV2 and ShuffleNet. Once-for-All adds one more idea: train a single large network once, then cut specialized sub-networks out of it for each device, which is what makes the 80.0% at 595 M affordable to produce as well as to run.

The frontier and why it moves left

The two arrows on the slide, "the higher the better" and "the lower the better", state the accuracy-efficiency trade-off as a partial order. Model A dominates model B if A is at least as accurate and at most as costly, with one of the two strictly better. The models nobody dominates form the Pareto frontier, the staircase you saw in the last visual. Every point below or to the right of it is a worse deal than something already available. Architecture search did not just add good models; it moved the entire staircase left, so that the best accuracy at 0.6 B MACs today exceeds the best at 8 B a few years earlier.

The slide names the payoff in two lines. Efficient deep learning enables daily-life applications on mobile phones, which is why the Once-for-All paper defines a "mobile setting" as anything under 600 M MACs and reports being the first to reach 80% inside it. And it enables private on-device Inference and training, the theme of the last concept in this part. Both follow from the same fact: a model that costs fourteen times less can live on a device that has fourteen times less to spend.

Slide 7 shows two photographs of a small camera board on a desk. In one it reports whether the person in front of it wears a face mask; in the other it detects people in an office. There is no phone, no laptop and no network cable in the picture. The board is an OpenMV Cam, whose processor is an ARM Cortex-M7 Microcontroller (MCU), and the software is MCUNet, a framework from the same lab as Once-for-All (Lin et al., 2020).

This is TinyML: machine learning on devices with kilobytes of memory. MCUNet's reference target, the STM32F746, has 320 kB of SRAM and 1 MB of flash, and its Table 1 lines that up against a phone and a cloud GPU. The OpenMV Cam H7 in the photos is a somewhat larger sibling: STM32H743 at 480 MHz, with 1 MB of RAM in total, of which MCUNet's experiments use the 512 kB main SRAM block, and 2 MB flash (OpenMV). Both are bare-metal devices with no operating system and no DRAM. Slide 23 returns to these exact numbers as the tiny AI column of the hardware tiers. Its cloud and mobile columns are updated to an 80 GB GPU and a 256 GB phone, so expect those two cells to differ from the table below while the microcontroller column stays the same.

Why nothing off the shelf fits

The paper gives three failures. ResNet-50's weight storage is about 102 MB, 100x the flash. MobileNetV2 needs about 6.8 MB of peak activation memory, 22x the SRAM (the paper rounds up). Even the int8-quantized MobileNetV2 peaks at 1.7 MB, still 5.3x over, and that is before the runtime's own buffers. Reaching the board required designing the network and the inference engine together: TinyNAS searches architectures under the exact memory limits and TinyEngine executes them without interpreter overhead, and MCUNet as a whole uses 3.5x less SRAM and 5.7x less flash than int8 MobileNetV2 and ResNet-18 at similar accuracy. The result is 70.7% ImageNet top-1 on an STM32H743 (the OpenMV H7 chip, which MCUNet evaluates under a 512 kB SRAM and 2 MB flash budget, although the board has 1 MB of SRAM in total), the first time that bar was reached on an off-the-shelf microcontroller, and about 62% inside the 320 kB / 1 MB budget of the F746 (Lin et al., 2020, Table 4, 4-bit).

The reasons to bother are the same four from the first concept, now at their most extreme. MCUNet points out that there are around 250 billion microcontrollers in the world, that processing raw sensor data at the source protects privacy because nothing is transmitted, and that inference on the node works in places without Internet access (Lin et al., 2020). A mask detector on a camera board costs a few dollars, runs on milliwatts and never uploads a face.

A smart speaker slowly gets better at recognizing the voices in one house. A phone learns its owner's face after the first enrollment. Both need data that arrives after the product shipped, and both kinds of data (a voice, a face) are exactly what users least want uploaded. The figure on slide 8 draws the consequence: new and sensitive data flows from the user to the Edge device, the arrow from the device to the cloud is crossed out for privacy, and a small loop above the device shows the model updating itself where the data is.

That loop is On-device training (learning). The slide lists four benefits. Better privacy, because nothing leaves the device. Lower cost, because there is no upload and no cloud training bill. Customization, because the model adapts to this user, this room, this camera angle. Life-long learning, because the adaptation never has to stop; AI systems, as the slide puts it, need to continually adapt to new data collected locally. Lin and colleagues summarize the goal as letting users benefit from customized models without transferring their data to the cloud (Lin et al., 2022).

Why training is so much more expensive than inference

The third line of the slide is the catch: training is more expensive than Inference, hard to fit in edge memory, and limited by power. All three follow from how backpropagation works. Inference is one forward pass through the layers, and each layer's activations can be discarded as soon as the next layer has consumed them. Training runs the same forward pass, then a backward pass that costs about twice as much: propagating the gradient through each layer is one multiply by the transposed weights, which Goodfellow, Bengio and Courville note has the same cost as the forward multiply (Goodfellow et al., 2016, Sec. 6.5.7), and computing the weight gradient is a second multiply of the same size. So one training step is roughly three times the arithmetic of one inference pass, the convention behind the 6N versus 2N per-token estimate used in scaling studies (Kaplan et al., 2020). The backward pass also needs the activations that the forward pass produced. Goodfellow and colleagues put it directly: the input to each hidden nonlinearity must be stored from the moment it is computed until the backward pass has returned to the same point, so memory grows with the number of examples in the batch and the width of the layers. On top of that, the optimizer keeps its own state: a momentum buffer, or two moment estimates for Adam, each the size of the model.

Power is the last constraint. A training step roughly triples the arithmetic of every example, and a device that runs on a battery or a solar cell cannot run it continuously. So on-device learning is not merely inference plus a little bookkeeping; it is a harder problem than the one MCUNet solved, and the course treats it as its own topic.

Picture a head-mounted camera looking at a door, a small label reading id 27, 52 cm, and a green mask painted over one region. A single click on the door produced that mask. That is what promptable means. The user supplies a hint, a point, a box or a phrase, and the model returns the object the hint refers to. The Segment Anything Model (SAM) from Kirillov et al. was trained on SA-1B, over one billion masks on 11 million images, so it can do this for objects it has never been told about.

The architecture matters more than the demo. SAM is three blocks. An image encoder, a ViT-H pre-trained with masked autoencoding, reads the image once and produces an image embedding. A prompt encoder turns sparse prompts (points, boxes, text) and dense prompts (masks) into prompt embeddings. A lightweight mask decoder combines the two and outputs a valid mask. The paper is explicit about the asymmetry: the image encoder runs once per image, and given a precomputed embedding, the prompt encoder and mask decoder run in a web browser, on a CPU, in about 50 ms. Encode once, expensively; prompt many times, cheaply.

Attack the block that dominates

Once you see that the encoder holds 2973 GMACs and the rest is a 50 ms CPU job, the efficiency strategy writes itself. EfficientViT-SAM (Zhang, Cai and Han) keeps SAM's prompt encoder and mask decoder unchanged and replaces only the image encoder with EfficientViT, the multi-scale linear attention backbone from Cai et al. (ICCV 2023). Training happens in two stages: first the new encoder is distilled from SAM's ViT-H encoder as the teacher, then the whole model is trained end to end on SA-1B. Nothing else in the pipeline moves.

The fair way to judge the swap is accuracy against speed on the same hardware: zero-shot COCO mAP against A100 throughput with TensorRT in FP16. Zero-shot here has a precise meaning: the models were never trained on COCO, and they are evaluated with boxes from a ViTDet detector as prompts. Two speedups get quoted, and they belong to different models. The 16.5x figure is EfficientViT-SAM-XL1, which is faster and more accurate than SAM (47.8 versus 46.5 mAP). The 48.9x headline belongs to EfficientViT-SAM-L2, which matches SAM at 46.6 mAP while running 538 images per second instead of 11. The full table from the paper makes the families visible.

The table carries a second lesson. EfficientSAM has 25.3 M encoder parameters, eight times fewer than XL1's 203.3 M, and yet the two run at the same throughput (183 versus 182 img/s), with EfficientSAM losing 3.4 mAP. Model size is about memory; speed is about MACs, memory traffic and kernels. Keep the three apart.

Type "teddy bears working as mad scientists", "a bowl of soup as a planet on a 1960s poster" or "an astronaut riding a horse on Mars" into Midjourney and a finished picture comes back. A diffusion model produced each one from a sentence, by starting from noise and removing a little of it at every step until an image appears. The results look effortless. The bill is not. Emad Mostaque, then CEO of Stability AI, said that training Stable Diffusion used 256 A100 GPUs for about 150,000 GPU-hours at a cost of about $600,000, the figure the lecture uses as Stable Diffusion's training cost.

Why parameters are not the cost

Training is paid once. Inference is paid on every request, so compare what one forward pass costs in MACs across four very different models. The natural expectation is that the biggest model costs the most. The numbers say otherwise, and the reason is the most important idea in this concept.

FLUX.1 has 12 B parameters and CogVideoX has 5.6 B, yet CogVideoX needs 8.9x the MACs per step. Llama 3 8B has more parameters than ViT-H by a factor of thirteen but costs eleven times more, not thirteen, and FLUX.1 with 1.5x the parameters of Llama 3 costs 3.3x as much. Parameters alone predict none of these ratios. What predicts them is how many tokens each model pushes through those parameters. A dense transformer spends roughly one multiply-accumulate (two FLOPs) per parameter per token in its linear layers, and on top of that attention compares every token with every other token.

Now the numbers make sense. ViT-H sees one 518 × 518 image cut into 14-pixel patches, 37 × 37 = 1,369 tokens: 632 M × 1,370 ≈ 0.87 T for the linear layers, and the attention term adds the rest of the 1.02 T. Llama 3 processes about 1,500 tokens. FLUX.1 works on the latent of a 1K × 1K image, about 4,000 tokens. CogVideoX compresses 49 frames of 480 × 720 video into 13 latent frames and denoises them all at once, about 17,500 tokens, roughly four times FLUX.1. Because the attention term grows with the square of that count, the bill grows far more than four times, and the smaller parameter count cannot save it. Read the linear term as the weights each token actually passes through, not the headline parameter count: for FLUX.1 the naive 12 B × 4,096 ≈ 49 T overshoots the 37.2 T bar because image and text tokens use separate weights in its double-stream blocks. One accounting rule explains the ordering of the chart: compute is parameters times tokens, plus attention.

One more multiplier hides in the words "single step" under the FLUX.1 and CogVideoX bars. A diffusion model repeats its forward pass once per denoising step. With a 50-step sampler, one 1K FLUX.1 image costs 37.2 T × 50 = 1,860 T MACs, and a CogVideoX clip costs 332 T × 50 = 16,600 T. A classifier pays its bar once; a generator pays it fifty times.

Resolution is worse than quadratic

Now hold the model fixed and vary only the image side length for FLUX.1, from 1024 to 4096. Going four times wider makes the image sixteen times larger in pixels, so a quadratic law would predict sixteen times the MACs. From the slide's left chart, one step at 1K x 1K costs 37.2 T, and the curve on the right ends near 1,950 T at 4096: a factor of about 52, far above the 16x a quadratic law predicts. This is what the lecture calls super-quadratic scaling, and the formula above explains it. Tokens grow with pixels (16x), the linear term grows with tokens (16x), and the attention term grows with tokens squared (256x). The blend lands at about 52x.

Ask SDXL for a "Romantic painting of a ship sailing in a stormy sea" at high resolution. One GPU paints one ship in 12.3 s. The obvious way to go faster is to cut the canvas into four vertical strips, hand each strip to its own GPU, and let them work without talking. The result is 3.9x faster and wrong: four strips, four ships, four different skies. Each patch generated its own picture because no patch could see the others.

DistriFusion (Li et al., CVPR 2024) keeps the split and restores the context. The trick rests on a property of diffusion: consecutive denoising steps produce similar activations, because each step removes only a little noise. So a patch can attend to its neighbours' activations from the previous step instead of waiting for the current one. The paper calls this displaced patch parallelism. The first few denoising steps (the warm-up: the first step plus four more in the paper) run synchronous patch parallelism so everyone starts from the same picture; after that, communication runs asynchronously and hides behind computation. No retraining is needed. The latency drops to 4.16 s, 3.0x faster than one GPU, and the ship is one ship.

The pattern is general in distributed systems: splitting work across machines helps only when the compute you save exceeds the communication you add. Here the saving is large because a high-resolution diffusion step is enormous (MACs in the hundreds of trillions), and the communication is hidden behind it. The paper also runs a scaling study on a larger image, 3840 × 3840, shown in the table below. Read down the efficiency column: every added GPU still helps, but each device contributes less than the last, because the communication grows while the per-device work shrinks.

A 2015 Toyota Prius V with a roof-mounted Velodyne LiDAR drives a wooded test track. A network reads raw 3D point clouds and produces steering. A car does not care how elegant the network is; it cares whether the answer arrives before the deadline the system sets. Three measurements tell the story: MinkowskiNet at 5 fps is "too slow to drive", the same network with optimised kernels reaches 18 fps, and Fast-LiDARNet reaches 47 fps, above the 30 fps line the slide marks as real time.

The most useful way to read fps is to invert it into a frame time. A stream at 30 fps gives you 33.3 ms per frame, and everything the model does must fit inside that budget.

The paper (Liu, Amini, Zhu, Karaman, Han and Rus, ICRA 2021) is explicit about the two levers. Their optimised sparse convolution kernel gives a 3.6x acceleration, and their hardware-aware model design reduces latency by another 2.6x. The two multiply: 3.6 × 2.6 ≈ 9.4, which is 47 / 5. The first lever is pure systems work: the same network, the same accuracy, better code for the sparse operations point clouds need. The second is algorithmic: a smaller architecture chosen with the hardware in mind. This pairing is what the lecture means by algorithm and system co-design. Neither lever alone reaches the line. Kernels take the network from 200 ms to 55.6 ms and still miss by 22 ms; the redesign brings it to 21.3 ms.

MT-NLG, the largest model on the lecture's chart, has 530 billion parameters. Store each one in FP16, two bytes, and the weights alone occupy 530 × 10⁹ × 2 = 1,060 GB. The largest GPU of that period, an A100 with 80 GB, holds 40 billion FP16 parameters. The gap between the two is 1,060 / 80 = 13.25, so 14 GPUs of weights before a single activation is computed.

To compare a model with a chip, put both on one log axis of model size by converting GPU bytes into a parameter count at FP16, which is what the lecture's assumption "data are FP16" means. 16 GB becomes 8 B parameters, 32 GB becomes 16 B, 80 GB becomes 40 B. On that shared axis the two curves tell the story of the large language model era: model size climbs from 0.05 B to 530 B, about 10,000x in roughly four years, while GPU memory climbs from 16 GB to 80 GB, 5x. Models have outgrown the chips they run on, and the distance keeps widening.

Parameters to bytes to GPUs

Every memory question in this course starts with one multiplication. Parameters times bytes per parameter gives the weight memory; divide by the memory of one device and round up to get the device count. The bytes per parameter depend on the numeric format: 4 for FP32, 2 for FP16 or BF16, 1 for INT8, 0.5 for INT4. Quantisation, which later lectures cover in depth, is simply the art of moving down that list without losing accuracy.

Try Llama 3 8B in the calculator. In FP16 it needs 16 GB, four times the 4 GB of RAM slide 23 gives a phone. Slide 23 lists that 4 GB as activation memory and 256 GB as weight storage; in practice LLM inference loads the weights into that same RAM, so the 4 GB is the binding limit. In INT4 the model needs 4 GB, exactly at the line, with nothing left for activations. Now try the 2017 Transformer at 0.05 B against a microcontroller's 1 MB of weight storage: 100 MB in FP16, a hundred times over. The gap between what models need and what edge hardware has is the subject of part 03.

Why run them at the edge anyway

If LLMs barely fit in a rack of cloud GPUs, why does the lecture insist on running them on laptops, cars and robots? Three reasons, each one a research direction in this course.

1. Copilot services, such as code completion, office assistants and in-game chat, running locally on the machine that uses them. Latency drops, and the service works in a car or a robot far from any data center.
2. Resource-constrained, low-power devices that are not always connected. An edge LLM must work when the link is slow or absent.
3. Privacy of user data. Prompts contain code, documents and conversations, and a local model never uploads them.

This is the design space for an on-device assistant, and it is why the rest of the course is about shrinking the memory (quantisation, pruning), the compute (efficient attention, distillation) and the deployment (compilers, kernels) of exactly the models on this chart. For a PhD project on embedded ML, the two-slope chart is the problem statement.

A single NVIDIA A100 from 2020 performs 312 trillion dense FP16 operations per second and draws 400 W. Four years earlier the P100 managed 18.7 trillion at 250 W. That is a 17x jump in throughput. How much of it came from simply burning more power? Only 1.6x (400 / 250). The other 10x came from architecture: Tensor Cores, faster HBM2e and a 7 nm process node. That single observation is the whole story of the cloud tier, and the rest of this concept just extends it across five generations.

Slide 17 repeats the three pillars from part 01 to mark the pivot: algorithm and data were the first two, and now we ask whether the third, hardware, can absorb the demand growth by itself. Slide 18 answers with four charts for the NVIDIA data center GPU line: P100, V100, A100, H100 and B100: the P100 as a PCIe card (as the errata below explains) and V100 to B100 in the SXM mezzanine form factor used in HGX and DGX servers, which allows a higher power budget than a PCIe card. The four series are performance, memory bandwidth, power and memory capacity, and the lesson lives in how differently they grow.

Reading the vertical axis: what a TOPS is

TOPS stands for trillions of operations per second. The unit hides two conventions you must know. First, one multiply-accumulate counts as two operations (one multiply, one add), so the MAC counts from part 01 convert with a factor of two. Second, NVIDIA quotes Tensor Core throughput at a specific precision, here FP16, and its spec sheets list two numbers: a dense figure and a figure "with sparsity" that is exactly double. The doubled number applies only when the weights have been pruned to a 2:4 structured pattern (two of every four values zero). The slide uses the dense figures, and so should you when comparing chips. The A100 page reads 312 TFLOPS dense and 624 TFLOPS with sparsity; the H100 SXM lists 1,979 sparse, which is 989 dense, rounded to 990 on the slide (NVIDIA A100 and H100 pages).

Five generations, four very different slopes

Read the table as a race between four quantities. Compute ran 94x. Efficiency ran 33x, which is why compute could run so far on so little extra power. Capacity and bandwidth ran only about 12x. So every generation, the arithmetic units can chew through data faster than the memory can deliver it, by a margin that widens each time.

Gholami and colleagues measured those three rates over two decades of FLOPS, DRAM bandwidth and interconnect bandwidth, and they conclude that memory, not compute, is now the primary bottleneck for AI serving (Gholami et al., 2024). They call it the memory wall.

Why bandwidth matters: the memory wall

Every inference streams weights and activations from the GPU's high-bandwidth memory (HBM) into the compute units. Memory bandwidth is the rate of that stream. If the chip can do 990 trillion operations per second but fetch only 3.35 trillion bytes per second, then it must perform about 295 operations on every byte it fetches to stay busy. A layer that does less arithmetic per byte than that sits idle, waiting for memory. The threshold is called the arithmetic intensity a workload needs to be compute bound.

Tie back to the LLM curve: capacity, not TOPS, sets the minimum machine

Part 02 ended with model size outrunning GPU memory. Put the two lines together. Megatron-Turing NLG has 530 billion parameters (Smith et al., 2022). GPU memory grew 12x over the eight years from 2016 to 2024; in roughly half that time, from the Transformer of 2017 to MT-NLG in 2021, LLM size grew from 0.05B to 530B, about 10,000x. The hardware pillar cannot close that by itself.

The Snapdragon 845 of 2018 delivered 3 TOPS at about 9 W. The Snapdragon 8 Gen 1 of 2022 delivers 52 TOPS at about 10 W. Seventeen times the performance, essentially no extra power. Why is the power line flat when the cloud line climbed to 700 W? Because a phone is a sealed slab held against skin with no fan. Its sustained power is capped by heat, not by engineering ambition, so every gain must come from efficiency. That is the rule for the whole edge tier: the power envelope is the constraint, and performance per watt is the design metric.

Slides 19 to 21 show three families that each obey this rule in their own way. The Qualcomm Hexagon DSP and the Apple Neural Engine live inside phones and share a lid near 10 W. NVIDIA Jetson modules live in robots and industrial boxes that have a fan and a battery budget, so their power is allowed to climb with each tier. Explore all three, and the cloud and microcontroller tiers, before reading on.

Qualcomm Hexagon: seventeen times the work under the same lid

Hexagon began as a digital signal processor for audio and camera pipelines. Qualcomm turned it into what its Snapdragon 888 brief calls a "fused AI-accelerator architecture": a tensor accelerator, vector extensions and a scalar unit sharing memory, supporting INT8, INT16 and FP16. The brief claims 26 TOPS for the whole AI Engine and "up to 3 times performance per watt improvement" over the 865; the 8 Gen 1 brief claims "up to 4x faster" and "best performance-per-watt in Snapdragon history" (Qualcomm product briefs). Notice the vendor language: every headline is a ratio to power. The products on the slide are the Qualcomm Robotics RB5 platform (865 class, 15 TOPS), the Galaxy Tab with the 888+ and the Galaxy Z Fold with the 8+ Gen 1.

Apple Neural Engine: the same lesson from a different vendor

Apple describes the Neural Engine as an energy-efficient, high-throughput engine for ML inference on its own silicon. Its newsroom gives the per-generation numbers the slide plots: 600 billion operations per second for the A11 in 2017, 5 trillion for the A12, 11 trillion for the 16-core A14, 15.8 trillion for the A15 and "nearly 17 trillion" for the A16 (Apple Newsroom). The A17 Pro's 35 TOPS comes from Apple's 2023 event rather than the newsroom text, which only says "up to 2x faster". The products on the slide span an iPhone (A14), an iPad (A15) and an iMac with the M1 Max, the same engine design scaled up.

From A11 to A17 the engine gained about 58x in throughput while the estimated power sat between 6 and 8 W. Two vendors, two architectures, one shape: a rising performance line under a flat power line.

NVIDIA Jetson: a ladder instead of a lid

A Jetson is a complete system on module: GPU, CPU, memory, power management and high-speed interfaces on one board that plugs into a carrier. It targets robots, drones and industrial edge boxes, and those have what a phone lacks: a fan and a battery or wall power. So NVIDIA sells a ladder of tiers, each with a configurable power range, and the slide plots the top of each range: Nano at 5 to 10 W, TX2 at 7.5 to 15 W, Xavier NX at 10 to 20 W, AGX Xavier at up to 30 W, AGX Orin 32GB at 15 to 40 W and AGX Orin 64GB at 15 to 60 W (NVIDIA Jetson pages, Connect Tech comparison).

Here performance grew about 550x (0.5 to 275 TOPS) while power grew 6x, so efficiency still rose about 90x. The difference from the phone families is not the efficiency trend but the freedom to spend more watts when the application can afford them. A warehouse robot on a 60 W budget gets a different chip than a drone on 10 W.

The trap: TOPS from different vendors are not one axis

It is tempting to line up the A17 Pro at 35, the Snapdragon 888 at 26 and the Jetson AGX Orin at 275 and rank them. Do not. NVIDIA's Xavier and Orin figures are INT8 (Orin sparse INT8), while Nano and TX2 are FP16 FLOPS (NVIDIA Jetson pages). The cloud numbers on slide 18 are dense FP16. Qualcomm sums INT8 throughput across the CPU, GPU and Hexagon together. Apple does not state a precision at all. Sze, Chen, Yang and Emer titled a whole paper on this: "TOPS/W (Alone) Considered Harmful". Their point is that peak numbers say nothing about utilization on a real network; a fair comparison needs the same precision, the same workload and a measured, not theoretical, throughput (Sze et al., 2020).

The STM32F746NG is one chip with a Cortex-M7 core at 216 MHz, 1 MB of flash and 320 kB of SRAM, drawing about 360 mW. It is the "off-the-shelf commercial microcontroller" on which MCUNet ran ImageNet-class vision in part 01 (slide 7). Against the smallest Jetson, the Nano at 10 W and 4 GB, it has 28x less power and 12,500x less memory. This is the tier that TinyML is named for.

A microcontroller is a compact integrated circuit designed for embedded systems, integrating a processor, memory and input/output peripherals on a single chip. The MCUNet paper adds what the definition implies: these are "bare-metal devices that do not have an operating system, nor do they have DRAM" (Lin et al., 2020). Everything a model needs, weights and activations alike, must live in the on-chip flash and SRAM, because there is nothing else.

The units drop two prefixes

Slide 22 quietly changes the axes. Power is in milliwatts, not watts. Memory is in kilobytes, not gigabytes. Performance is in DMIPS, not TOPS. Each change is a signal that this tier is a different world, and the last one deserves the most care.

The core data come from the vendors: the Arduino Zero's SAMD21 is a Cortex-M0+ at 48 MHz with 32 kB SRAM and 256 kB flash; the Due's SAM3X8E is a Cortex-M3 at 84 MHz with 96 kB SRAM; the STM32F407VG is a Cortex-M4 at 168 MHz rated 210 DMIPS with 192 kB SRAM plus 4 kB backup and 1 MB flash; the STM32F746NG is a Cortex-M7 at 216 MHz rated 462 DMIPS with 320 kB SRAM and 1 MB flash (Arduino, Keil and ST documentation).

What DMIPS measures, and what it does not

Dhrystone is a synthetic benchmark that Reinhold Weicker published in 1984 to exercise typical systems code: integer arithmetic, string handling, pointer chasing, no floating point (Weicker, 1984). A DMIPS is Dhrystones per second divided by 1757, the score of a VAX 11/780, which was nominally a one-MIPS machine. Arm publishes DMIPS per MHz for each core so that a clock frequency turns into a rating: Cortex-M0+ 0.95, Cortex-M3 and M4 1.25, Cortex-M7 2.14 (Arm Cortex-M comparison table).

So why can you not convert DMIPS into a small TOPS? Because the benchmark contains no MACs, and a neural network is almost nothing but MACs. You can only bound the MCU from above by counting what its SIMD unit could do at best.

One more pattern in the table is worth noticing: DMIPS per milliwatt falls as the boards get faster, from 7.5 on the Zero to 1.3 on the F746. The faster core pays more energy per instruction. At this tier nobody buys the F746 for efficiency; they buy it because 320 kB is the smallest SRAM in which a useful vision model can breathe.

Take ResNet-50, the workhorse image classifier from part 01, with 25,557,032 parameters (torchvision). At FP32 its weights are 102 MB; at FP16, 51 MB; at INT8, 25.6 MB. A cloud GPU holds the FP32 version 783 times over in its HBM. A phone stores it in 256 GB of flash with room for thousands of copies. A microcontroller's 1 MB flash cannot hold even the INT8 version, which is 25.6x too big. Slide 23 turns this into three tiers with two numbers each, and the ratios between them are the thesis of the course.

The lecture calls the tiers cloud AI, mobile AI and tiny AI, and gives each tier two budgets. Activation memory is the working memory that holds intermediate results during computation: 80 GB of HBM on an A100-class cloud GPU, 4 GB of DRAM on a phone, 320 kB of SRAM on the STM32F746. Weight storage is the persistent memory that holds the model: terabytes to petabytes in a data center, 256 GB of phone flash, 1 MB of on-chip flash.

MCUNet states the same gap in words: microcontroller memory is "3 orders of magnitude smaller than mobile devices, and 5-6 orders of magnitude smaller than cloud GPUs" (Lin et al., 2020). Its 2020 table used a V100 with 16 GB and an iPhone 11 with more than 64 GB; the slide updates the cloud and mobile columns to an 80 GB A100 and a 256 GB phone, while the tiny column is unchanged at 320 kB and 1 MB. The top tiers moved; the bottom did not.

Two budgets, not one

The two rows on slide 23 are not interchangeable, and confusing them is the most common mistake in sizing a TinyML deployment. Storage holds the weights persistently, so what matters is the total model size, and on an MCU that budget is the read-only flash. Activation memory is the read-write scratch space used while a layer runs, so what matters is the peak across layers, because a layer's input and output tensors must coexist at the moment it executes. MCUNet puts it in one line: "SRAM constrains the activation size (read and write); Flash constrains the model size (read-only)". A model can fit in flash and still crash at runtime because one wide early layer's activations exceed SRAM, and a model with small activations can still be unloadable because its weights exceed flash.

Why the rest of the course exists

Put the two halves of this lecture side by side. The largest model in part 02, MT-NLG at 530B parameters, needs 1.06 TB of FP16 weights. The smallest tier here offers 1 MB. That is a factor of about one million, six orders of magnitude. GPU memory grew 12x in eight years, and phone memory grew 4x across the Snapdragon generations on slide 19. No hardware trend closes six orders of magnitude on any timescale that matters. So the model must change.

Each later lecture is one lever on the model side of this gap. Pruning removes weights; quantization shrinks each remaining weight from 4 bytes to 1 or fewer; neural architecture search, including Once-for-All from part 01, designs networks under an explicit memory budget; knowledge distillation transfers a large model's accuracy into a small one; and algorithm and system co-design, as in TinyEngine, makes the runtime spend the budget wisely. Slide 24 credits the source of this framing: the deck follows Prof. Song Han's MIT 6.5940 course, TinyML and Efficient Deep Learning Computing, whose introductory lecture carries the same charts (MIT HAN Lab).