Accuracy comes at a cost

Open the camera on your phone and it finds faces before you press anything. Ask a speaker a question and a language model answers. Paste a screenshot into a chatbot and it reads the text and the picture together. Vision, language and multimodal tasks are all served today by deep networks, and every one of those answers costs billions of arithmetic operations. Classifying a single image with ResNet-50 takes about 3.9 billion multiply-accumulate operations (Sze et al., 2017), and that is a model from 2015 doing the easiest of the three tasks.

That is the whole tension of this course, and slide 2 states it in one line. Machine learning (ML) and deep learning are applied everywhere, they are computationally expensive, and the challenge is to make them lightweight and fast without losing accuracy. Every technique you will learn, from pruning and quantization to architecture search, hardware accelerators and tiny Inference engines, is one answer to that single question. It helps to write the question the way an optimization course would: keep accuracy as high as possible while staying under whatever budget the target device imposes.

The budgets are what change from device to device. A cloud GPU has a generous C and M; a phone has a tight Latency T and a battery; a sensor node has a memory M measured in kilobytes. The accuracy you can afford depends on which row of that constraint list binds, and part 03 puts real numbers on each row.

Three pillars, one lens for the whole course

Slide 3 gives you the lens to organize everything that follows: deep learning advances when three things improve together. Its three pillars are the algorithm (the model and how it is trained), the hardware it runs on, and the data it learns from. None of them is enough alone. The 2012 breakthrough on ImageNet, which the next concept quantifies, needed all three at once: a new deep convolutional architecture, two GPUs able to run it for a week, and a dataset of about 1.2 million labeled images (Krizhevsky et al., 2012).

The same three words tell you where efficiency work happens. Compressing, quantizing and searching for architectures are algorithm-pillar work: the model changes, the chip does not. Building accelerators, from data-center GPUs down to the Hexagon DSP and the Neural Engine, is hardware-pillar work: the chip changes so that the same model runs faster or cooler. The deck returns to this exact slide as slide 17 to open the hardware section, so treat slide 3 as the introduction of the framework and slide 17 as the pivot to its second pillar. The data pillar makes its own appearance on slide 8, where data that cannot leave a device forces learning to happen on the device.

Why efficiency matters beyond speed

Speed is the obvious reason to want small models, but the exam will ask for more, and the deck gives you at least four. Privacy: a model that fits on the device can process a face or a voice without uploading it. Cost: training and serving large models is billed in GPU hours and, increasingly, in carbon. The Once-for-All paper notes that training a separate specialized network for every deployment target emits as much CO2 as five cars over their lifetimes (Cai et al., 2020). Connectivity: sensors in fields, mines and rural clinics may have no network at all, and MCUNet argues that on-device processing extends AI to areas without Internet access (Lin et al., 2020). Energy: a battery-powered node has a milliwatt budget, and every unnecessary multiply drains it.

• Privacy: raw data stays where it was collected.
• Cost: fewer GPU hours, smaller cloud bills, less carbon.
• Connectivity: the model works where the network does not.
• Energy: milliwatt devices need models sized for milliwatts.

Look at the left panel of slide 4 before the bar chart. It shows six test images (a crop of the eight in the AlexNet paper) with the five classes AlexNet ranked highest for each. For the mushroom photo the top guess is "agaric" and the true label "mushroom" sits second. For the cherry photo the top guess is "dalmatian", followed by grape, elderberry, a bull terrier and currant; "cherry" is nowhere in the list. Under one scoring rule the mushroom counts as correct and the cherry does not; under another rule both are wrong. Which rule you use decides every number on the right of the slide.

The rule the contest used is Top-5 error rate. The ImageNet challenge (ILSVRC) let each algorithm return five labels per image and counted the image as correct if any one of them matched the single ground-truth label; the error is the fraction of test images with no match (Russakovsky et al., 2015, Eq. 1). The organizers allowed five guesses because ImageNet photos often contain several objects but carry only one label, and a system that names a visible second object should not be punished. From 2012 the contest reported only top-5 error, since all their measures ranked entries the same way.

The other rule, Top-1 accuracy, keeps only the single highest-scoring class and counts the fraction of images where it matches. It is stricter, so the same model always scores worse on top-1 than on top-5. Slides 5 and 6 use top-1 accuracy; slide 4 uses top-5 error. Keeping the two apart is the single most common slip in reading this part of the deck.

Eight years of the winning entry

With the metric fixed, the bar chart reads as a history. The winning top-5 error went from 28.2% in 2010 and 25.8% in 2011 to 16.4% in 2012, when SuperVision, the entry now called AlexNet, was the first deep convolutional network to win. The error kept falling as the networks deepened: Clarifai at 11.7% in 2013, GoogLeNet at 6.7% in 2014, ResNet at 3.57% in 2015 with an ensemble of residual networks up to 152 layers deep (He et al., 2016), Trimps-Soushen at 2.99% in 2016 and SENet at 2.25% in 2017 (Hu et al., 2018). The slide rounds those to one decimal.

What "super-human" actually means here

The human bar is not an average person. It is one expert annotator who trained on 500 images and then labeled 1500 test images at roughly one per minute, reaching 5.1% top-5 error; a second annotator with less training scored 12.0%, and GoogLeNet scored 6.8% on the same sample (Russakovsky et al., 2015, Sec. 6.4). So "DNNs achieve super-human accuracy" means that from 2015 the winning network beats a well-trained, patient human on the top-5 rule for 1000 fine-grained classes, many of them dog breeds. It says nothing about general vision, and it says nothing about top-1.

Take two models from the same family. ResNet-50 needs about 3.8 billion multiply-adds per image and ResNet-101 about 7.6 billion, twice as many (He et al., 2016, Table 1). On slide 5 they sit at about 75.3% and 76.4% top-1 accuracy. Doubling the work bought about one percentage point. That one pair is the message of the slide: high accuracy comes at the cost of high computation, and the exchange rate gets worse as you climb.

To read the chart you need its currency. The horizontal axis counts MACs, multiply-accumulate operations: one multiplication of an input by a weight, immediately added into a running sum. Sze and colleagues call the MAC "the fundamental component" of both convolutional and fully connected layers (Sze et al., 2017). Counting MACs gives a hardware-neutral cost: the number is a property of the model and the input size, not of the chip, unlike latency or energy which change with every device. A convolutional layer's count follows directly from its shape.

How to read a bubble chart

Three quantities share the chart. Horizontal position is cost in billions of MACs, so left is cheaper. Vertical position is Top-1 accuracy on ImageNet, so up is better. Bubble area is Model size, the number of parameters, from 2 M to 64 M according to the legend. Parameters measure storage (how many bytes the weights occupy) and MACs measure work (how many operations each input triggers), and the two are related but not the same. Circles are handcrafted architectures and stars are found by AutoML, which becomes important on the next slide. The goal region is the upper left: high accuracy at low cost.

Diminishing returns

Trace the chart from the cheapest model upward and the exchange rate collapses. For each step, divide the accuracy gained by the extra MACs spent. MobileNetV1 is the cheap starting point at 70.6% for 0.57 B MACs (Howard et al., 2017); every other number below is read from the chart.

The marginal value of computation falls almost fivefold across the chart, from 1.41 to 0.30 points per billion, and the worst trade is inside a single family. Notice also that Xception is itself dominated: DPN-92 is more accurate (79.3%) with fewer MACs (6.5 B).

Now put the stars back on the chart. Once-for-All (OFA) reaches 80.0% top-1 with 595 M MACs (Cai et al., 2020). Xception, the most expensive handcrafted model on the previous slide and within 0.3 points of the most accurate (DPN-92), needed about 8.4 B for 79.0%. Divide and you get 8400 / 595 = 14.1: the same accuracy, or one point more, for one fourteenth of the computation. That is the dashed arrow at the top of slide 6, and it is drawn straight from Figure 2 of the Once-for-All paper.

The MobileNetV1 row makes the point from the other side: at essentially the same budget, the searched model is 9.4 points more accurate. The stars are all products of Neural architecture search (NAS): rather than a person choosing kernel sizes, widths and depths, an algorithm searches over them under an explicit MAC or latency budget. EfficientNet, NASNet-A, MobileNetV3, ProxylessNAS, AmoebaNet, PNASNet and DARTS all sit inside the boxed region below about 2.5 B MACs, next to two efficient handcrafted models, MobileNetV2 and ShuffleNet. Once-for-All adds one more idea: train a single large network once, then cut specialized sub-networks out of it for each device, which is what makes the 80.0% at 595 M affordable to produce as well as to run.

The frontier and why it moves left

The two arrows on the slide, "the higher the better" and "the lower the better", state the accuracy-efficiency trade-off as a partial order. Model A dominates model B if A is at least as accurate and at most as costly, with one of the two strictly better. The models nobody dominates form the Pareto frontier, the staircase you saw in the last visual. Every point below or to the right of it is a worse deal than something already available. Architecture search did not just add good models; it moved the entire staircase left, so that the best accuracy at 0.6 B MACs today exceeds the best at 8 B a few years earlier.

The slide names the payoff in two lines. Efficient deep learning enables daily-life applications on mobile phones, which is why the Once-for-All paper defines a "mobile setting" as anything under 600 M MACs and reports being the first to reach 80% inside it. And it enables private on-device Inference and training, the theme of the last concept in this part. Both follow from the same fact: a model that costs fourteen times less can live on a device that has fourteen times less to spend.

Slide 7 shows two photographs of a small camera board on a desk. In one it reports whether the person in front of it wears a face mask; in the other it detects people in an office. There is no phone, no laptop and no network cable in the picture. The board is an OpenMV Cam, whose processor is an ARM Cortex-M7 Microcontroller (MCU), and the software is MCUNet, a framework from the same lab as Once-for-All (Lin et al., 2020).

This is TinyML: machine learning on devices with kilobytes of memory. MCUNet's reference target, the STM32F746, has 320 kB of SRAM and 1 MB of flash, and its Table 1 lines that up against a phone and a cloud GPU. The OpenMV Cam H7 in the photos is a somewhat larger sibling: STM32H743 at 480 MHz, with 1 MB of RAM in total, of which MCUNet's experiments use the 512 kB main SRAM block, and 2 MB flash (OpenMV). Both are bare-metal devices with no operating system and no DRAM. Slide 23 returns to these exact numbers as the tiny AI column of the hardware tiers. Its cloud and mobile columns are updated to an 80 GB GPU and a 256 GB phone, so expect those two cells to differ from the table below while the microcontroller column stays the same.

Why nothing off the shelf fits

The paper gives three failures. ResNet-50's weight storage is about 102 MB, 100x the flash. MobileNetV2 needs about 6.8 MB of peak activation memory, 22x the SRAM (the paper rounds up). Even the int8-quantized MobileNetV2 peaks at 1.7 MB, still 5.3x over, and that is before the runtime's own buffers. Reaching the board required designing the network and the inference engine together: TinyNAS searches architectures under the exact memory limits and TinyEngine executes them without interpreter overhead, and MCUNet as a whole uses 3.5x less SRAM and 5.7x less flash than int8 MobileNetV2 and ResNet-18 at similar accuracy. The result is 70.7% ImageNet top-1 on an STM32H743 (the OpenMV H7 chip, which MCUNet evaluates under a 512 kB SRAM and 2 MB flash budget, although the board has 1 MB of SRAM in total), the first time that bar was reached on an off-the-shelf microcontroller, and about 62% inside the 320 kB / 1 MB budget of the F746 (Lin et al., 2020, Table 4, 4-bit).

The reasons to bother are the same four from the first concept, now at their most extreme. MCUNet points out that there are around 250 billion microcontrollers in the world, that processing raw sensor data at the source protects privacy because nothing is transmitted, and that inference on the node works in places without Internet access (Lin et al., 2020). A mask detector on a camera board costs a few dollars, runs on milliwatts and never uploads a face.

A smart speaker slowly gets better at recognizing the voices in one house. A phone learns its owner's face after the first enrollment. Both need data that arrives after the product shipped, and both kinds of data (a voice, a face) are exactly what users least want uploaded. The figure on slide 8 draws the consequence: new and sensitive data flows from the user to the Edge device, the arrow from the device to the cloud is crossed out for privacy, and a small loop above the device shows the model updating itself where the data is.

That loop is On-device training (learning). The slide lists four benefits. Better privacy, because nothing leaves the device. Lower cost, because there is no upload and no cloud training bill. Customization, because the model adapts to this user, this room, this camera angle. Life-long learning, because the adaptation never has to stop; AI systems, as the slide puts it, need to continually adapt to new data collected locally. Lin and colleagues summarize the goal as letting users benefit from customized models without transferring their data to the cloud (Lin et al., 2022).

Why training is so much more expensive than inference

The third line of the slide is the catch: training is more expensive than Inference, hard to fit in edge memory, and limited by power. All three follow from how backpropagation works. Inference is one forward pass through the layers, and each layer's activations can be discarded as soon as the next layer has consumed them. Training runs the same forward pass, then a backward pass that costs about twice as much: propagating the gradient through each layer is one multiply by the transposed weights, which Goodfellow, Bengio and Courville note has the same cost as the forward multiply (Goodfellow et al., 2016, Sec. 6.5.7), and computing the weight gradient is a second multiply of the same size. So one training step is roughly three times the arithmetic of one inference pass, the convention behind the 6N versus 2N per-token estimate used in scaling studies (Kaplan et al., 2020). The backward pass also needs the activations that the forward pass produced. Goodfellow and colleagues put it directly: the input to each hidden nonlinearity must be stored from the moment it is computed until the backward pass has returned to the same point, so memory grows with the number of examples in the batch and the width of the layers. On top of that, the optimizer keeps its own state: a momentum buffer, or two moment estimates for Adam, each the size of the model.

Power is the last constraint. A training step roughly triples the arithmetic of every example, and a device that runs on a battery or a solar cell cannot run it continuously. So on-device learning is not merely inference plus a little bookkeeping; it is a harder problem than the one MCUNet solved, and the course treats it as its own topic.