Why efficient deep learning

AOverview

Modern deep learning is expensive and it is everywhere. This lecture measures both halves of that sentence: how much computation and memory the models you already use actually cost, and how little of either the devices you want to deploy on can supply. Between those two measurements sits the question every later lecture answers, which is how to shrink a model without giving away the accuracy that made it worth running.

As a PhD student you need this lecture in three places. Exams turn it into arithmetic: a reduction ratio between two models, bytes per parameter at a given precision, a frame rate converted into milliseconds and checked against a budget, or TOPS per watt for a GPU generation. Your research project puts a model on a board with kilobytes of SRAM, and the sizing step you defend there is this lecture's comparison of demand against supply. And every efficient-model paper you read from now on, from MCUNet to an edge language model, opens by making exactly the trade you will learn to state in one line: what the model costs, what the target can pay, and how large the gap is.

The path has five stops. You watch ImageNet error fall from 28.2% to 2.3% and ask what each drop cost, then read the frontier chart where accuracy is bought with MACs and parameters and see architecture search bend that frontier downward. You follow the same question into models that are not classifiers at all, promptable segmentation, video diffusion, 3D perception and language models, where the bill is set by tokens and denoising steps rather than parameter count. You then measure supply, reading a data center GPU trajectory of 94x more compute on 2.8x the power against phone NPUs, Jetson modules and a microcontroller at 320 kB of SRAM. The lecture closes on the ratio between the two measurements, 80 GB of cloud activation memory against 320 kB on a microcontroller, about 250,000x end to end.

The numbers this lecture anchors

Each row is introduced with its own worked arithmetic in one of the three parts, so that by the end of the lecture no number here is a fact you memorized. They are the anchors the exam and the reference sheet return to, and the vocabulary every later lecture assumes.

Success looks like

• State the course challenge in one sentence and name the three pillars, algorithm, hardware and data, with one efficiency lever for each.
• Define top-5 error and top-1 accuracy precisely, say why the two cannot be compared directly, and read a bubble chart of accuracy against MACs and parameters.
• Compute a computation reduction ratio, such as 8400 / 595 = 14.1x, and explain the Pareto frontier shift that architecture search produces.
• Convert parameters and precision into bytes, divide by a device budget, and report the overshoot as a ratio, for example 102 MB of FP32 ResNet-50 weights against 1 MB of flash.
• Turn a rate into a time, compare it against a budget, and say which of two levers crosses the line, as in 5 fps to 47 fps against a 33.3 ms frame period.
• Read a GPU spec sheet: define TOPS, memory bandwidth and TDP, and compute TOPS per watt for any generation.
• Compare cloud, mobile and tiny tiers on activation memory and weight storage, separate the two budgets, and decide whether a given model fits each tier.

How to study this lecture

1. Read the parts in order. Demand comes before supply: the model costs of the first two parts only become meaningful once the third part shows what a device can pay.
2. Keep a pencil next to every worked example. Compute the slide's number before you reveal it, then check yourself against the arithmetic in the concept.
3. Answer every recall prompt in your head before opening it. Ratios that you retrieved stay; the ones you only read do not.
4. Take each quiz and read the explanation even when you are right; the wrong options are the exam traps.
5. Read the slide errata carefully. Several slides carry inherited arithmetic slips, and spotting why a number is wrong is the best proof that you own the computation behind it.
6. Mark a concept understood only when you could redo its number and its ratio from a blank page. Unmarked concepts show you where to return.
7. Come back after a few days and size a model cold on a device you have not used, such as the board your own project targets. Spaced practice on a new example is what turns a formula into a skill.

BThe 3 parts

1. 01Accuracy comes at a costDeep learning now beats humans on ImageNet, but every accuracy gain was paid for in computation and model size, and efficient design (NAS, MCUNet, on-device training) is what brings it to phones and IoT devices.6 conceptsSlides 1-836 min
   1. 1.1Deep learning is everywhere and expensive: the challenge of this course
   2. 1.2ImageNet: eight years from 28.2% to below human
   3. 1.3Reading the bubble chart: accuracy is bought with MACs and parameters
   4. 1.4Architecture search moves the frontier: 14x less compute at equal accuracy
   5. 1.5TinyML: deep learning on a microcontroller with kilobytes of memory
   6. 1.6Learning on the device: why it is wanted and why it is hard
2. 02Segmentation, generative models and LLMsPromptable segmentation, diffusion models, 3D perception and large language models push computation and memory far beyond classification, and each has an efficient counterpart that recovers speed without losing quality.5 conceptsSlides 9-1630 min
   1. 2.1Promptable segmentation: one heavy encoder, one cheap decoder
   2. 2.2Generative models: tokens and steps set the bill, not parameters
   3. 2.3DistriFusion: splitting one image across GPUs without splitting the story
   4. 2.4Fast-LiDARNet: reaching real time by co-designing kernels and model
   5. 2.5LLMs outgrow GPU memory, and why they must still run at the edge
3. 03Cloud and edge AI hardware, and the gap between themHow NVIDIA data center GPUs, mobile NPUs, Jetson modules and microcontrollers compare in performance, power and memory, and why the orders-of-magnitude gap from cloud to tiny devices demands efficient models.4 conceptsSlides 17-2424 min
   1. 3.1The hardware pillar: cloud GPUs grew 94x in compute on 2.8x the power
   2. 3.2Phones, Apple silicon and Jetson: performance inside a power envelope
   3. 3.3Microcontrollers: milliwatts, kilobytes and a different yardstick
   4. 3.4The gap: 250,000x in activation memory, and why the rest of the course exists

CGlossary and reference

• GlossaryEvery term from the lecture, defined once, with links back to the slides that use it.
• Reference sheetThe lecture on one printable page, for revision before a quiz or exam.