Cloud and edge AI hardware, and the gap between them

A single NVIDIA A100 from 2020 performs 312 trillion dense FP16 operations per second and draws 400 W. Four years earlier the P100 managed 18.7 trillion at 250 W. That is a 17x jump in throughput. How much of it came from simply burning more power? Only 1.6x (400 / 250). The other 10x came from architecture: Tensor Cores, faster HBM2e and a 7 nm process node. That single observation is the whole story of the cloud tier, and the rest of this concept just extends it across five generations.

Slide 17 repeats the three pillars from part 01 to mark the pivot: algorithm and data were the first two, and now we ask whether the third, hardware, can absorb the demand growth by itself. Slide 18 answers with four charts for the NVIDIA data center GPU line: P100, V100, A100, H100 and B100: the P100 as a PCIe card (as the errata below explains) and V100 to B100 in the SXM mezzanine form factor used in HGX and DGX servers, which allows a higher power budget than a PCIe card. The four series are performance, memory bandwidth, power and memory capacity, and the lesson lives in how differently they grow.

Reading the vertical axis: what a TOPS is

TOPS stands for trillions of operations per second. The unit hides two conventions you must know. First, one multiply-accumulate counts as two operations (one multiply, one add), so the MAC counts from part 01 convert with a factor of two. Second, NVIDIA quotes Tensor Core throughput at a specific precision, here FP16, and its spec sheets list two numbers: a dense figure and a figure "with sparsity" that is exactly double. The doubled number applies only when the weights have been pruned to a 2:4 structured pattern (two of every four values zero). The slide uses the dense figures, and so should you when comparing chips. The A100 page reads 312 TFLOPS dense and 624 TFLOPS with sparsity; the H100 SXM lists 1,979 sparse, which is 989 dense, rounded to 990 on the slide (NVIDIA A100 and H100 pages).

Five generations, four very different slopes

Read the table as a race between four quantities. Compute ran 94x. Efficiency ran 33x, which is why compute could run so far on so little extra power. Capacity and bandwidth ran only about 12x. So every generation, the arithmetic units can chew through data faster than the memory can deliver it, by a margin that widens each time.

Gholami and colleagues measured those three rates over two decades of FLOPS, DRAM bandwidth and interconnect bandwidth, and they conclude that memory, not compute, is now the primary bottleneck for AI serving (Gholami et al., 2024). They call it the memory wall.

Why bandwidth matters: the memory wall

Every inference streams weights and activations from the GPU's high-bandwidth memory (HBM) into the compute units. Memory bandwidth is the rate of that stream. If the chip can do 990 trillion operations per second but fetch only 3.35 trillion bytes per second, then it must perform about 295 operations on every byte it fetches to stay busy. A layer that does less arithmetic per byte than that sits idle, waiting for memory. The threshold is called the arithmetic intensity a workload needs to be compute bound.

Tie back to the LLM curve: capacity, not TOPS, sets the minimum machine

Part 02 ended with model size outrunning GPU memory. Put the two lines together. Megatron-Turing NLG has 530 billion parameters (Smith et al., 2022). GPU memory grew 12x over the eight years from 2016 to 2024; in roughly half that time, from the Transformer of 2017 to MT-NLG in 2021, LLM size grew from 0.05B to 530B, about 10,000x. The hardware pillar cannot close that by itself.

The Snapdragon 845 of 2018 delivered 3 TOPS at about 9 W. The Snapdragon 8 Gen 1 of 2022 delivers 52 TOPS at about 10 W. Seventeen times the performance, essentially no extra power. Why is the power line flat when the cloud line climbed to 700 W? Because a phone is a sealed slab held against skin with no fan. Its sustained power is capped by heat, not by engineering ambition, so every gain must come from efficiency. That is the rule for the whole edge tier: the power envelope is the constraint, and performance per watt is the design metric.

Slides 19 to 21 show three families that each obey this rule in their own way. The Qualcomm Hexagon DSP and the Apple Neural Engine live inside phones and share a lid near 10 W. NVIDIA Jetson modules live in robots and industrial boxes that have a fan and a battery budget, so their power is allowed to climb with each tier. Explore all three, and the cloud and microcontroller tiers, before reading on.

Qualcomm Hexagon: seventeen times the work under the same lid

Hexagon began as a digital signal processor for audio and camera pipelines. Qualcomm turned it into what its Snapdragon 888 brief calls a "fused AI-accelerator architecture": a tensor accelerator, vector extensions and a scalar unit sharing memory, supporting INT8, INT16 and FP16. The brief claims 26 TOPS for the whole AI Engine and "up to 3 times performance per watt improvement" over the 865; the 8 Gen 1 brief claims "up to 4x faster" and "best performance-per-watt in Snapdragon history" (Qualcomm product briefs). Notice the vendor language: every headline is a ratio to power. The products on the slide are the Qualcomm Robotics RB5 platform (865 class, 15 TOPS), the Galaxy Tab with the 888+ and the Galaxy Z Fold with the 8+ Gen 1.

Apple Neural Engine: the same lesson from a different vendor

Apple describes the Neural Engine as an energy-efficient, high-throughput engine for ML inference on its own silicon. Its newsroom gives the per-generation numbers the slide plots: 600 billion operations per second for the A11 in 2017, 5 trillion for the A12, 11 trillion for the 16-core A14, 15.8 trillion for the A15 and "nearly 17 trillion" for the A16 (Apple Newsroom). The A17 Pro's 35 TOPS comes from Apple's 2023 event rather than the newsroom text, which only says "up to 2x faster". The products on the slide span an iPhone (A14), an iPad (A15) and an iMac with the M1 Max, the same engine design scaled up.

From A11 to A17 the engine gained about 58x in throughput while the estimated power sat between 6 and 8 W. Two vendors, two architectures, one shape: a rising performance line under a flat power line.

NVIDIA Jetson: a ladder instead of a lid

A Jetson is a complete system on module: GPU, CPU, memory, power management and high-speed interfaces on one board that plugs into a carrier. It targets robots, drones and industrial edge boxes, and those have what a phone lacks: a fan and a battery or wall power. So NVIDIA sells a ladder of tiers, each with a configurable power range, and the slide plots the top of each range: Nano at 5 to 10 W, TX2 at 7.5 to 15 W, Xavier NX at 10 to 20 W, AGX Xavier at up to 30 W, AGX Orin 32GB at 15 to 40 W and AGX Orin 64GB at 15 to 60 W (NVIDIA Jetson pages, Connect Tech comparison).

Here performance grew about 550x (0.5 to 275 TOPS) while power grew 6x, so efficiency still rose about 90x. The difference from the phone families is not the efficiency trend but the freedom to spend more watts when the application can afford them. A warehouse robot on a 60 W budget gets a different chip than a drone on 10 W.

The trap: TOPS from different vendors are not one axis

It is tempting to line up the A17 Pro at 35, the Snapdragon 888 at 26 and the Jetson AGX Orin at 275 and rank them. Do not. NVIDIA's Xavier and Orin figures are INT8 (Orin sparse INT8), while Nano and TX2 are FP16 FLOPS (NVIDIA Jetson pages). The cloud numbers on slide 18 are dense FP16. Qualcomm sums INT8 throughput across the CPU, GPU and Hexagon together. Apple does not state a precision at all. Sze, Chen, Yang and Emer titled a whole paper on this: "TOPS/W (Alone) Considered Harmful". Their point is that peak numbers say nothing about utilization on a real network; a fair comparison needs the same precision, the same workload and a measured, not theoretical, throughput (Sze et al., 2020).

The STM32F746NG is one chip with a Cortex-M7 core at 216 MHz, 1 MB of flash and 320 kB of SRAM, drawing about 360 mW. It is the "off-the-shelf commercial microcontroller" on which MCUNet ran ImageNet-class vision in part 01 (slide 7). Against the smallest Jetson, the Nano at 10 W and 4 GB, it has 28x less power and 12,500x less memory. This is the tier that TinyML is named for.

A microcontroller is a compact integrated circuit designed for embedded systems, integrating a processor, memory and input/output peripherals on a single chip. The MCUNet paper adds what the definition implies: these are "bare-metal devices that do not have an operating system, nor do they have DRAM" (Lin et al., 2020). Everything a model needs, weights and activations alike, must live in the on-chip flash and SRAM, because there is nothing else.

The units drop two prefixes

Slide 22 quietly changes the axes. Power is in milliwatts, not watts. Memory is in kilobytes, not gigabytes. Performance is in DMIPS, not TOPS. Each change is a signal that this tier is a different world, and the last one deserves the most care.

The core data come from the vendors: the Arduino Zero's SAMD21 is a Cortex-M0+ at 48 MHz with 32 kB SRAM and 256 kB flash; the Due's SAM3X8E is a Cortex-M3 at 84 MHz with 96 kB SRAM; the STM32F407VG is a Cortex-M4 at 168 MHz rated 210 DMIPS with 192 kB SRAM plus 4 kB backup and 1 MB flash; the STM32F746NG is a Cortex-M7 at 216 MHz rated 462 DMIPS with 320 kB SRAM and 1 MB flash (Arduino, Keil and ST documentation).

What DMIPS measures, and what it does not

Dhrystone is a synthetic benchmark that Reinhold Weicker published in 1984 to exercise typical systems code: integer arithmetic, string handling, pointer chasing, no floating point (Weicker, 1984). A DMIPS is Dhrystones per second divided by 1757, the score of a VAX 11/780, which was nominally a one-MIPS machine. Arm publishes DMIPS per MHz for each core so that a clock frequency turns into a rating: Cortex-M0+ 0.95, Cortex-M3 and M4 1.25, Cortex-M7 2.14 (Arm Cortex-M comparison table).

So why can you not convert DMIPS into a small TOPS? Because the benchmark contains no MACs, and a neural network is almost nothing but MACs. You can only bound the MCU from above by counting what its SIMD unit could do at best.

One more pattern in the table is worth noticing: DMIPS per milliwatt falls as the boards get faster, from 7.5 on the Zero to 1.3 on the F746. The faster core pays more energy per instruction. At this tier nobody buys the F746 for efficiency; they buy it because 320 kB is the smallest SRAM in which a useful vision model can breathe.

Take ResNet-50, the workhorse image classifier from part 01, with 25,557,032 parameters (torchvision). At FP32 its weights are 102 MB; at FP16, 51 MB; at INT8, 25.6 MB. A cloud GPU holds the FP32 version 783 times over in its HBM. A phone stores it in 256 GB of flash with room for thousands of copies. A microcontroller's 1 MB flash cannot hold even the INT8 version, which is 25.6x too big. Slide 23 turns this into three tiers with two numbers each, and the ratios between them are the thesis of the course.

The lecture calls the tiers cloud AI, mobile AI and tiny AI, and gives each tier two budgets. Activation memory is the working memory that holds intermediate results during computation: 80 GB of HBM on an A100-class cloud GPU, 4 GB of DRAM on a phone, 320 kB of SRAM on the STM32F746. Weight storage is the persistent memory that holds the model: terabytes to petabytes in a data center, 256 GB of phone flash, 1 MB of on-chip flash.

MCUNet states the same gap in words: microcontroller memory is "3 orders of magnitude smaller than mobile devices, and 5-6 orders of magnitude smaller than cloud GPUs" (Lin et al., 2020). Its 2020 table used a V100 with 16 GB and an iPhone 11 with more than 64 GB; the slide updates the cloud and mobile columns to an 80 GB A100 and a 256 GB phone, while the tiny column is unchanged at 320 kB and 1 MB. The top tiers moved; the bottom did not.

Two budgets, not one

The two rows on slide 23 are not interchangeable, and confusing them is the most common mistake in sizing a TinyML deployment. Storage holds the weights persistently, so what matters is the total model size, and on an MCU that budget is the read-only flash. Activation memory is the read-write scratch space used while a layer runs, so what matters is the peak across layers, because a layer's input and output tensors must coexist at the moment it executes. MCUNet puts it in one line: "SRAM constrains the activation size (read and write); Flash constrains the model size (read-only)". A model can fit in flash and still crash at runtime because one wide early layer's activations exceed SRAM, and a model with small activations can still be unloadable because its weights exceed flash.

Why the rest of the course exists

Put the two halves of this lecture side by side. The largest model in part 02, MT-NLG at 530B parameters, needs 1.06 TB of FP16 weights. The smallest tier here offers 1 MB. That is a factor of about one million, six orders of magnitude. GPU memory grew 12x in eight years, and phone memory grew 4x across the Snapdragon generations on slide 19. No hardware trend closes six orders of magnitude on any timescale that matters. So the model must change.

Each later lecture is one lever on the model side of this gap. Pruning removes weights; quantization shrinks each remaining weight from 4 bytes to 1 or fewer; neural architecture search, including Once-for-All from part 01, designs networks under an explicit memory budget; knowledge distillation transfers a large model's accuracy into a small one; and algorithm and system co-design, as in TinyEngine, makes the runtime spend the budget wisely. Slide 24 credits the source of this framing: the deck follows Prof. Song Han's MIT 6.5940 course, TinyML and Efficient Deep Learning Computing, whose introductory lecture carries the same charts (MIT HAN Lab).