Glossary

A pruning pattern where two of every four weights are zero, which NVIDIA Tensor Cores from Ampere on can skip; spec sheets quote a sparse TOPS figure exactly double the dense one, and only pruned models reach it.

• Slide 18

Working memory that holds intermediate results during computation: 80 GB for cloud AI, 4 GB for mobile AI, 320 kB for tiny AI.

• Slide 23

Jointly optimizing the model and the system software or hardware it runs on, such as kernel optimization plus model redesign, to reach real-time performance.

• Slide 14

Energy-efficient, high-throughput engine for ML inference on Apple silicon, from 0.6 TOPS (A11) to 35 TOPS (A17).

• Slide 20

Operations performed per byte fetched from memory; a workload below the chip's break-even ratio (about 295 operations per byte on an H100 at 990 TOPS and 3.35 TB/s) is memory bound, and LLM decoding at batch size one sits near 1.

• Slide 18

Arm's family of 32-bit microcontroller processor cores (M0+, M3, M4, M7) used on boards such as the Arduino Zero and the STM32F746; Arm rates each core in DMIPS per MHz, and the Cortex-M7 is the core MCUNet targets.

• Slide 7
• Slide 22

The transformer operation that compares every token with every other token, so its compute and memory grow with the square of the token count; it is why diffusion cost is super-quadratic in resolution and why efficient attention is a research area.

• Slide 12

The training algorithm that follows a forward pass with a backward pass computing gradients layer by layer; it costs about twice the forward pass and must keep every layer's activations until the backward pass returns, which is why training needs more compute and memory than inference.

• Slide 8

Data center accelerator such as NVIDIA P100, V100, A100, H100 and B100, with up to 192 GB memory and 700 W power.

• Slide 18
• Slide 23

Three hardware tiers of decreasing resources, showing that edge AI devices still have a huge gap to cloud processors.

• Slide 23

Machine learning with deep neural networks of many layers, applied to vision, language and multimodal tasks, whose accuracy comes at the cost of high computation.

• Slide 2
• Slide 4
• Slide 5

One forward pass of a diffusion model that removes a little noise from the current image or latent; a sampler runs tens of them per image, so the real cost of one image is the per-step MACs times the step count.

• Slide 12
• Slide 13

Generative model that creates realistic images or video from a natural language description by iterative denoising; very computationally expensive to train and run.

• Slide 11
• Slide 12
• Slide 13

DistriFusion's mechanism: each GPU denoises its own image patch but attends to the neighbouring patches' activations from the previous denoising step, which are similar enough to the current ones, so communication is asynchronous and hidden behind computation.

• Slide 13

Distributed parallel inference for high-resolution diffusion models that splits work across 4 GPUs for 4.0x fewer MACs per device and 3.0x lower latency without the duplicated-subject artifacts of naive patch parallelization.

• Slide 13

Dhrystone million instructions per second, an index of integer computation used to rate microcontroller performance.

• Slide 22

Computing device close to the user or sensor (phone, smart speaker, car, robot, microcontroller) that processes data locally instead of in a cloud server.

• Slide 8
• Slide 16
• Slide 19
• Slide 20
• Slide 21
• Slide 23

Running LLMs locally on laptops, cars and robots for copilot services, for devices that may not be connected and to keep personal data private.

• Slide 16

SAM accelerated by replacing its image encoder with EfficientViT, matching SAM-ViT-H zero-shot segmentation performance with a 48.9x speedup on an A100 GPU.

• Slide 10

3D LiDAR perception network accelerated by algorithm/system co-design to 47 fps, above the 30 fps real-time threshold, versus 5 fps for MinkowskiNet.

• Slide 14

The persistent, read-only-during-inference storage of a microcontroller, e.g. 1 MB on the STM32F746, which holds the model weights and therefore constrains the total model size.

• Slide 23

Count of arithmetic operations where a multiply and an add are counted separately, so one MAC equals two FLOPs; some papers label multiply-adds as FLOPs, so check the convention before comparing, and TOPS and TFLOPS ratings use the two-per-MAC count.

• Slide 5
• Slide 18

16-bit floating point precision, 2 bytes per value, assumed when converting model parameters to GPU memory.

• Slide 15
• Slide 16
• Slide 18

The time budget for one frame, 1000 ms divided by the frame rate: 33.3 ms at 30 fps, 200 ms at 5 fps, 21.3 ms at 47 fps; a model is real time only when its per-frame latency fits inside it.

• Slide 14

Inference throughput for video or sensor streams; 30 fps is the real-time threshold used for driving.

• Slide 14

The stacked DRAM packaged next to a data center GPU that holds weights and activations: 16 GB at 732 GB/s on the P100, 80 GB at 2039 GB/s on the A100, 192 GB at 8192 GB/s on the B100.

• Slide 18
• Slide 23

Large-scale image classification benchmark and annual contest used to track progress of deep networks, where winners' top-5 error fell from 28.2% in 2010 to 2.3% in 2017, below the human 5.1%.

• Slide 4
• Slide 5
• Slide 6

Using a trained model to produce predictions for new inputs; cheaper than training.

• Slide 6
• Slide 8
• Slide 20

8-bit integer precision, 1 byte per value, 4x smaller than FP32; the format MCUNet uses for its quantized models and the one NVIDIA quotes for Jetson Xavier and Orin TOPS.

• Slide 21
• Slide 23

Transformer-based language model with billions of parameters, growing from 0.05B (Transformer, 2017) to 530B (MT-NLG), far faster than GPU memory.

• Slide 15
• Slide 16

Time to produce one output, e.g. 12.3 s for one high-resolution image on a single GPU versus 4.16 s with DistriFusion on 4 GPUs.

• Slide 13

Algorithms that learn patterns from data to make predictions or decisions instead of being explicitly programmed; the slides pair it with deep learning as widely used but computationally expensive.

• Slide 2

Count of multiply-accumulate operations a model performs for one input, a hardware-neutral measure of computational cost reported in billions (B) or trillions (T).

• Slide 5
• Slide 6
• Slide 12
• Slide 13

The COCO instance segmentation accuracy metric: average precision over recall levels and mask overlap thresholds, averaged over classes; SAM-ViT-H scores 46.5 and EfficientViT-SAM-L2 46.6 zero-shot.

• Slide 10

Tiny deep learning system that enables vision tasks like facial mask detection and person detection on IoT devices such as an ARM Cortex-M7 with an OpenMV Cam.

• Slide 7

Rate at which data can be moved to and from accelerator memory in GB/s, rising from 732 GB/s (P100) to 8192 GB/s (B100).

• Slide 18

The widening gap between compute and data movement: P100 to B100 compute grew 94x while memory capacity and bandwidth grew about 12x, so memory-bound work such as LLM token decoding is limited by bandwidth, not TOPS.

• Slide 18

Compact integrated circuit for embedded systems that includes a processor, memory and I/O peripherals on a single chip, with milliwatts of power and kilobytes of memory.

• Slide 7
• Slide 22
• Slide 23

Number of parameters (weights) in a model, e.g. 2M to 64M for ImageNet CNNs and up to 530B for LLMs; memory needed depends on the numeric precision per parameter.

• Slide 5
• Slide 15
• Slide 16

Automated (AutoML) search for network architectures that reach high accuracy with low computation, as opposed to handcrafted designs.

• Slide 6

The number format used to store each parameter or activation, which sets the bytes per value: 4 for FP32, 2 for FP16 or BF16, 1 for INT8, 0.5 for INT4; memory is parameters times bytes per parameter, and TOPS ratings are quoted at a specific precision.

• Slide 15
• Slide 16
• Slide 18
• Slide 21

Edge AI family of systems on module, from Nano (0.5 TOPS, 10 W, 4 GB) to AGX Orin 64GB (275 TOPS, 60 W, 64 GB).

• Slide 21

Updating a model directly on the edge device with locally collected data, giving better privacy, lower cost, customization and life-long learning, but harder because training costs more than inference and edge memory and power are limited.

• Slide 8

NAS approach that trains one network and specializes sub-networks for efficient deployment, reaching 80.0% ImageNet top-1 at 595M MACs, 14x less computation than comparable models.

• Slide 6

Speedup divided by device count, e.g. 3.0x on 4 GPUs is 75% for DistriFusion; the shortfall from 100% is the cost of communication between devices.

• Slide 13

On accuracy-versus-MACs plots higher accuracy is better and lower computation is better, so the best models sit toward the upper left; the Pareto frontier is the set of models no other model beats on both accuracy and MACs, drawn as a staircase toward the upper left that architecture search shifts further left.

• Slide 5
• Slide 6

The sustained power a device can dissipate: about 10 W for a fanless phone, fixed by heat, so phone accelerators gain only through efficiency; 10 to 60 W across Jetson tiers with cooling; 250 to 700 W for a cloud GPU.

• Slide 18
• Slide 19
• Slide 20
• Slide 21

Segmentation task where the user supplies a prompt (a point, a box, a mask or text) and the model returns a valid mask for the object the prompt refers to, even for objects it was never told about.

• Slide 9

Digital signal processor in Snapdragon chips designed to deliver AI performance at low power, from 3 to 52 TOPS at about 10 W.

• Slide 19

Promptable image segmentation model with an image encoder, a prompt encoder and a lightweight mask decoder that outputs a valid mask for a given prompt.

• Slide 9
• Slide 10

The on-chip read-write memory of a microcontroller, e.g. 320 kB on the STM32F746, which holds activations during inference and therefore constrains the peak activation size.

• Slide 22
• Slide 23

About $600,000 of compute, 256 A100 GPUs for 150k GPU hours, illustrating the expense of training generative models.

• Slide 11

Diffusion model computation grows faster than the square of image resolution, e.g. FLUX.1 MACs rising from tens of trillions at 1024 to nearly 2000T at 4096.

• Slide 12

Complete compute board that integrates a GPU, CPU, memory, power management and high-speed interfaces in one module.

• Slide 21

Algorithm, hardware and data: the three factors whose joint progress drives deep learning capability and whose balance determines efficiency.

• Slide 3
• Slide 17

Number of inputs processed per second, e.g. images per second on an A100 (11 for SAM-ViT-H, 538 for EfficientViT-SAM-L2) or frames per second for a sensor stream; it depends on MACs, memory traffic and kernels, not on parameter count alone.

• Slide 10
• Slide 14

MCUNet's inference engine for microcontrollers: a code-generating runtime that executes the searched network without interpreter overhead, cutting SRAM use and flash size compared with generic frameworks.

• Slide 7

Running machine learning on tiny, resource-constrained IoT devices such as microcontrollers with kilobytes of memory.

• Slide 7
• Slide 23
• Slide 24

MCUNet's architecture search stage: a two-stage neural architecture search that first picks a search space fitting the microcontroller's SRAM and flash limits, then searches a network inside it.

• Slide 7

The units a transformer processes: text pieces for a language model, image patches for a ViT, or latent patches and frames for a diffusion model; linear-layer compute scales with the token count and attention with its square.

• Slide 12

Fraction of test images whose single highest-scoring predicted class matches the true label.

• Slide 5
• Slide 6

Fraction of test images whose true label is not among the model's five highest-scoring predicted classes.

• Slide 4

Trillions of operations per second, a performance metric for AI accelerators, e.g. 1750 TOPS for NVIDIA B100 and 52 for Snapdragon 8 Gen 1.

• Slide 18
• Slide 19
• Slide 20
• Slide 21

Peak TOPS divided by board power, the efficiency metric behind every accelerator chart: 0.075 for the P100 rising to 2.5 for the B100, and 0.33 to 5.2 across Snapdragon 845 to 8 Gen 1 under a fixed 10 W lid.

• Slide 18
• Slide 19
• Slide 20
• Slide 21

Persistent storage that holds model weights: about TB/PB for cloud AI, 256 GB for mobile AI, 1 MB for tiny AI.

• Slide 23

Accuracy of a model on a task or dataset it was not specifically trained on, such as SAM's zero-shot COCO mAP.

• Slide 10