Segmentation, generative models and LLMs

Picture a head-mounted camera looking at a door, a small label reading id 27, 52 cm, and a green mask painted over one region. A single click on the door produced that mask. That is what promptable means. The user supplies a hint, a point, a box or a phrase, and the model returns the object the hint refers to. The Segment Anything Model (SAM) from Kirillov et al. was trained on SA-1B, over one billion masks on 11 million images, so it can do this for objects it has never been told about.

The architecture matters more than the demo. SAM is three blocks. An image encoder, a ViT-H pre-trained with masked autoencoding, reads the image once and produces an image embedding. A prompt encoder turns sparse prompts (points, boxes, text) and dense prompts (masks) into prompt embeddings. A lightweight mask decoder combines the two and outputs a valid mask. The paper is explicit about the asymmetry: the image encoder runs once per image, and given a precomputed embedding, the prompt encoder and mask decoder run in a web browser, on a CPU, in about 50 ms. Encode once, expensively; prompt many times, cheaply.

Attack the block that dominates

Once you see that the encoder holds 2973 GMACs and the rest is a 50 ms CPU job, the efficiency strategy writes itself. EfficientViT-SAM (Zhang, Cai and Han) keeps SAM's prompt encoder and mask decoder unchanged and replaces only the image encoder with EfficientViT, the multi-scale linear attention backbone from Cai et al. (ICCV 2023). Training happens in two stages: first the new encoder is distilled from SAM's ViT-H encoder as the teacher, then the whole model is trained end to end on SA-1B. Nothing else in the pipeline moves.

The fair way to judge the swap is accuracy against speed on the same hardware: zero-shot COCO mAP against A100 throughput with TensorRT in FP16. Zero-shot here has a precise meaning: the models were never trained on COCO, and they are evaluated with boxes from a ViTDet detector as prompts. Two speedups get quoted, and they belong to different models. The 16.5x figure is EfficientViT-SAM-XL1, which is faster and more accurate than SAM (47.8 versus 46.5 mAP). The 48.9x headline belongs to EfficientViT-SAM-L2, which matches SAM at 46.6 mAP while running 538 images per second instead of 11. The full table from the paper makes the families visible.

The table carries a second lesson. EfficientSAM has 25.3 M encoder parameters, eight times fewer than XL1's 203.3 M, and yet the two run at the same throughput (183 versus 182 img/s), with EfficientSAM losing 3.4 mAP. Model size is about memory; speed is about MACs, memory traffic and kernels. Keep the three apart.

Type "teddy bears working as mad scientists", "a bowl of soup as a planet on a 1960s poster" or "an astronaut riding a horse on Mars" into Midjourney and a finished picture comes back. A diffusion model produced each one from a sentence, by starting from noise and removing a little of it at every step until an image appears. The results look effortless. The bill is not. Emad Mostaque, then CEO of Stability AI, said that training Stable Diffusion used 256 A100 GPUs for about 150,000 GPU-hours at a cost of about $600,000, the figure the lecture uses as Stable Diffusion's training cost.

Why parameters are not the cost

Training is paid once. Inference is paid on every request, so compare what one forward pass costs in MACs across four very different models. The natural expectation is that the biggest model costs the most. The numbers say otherwise, and the reason is the most important idea in this concept.

FLUX.1 has 12 B parameters and CogVideoX has 5.6 B, yet CogVideoX needs 8.9x the MACs per step. Llama 3 8B has more parameters than ViT-H by a factor of thirteen but costs eleven times more, not thirteen, and FLUX.1 with 1.5x the parameters of Llama 3 costs 3.3x as much. Parameters alone predict none of these ratios. What predicts them is how many tokens each model pushes through those parameters. A dense transformer spends roughly one multiply-accumulate (two FLOPs) per parameter per token in its linear layers, and on top of that attention compares every token with every other token.

Now the numbers make sense. ViT-H sees one 518 × 518 image cut into 14-pixel patches, 37 × 37 = 1,369 tokens: 632 M × 1,370 ≈ 0.87 T for the linear layers, and the attention term adds the rest of the 1.02 T. Llama 3 processes about 1,500 tokens. FLUX.1 works on the latent of a 1K × 1K image, about 4,000 tokens. CogVideoX compresses 49 frames of 480 × 720 video into 13 latent frames and denoises them all at once, about 17,500 tokens, roughly four times FLUX.1. Because the attention term grows with the square of that count, the bill grows far more than four times, and the smaller parameter count cannot save it. Read the linear term as the weights each token actually passes through, not the headline parameter count: for FLUX.1 the naive 12 B × 4,096 ≈ 49 T overshoots the 37.2 T bar because image and text tokens use separate weights in its double-stream blocks. One accounting rule explains the ordering of the chart: compute is parameters times tokens, plus attention.

One more multiplier hides in the words "single step" under the FLUX.1 and CogVideoX bars. A diffusion model repeats its forward pass once per denoising step. With a 50-step sampler, one 1K FLUX.1 image costs 37.2 T × 50 = 1,860 T MACs, and a CogVideoX clip costs 332 T × 50 = 16,600 T. A classifier pays its bar once; a generator pays it fifty times.

Resolution is worse than quadratic

Now hold the model fixed and vary only the image side length for FLUX.1, from 1024 to 4096. Going four times wider makes the image sixteen times larger in pixels, so a quadratic law would predict sixteen times the MACs. From the slide's left chart, one step at 1K x 1K costs 37.2 T, and the curve on the right ends near 1,950 T at 4096: a factor of about 52, far above the 16x a quadratic law predicts. This is what the lecture calls super-quadratic scaling, and the formula above explains it. Tokens grow with pixels (16x), the linear term grows with tokens (16x), and the attention term grows with tokens squared (256x). The blend lands at about 52x.

Ask SDXL for a "Romantic painting of a ship sailing in a stormy sea" at high resolution. One GPU paints one ship in 12.3 s. The obvious way to go faster is to cut the canvas into four vertical strips, hand each strip to its own GPU, and let them work without talking. The result is 3.9x faster and wrong: four strips, four ships, four different skies. Each patch generated its own picture because no patch could see the others.

DistriFusion (Li et al., CVPR 2024) keeps the split and restores the context. The trick rests on a property of diffusion: consecutive denoising steps produce similar activations, because each step removes only a little noise. So a patch can attend to its neighbours' activations from the previous step instead of waiting for the current one. The paper calls this displaced patch parallelism. The first few denoising steps (the warm-up: the first step plus four more in the paper) run synchronous patch parallelism so everyone starts from the same picture; after that, communication runs asynchronously and hides behind computation. No retraining is needed. The latency drops to 4.16 s, 3.0x faster than one GPU, and the ship is one ship.

The pattern is general in distributed systems: splitting work across machines helps only when the compute you save exceeds the communication you add. Here the saving is large because a high-resolution diffusion step is enormous (MACs in the hundreds of trillions), and the communication is hidden behind it. The paper also runs a scaling study on a larger image, 3840 × 3840, shown in the table below. Read down the efficiency column: every added GPU still helps, but each device contributes less than the last, because the communication grows while the per-device work shrinks.

A 2015 Toyota Prius V with a roof-mounted Velodyne LiDAR drives a wooded test track. A network reads raw 3D point clouds and produces steering. A car does not care how elegant the network is; it cares whether the answer arrives before the deadline the system sets. Three measurements tell the story: MinkowskiNet at 5 fps is "too slow to drive", the same network with optimised kernels reaches 18 fps, and Fast-LiDARNet reaches 47 fps, above the 30 fps line the slide marks as real time.

The most useful way to read fps is to invert it into a frame time. A stream at 30 fps gives you 33.3 ms per frame, and everything the model does must fit inside that budget.

The paper (Liu, Amini, Zhu, Karaman, Han and Rus, ICRA 2021) is explicit about the two levers. Their optimised sparse convolution kernel gives a 3.6x acceleration, and their hardware-aware model design reduces latency by another 2.6x. The two multiply: 3.6 × 2.6 ≈ 9.4, which is 47 / 5. The first lever is pure systems work: the same network, the same accuracy, better code for the sparse operations point clouds need. The second is algorithmic: a smaller architecture chosen with the hardware in mind. This pairing is what the lecture means by algorithm and system co-design. Neither lever alone reaches the line. Kernels take the network from 200 ms to 55.6 ms and still miss by 22 ms; the redesign brings it to 21.3 ms.

MT-NLG, the largest model on the lecture's chart, has 530 billion parameters. Store each one in FP16, two bytes, and the weights alone occupy 530 × 10⁹ × 2 = 1,060 GB. The largest GPU of that period, an A100 with 80 GB, holds 40 billion FP16 parameters. The gap between the two is 1,060 / 80 = 13.25, so 14 GPUs of weights before a single activation is computed.

To compare a model with a chip, put both on one log axis of model size by converting GPU bytes into a parameter count at FP16, which is what the lecture's assumption "data are FP16" means. 16 GB becomes 8 B parameters, 32 GB becomes 16 B, 80 GB becomes 40 B. On that shared axis the two curves tell the story of the large language model era: model size climbs from 0.05 B to 530 B, about 10,000x in roughly four years, while GPU memory climbs from 16 GB to 80 GB, 5x. Models have outgrown the chips they run on, and the distance keeps widening.

Parameters to bytes to GPUs

Every memory question in this course starts with one multiplication. Parameters times bytes per parameter gives the weight memory; divide by the memory of one device and round up to get the device count. The bytes per parameter depend on the numeric format: 4 for FP32, 2 for FP16 or BF16, 1 for INT8, 0.5 for INT4. Quantisation, which later lectures cover in depth, is simply the art of moving down that list without losing accuracy.

Try Llama 3 8B in the calculator. In FP16 it needs 16 GB, four times the 4 GB of RAM slide 23 gives a phone. Slide 23 lists that 4 GB as activation memory and 256 GB as weight storage; in practice LLM inference loads the weights into that same RAM, so the 4 GB is the binding limit. In INT4 the model needs 4 GB, exactly at the line, with nothing left for activations. Now try the 2017 Transformer at 0.05 B against a microcontroller's 1 MB of weight storage: 100 MB in FP16, a hundred times over. The gap between what models need and what edge hardware has is the subject of part 03.

Why run them at the edge anyway

If LLMs barely fit in a rack of cloud GPUs, why does the lecture insist on running them on laptops, cars and robots? Three reasons, each one a research direction in this course.

1. Copilot services, such as code completion, office assistants and in-game chat, running locally on the machine that uses them. Latency drops, and the service works in a car or a robot far from any data center.
2. Resource-constrained, low-power devices that are not always connected. An edge LLM must work when the link is slow or absent.
3. Privacy of user data. Prompts contain code, documents and conversations, and a local model never uploads them.

This is the design space for an on-device assistant, and it is why the rest of the course is about shrinking the memory (quantisation, pruning), the compute (efficient attention, distillation) and the deployment (compilers, kernels) of exactly the models on this chart. For a PhD project on embedded ML, the two-slope chart is the problem statement.