Latency, throughput and energy

Suppose the detector you trained in the cloud is to run on a Jetson-class board mounted on a vehicle. Before the board can use it, three separate questions must be answered. Does the model fit in the flash and RAM you have? Does one inference finish before the next camera frame lands, roughly every 33 ms at 30 frames per second? And how much charge does each inference pull from the battery? Slide 4 draws these as the three outer goals of a triangle, Smaller, Faster and Greener, and the three costs that quantify them: Storage, Latency and Energy.

The part of the triangle that matters is the middle. All three costs are driven by only two resources, Computation and Memory. Storage is purely a memory question: how many bytes the weights occupy. Latency and energy depend on both, because an inference both performs arithmetic and moves data, and each of those takes time and burns charge. That is why the metric list on the side of the slide has exactly two groups. The efficiency metrics that are memory-related are the number of parameters, the model size and the total and peak number of activations. The computation-related ones are MACs, FLOPs with their rate FLOPS, and OPs with their rate OPS.

This taxonomy is the spine of the whole lecture. Parts 02 and 03 fill the memory group (parameters, model size, activations), part 04 fills the computation group (MACs, FLOPs, OPs), and this part explains the three costs those metrics are meant to predict. Whenever a later metric appears, place it here first: which group is it in, and which cost does it feed?

Metrics you count versus costs you measure

One distinction keeps the rest of the part honest. Parameters and MACs are properties of the network alone. You can count them on paper from the layer shapes, and they do not change when you move the model to another chip. Latency, throughput and energy are properties of the network running on a specific processor with a specific runtime. They must be measured, or at least estimated from hardware numbers. The same split reappears in the latency model of concept 4 as NN specification versus hardware specification.

Slide 5 shows the same Cityscapes street segmented twice. The two masks look almost identical, and by the usual accuracy metric they are: SegFormer-B5 reaches 82.4 mIoU and EfficientViT-L1 reaches 82.7 mIoU. The difference is time. SegFormer needs 638 ms per frame and EfficientViT needs 45.9 ms (the slide rounds to 46 ms). At 30 frames per second a new frame arrives every 33 ms, so one model falls nineteen frames behind for every frame it finishes, and the other almost keeps up.

That per-frame time is latency: the delay between starting one specific task and finishing it. Patterson and Hennessy call the same quantity response time or execution time, the time between the start and completion of a task. It is a property of one task, not of a stream of tasks, and the next concept shows why that restriction matters. The 13.9x gap (638 / 45.9) is exactly the headline of the EfficientViT paper, which reports up to 13.9x GPU latency reduction over SegFormer at the same or better accuracy (Cai et al., ICCV 2023).

Read the ratio row carefully, because it plants the seed for concept 4. The MAC count falls 5.2x, but latency falls 13.9x. If latency were simply proportional to arithmetic, the two ratios would match. They do not, because SegFormer's softmax attention moves large intermediate tensors through memory, and that data movement, not its MACs, is what makes it slow on an edge GPU. MACs alone do not predict latency; they are one input to it.

Four conditions every latency must state

The slide's footnote is not fine print. Latency is measured, so it depends on everything in the measurement. The paper's setup sentence names four things: the hardware, the runtime, the numeric precision and the batch size. Change any one of them and the number changes, sometimes by an order of magnitude. The same paper measures EfficientViT-B1 at 24.3 ms on Orin and 0.82 s on a Jetson Nano, and SegFormer-B1 at 146 ms on Orin and 5.6 s on Nano. Same model, same precision, same batch size, different board: 34x to 38x apart.

Slide 6 replaces one street with two mosaics of video clips and asks a different question: not how long one clip takes, but how many clips get done per second. One system manages 6.1 videos per second, the other 77.4. Those two numbers come from Table 5 of the TSM paper (Lin, Gan and Han, ICCV 2019): I3D at 306 GFLOPs versus TSM8F at 33 GFLOPs, both on a single Tesla P100.

That rate is throughput: the amount of work completed per unit time, what Patterson and Hennessy also call bandwidth. The TSM paper is unusually helpful here because it states its protocol: latency was measured with batch size 1 and throughput with batch size 16. TSM8F's latency is 17.4 ms. If throughput were the reciprocal of latency you would expect 1000 / 17.4 = 57.5 videos per second, yet the paper reports 77.4. The extra comes from batching, and it is the whole point of slide 7.

Two designs, same processor, opposite answers

Slide 7 pits two designs against each other. Design 1 handles one image at a time and takes 50 ms per image. Design 2 takes four images together and finishes the whole group in 100 ms. Each image in Design 2 waits the full 100 ms, because nothing leaves the processor until the batch is done.

A second scenario for practice

A cloud server serves the same model at two batch sizes. At batch 1 a request takes 2 ms. At batch 32 the whole batch takes 8 ms, because the GPU can spread thirty-two images over its cores far more efficiently than one.

Batching paid 4x latency to buy 8x throughput. Whether that is a good trade depends on who is waiting. A brake camera needs one frame answered now: latency, at batch size 1, which is why slide 5 measured that way. A data center indexing videos overnight needs clips per hour: throughput, at whatever batch size fills the GPU. Your research project sits on the first side of that line.

Concept 2 left a puzzle: MACs fell 5.2x but latency fell 13.9x. To resolve it you need a model of where the milliseconds go. Watch one layer execute. The processor loads the input activations, loads the layer's weights, multiplies and accumulates, and stores the output activations. Two of those steps are arithmetic; the other two are data movement. Real hardware does not wait for one to finish before starting the other. While the multipliers chew on the current tile, the memory system is already fetching the next weights, so the two streams run side by side and the layer finishes when the slower stream finishes.

Computation time is the operation count divided by the rate at which the processor retires operations. Memory time is the bytes that must move divided by the memory bandwidth, once for the weights and once for the activations in and out. Now look at the colour coding on the slide, because it is the sentence the exam wants. Every numerator is a property of the network: operations, model size, activation sizes. Every denominator is a property of the processor: operations per second, bandwidth. That split is the NN specification versus hardware specification distinction, and it makes efficiency a co-design problem. You change numerators by redesigning the network, denominators by choosing hardware, and latency by whichever of the two you can actually afford to change.

The same model appears in the industry literature under different names. NVIDIA's GPU performance guide states that memory time is bytes accessed divided by memory bandwidth, math time is operations divided by math bandwidth, and the longer of the two shows what limits performance: math-limited if math time is longer, memory-limited if memory time is longer. The roofline model of Williams, Waterman and Patterson (CACM 2009) writes the same idea as a rate, attainable performance = min(peak compute, bandwidth x operational intensity). The vocabulary to adopt is compute-bound when T_computation wins and memory-bound when T_memory wins. One caution before using datasheet numbers: dividing by a peak OPS figure gives a floor on T_computation, not an estimate of it, because real kernels rarely keep every multiplier busy. The measured time over that floor is the inverse of the utilisation the runtime achieved.

Worked example A: a toy model and two processors

Worked example B: back to slide 5

The third cost is energy, and slide 9 settles where it goes with one table. In a 45 nm process at 0.9 V, a 32-bit integer add costs 0.1 pJ and a 32-bit integer multiply 3.1 pJ. Reading one 32-bit word from DRAM costs 640 pJ. The multiply is the most expensive arithmetic on the list, and the memory access is two hundred times dearer than it.

Three ratios from that table are worth memorizing. DRAM over integer multiply is 640 / 3.1 ≈ 206, which the slide rounds to 200x. DRAM over SRAM is 640 / 5 = 128, so keeping data on-chip is worth two orders of magnitude on its own. And float add over integer add is 0.9 / 0.1 = 9, a seed for quantization: the same arithmetic in integers is nine times cheaper per add. Han et al. summarize the whole table in one line, that memory access is three orders of magnitude more energy expensive than simple arithmetic (640 / 0.1 = 6400).

The chain the slide writes across the top follows directly: data movement means more memory references, and more memory references mean more energy. Horowitz's own framing in the ISSCC 2014 talk is that a DRAM access at 1 to 2 nJ is a couple of orders of magnitude above a cache access or a functional operation at around 10 pJ, and that even with better I/O circuits DRAM will stay expensive, around 0.6 nJ per 8 bytes, because requests and data still have to travel a long physical distance. The slide's 640 pJ per 32-bit word is about half of the 1.3 nJ per 64-bit access in his Figure 1.1.9, as the provenance note below explains.

The log axis of the slide's chart is not decoration either. On a linear axis the six small bars would be invisible next to DRAM; a 0.1 pJ bar next to a 640 pJ bar is a pixel next to a page. Only a logarithmic axis can show both, and it is the honest way to read any table whose entries span four decades.

Worked example: AlexNet's energy bill

Every efficiency technique later in the course can be read as a way of cutting that memory line. Keep weights in on-chip SRAM, which is 128x cheaper per access than DRAM. Reuse each fetched value as many times as possible before it leaves the chip; Horowitz notes that efficient designs complete on the order of a thousand operations per DRAM fetch. And make the model small enough to fit on-chip in the first place, which is what model size in part 02, peak activations in part 03, and pruning and quantization later in the course are all about.