NetAdapt: platform-aware pruning to a latency budget

Suppose a pretrained MobileNet runs in 10.0 ms on one core of a Pixel 1 and the product needs 7.0 ms. The Sensitivity analysis of part 03 cannot help directly, because its curves are in accuracy versus Pruning ratio, not milliseconds, and it ignores Layer interaction. AMC (AutoML for Model Compression) from part 04 could be pointed at latency, but it needs a reinforcement learning agent and a training loop of its own. NetAdapt asks a much simpler question, and asks it many times: which single layer can I thin right now so that latency falls by a fixed step while accuracy falls the least?

The slides call it a rule-based iterative, or progressive, method, and every word matters. Rule-based means no learned policy: the decision each round is a comparison of measured numbers. Iterative means the budget is reached in many small steps rather than one cut. Progressive means each step starts from the winner of the previous one. It is Automated pruning in the sense of part 04, because nobody hand-picks per-layer ratios, but the automation is a loop you could run with a spreadsheet.

The problem, and the trick of tightening it slowly

Yang et al. state the goal as a constrained optimization. Accuracy should be as high as possible while every resource of interest (latency, energy, model size) stays under its budget.

Solving that in one shot is hopeless, so the paper breaks it into a chain of easier problems. At iteration i the constraint is not the final budget but the previous network's resource use minus a step ΔR.

The paper calls ΔR the resource reduction schedule and compares it to a learning-rate schedule. In the small MobileNetV1 experiments it starts at 0.5 ms and decays by 0.96 every iteration; for larger networks the initial step is scaled with the network's latency. The slides simplify this to a manually defined constant, and the worked example below does the same. Think of it as a progressive barrier: each round the wall moves in a little, and the network only has to squeeze past the new wall, never the final one.

One iteration, step by step

Read slides 34 to 39 as a single loop body. The paper's Algorithm 1 has the same shape, with K equal to the number of convolutional and fully connected layers, so a network with K such layers produces K proposals per iteration.

1. Set the constraint for this round: Con = Res_i minus ΔR_i.
2. For each layer L_k, choose the number of filters to keep, from the lookup table, so the whole network meets Con; then choose which filters to keep by l2 magnitude.
3. Short-term fine-tune that proposal (the slides say about 10k iterations) and measure its accuracy on a holdout set.
4. Keep the proposal with the highest accuracy; it becomes the starting point of the next iteration.
5. Repeat while the latency is still above the budget.
6. Long-term fine-tune the final network until convergence.

Why a lookup table and not FLOPs

The constraint in Eq. 2 is a latency, so the loop needs to know the latency of every proposal, and there are K of them per iteration. Timing each one on the phone is slow and hard to parallelize. NetAdapt instead builds a Latency lookup table per layer before the loop starts: for every shape a layer might take (input channels, output channels, resolution), measure it once on the device and store the number. Layers that share a shape share an entry. The latency of a whole proposal is then the sum of its layers' entries. The paper's own illustration is a two-layer network where layer 1 with 4 filters on a 3-channel input costs 6 ms, and layer 2 with 6 filters on the 4 channels that layer 1 now produces costs 4 ms, so the network is estimated at 6 + 4 = 10 ms. Notice that the second entry is indexed by the first layer's filter count, which is how the table accounts for the removed input channels. A table indexed by whole networks would grow exponentially with depth; a table per layer grows linearly.

Why not skip the measurement and count multiply-accumulates? Because MACs are an indirect metric, and their relation to latency is neither linear nor the same across devices. The paper makes the point with a striking table: a network guided by MACs ended up with 19% fewer MACs than the baseline and 29% longer latency on a Pixel 1 CPU.

The lesson carries over to your own boards. Memory traffic, kernel launch overheads, cache behaviour and how well a library tiles a particular layer shape all change latency without changing the MAC count. The honest number is the one the device reports, and the lookup table is a cheap way to have that number on hand for every candidate. The paper checks the approximation on a Pixel 1 and finds the summed estimate highly correlated with the real measurement. Part 04 noted that AMC can also use a pre-built lookup table to optimize latency; NetAdapt makes it the default rather than an option.

Why fine-tune twice

There are two kinds of Fine-tuning in the loop and they serve different purposes. The short-term fine-tune happens once per proposal, so K times per iteration. Its only job is to make the K accuracies comparable. The paper reports that without it, the accuracy of a small network after pruning rapidly drops to nearly zero, and the algorithm then picks the best proposal solely based on noise. With 10k iterations the accuracy stays above 20% and the ranking becomes meaningful.

The long-term fine-tune happens once, after the loop has stopped, and runs until convergence. This is the step that repairs the damage of all the cuts on the final architecture. In the paper it adds between 1.8 and 4.5 points, 3.4 on average, on top of the last short-term number.

A run you can redo by hand

Take a four-layer network with 32, 64, 64 and 128 filters (layers A to D), a measured latency of 10.0 ms and a budget of 7.0 ms. Fix ΔR = 1.0 ms, ten percent of the starting latency. The lookup tables below are illustrative, the accuracies are synthetic, and the effect of one layer's cut on the next layer's input channels is ignored so the sums stay readable. The paper decays ΔR and accounts for the next layer.

Now run it yourself. The simulator uses the same tables and the same synthetic accuracy model, so its default run reproduces the example above. Then change the budget or the step and watch which layer gets picked: a smaller ΔR takes more iterations, produces more intermediate models, and tends to end slightly higher in accuracy, which is what the paper's schedule study found.

NetAdapt against AMC

Both methods answer the question of part 02, which Pruning ratio for which layer, without a human in the loop, and both were published at the same conference. They differ in almost every mechanism, and the NetAdapt paper even uses AMC (under its earlier name, ADC) as a baseline, beating it by 1.2x in latency on the large MobileNetV1.

Look back at the worked example. The run was aiming for one network at 7.0 ms, but on the way it produced Net_1 at 9.0 ms and Net_2 at 7.6 ms, and neither took any extra work. Each was the best of its round and each already satisfied its own, slightly looser, constraint. Slide 40 is that observation at full scale: the number of models equals the number of iterations.

The paper states it directly. Besides the final network, NetAdapt can generate a sequence of simplified networks, the highest-accuracy network from each iteration Net_1 to Net_i, which together provide the efficient frontier of accuracy and resource consumption trade-offs. The authors call it a family of simplified networks that allows dynamic network selection. With the 0.5 ms initial step and 0.96 decay, the small MobileNetV1 run in Table 2 took 28 iterations, so it delivered 28 networks. The red dots on slide 40 are a series of the same kind plotted the same way, one dot per iteration winner, from about 3.5 ms to 11 ms.

Reading the chart

The horizontal axis is latency on a single large core of a Pixel 1 CPU, measured with TensorFlow Lite as the median of eleven runs. The vertical axis is ImageNet top-1 accuracy. The red dots are the NetAdapt model series for the small MobileNetV1; the green triangles are MobileNetV1 shrunk with the width and resolution multipliers, which is Uniform shrinking from part 02; the blue diamond is MorphNet. The two arrows pick a multiplier point and the MorphNet point and slide left along a horizontal line to the NetAdapt curve, which is where the speedups on the slide come from.

From about 12.1 ms to about 7.3 ms is 1.7x faster with 0.3% higher accuracy than the multiplier baseline. From about 7.0 ms to about 4.4 ms is 1.6x faster with 0.3% higher accuracy than MorphNet. Across the whole curve the paper summarises it as up to 1.7x faster with the same or higher accuracy. On a mobile GPU (Samsung Galaxy S8 through SNPE) the gain drops to about 1.2x, because about 6.2 ms of that pipeline's latency is overhead that no amount of pruning can remove.

Why does the frontier beat the multipliers at all? A width multiplier thins every layer by the same factor, so it cannot express that some layers matter more. NetAdapt's greedy picks build a non-uniform architecture: the paper's Fig. 10 shows it keeping more filters in layer 6, where the feature map resolution drops, and in the last convolution that feeds the 1000-class classifier, while cutting layers 7 to 10 harder. That is the same lesson as part 02, now discovered by measurement rather than by a sensitivity sweep.

What the schedule does to the series

Because the series is a by-product of the loop, the step size shapes it. The paper compares three schedules for the same target latency.