Neural network fundamentals

AOverview

Before you can shrink a network you have to be able to read one. This lecture is the anatomy lesson for the whole course: it takes every block a modern model is built from, the neuron, the fully connected layer, the convolution with its padding, stride and groups, pooling, the normalization family, the activation function and the transformer block, and teaches you to write each one as a tensor equation whose parameters, activations and output shapes you can count from a diagram.

As a PhD student you need this in three places. Exams hand you a layer list and ask for shapes, parameter counts, output sizes after padding and stride, a batch norm or an attention computed by hand, and the reason a microcontroller prefers ReLU6 to swish. Your research project puts a model on a board with a few hundred kilobytes of SRAM, and every pruning, quantization and architecture decision you defend there is a decision about the counts this lecture defines. And any paper or codebase you read from now on, from MobileNet to a transformer, assumes you can look at W (c_o, c_i, k_h, k_w) or softmax(Q Kᵀ / √d_k) V and see both the arithmetic and the memory it costs.

The path has six stops. You start with one neuron as a weighted sum and stack neurons into fully connected layers and multilayer perceptrons, learning to count parameters against flash and activations against SRAM. You then trade full connectivity for a small shared filter and get the convolution, with its four tensors and its shrinking feature map. Padding, stride and groups follow: the three knobs that set output size, receptive field and cost without touching the idea. Pooling shrinks every map with zero parameters, and the feature hierarchy explains what the stacked filters end up learning. Normalization recentres activations so training works, then folds away at inference. The lecture closes with the activation functions an int8 device can afford and the transformer block, whose attention matrix grows with the square of the sequence length.

Formulas you will own by the end

Each row is derived, not quoted, in one of the six parts, and each is checked against a number from the slides (the 47 parameters of the 5-4-3-2 network, the 1216 of the ConvNetJS first layer, the 576 depthwise weights against 36,864 standard). They are the anchors the exam and the reference sheet return to.

Success looks like

• Write a neuron as y = f(Σ w_i x_i + b), name every symbol in both the biological and the tensor vocabulary, and count layers without counting the input.
• State the shapes X (n, c_i), W (c_o, c_i), b (c_o,), Y (n, c_o) of a fully connected layer, and say which count the batch size touches and which memory each lands in.
• Write the four Conv2D tensors in PyTorch order, compute the output size from h_i, k, p and s, and evaluate one output value by hand from a multi-channel input.
• Compute the receptive field after L layers with and without stride, and count the weights of standard, grouped and depthwise convolutions to justify a MobileNet-style block for a device.
• Chain shapes through a CONV-RELU-POOL network, separate the layers that cost parameters from those that cost only activations, and explain the edges-to-objects hierarchy the filters learn.
• Write the batch norm transform with every tensor shape, derive the fused W' and b' at inference, and name which axes BN, LN, IN and GN average over.
• Draw the transformer block, write Attention(Q, K, V) = softmax(Q Kᵀ / √d_k) V with shapes, compute a small attention by hand, and argue why N² scores and exponentials are the embedded bottlenecks.

How to study this lecture

1. Read the parts in order. Each block reuses the counting recipe of the one before it: the FC layer teaches the recipe, convolution adds spatial dimensions, and every later block is a variation.
2. Keep a pencil next to every worked example. Compute the slide's number before you reveal it, then check yourself in the layer builders, the activation plotter and the attention stepper.
3. Answer every recall prompt in your head before opening it. Shapes and formulas that you retrieved stay; the ones you only read do not.
4. Take each quiz and read the explanation even when you are right; the wrong options are the exam traps.
5. Read the slide errata carefully. Several slides carry inherited arithmetic slips, and spotting why they are wrong is the best test that you own the formula.
6. Mark a concept understood only when you could redo its formula and its slide number from a blank page. Unmarked concepts show you where to return.
7. Come back after a few days and redo the counts cold on a network you have not seen, such as the backbone of your project detector. Spaced practice on a new example is what turns a formula into a skill.

BThe 6 parts

1. 01Neurons, weights and the fully connected layerThe vocabulary of neural networks and the simplest layer, where every output neuron sees every input neuron.5 conceptsSlides 1-730 min
   1. 1.1Why an embedded-ML course starts by counting neurons
   2. 1.2One neuron: weighted inputs, a bias, and a squashing function
   3. 1.3Layers, width, depth, and the two synonym families
   4. 1.4The fully connected layer as one matrix product
   5. 1.5Stacking FC layers into a multilayer perceptron
2. 02Convolution layers and their tensor shapesLocal receptive fields and weight sharing, from 1D to 2D convolution, the no-padding output size and a fully worked two-channel example.4 conceptsSlides 8-1524 min
   1. 2.1Local connections and one shared filter: the two ideas behind convolution
   2. 2.2Reading the four Conv2D tensors: X, Y, W and b
   3. 2.3Sliding the window, and why the feature map shrinks
   4. 2.4Computing one output value by hand, then counting the parameters
3. 03Padding, stride, receptive field and grouped convolutionHow padding and stride control the output size, how the receptive field grows with depth, and how grouped and depthwise convolution cut parameters.4 conceptsSlides 16-2024 min
   1. 3.1Padding: choosing the output size without touching the weights
   2. 3.2Receptive field: why deep nets stride to see the whole image
   3. 3.3Grouped convolution: g narrower convolutions side by side
   4. 3.4Depthwise convolution and the question the slide asks
4. 04Pooling and what CNN filters learnParameter-free downsampling with max and average pooling, the feature hierarchy a CNN learns, and two interactive demos.4 conceptsSlides 21-2624 min
   1. 4.1Pooling shrinks every map, and only the map
   2. 4.2Two hyperparameters, zero parameters: the pooling arithmetic
   3. 4.3From edges to objects: what the filters learn
   4. 4.4Two browser demos as counting practice
5. 05Batch, layer, instance and group normalizationWhy nicely scaled activations make optimization easier, how batch normalization computes, scales and shifts, how it changes at inference, and the axes each normalization variant averages over.5 conceptsSlides 27-3830 min
   1. 5.1Why a layer wants zero-centered, evenly scaled inputs
   2. 5.2The batch norm transform: normalize per feature, then let gamma and beta choose the range
   3. 5.3At inference the statistics are frozen, so BN folds into the layer before it
   4. 5.4Batch, layer, instance and group norm differ only in which cells share one mean
   5. 5.5Batch norm by hand: four samples, three features, every number
6. 06Activation functions and the transformerThe non-linearities used in embedded networks, then the transformer block: scaled dot-product attention worked by hand and the position-wise feed-forward network.6 conceptsSlides 39-4536 min
   1. 6.1Why the neuron needs a bend, and which bends an MCU can afford
   2. 6.2The transformer block: attention, Add and Norm, feed-forward, positional encoding
   3. 6.3Attention as retrieval: query, key, value and the N x N cost
   4. 6.4Scaled dot-product attention computed by hand on four tokens
   5. 6.5From attention weights to a position-wise feed-forward network
   6. 6.6Where this lecture's ideas come from

CGlossary and reference

• GlossaryEvery term from the lecture, defined once, with links back to the slides that use it.
• Reference sheetThe lecture on one printable page, for revision before a quiz or exam.