COE 592Lecture 00Reference

Reference sheet

Course overview compressed onto one page: the definitions, formulas and numbers to have in your head before a quiz or exam.

Course facts and week one

Everything the deck states about the course, the instructor and the first evening. Part 01: The instructor and edge AI in practice, Part 02: Course takeaways and getting started

Facts as stated on slides 1, 2, 9, 10, 12 and the course registry

Course: COE 592 Machine Learning on Embedded Systems, term T261
Instructor: Dr. Abdul Jabbar Siddiqui, Computer Engineering, KFUPM; SERV Lab lead; SDAIA-KFUPM JRC-AI scholar
Degrees: PhD uOttawa 2021, MASc uOttawa 2015, BSc KFUPM 2012; NRC Canada 2020 to 2022
Meetings: Monday and Wednesday, 20:10 to 21:25, Building 59, Room 2004
Icebreaker: Name, educational background, work background
Survey: https://forms.cloud.microsoft/r/5ssu8gkcS9
Syllabus, grading, deliverables: On Blackboard only; not in the deck

Fill in the survey and introduce yourself with the three items.
Pick a device tier first; its memory budget is an input to model selection, not an output.
Pick a candidate model that fits or is one or two optimizations from fitting; record parameters and float32 size.
Start a metrics log before the first optimization. A baseline taken after changes is not a baseline.

When	What to record
Week 1	Device tier and its flash, SRAM or RAM budget from the datasheet
Week 1	Candidate model, parameter count, float32 weight size
Week 1	Baseline accuracy on a held-out set you never change
First run on hardware	Latency per inference and peak memory as the runtime reports them
Every optimization	The same numbers again next to the baseline, plus what it cost

Starter metrics log

SERV

Secure, Efficient, Robust Vision: the lab name is the syllabus in miniature. Part 01

Letter	Word	Course theme	Example in the deck
S	Secure	Adversarial machine learning, intrusion detection	PNet-IDS (slide 6)
E	Efficient	Compression, pruning, quantization, nano detectors	EcoWeedNet (slide 4), PNet-IDS (slide 6)
R	Robust	Rain, noise, blur, distribution shift	YOLO-RAW (slide 5)
V	Vision	Cameras, thermal imagers, LiDAR as input	Slides 3 to 8

SERV mapped onto the course

Inspection rigs: shrink the model or ship the data

A UAV lifts a fixed mass, and every gram of sensor or computer is a gram less battery. Two options exist: run a model small enough on board, or ship raw data. The bandwidth arithmetic rules out the second. Part 01

Sensor	Measures	Good at	Weak at	Data rate
Visual (RGB) camera	Reflected visible light	Colour and texture	Darkness, rain, glare	Highest
Thermal (IR) camera	Emitted infrared, so temperature	Hot bearings, people at night	No colour, low resolution	Moderate
LiDAR	Pulsed-laser time of flight, so range	Geometry, clearance, 3D point clouds	Sparse at range, cost, weight	High
GPS receiver	Position and time	Georeferencing frames and points	Nothing about the scene	Negligible

The four modalities on the slide 3 rigs

Five-camera rig, worked numbers

Assumption: 5 cameras at 1080p, 8 Mbit/s each (illustrative)
Stream leaving the rig: 5 × 8 = 40 Mbit/s
One hour of driving: 40 × 3600 = 144,000 Mbit, about 18 GB
Upload at 5 Mbit/s: 144,000 / 5 = 28,800 s, about 8 hours per hour recorded
Satyanarayanan 2017: 12,000 users at 1080p need about 100 Gbit/s ingress; a million users need 8.5 Tbit/s

Nano detectors: efficiency measured

YOLO size classes n, s, m, l, x share one architecture and differ in width and depth. Figures are Ultralytics, COCO validation at 640 px. Part 01

Model	Parameters	FLOPs	mAP50-95	CPU ONNX	T4 TensorRT
YOLO11n	2.6M	6.5B	39.5	56.1 ms	1.5 ms
YOLO12n	2.6M	7.5B	40.6	not listed	1.64 ms
YOLO11m	20.1M	68.1B	51.5	183.2 ms	4.7 ms
YOLO11x	56.9M	195.3B	54.7	462.8 ms	11.3 ms

YOLO size classes

Nano versus medium, and EcoWeedNet's result

Parameters: 20.1 / 2.6 ≈ 7.7x fewer in YOLO11n
FLOPs: 68.1 / 6.5 ≈ 10.5x fewer per image
CPU latency: 183.2 / 56.1 ≈ 3.3x lower; below the FLOP ratio because small models leave hardware idle
Price: 51.5 − 39.5 = 12.0 mAP50-95 points
EcoWeedNet: 95.2% mAP@0.5 on CottonWeedDet12 with about 4.21% of YOLOv4 parameters and 6.59% of its GFLOPs

Number	Measures	Unit	Kind
Parameters	Size, so memory	millions (M)	Metric
FLOPs	Compute per inference	billions (B) or GFLOPs	Metric
Latency	Time per inference on a named device	ms	Metric
mAP	Detection quality on a dataset	percent or points	Metric
GradCAM++ heatmap	Which pixels drove a class score	none	Attribution, not a metric

What each number measures

Rain scene: robustness measured

YOLOv5-m (21.2M parameters, 49.0B FLOPs) against YOLO-RAW on rainy, Gaussian-noise and motion-blur test sets. Gulls numbered top to bottom in the YOLO-RAW panel. Part 01

Object	YOLOv5-m	YOLO-RAW
Gull 1 (top left)	no box	Bird 0.50
Gull 2 (centre)	UAV 0.29 (false positive)	Bird 0.63
Gull 3	no box	Bird 0.54
Gull 4	no box	Bird 0.73
Gull 5 (bottom)	Bird 0.35	Bird 0.37 (approximate)
Real drone, rain field	UAV 0.90	UAV 0.93

Every label on slide 5

\text{precision} = \frac{TP}{TP + FP}, \qquad \text{recall} = \frac{TP}{TP + FN}

The two numbers a confidence threshold trades against each other

Can a threshold fix the baseline?

Ultralytics default: 0.25
At 0.30: YOLOv5-m loses the false UAV and nothing else, but still sees one bird in five
At 0.60: YOLOv5-m reports nothing; YOLO-RAW has lost three of five correct birds
Rule: A threshold slices the confidence mass a model produced. A better model moves that mass.

PNet-IDS: line rate on a gateway

A network-based IDS (NIST SP 800-94) running on the gateway CPU that forwards the packets. The deadline is set by the link, not the designer. IEEE Access 2025, DOI 10.1109/ACCESS.2025.3575705. Part 01

t_{\text{budget}} = \frac{1}{\text{packets/s}} = \frac{L_{\text{packet}} \times 8}{R}

Time per packet on a link of rate R bits per second

Line-rate arithmetic (illustrative numbers)

Packets per second: 100 × 10^6 / (500 × 8) = 25,000 on a 100 Mbit/s link with 500-byte packets
Budget per packet: 1 / 25,000 = 40 µs
Heavy model: 10 MFLOPs at 1 GFLOP/s is 10 ms, about 250x too slow
Light model: 0.02 MFLOPs at 1 GFLOP/s is 20 µs, inside the budget
Generalizable: Accuracy held under traffic distribution shift, via knowledge distillation (student reproduces a teacher)
Benchmarks: BoT-IoT and CIC-IDS2017; wins are fewer parameters, fewer FLOPs, smaller size at maintained accuracy

Physics sets the deadline

Distance travelled during the reply latency is the safety margin lost. Deadline first, then accuracy. Part 01

d_{\text{lost}} = v \times t_{\text{latency}}

Metres of travel before the detector answers

Scenario	Speed	Travel per latency	Worked number	Cost of a miss
Shuttle at a T-intersection	10 m/s	1 m per 100 ms	200 ms cloud round trip is 2 m of travel	A pedestrian is struck
Wagon past a wayside camera	15 m/s	1.5 m per 100 ms	33 ms per frame at 30 fps, wagon moves 0.5 m	A defect leaves the yard

Two safety-critical edge inference problems

Five takeaways as one loop

Measure, optimize, compare, explore, deploy, then measure again. Deep Compression (Han, Mao and Dally, 2016) is the canonical worked loop. Part 02

Pillar	Verb	Question	Example
1 Measure efficiency	Measure	How big, slow, hungry is the baseline?	AlexNet 240 MB, 61 M parameters, 32-bit
2 Accelerate	Optimize	Which technique shrinks the cost that hurts most?	Prune 9x, quantize to 5 bits, Huffman code
3 Analyse trade-offs	Compare	What did each step cost in accuracy or effort?	35x smaller, no accuracy loss, after retraining
4 Track the frontier	Explore	What has research or industry done since?	MCUNet, TinyTL, training under 256 KB
5 Build and deploy	Deploy	Does it really run on the target hardware?	3x to 4x speedup, 3x to 7x energy efficiency

Pillar, verb, question, Deep Compression example

Efficiency metrics and the size formula

Seven metrics that can disagree: a pruned model can be smaller yet no faster on hardware that cannot skip zeros. Part 02

Metric	What it counts	Unit
Parameter count	Learnable weights and biases	M
Model size	Parameters × bits per weight / 8	MB or KB
Peak memory	Weights plus the largest activation tensors at once	MB or KB
Compute	MACs or FLOPs per inference	MMACs or GFLOPs
Latency	Time from one input to its output	ms
Throughput	Inferences per second	inferences/s
Energy	Joules per inference or average power	mJ or mW

Efficiency metrics (preview of later lectures)

\text{size (bytes)} = \frac{N_{\text{params}} \times b}{8}

N parameters at b bits each; MB means one million bytes

Bits	Calculation	Size	Pico (2 MB)?	Nano 33 BLE Sense (1 MB)?
32-bit	3,000,000 × 32 / 8	12 MB	No	No
16-bit	3,000,000 × 16 / 8	6 MB	No	No
8-bit	3,000,000 × 8 / 8	3 MB	No	No
4-bit	3,000,000 × 4 / 8	1.5 MB	Yes	No

3 M parameters at four bit widths against two flash budgets

Inference versus training on device

Item	Inference	Training
Weights	Yes	Yes
Current activations	Yes	Yes
Stored activations for the backward pass	No	Yes, the bottleneck (TinyTL)
Gradients for every weight	No	Yes
Optimizer state	No	Yes
LiteRT for Microcontrollers	Supported	Not supported

What each holds in memory

Device tiers

Datasheet capacities, not what is free once the runtime is loaded. MCUNet reached over 70% ImageNet top-1 on a microcontroller, after the whole loop on a purpose-built architecture. Part 02

Tier	Board	Memory	Typically runs
Microcontroller	Arduino Nano 33 BLE Sense Rev2 (nRF52840)	1 MB flash, 256 KB SRAM	Keyword spotting, gesture, tiny classifiers
Microcontroller	Raspberry Pi Pico / Pico 2	2 MB flash, 264 KB SRAM / 4 MB, 520 KB	Same class, more headroom
Single-board computer	Raspberry Pi 4 Model B	1 to 8 GB LPDDR4	Quantized MobileNet-class vision, small detectors
GPU module	Jetson Orin Nano Super	8 GB LPDDR5, 102 GB/s, 67 INT8 TOPS	Real-time detection, small transformers, fine-tuning

One example board per tier

Slide errata and reading notes

Answer with the corrected fact; do not attribute results to uncaptioned photographs. Part 01, Part 02

What the slides get wrong or leave out

Slide 2: Degree punctuation is inconsistent, (UOttawa 2021) versus (UOttawa, 2015). JRC is unexpanded: SDAIA-KFUPM Joint Research Center for AI.
Slide 3: No caption, no citation. Describe the rig (RGB, thermal, LiDAR, GPS); attribute no measured result.
Slide 5: RAW is never expanded, on the slide or in the paper. The fifth YOLO-RAW label is occluded and reads about Bird 0.37.
Slides 7, 8: Photographs with no paper and no result. Cite FHWA HRT-18-044 and the NRC rail review for the problem class only.
Slide 11: "Resource-constraint platforms" should be "resource-constrained". Spelling mixes British "Analyse" with American "Optimize".
Slide 12: Only "View on Blackboard". No grading weights, exam dates, textbook or office hours are in the deck.