Latency budget, exercises and summary

Picture a video call. When you speak, your voice should reach the other person quickly enough that the two of you do not talk over each other. The ITU-T telephony standard G.114 puts a number on “quickly enough”: if one-way delay stays below 150 ms, most applications experience essentially transparent interactivity, and 400 ms should not be exceeded for general network planning. So for one direction of that call, you have about 150 ms to spend, and every hop, queue and processing step you place in the path takes some of it.

That allowance is the Latency budget. Its upper limit is written τ (tau): the maximum acceptable end-to-end Response time (RT) of a service. The budget is the interval $[0, \tau]$, and a design is inside it when $RT \le \tau$. Notice who sets τ. It is not the network, and it is not the hardware you happen to own. It comes from the user and the application: human perception for a chat or a game, physics for a headset that must keep up with your head. The network and the servers only decide how much of the budget you spend.

What counts against the budget

It is tempting to compare τ with the Latency to a layer: the client reaches the edge with small latency, the fog with medium latency and the cloud with large latency. But latency is only part of the bill. Recall from part 9 that response time is the trip out, plus the Processing time (PT) at the server, plus the trip back:

A per-layer latency figure is one-way, so never compare a single arrow directly with τ. When a pipeline splits its tasks across several layers, every hop between them joins the sum as well, together with every task’s processing time. The gauge below makes the consequence visible: the same request, three placements, one deadline.

The last step of that example hides the real tension of this whole part. Moving closer shrinks the latency terms, but closer sites are smaller, so the processing term can grow. Neither “always edge” nor “always cloud” is a rule; the inequality decides. The simulator lets you feel that tension. Its three tasks are an image resize, object detection and drawing boxes, and the presets map them to different layers. Set τ to 100 ms and find the one preset that stays in budget.

Where real values of τ come from

Because τ belongs to the user, you should never invent it. Studies of human perception and device physics give defensible numbers, and they span four orders of magnitude: a headset that must track your head has a couple of tens of milliseconds, a conversation has about a hundred and fifty, a recognition assistant has most of a second, and a user waiting for a report keeps attention for about ten seconds. The table collects the values this course relies on, roughly from the strictest to the most relaxed. You will use every row in the design exercises that follow.

One more rule turns these numbers into a real test. A budget is a promise about the experience of users, and users remember the slow requests, not the average one. So published budgets are stated at a percentile.

Knowing the inequality is not the same as designing a system. Real services are not one request and one server; they are pipelines of tasks with different needs. Design questions of this kind have no single right answer, but they do have a right method, and it is the same four steps every time: list the tasks, set τ per task, place each task on a layer, then check Response time (RT) and the trade-offs.

Smart glasses are a good first case because every constraint pulls hard. The device is small, runs on a tiny battery and sits on your face, so it cannot get hot. Yet the job, recognizing the faces and objects in front of you, is heavy vision computing. Before reading on, take a minute and list the tasks a face-recognition assistant on glasses would need. That effort is what makes the model answer stick.

Step 4 is worth a second look, because it shows the method working honestly. The cloudlet design does not magically hit every bound: it meets the loose face-recognition deadline and misses the tight one. That is a real, measured result at the tail, and it is exactly the kind of statement a good design report makes. The cloud alternative misses by more, so the edge is the better placement, not a perfect one.

The trade-offs behind the placement

Latency is not the only reason for this split. Shi et al., summarizing Ha et al.’s cloudlet study for wearable cognitive assistance, report that offloading can cut energy use by 30 to 40%, which matters on a device with a battery the size of a stick of gum. They also argue that processing at the edge protects privacy better than uploading raw data to the cloud, and faces are among the most sensitive data there is.

The price is dependence. The glasses now need a working wireless link and an edge site near wherever the wearer walks. Edge sites are few and expensive, so scalability and reach are limited. The cloud has global reach and elastic capacity, but it misses the tight end of the budget. That is exactly why the design keeps both, and why every placement you propose should name what it gains and what it gives up.

Picture a warehouse worker holding a tablet. Overlays mark the next shelves to pick from, a ring counts progress, and a green arrow on the floor says the target is 12 m ahead. It looks like one app. For design purposes it is at least six tasks with deadlines that range from milliseconds to hours. That is the central idea of this concept: τ belongs to a task, not to an app, and a single app routinely needs every layer of the continuum at once.

Motion-to-photon: the tightest budget in the course

Both augmented and virtual reality are governed by one deadline, so it pays to understand it before designing either. When you move your head, or the tablet, the image must move with you. The time from the movement to the matching light leaving the display is motion-to-photon (MTP) latency. If it is too long, your eyes and your inner ear disagree. Elbamby et al. explain that high MTP sends conflicting signals to the vestibulo-ocular reflex, which can cause motion sickness. In AR the failure is also visible: a virtual label that lags behind a real shelf is noticed at once, because the real world is the reference.

The research community broadly agrees on the ceiling. Elbamby et al. report a broad consensus that MTP must stay below 15 to 20 ms. RFC 9699 uses 20 ms at most for XR as a whole, and shows how little of it is left for computing once the display has taken its share:

Estimates of the split vary. Elbamby et al. put display delay at about 10 to 15 ms, expected to fall to 5 ms, which leaves 14 ms for computing and communication. Either way, only a few milliseconds remain for any network trip, and a trip to a distant cloud does not fit even once.

The standard escape is reprojection, also called time warp: the device takes the last rendered frame and shifts it to match the newest head pose just before display. Reprojection is cheap, so it can run locally inside the MTP budget, and it means a heavy frame rendered elsewhere can arrive a little late without the image lagging behind your head. This one technique is what lets the heavy work leave the device at all.

Warehouse AR: six tasks, four layers

Notice the new role of the fog layer. Nothing in the smart-glasses design was shared between users, but a warehouse is a team. The pick list and the route optimizer serve every picker in the building, their deadline is around a second rather than a frame, and keeping them on site avoids sending every lookup to the cloud. Shared site state is the natural job of fog.

Interactive VR: every pixel under the deadline

Virtual reality shares the same budget, but raises the stakes: in AR only the overlay depends on MTP, while in VR every pixel you see does. The same four steps apply, and the same pattern appears, with a GPU-hungry rendering task pulled between a light headset and a distant data center.

The trade-offs are sharp. An edge GPU per site is expensive, so each site serves a limited number of users. The alternative, a headset tethered by cable to a local PC, meets the budget but gives up mobility. In Latency budget terms, every design is judged the same way: does the per-frame Response time (RT) fit inside MTP, measured at the tail rather than the average?

Put the three designs side by side and one pattern falls out. The lecturer’s reference answers below are a compact version of the tables you built above. If you placed recognition, sensor interaction and rendering at the edge and storage and analytics in the cloud, you have the same architecture.

Every app keeps the Cloud, and every app adds the Edge. What decides the Mapping of a task is a single comparison: is its deadline shorter than the cloud round trip plus the processing it needs? Recognizing a face, reacting to a sensor and drawing the next frame all have deadlines the cloud cannot meet. Storing photos, crunching datasets and training models have none, and they want exactly what the cloud is good at: scale, durability and reach.

Latency is the strongest force in that split, but not the only one. Satyanarayanan’s SEC 2017 keynote gives three reasons to use the edge, and all three appeared in the exercises. Latency, in both mean and tail, drives rendering and recognition. Bandwidth, both peak and average, explains why glasses and AR tablets should not stream raw video to a distant data center. Privacy, which he calls an exposure firewall for the IoT, explains keeping faces on a nearby server.

The whole lecture in one picture

That pattern is the lecture’s conclusion in miniature. Before, you architected an application on a single Server layer: pick a machine, deploy the backend, done. Now, because data, IoT devices and machine learning produce and consume so much close to the user, the server is spread across the Edge-to-cloud (E2C) continuum.

That changes the unit of design. You no longer place a server; you place each task of each service Service pipeline. And a placement needs a criterion. The lecture chooses response time, written as a function of Latency and Processing time (PT). Its concrete form generalizes the budget condition from the start of this part to a pipeline that crosses several layers:

Response time is the right criterion because it is what users actually experience, and because it keeps both levers in view. A faster server only shrinks the processing terms; placement is what shrinks the latency terms. Every argument in this part, from the fog example that missed by 10 ms to the headset that cannot afford one 4G round trip, was an argument about which lever was available.

The design loop

1. Break the service into a pipeline of tasks.
2. Set τ for each task from a published source.
3. Choose a Mapping of tasks to Device, Edge, Fog and Cloud.
4. Compute RT from latency and processing time, at a percentile.
5. Check RT ≤ τ and weigh battery, privacy, cost and reach. If it fails, remap and repeat.

Where the course goes next

Placing a task on a layer is still abstract. In practice it means running that task inside a virtual machine or a container on that layer. Satyanarayanan’s keynote notes that cloudlets can run lighter-weight containers such as Docker inside VMs, which is exactly what the virtualization and container lectures build.

Renting those layers means choosing a cloud model. NIST SP 800-145 defines the service models (SaaS, PaaS, IaaS) and the deployment models (private, community, public, hybrid). Those definitions are the subject of the cloud models lecture, and they decide who operates each layer in your mapping.