Hypervisors, containers, namespaces and cgroups

Start with a laptop running VirtualBox with Ubuntu inside it. When you press the power button, macOS boots first. VirtualBox is then just another application, next to Safari, and the whole Ubuntu guest lives inside that application's process. Now compare a server in an AWS data center that hosts EC2 instances. Nothing general purpose boots before the hypervisor. It is the first thing on the machine, and it hands out CPU and memory to the guests directly.

Both machines do Virtualization, and both run a Virtual machine monitor (VMM) that makes each Virtual machine (VM) believe it has a real computer. The difference is where the VMM sits, and that comes down to one question: who owns the hardware?

Two placements

• A Type 1 hypervisor (bare metal, or native) is the lowest software layer on the machine. It runs directly on the hardware and acts as a specialized OS whose only job is to run guests. Red Hat describes it as a hypervisor that "runs directly on the host's hardware to manage guest operating systems".
• A Type 2 hypervisor (hosted) runs on a conventional OS "as a software layer or application". The host OS owns the hardware, and the VMM is one of its processes, with less privilege than the OS itself. VirtualBox's own manual calls it "a so-called hosted hypervisor, sometimes referred to as a type 2 hypervisor".

Real examples, and where the labels get blurry

VMware's Type 1 hypervisor is ESXi, described as "the hypervisor [that] runs virtual machines"; vCenter Server is the central management service for many ESXi hosts, and vSphere is the name of the whole suite. On the hosted side sit VirtualBox, VMware Workstation and Parallels Desktop.

Picture a lab PC running VirtualBox for a coursework VM, next to an EC2 c7i instance. When the coursework VM compiles code, its virtual CPU is just a thread that competes with Chrome and the antivirus for time on the host scheduler. The EC2 instance gets CPUs and memory partitioned for it by a minimal hypervisor, with no desktop OS in the way.

That picture gives the whole comparison. Because a Type 1 hypervisor owns the hardware, it adds less overhead (the slide says about 5%), manages resources directly and isolates guests more strongly, but it needs more skill to install and operate. A Type 2 hypervisor reaches the hardware only through the host OS, so it costs more (about 10%), controls resources indirectly and isolates less, but anyone can install it.

Why cloud providers pick Type 1

A Cloud service provider (CSP) sells slices of the same physical server to strangers. Every percent of overhead is capacity it cannot sell, and every extra layer between tenants is attack surface. AWS shows where this leads. Its Nitro Hypervisor is "a custom-developed, minimized hypervisor based on KVM", cut down on purpose: it has "no networking stack, no general-purpose file system implementations, and no peripheral device driver support". Networking and storage are offloaded to dedicated Nitro Cards and exposed to guests through SR-IOV. The hypervisor does little more than partition CPU and memory, which is the Type 1 idea taken to its limit, and the design is lean enough that AWS can also sell bare metal instances.

On one Ubuntu server, run docker run nginx and then docker run mongo. Each starts in well under a second, because neither boots an operating system. Both are ordinary Linux processes making system calls into the same host kernel. Run uname -r inside either one and it prints the host's kernel version, because there is no other kernel to ask.

That is OS-level virtualization. There is no hypervisor and no guest kernel. The line that matters is the System call interface, the entry point through which user-space code asks the kernel for anything. Above it, each environment has its own isolated user space: the application and its libraries. Below it, everything is shared: the kernel, and the hardware (or the virtual hardware, if the host is itself a VM). These isolated environments are called a Container.

The design has three consequences. Containers are light and fast, because starting one is starting a process. They are tied to one OS family, because the shared kernel must understand their system calls. NIST is blunt: "a Linux host can only run containers built for Linux". And their boundary is weaker than a VM's, because a shared kernel "invariably results in a larger inter-object attack surface than seen with hypervisors". One kernel bug can be a door between every container on the host.

Take the slide's own example: one server runs a database and a web server as plain processes. Two things can go wrong. First, the web server can list the database's processes, send them signals, read its files and bind its ports, so whoever compromises one has a view of the other. Second, a heavy query makes the database use every CPU core, and the website stalls.

These are different problems, and Linux fixes each with a different kernel feature. The first is about visibility. A Namespace gives a group of processes its own view of a kernel resource, so the web server cannot even see the database's processes, network stack or mount points. The second is about consumption. Control groups (cgroups) put quotas, weights and accounting on how much CPU, memory and I/O a group can use. Docker's security guide gives the same split: namespaces "provide the first and most straightforward form of isolation", while cgroups "implement resource accounting and limiting".

Neither feature is new. The first namespace (mount) arrived in Linux 2.4.19 in 2002, most of the rest landed between 2.6.15 and 2.6.26 (July 2008), and user namespaces were only completed in 3.8 (2013), which made unprivileged containers practical. Cgroups were started in 2006 and merged in 2.6.24. A Container is simply these two combined, plus a root filesystem image. NIST describes the second half with a concrete case: a host with 10 GB of memory can give 1 GB to each of nine containers, so no single one can starve the rest.

You can make a Namespace by hand on any Linux machine. The command below starts a shell in a new PID namespace and mounts a fresh /proc for it. Inside, ps a shows only two processes, and the shell believes it is PID 1.

sudo unshare --fork --pid --mount-proc bash
ps a

PID TTY      STAT   TIME COMMAND
      1 pts/0    S      0:00 bash
      8 pts/0    R+     0:00 ps a

From a second terminal on the host, the same bash appears in ps -A under a large global PID such as 24517. Nothing was copied and no second kernel was started. The kernel simply shows that process a different numbering. NIST gives the container version: ps -A inside an Apache Container lists only httpd.

The rule

The Linux manual defines a namespace as something that "wraps a global system resource in an abstraction that makes it appear to the processes within the namespace that they have their own isolated instance". There is one kernel and many views. Every process belongs to exactly one namespace of each type, and a container runtime such as Docker creates a fresh set (all but user and time, by default) for every container. You can see a process's memberships as symbolic links in /proc/<pid>/ns/ (since Linux 3.8), and a tool can join an existing namespace with setns(2), which is how docker exec gets a shell into a running container.

The network namespace is what gives each container "its own networking stack, including unique interfaces and IP addresses", in NIST's words, so two containers can both listen on port 80. The user namespace, when enabled, lets a process be root inside its container while mapping to an unprivileged user on the host.

On a host with 2 CPUs, start MongoDB with docker run --cpus=1.5 --memory=512m mongo. Docker writes two numbers into the container's cgroup: a CPU quota of 150000 µs per 100000 µs period, and a memory ceiling of 512 MiB. If MongoDB grows past that ceiling and the kernel cannot reclaim memory, the OOM killer kills a process inside that cgroup only. The web server next to it never notices.

docker run -d --name db --cpus=1.5 --memory=512m mongo
cat /sys/fs/cgroup/system.slice/docker-$(docker inspect -f '{{.Id}}' db).scope/cpu.max
cat /sys/fs/cgroup/system.slice/docker-$(docker inspect -f '{{.Id}}' db).scope/memory.max

150000 100000
536870912

How cgroups work

The manual says cgroups "allow processes to be organized into hierarchical groups whose usage of various types of resources can then be limited and monitored". Each kind of resource is enforced by a controller, which Red Hat defines as "a kernel subsystem that represents a single resource, such as CPU time, memory, network bandwidth or disk I/O". The main controllers are cpu, memory, io, pids and cpuset; freezing, a separate freezer controller in cgroup v1, is the core file cgroup.freeze in v2. Cgroup v2, official since Linux 4.5, puts everything in one tree mounted at /sys/fs/cgroup, where every setting is a small file you can read and write.

The slide lists four functions, and each maps to concrete files: