MOV EAX, EDX

It copies one processor register into another.

Every program you’ve ever written ends up as a list of instructions like that one, sitting in memory. The processor keeps a handful of tiny, fast storage slots of its own, called registers, and one of them always holds the address of the next instruction to run. Running a program is the processor fetching that instruction, doing what it says, and moving on to the next one, billions of times a second. Most instructions are that small: they change a register, or they read or write memory at an address another register holds.

Run MOV EAX, EDX inside a virtual machine, and nothing happens to the hypervisor. The processor executes it, moves on to the next instruction, and the software in charge of the virtual machine never finds out.

If you picture a VM as a layer that reads the guest’s code and carries it out, that’s the picture this post is here to replace. The walk I’m going to take through it is borrowed from a talk a colleague gave to Red Hat’s KVM team in 2013.¹ His slide for this instruction ends with five words that are still the best summary I know:

KVM is unaware of it.

The processor did the work, of course, but no software stood between it and the guest while it did.

Your VM is a process

Now look at the host. On a Linux machine running KVM, ps shows each virtual machine as an ordinary process, usually qemu-kvm or qemu-system-x86_64. Each virtual CPU is a thread inside it. Most of that process’s memory is the guest’s RAM (not all of it, and not always resident, but most). kill -9 works on it about the way pulling the power cord works on a real machine.

Each vCPU thread spends its life in a loop. This is the real KVM API, reduced to its skeleton:

for (;;) {
    ioctl(vcpu_fd, KVM_RUN, 0);
    switch (run->exit_reason) {
        /* handle whatever KVM handed back to userspace */
    }
}

That ioctl is where the thread hands the processor to the guest. It returns only when something needs QEMU’s attention. The switch is the part people picture as “the hypervisor,” and it only ever sees the expensive cases. Plenty of events stop in KVM, inside the kernel, get handled there, and the guest resumes without QEMU waking up at all.

So instead of a layer, picture three paths: a fast path where the guest runs directly on the processor, a kernel path where KVM steps in briefly, and a userspace path where QEMU does real work. Nearly everything interesting in virtualization is about which of those paths an event takes, and how often.

A word about where I’m standing. From 2011 to 2022 I was one of the managers in the Red Hat group that worked on the Linux virtualization stack, upstream and in RHEL and the products built on it. I started with one small team and left with about fifty people. I was in the design discussions, the patch reviews, the release planning, and the arguments about what belonged upstream. I didn’t write this code, and my teams wrote only part of it. It’s open source, and much of it came from the wider community, from teams before mine, and from teams under other managers. These days I lead engineering for a managed Kubernetes service on Azure. Its nodes are virtual machines too, on Microsoft’s hypervisor rather than KVM, and most of the people I work with meet all of this only from above. This post is for them.

The trap door

Modern x86 processors have a mode built for this. The host fills in a control structure that lists the events it wants to hear about, then enters the guest. From then on, the guest’s instructions run natively, and the processor watches for the listed events. When one happens, it saves the guest’s state and jumps back to the host. That’s a VM exit. Going back in is a VM entry.

It wasn’t always possible. In 1974, Popek and Goldberg wrote down what an architecture needs in order to be virtualized this way: every instruction that could reveal or change the real machine’s state has to trap when a guest runs it, and x86 didn’t meet that bar. A handful of sensitive instructions simply behaved differently for a guest, with no trap. So VMware rewrote guest code on the fly (binary translation), and Xen modified the guest OS to call the hypervisor instead of touching the hardware (paravirtualization). Intel’s VT-x and AMD-V added the trap door to the processor around 2005 and 2006, and an unmodified guest could run without either trick.

Binary translation mostly went away after that, but paravirtualization stayed. It stopped being how the CPU works and became how devices, clocks, and scheduling work, and it comes back three more times in this post.

Memory: the cost moved

Here’s the second instruction:

MOV [addr], EDX

The same MOV, but now it writes a register into memory, at the address addr. In an ordinary process, that address isn’t a real location in RAM. Every process gets its own private view of memory, so two processes can use the same address and mean different bytes. The OS keeps a map for each one, called page tables, from these virtual addresses to physical memory, and the processor follows that map by itself. Following it takes several memory reads (four, on a typical x86-64 machine), so the processor keeps a small cache of recent answers, the TLB, and most accesses never walk the map at all.

A guest adds a second layer. The guest OS believes it’s translating to physical memory, but its “physical” memory is just memory inside the QEMU process, which the host translates again.

The first answer was shadow page tables. KVM watched the guest edit its own page tables and maintained a combined copy the real processor could use. It worked, but keeping the copy in sync cost a steady stream of exits.

In 2007 and 2008, AMD’s NPT and Intel’s EPT moved the second translation into hardware. The processor walks the guest’s tables and the host’s tables by itself, and an ordinary guest page fault no longer needs KVM at all.

That didn’t make translation free. On a TLB miss, every step of the guest’s walk now needs a walk of the host’s tables too. Intel’s own engineers wrote that this can take the number of memory references needed to fill one TLB entry from four to twenty-four.²

Huge pages help here, because a bigger page means fewer TLB misses and a shorter walk. If you’ve ever requested them in a pod spec, you made a decision about translation pressure, whether or not anyone called it that.

Silicon took the work off the software path and left a cheaper version of the same cost behind, somewhere the guest can’t see or account for. That’s the pattern of the whole post, which is why this section can be short.

I/O: one store, five endings

Here’s the third instruction, and the one this post is really about:

MOV [addr], EDX

The same bytes as the last one. This time, though, addr isn’t memory. Devices have addresses too: a network card or a disk controller answers at a small range of them, and writing there is how a driver tells the device to do something. In a VM there is no card at that address, so the host leaves it out of its own page tables on purpose, and the store causes a VM exit. Somebody has to pretend to be the card.

That store is where the host has to step in, and what happened to it over the last twenty years is the story of virtualized I/O. It has had five answers.

The oldest answer is to let it trap and emulate a real card. Say the guest has an Intel e1000 network card, because nearly every OS ships a driver for one. The driver talks to the card through its registers: a write to say there’s a packet waiting, then more reads and writes to handle the interrupt once it’s sent. Each of those is a store or a load like the one above. Each one exits to KVM, KVM sees a device address and returns to QEMU, and QEMU runs code that behaves the way the real card’s register would. That’s a round trip all the way out to userspace for every register access, and a busy card makes a lot of them.

It’s perfectly compatible, and slow.

The second answer, virtio, stops pretending to be real hardware. virtio is a family of devices designed for virtual machines, and the guest knows it’s talking to one. Instead of imitating a real card’s registers, the guest and host agree on a simple contract: rings of descriptors (“here’s a buffer, send it”) in memory they share, and a notification that either side can ask the other to skip while it’s already busy. Under load, a stream of packets can flow with hardly any exits at all. The guest’s store to a device register, that third MOV, mostly stops happening. That’s the whole trick: virtio didn’t make exits cheaper, it made them rare. This is the first place paravirtualization comes back: the guest cooperates with the host because it knows it’s in a VM.

The third answer, vhost-net, deals with where that notification lands. With plain virtio it still wakes QEMU, which then pushes the packets through the host kernel. vhost-net moves the backend into the host kernel itself, right where the packets are headed anyway. On the common path, QEMU never wakes up.

The fourth, vhost-user, keeps the idea and changes the destination. The backend lives in some other userspace process, usually one somebody chose on purpose: a packet-processing data plane like Open vSwitch with DPDK, or a storage engine like SPDK.

The fifth answer is to give the guest a real device. With VFIO, often paired with SR-IOV (a card that presents itself as many small cards), the guest’s MOV to a device register lands on actual hardware. The device then writes straight into guest memory, and the IOMMU, which does for devices what page tables do for programs, keeps it from writing anywhere else. Nothing is emulated on the data path.

Every one of those steps moved the mediation somewhere cheaper, or handed it to something more specialized. And every one paid for that with something: portability, visibility into what the guest is doing, compatibility with guests that don’t know they’re virtualized, or the host’s freedom to change its mind later.

You’ve probably picked a rung already. On EC2, enhanced networking uses SR-IOV, and so does accelerated networking on Azure. If either one is on, your VM’s network card is a slice of a real one, and the host is mostly out of the way.

Live migration shows the price most clearly. When every device is emulated, moving a running VM to another host is a property of the platform: copy the memory and the device state, and resume on the other side. With an assigned device, part of the VM’s state lives inside a real card the host doesn’t control. It can still work (some drivers support migrating that state, and both KubeVirt and Azure unplug the SR-IOV interface before a move and plug a fresh one in after it), but migration becomes a property of the device and the stack around it.

This is also where KVM’s design paid off. KVM is a module in the Linux kernel, and QEMU is a Linux process, so both could lean on everything Linux already had: its scheduler, its memory manager, its I/O stack, its drivers. A hypervisor that runs beneath the OS, instead of inside it, has to bring its own scheduler and memory manager. Containers made the same bet on reusing the kernel and took it further, because every container on a host shares that one kernel. What keeps them apart is the kernel’s own bookkeeping (namespaces and cgroups), while VMs are kept apart by the processor’s virtualization hardware.

Time and scheduling

A vCPU is a thread on the host. The host can stop running it at any moment, to run another VM or anything else, and the guest gets no say.

The guest’s clocks feel that first. A native OS reads a counter on the processor and trusts it. In a VM, that counter has to keep making sense while the vCPU is paused, when its thread moves to another physical CPU, and when the whole VM moves to another host with a different processor. The hardware can offset and scale the counter. KVM also gives the guest a paravirtual clock, kvmclock, in a page of shared memory, so the guest can compute a time the host agrees with. That’s the second place paravirtualization comes back.

Scheduling feels it next. When a vCPU has nothing to do, the guest runs HLT, which exits, and the host gets the real CPU back. That part is easy. Locks and interrupt handling are the hard cases. An OS is full of tricks for using real hardware efficiently, and a good virtualization stack has to keep all of them working, invisibly when it can and with the guest’s help when it can’t.

Take a spinlock inside the guest: one vCPU takes it, and then the host deschedules that vCPU’s thread. Another vCPU in the same guest starts spinning on the lock, burning a real host CPU while it waits for a holder that isn’t running. Paravirtual spinlocks let the waiter give up after a while and sleep, and the vCPU that releases the lock wakes it with a call to the host. That’s the third place paravirtualization comes back.

Then there’s steal time. For every vCPU, the host counts how long it was ready to run but wasn’t, because the host had something else on the real CPU. It writes that count into guest memory, and the guest kernel reports it as the st column in top. It doesn’t count time the vCPU spent halted with nothing to do. That’s idle, and idle is the guest’s own business.

Steal time is where the abstraction admits it’s an abstraction. virtio, kvmclock, and the spinlock trick are private deals between the guest kernel and the host, and nothing above the kernel sees them. %st is different. It’s the guest being told, as a number anyone can read in top, that for that slice of time it didn’t own the processor at all.

If your service runs on cloud VMs, that number is already part of your latency. A request that arrives while your vCPU is stolen just waits, and nothing in your own process’s CPU graphs explains the delay. Most dashboards don’t graph %st, and if you’re ever digging into service latency, I’d consider putting it right next to p99.

Why it’s under Kubernetes again

So silicon moved the boundary rather than finishing the job: once ordinary instructions ran natively, the hard problems went out into memory, devices, time, scheduling, migration, and trust.

You can watch that happen in the KVM Forum programs. In 2011 the program was full of device assignment, virtio, qcow2, and performance work. By 2021 there were talks on secure migration of encrypted VMs, on Kata Containers, and on rust-vmm, a set of Rust building blocks for new VMMs. The old topics were still there, next to the new ones.

Meanwhile, Kubernetes made the VM layer easy to forget. Your node is probably a VM, but you think in pods. Your pod’s CPU limit is a cgroup quota, enforced by a kernel that is itself a guest, on vCPUs that can be stolen from.

Then multi-tenancy brought the VM back from the other side. Containers share a kernel, and a shared kernel is a thin wall between two customers who don’t trust each other. So the hardware boundary came back behind a container interface. Kata Containers runs each pod inside a lightweight VM. Firecracker, which AWS built for Lambda, is a VMM with a deliberately small device model and a streamlined way of loading the kernel, built for workloads that live for seconds. libkrun, whose author was on one of my teams, turns the idea into a library: any program can use it to run a process inside its own lightweight VM, and the container runtime crun uses it to put a container behind that boundary.

It went the other direction too. KubeVirt, which Red Hat ships as OpenShift Virtualization, makes a full virtual machine a Kubernetes resource, scheduled and managed next to pods. These stacks include upstream work from the teams I used to manage.

Confidential computing

Everything above assumes a host that can see everything. It reads guest memory to emulate a device, copies guest pages to migrate a VM, and looks inside a hung guest to debug it. Most of the ladder was built on that visibility.

Confidential computing takes it away on purpose. AMD’s SEV-SNP and Intel’s TDX encrypt a guest’s memory with keys the host doesn’t have, and use attestation so the guest’s owner can check what it’s running on. Trust moves to the processor, its firmware, and whoever verifies the measurements.

The machinery in this post gets harder. The host can’t read the guest’s buffers, so I/O goes through pages the guest explicitly shares, usually with an extra copy. Migration needs a way to move protected state without exposing it. Debugging a guest you can’t look inside is its own problem.

I saw the beginning of this turn before I left virtualization in 2022, and everything since is reading, not experience.

The last twenty years were about making mediation cheaper by leaning on a host that could see everything. There’s work now, under the name TDISP, on letting a trusted device write straight into a protected guest’s memory. Does that become the ordinary path, the way virtio did, or does confidential I/O keep paying for the extra copy? And if the fast path does come back, how much trust does it have to hand the device to get there? I’m following it from a distance now, as a reader, and mostly for the fun of it.

Human Directed, AI Articulated. This post was co-written with Claude. The judgments, and any mistakes, are mine.