How KVM Works: VM Exits, EPT, and Why a Guest Is Scheduled Like a Process

A process that starts a virtual machine does not execute guest instructions the way it executes its own. When the guest is running, the CPU is in a special non-root mode. The host kernel only regains control when the hardware decides an exit is required. That boundary is what makes KVM possible without rewriting every guest instruction.

This article explains that contract: how the Linux kernel becomes a hypervisor, what a VM exit is, how memory translation works with EPT, and why the guest is still scheduled like any other process on the host.

Related reading: How Linux Namespaces and cgroups Work, How Virtual Memory Works, and How Linux CFS Works.

The contract

KVM turns the Linux kernel into a type-2 hypervisor. User space (almost always QEMU) creates a VM, allocates guest memory, and asks the kernel to run a virtual CPU. The kernel uses hardware virtualization extensions (Intel VT-x or AMD-V) so that guest code runs directly on the physical CPU until something forces an exit back to the host.

The guest sees its own kernel, its own page tables, and its own devices. From the host scheduler's point of view, the vCPU is just a thread that spends most of its time inside a long-running ioctl.

The QEMU / KVM split

QEMU runs in user space. It owns the device models (disks, NICs, the firmware image, the boot process) and the memory backing for the guest. KVM is a kernel module (kvm.ko plus kvm-intel.ko or kvm-amd.ko). It owns the virtual CPU and the memory-translation machinery that the hardware needs.

The interface is a file descriptor. Opening /dev/kvm and issuing ioctls creates a VM, creates vCPUs, registers memory regions, and finally runs the vCPU. The run loop looks like this:

vm_fd = open("/dev/kvm", O_RDWR);
ioctl(vm_fd, KVM_CREATE_VM, 0);
vcpu_fd = ioctl(vm_fd, KVM_CREATE_VCPU, 0);
// set memory regions, registers, etc.
for (;;) {
    ioctl(vcpu_fd, KVM_RUN, 0);
    // examine kvm_run.exit_reason and handle it
}

Most of the time KVM_RUN does not return. The thread is blocked in the kernel while the guest executes. It returns only when a VM exit occurs that KVM cannot (or chooses not to) handle entirely inside the kernel.

Hardware modes and the VMCS

Intel VT-x adds two modes of operation: VMX root mode and VMX non-root mode. The host kernel (KVM) runs in root mode. The guest runs in non-root mode. A special data structure called the Virtual Machine Control Structure (VMCS) holds the guest state that must be saved and the host state that must be restored on every transition. It also holds the controls that decide which events cause an exit.

AMD's equivalent is the VMCB. The principles are the same.

Entering the guest is a VMLAUNCH or VMRESUME instruction. Leaving it is a VM exit. On exit the hardware loads the host state from the VMCS and jumps to the host's exit handler. KVM then decides what to do with the exit reason.

What causes a VM exit

Not every guest instruction exits. Ordinary computation, memory accesses that hit in the guest page tables and the EPT, and many privileged operations that the hardware can virtualize stay inside non-root mode. Exits happen for events the hardware has been configured to trap:

• I/O port instructions (IN/OUT) when the device is emulated
• Certain MSR reads and writes
• CPUID
• EPT violations (the memory translation for a guest physical address is missing or has the wrong permissions)
• External interrupts that the host wants to handle
• Halt, pause, or other special instructions
• Hypercalls from the guest

Each exit has a cost. The CPU must save and restore a large amount of architectural state. High-exit-rate workloads (old device emulation, frequent page-table updates on shadow paging) show it. Hardware features such as APICv, posted interrupts, and EPT exist precisely to keep the exit rate down.

Memory virtualization: shadow paging versus EPT

A guest has its own page tables that translate guest virtual addresses (GVA) to guest physical addresses (GPA). The host must still translate those GPAs to host physical addresses (HPA). Two approaches exist.

Shadow paging (software). KVM builds a second set of page tables that map GVA directly to HPA. It intercepts every guest write to a page table so it can keep the shadow consistent. Every guest page fault and every CR3 change can become a VM exit. The mechanism works on older hardware but is expensive.

Extended Page Tables (EPT) / Nested Page Tables (NPT) (hardware). The CPU walks two tables. The guest's own page tables do GVA → GPA. The EPT (controlled by the host and pointed to by a field in the VMCS) does GPA → HPA. Guest page-table updates no longer force exits. An access that misses in the EPT causes an EPT violation exit; KVM installs the missing mapping and re-enters. On modern processors this is the normal path.

The guest still believes it owns physical memory. It does not. The addresses it programs into its page tables are GPAs. The real frames live wherever the host's memory allocator and the EPT say they live.

The guest is still a process

Each vCPU is a thread inside the QEMU process (or a dedicated thread). When that thread is inside KVM_RUN the scheduler sees a runnable task that is currently executing in kernel mode. When an exit returns to user space, the thread becomes an ordinary user-mode task again and can be preempted, migrated, or descheduled exactly like any other process.

Host NUMA placement, cgroup limits, and the CFS (or EEVDF) scheduler all apply to the QEMU threads. A noisy neighbor on the host can still steal cycles from a guest. Transparent huge pages, ballooning, and memory overcommit all interact with the same page-reclaim and allocator paths the host uses for ordinary processes.

This is why a container and a VM are not the same thing. A container shares the host kernel and the host page tables. A KVM guest has its own kernel, its own page tables, and a hardware-enforced boundary that only the hypervisor can cross. The cost of that boundary is the VM exit.

Misconceptions

KVM is a full hypervisor by itself. It is the CPU and memory virtualization engine. Device emulation and the management plane live in QEMU (or Firecracker, or cloud-hypervisor).

Every guest instruction traps. Ordinary arithmetic and most memory accesses do not. The whole point of hardware virtualization is that the common path stays in the guest.

The guest sees real physical addresses. It sees guest physical addresses. The mapping from those to host frames is under host control.

VM exits are the same as system calls. A system call is a deliberate request from user mode to kernel mode inside one operating system. A VM exit is a hardware-forced transition from non-root to root mode. The guest kernel may not even be aware that an exit occurred if KVM handled it transparently.

Takeaways

KVM makes the Linux kernel a hypervisor by using hardware virtualization extensions. Guest code runs directly until a configured event forces a VM exit. Memory translation is two-dimensional: the guest's page tables plus the host's EPT (or a software shadow). From the host's perspective each vCPU is still a schedulable thread; the only difference is that most of its CPU time is spent inside a long kernel ioctl that the hardware itself is executing.

The exit is the price of isolation. Hardware features exist to make that price as small as possible for the common case.

Further reading on this site: How the Linux Page Reclaim and OOM Killer Work, How NUMA Works, and How DMA and the IOMMU Work.