How TLB Shootdowns Work: IPIs, PCIDs, and Why Unmapping a Page Is Not Free

A process that calls munmap on a range it no longer needs expects the virtual addresses to disappear. On a single-core machine that is almost true: the kernel updates the page tables and invalidates the local TLB. On a multi-socket server the same call must also convince every other core that might still hold a stale translation for those addresses. That cross-core work is a TLB shootdown. It is why unmapping a page, changing protection, or performing a copy-on-write fault can cost more than the page-table update itself.

This article explains the hardware contract that makes shootdowns necessary, how Linux organizes them with IPIs, PCIDs, and lazy mode, and why the cost shows up in profiles even when the application is not thrashing.

It is not a second tour of page-table walks, the page cache, or NUMA placement. Those topics already cover how a translation is created and where the frames live. This one covers how the cached translations stay consistent after a change.

Why the hardware does not do it for you

A TLB is a small, private cache of virtual-to-physical translations inside each core. It is not coherent the way data caches are. There is no snoop or directory that automatically invalidates a remote TLB entry when a page-table entry is written. On x86 the only local instructions that affect the TLB are invlpg, invpcid, a write to CR3, or a change to certain CR4 bits. None of them reach another socket.

That design is deliberate. Coherence for TLBs would require extra traffic on every page-table write and would complicate the MMU. The operating system is therefore responsible for the protocol: after it has changed a PTE that other cores might have cached, it must explicitly request those cores to drop the old entry before it reuses the physical page or relies on the new permissions.

The same rule applies on ARM, POWER, and RISC-V, though the invalidation instructions and the broadcast facilities differ. Linux abstracts the common case behind flush_tlb_* helpers.

What triggers a shootdown

Any change that can make a previously valid translation incorrect needs a flush:

• Unmapping a page (munmap, process exit, madvise(MADV_DONTNEED))
• Changing protection (mprotect, COW fault that installs a new writable page)
• Installing a new mapping over an old one
• Kernel mappings that are global (they must be flushed on every CPU)

The kernel records which CPUs have ever run the affected address space in mm_cpumask. Only those CPUs are candidates. A CPU that has never seen the process can be skipped.

After the page-table update is visible, the initiating CPU builds a flush request that names the address range and the mm, then sends it to the target CPUs.

The IPI path

On x86 the ordinary mechanism is an inter-processor interrupt. The initiator calls something equivalent to smp_call_function_many with the target mask and a handler flush_tlb_func. Each remote CPU receives the IPI, enters the handler, executes the appropriate local invalidation (invlpg for a single page, a CR3 reload or invpcid for a range or the whole mm), and acknowledges. The initiator waits for the acknowledgements before it considers the shootdown complete.

That wait is the expensive part. On a quiet system a single IPI round-trip is a few hundred nanoseconds to a couple of microseconds. On a loaded 64-core machine with NUMA hops the same broadcast can take tens of microseconds, and the remote CPUs lose whatever work they were doing to service the interrupt. Batching helps: tlb_gather_mmu / tlb_finish_mmu accumulate multiple unmaps and issue one shootdown for the whole batch.

For very large ranges the kernel may choose a full TLB flush instead of per-page invlpg. The tunable /sys/kernel/debug/x86/tlb_single_page_flush_ceiling controls the cutoff; above it the collateral damage of a full flush is considered cheaper than thousands of individual invalidations.

PCIDs and ASIDs reduce the local cost

Without address-space tags, every context switch must flush the entire TLB, because the next process must not see the previous process\u0027s translations. PCIDs (Process-Context Identifiers) on x86, and the analogous ASIDs on other architectures, tag each TLB entry with the address space that created it. A core can keep entries from several processes and simply ignore those that do not match the current PCID in CR3.

Linux does not assign a permanent PCID to every process. It maintains a small per-CPU array of dynamic ASIDs (currently six) that cache the most recent address spaces that ran on that CPU. When a process is switched in, the kernel either reuses an existing slot or allocates a new one and flushes the old occupant. The generation counters attached to those slots let the shootdown code know whether a particular CPU still holds potentially stale entries for a given mm.

PCIDs do not eliminate shootdowns. They only make the local invalidation cheaper and allow the kernel to skip a full flush on context switch. Remote cores still need to be told when a mapping they might hold has changed.

Lazy TLB mode

When a CPU is idle or running a kernel thread that borrowed the previous process\u0027s address space (active_mm), it is in lazy TLB mode. It is not executing user code for that mm, so a shootdown can sometimes be deferred. The kernel records the lazy state; the next time that CPU returns to user mode for the same mm it performs the pending flush itself. Shootdown IPIs can therefore be omitted for CPUs that are currently lazy, provided the kernel can prove the CPU will not use the stale translation before it notices the pending flush.

This optimization matters for systems that run many short-lived kernel threads or that have idle cores. It is one reason a single-threaded process on a large machine does not always pay a full broadcast on every munmap.

Hardware broadcast and global ASIDs

Recent AMD processors support INVLPGB, a broadcast invalidation instruction. When a process is active on enough CPUs, Linux can assign it a global ASID that is identical on every core. The kernel can then issue a single invlpgb followed by a tlbsync instead of sending IPIs. Intel has analogous remote-atomics facilities that are being integrated. On machines that support the feature, heavily multi-threaded processes see a large reduction in shootdown latency.

Until those instructions are universal, the IPI path remains the common case.

What the cost looks like in practice

A database that repeatedly creates and destroys temporary mappings, a JIT that flips page permissions, or a server that munmaps large buffers under load will show the shootdown in two places: time spent in the initiating CPU\u0027s flush path, and time spent in IPI handlers on the remote CPUs. Tools such as perf with the tlb or ipi events, or ftrace on flush_tlb_*, make the cost visible. On a large machine the remote interrupts can also perturb unrelated workloads that happen to share the cores.

Reducing the frequency of mapping changes is usually more effective than tuning the flush ceiling. Mapping large regions once and reusing them, or using madvise hints that the kernel can handle with less precise flushes, keeps the shootdown rate down.

Misconceptions

The hardware keeps TLBs coherent. It does not. Data-cache coherence does not extend to the TLB. The OS must drive the protocol.

A CR3 write on one CPU invalidates every other CPU. It invalidates only the local TLB (for the matching PCID). Remote CPUs are unaffected until they receive an explicit flush.

PCIDs eliminate shootdowns. They eliminate most context-switch flushes and make selective invalidation possible. Shootdowns for changed mappings remain.

Only the CPU that ran the process needs a flush. Any CPU that has the mm in its mm_cpumask and is not safely lazy may hold a stale entry.

Where this sits next to other machinery

Copy-on-write faults and mprotect both end in a shootdown for the affected range; the page-table update is only half the work. The page cache and the buddy allocator decide which frames exist; the TLB decides whether a core is allowed to use a cached translation for them. NUMA placement affects which memory controller answers a miss after the TLB has been filled, but it does not change the shootdown protocol.

Related articles:

• How Virtual Memory Works covers the page-table walk that a TLB miss performs.
• How fork and Copy-on-Write Work shows the TLB invalidation that follows a COW fault.
• How CPU Caches Work describes the coherent data caches that sit beside the non-coherent TLB.

Takeaways

A TLB is a private, non-coherent cache of translations. Changing a page-table entry that other cores might have cached requires an explicit shootdown. Linux implements that shootdown with IPIs to the CPUs recorded in mm_cpumask, using PCIDs and lazy mode to avoid unnecessary work, and increasingly using hardware broadcast instructions when they are available. The cost is real, scales with the number of cores that have seen the address space, and is why frequent mapping changes remain expensive even when the page tables themselves are cheap to update.