How NUMA Works: Nodes, Distance, and Why First-Touch Allocation Matters

A process that calls malloc for a multi-gigabyte buffer on a two-socket server does not receive equal-cost memory. The physical frames behind those virtual pages land on one memory controller or the other. The core that first writes a page usually decides which controller owns it. Later loads from a core on the opposite socket cross the interconnect and pay extra latency and bandwidth. That asymmetry is Non-Uniform Memory Access.

This article explains the hardware contract, how Linux places pages, and why the first touch of a page is the moment that matters. It is not a second tour of page-table walks, cache-line coherence, or the CFS load balancer. Those layers sit above or beside this one. Related reading: How Virtual Memory Works, How CPU Caches Work, How Linux CFS Works, and How the Linux Buddy and SLUB Allocators Work.

The contract

A NUMA system divides the machine into nodes. Each node contains one or more CPU cores, a local memory controller, and the DRAM attached to that controller. Cores on the same node reach their local DRAM with lower latency and higher bandwidth than cores that must cross an interconnect to a remote node. The interconnect (QPI, UPI, Infinity Fabric, or the equivalent on the platform) is a shared resource; remote traffic contends with other traffic.

Distances are not always two values. On a four-socket or multi-hop machine some remote nodes are farther than others. Linux exposes the matrix through numactl --hardware or /sys/devices/system/node/node*/distance. A distance of 10 is local; higher numbers indicate hops.

What the hardware actually does

Physical addresses are not flat. The memory controller that owns a range of physical frames is fixed by the platform. When the kernel allocates a page it chooses a node, then asks that node's buddy allocator for a free frame. The page-table entry records the physical frame number; the MMU and the interconnect together decide whether the access stays local or travels.

Cache coherence still works across nodes. A line can be in Modified state on a remote core. The cost of a miss or an ownership transfer simply includes the interconnect round trip. False sharing that crosses nodes is more expensive than false sharing that stays inside one socket.

How Linux decides which node gets the page

The default policy for anonymous memory is first-touch. The page is not allocated when malloc returns a virtual address. It is allocated on the first write (or sometimes the first read, depending on the kernel and the mapping). The allocating core's current node is preferred. If that node is out of memory the kernel falls back according to the zone and the allowed nodes.

Explicit policies override first-touch:

• MPOL_PREFERRED or MPOL_PREFERRED_MANY — prefer a node but allow fallback.
• MPOL_BIND — restrict allocation to a set of nodes.
• MPOL_INTERLEAVE — round-robin across nodes, useful when a single process must touch more memory than one node holds.
• MPOL_LOCAL — force local when possible.

Userspace sets these with mbind, set_mempolicy, or the numactl wrapper. cpusets and cgroup cpuset controllers can further restrict the allowed nodes for a process tree. Automatic NUMA balancing (enabled by default on many distributions) periodically migrates pages toward the nodes that touch them and migrates threads toward the nodes that hold their pages. It is a heuristic; it does not replace an application that knows its own access pattern.

Why first-touch surprises people

A main thread that touches every page of a large array before handing the array to worker threads on other sockets pins the entire working set to the main thread's node. Subsequent remote accesses dominate. The fix is either to touch the pages from the threads that will own them, or to use explicit binding or interleaving before the workers start.

Allocators that pre-fault or that use a single thread for initialization inherit the same problem. Databases, JVMs, and language runtimes that allocate large heaps often expose NUMA-aware options precisely because the default first-touch policy plus a single initializing thread produces remote memory by accident.

What still costs cycles

Even with perfect placement, remote cache-to-cache transfers and the interconnect's bandwidth limit remain. A workload that streams more data than one node's memory controllers can supply will saturate regardless of which core runs it. Huge pages reduce TLB pressure but do not change the node that owns the frames. Transparent huge pages can also interact with NUMA balancing; the kernel may split or migrate them under pressure.

Misconceptions

• NUMA only matters on machines with more than 64 cores. Two-socket machines already show measurable remote penalties on latency-sensitive paths.
• Virtualization hides NUMA. The hypervisor can present a flat topology or a virtual NUMA topology. The guest still pays if its vCPUs and memory are not co-located on the host, and the guest's own first-touch policy still applies inside the virtual machine.
• Interleaving is always better. Interleaving spreads bandwidth and avoids one-node saturation, but every access becomes remote on average. Local placement wins when the working set fits and the threads can be pinned or scheduled with it.
• The page cache is NUMA-aware by magic. File pages are allocated on the node that first faults them in, just like anonymous pages. A single reader thread can pin a large file's cache to one node.

Takeaways

NUMA makes physical memory location visible to software. The kernel's default is first-touch: the core that writes a page usually owns the frame. Remote accesses cross an interconnect and cost more latency and bandwidth. Explicit policies, thread placement, and automatic balancing exist because the default is correct only when the first touch and the later use happen on the same node. Measure with numastat, perf mem events, or the topology in /sys before assuming a machine is uniform.