How System Calls Work: User Mode, Kernel Mode, and Crossing the Boundary

A user program cannot talk to a disk controller, change another process's page tables, or bind a TCP port by executing ordinary instructions. Those operations live behind a privilege boundary. A system call is the documented way to ask the kernel to do one of them and return.

This article is about that crossing: privilege rings, the syscall instruction on x86-64, the register ABI, how libc wraps the trap, why errno exists, what the vDSO is for, and why a tight loop of tiny syscalls is expensive. It is not a second tour of virtual memory translation, process versus thread scheduling, or container namespaces. Those layers use system calls. They are not the call itself.

The contract

The CPU runs in at least two privilege levels. On x86 they are historically called rings. User code runs at ring 3. The kernel runs at ring 0. A ring-3 instruction that tries to execute hlt, write a control register, or touch kernel memory is a fault, not a suggestion.

The kernel therefore publishes a numbered interface. Each number names an operation: open a file, map memory, create a process, wait for I/O. The caller places the number and arguments in agreed registers, executes a special instruction that transfers control to a kernel entry stub, and later resumes at the next user instruction with a return value in a register.

The contract is an ABI, not a C function. The C library is a convenience wrapper around that ABI. Raw syscall(2) exists because the wrapper and the kernel entry are separate objects.

Why the boundary exists

Isolation is cheaper to enforce if most instructions cannot touch shared hardware or another process's address space. Virtual memory already hides physical frames. Privilege hides privileged instructions and kernel mappings. Together they let an untrusted binary run without rewriting the page tables of the process next to it.

The cost is that every "please do something I am not allowed to do" becomes a controlled transition: save user state, switch address-space view if needed, validate arguments, perform the work, restore user state. That transition is the system call.

How the trap actually happens on x86-64 Linux

Modern Linux on x86-64 uses the syscall instruction, defined with the companion sysret (or, in some paths, iret). Older 32-bit paths used int 0x80 or sysenter. Those still exist for compatibility. New 64-bit code should not use them.

A typical libc wrapper for write ends up doing the equivalent of:

mov rax, 1          ; __NR_write
mov rdi, fd
mov rsi, buf
mov rdx, count
syscall

On entry the CPU:

• saves the user instruction pointer and flags into registers the kernel will later read,
• loads a kernel instruction pointer and a kernel stack from MSRs programmed at boot (LSTAR and related registers),
• clears certain user flags so the kernel does not inherit a surprising interrupt or direction state,
• continues execution in kernel mode at the system-call entry stub.

The stub saves the rest of the user register file onto the kernel stack, looks up rax in a table of handlers, and calls that handler with the arguments still sitting in rdi, rsi, rdx, r10, r8, and r9. Note r10, not rcx: syscall itself overwrites rcx with the saved user RIP, so the fourth argument moves one register over relative to the ordinary System V user-space calling convention.

On the way out the kernel places a result in rax. A negative value in the kernel's internal convention means an error code. The libc wrapper turns that into -1 and writes the positive errno into errno. The raw syscall ABI and the POSIX C ABI are therefore not identical. Mixing them without care is a common source of "it returned -2 and I treated it as a file descriptor" bugs.

What the handler is allowed to do

A handler does not trust the caller. Pointer arguments are user virtual addresses. The kernel must copy them with helpers that check the address is mapped and belongs to the process, or it must probe the range before a write. A raw dereference of a user pointer inside the kernel is a security bug.

Many calls block. read on an empty pipe, futex wait, accept with no connection: the thread is taken off the run queue and a wait entry is recorded. When the event arrives, the thread is made runnable again and the syscall returns. From user code this looks like a slow function. From the scheduler it is a voluntary sleep, not a spin.

Some calls change the process's memory map: mmap, munmap, mprotect, brk. Those update page tables and may flush TLB entries. The next user-mode instruction after return runs under the new map.

A compact example

Consider open("/tmp/x", O_RDONLY) on Linux amd64.

1. The program calls the libc symbol open.
2. glibc (or musl) places __NR_openat in rax because modern Linux prefers openat with AT_FDCWD over the older open number.
3. syscall lands in entry_SYSCALL_64.
4. The handler copies the path string from user memory, walks the directory cache starting at the current working directory, checks permissions, allocates a file descriptor in the process's fd table, and attaches a file object.
5. Success returns a small non-negative integer in rax. Failure returns a negative errno internally; libc converts it.

Nothing about that sequence is a "function call" in the C sense. There is no shared stack frame between user open and the kernel function that implements it. There are two stacks: the user stack still holds the C frame; the kernel stack holds the entry frame.

The vDSO: syscalls that are not traps

Some queries are so frequent and so safe that trapping is wasteful. Reading the current time with clock_gettime(CLOCK_MONOTONIC, ...) is the usual example. Linux maps a small shared page into every process: the virtual dynamic shared object, or vDSO. The C library jumps to a function in that page. The function reads a sequence counter and a time base the kernel updates, then returns. No privilege change occurs.

The same idea covers some getcpu and (historically) gettimeofday paths. If the requested clock cannot be answered from the page, the vDSO function falls back to a real syscall. Programs that benchmark "syscall cost" with gettimeofday are often measuring a function call, not a trap.

Cost

A round trip is not free even when the handler is trivial. The CPU leaves the user pipeline, changes privilege, often switches the stack, may change PCID or address-space identifiers, and later restores user registers. Spectre-era mitigations added extra fencing and sometimes extra address-space work on some kernels. The exact nanoseconds move with microarchitecture and kernel version; the shape does not: a million tiny syscalls will lose to one syscall that moves more data, or to a batch interface.

That is why readv/writev, epoll or io_uring, sendmmsg, and clone3 exist. They keep the privilege crossing but raise the work per crossing. User-space networking stacks and some language runtimes go further and try to stay on one side of the boundary for long stretches.

Other architectures, same idea

AArch64 uses svc (supervisor call) and a similar register convention. RISC-V uses ecall. The number assignment is architecture-specific; Linux does not promise that __NR_write is 1 everywhere. Portable code uses libc or generated tables from the kernel UAPI headers, not hardcoded numbers copied from an amd64 man page.

What system calls are not

A system call is not an interrupt from a device. Device interrupts also enter the kernel, but they are not requested by the current user instruction and they do not return a value to that instruction. A page fault is also a kernel entry; it may restart the faulting instruction after the kernel installs a mapping. That is demand paging, covered with virtual memory, not a syscall ABI.

A libc function is not automatically a syscall. strlen never leaves user mode. malloc may run for a long time on a user free list and only trap when the heap must grow. fprintf buffers; the trap happens later inside write.

Containers do not invent a new kind of syscall. They filter or virtualize the existing table with seccomp, namespaces, and cgroups. The instruction is still syscall.

Where this shows up when something breaks

strace prints the ABI, not your C source. If you call open and see openat, the wrapper remapped the request. If a program works under glibc and fails with a raw syscall you wrote, check the fourth argument register and the sign of the return.

Permission errors, EFAULT from a bad pointer, and EINTR from a signal arriving during a blocked call are all products of this path. Restartable calls and SA_RESTART exist because the kernel can return before the work finished.

Misconceptions

System calls are just function calls into the operating system. They change privilege and stack. The calling convention is not the user C convention.

Every POSIX function is one trap. Many are several, or none. fork plus execve is two. system(3) is more.

Returning -1 is the kernel ABI. On Linux the kernel returns a negative errno in rax. Libc translates.

Faster clocks need faster traps. The vDSO exists so the common clock read is not a trap.

Seccomp stops the instruction. Seccomp inspects the request after the trap. The CPU still entered the kernel; the filter decides whether the handler runs.

Takeaways

User mode cannot perform privileged work. A system call is a numbered, register-based request that uses a dedicated instruction to enter the kernel, run a validated handler, and resume the next user instruction with a result. Libc wraps that ABI and hides errno conversion. The vDSO answers a few hot queries without a privilege change. Cost lives in the crossing, which is why batching interfaces exist.

If you already know how a process owns an address space and how a thread is scheduled, the system call is the remaining door between that thread and everything it is not allowed to touch directly.

Related reading on this site: How Virtual Memory Works, Processes vs Threads Explained, and How Linux Containers Work.