How epoll Works: Readiness, Level vs Edge Trigger, and Why select Doesn't Scale

A process that accepts ten thousand TCP connections cannot afford to ask the kernel about each socket in turn. select(2) and poll(2) copy an interest set into the kernel, walk it, and copy a ready set back. Cost grows with the number of watched descriptors, including the idle majority.

epoll is Linux's readiness multiplexer. You create one instance, add file descriptors to an interest list once, and wait only for the ready list. The kernel wakes you when a registered descriptor becomes readable, writable, or hung up. You still issue read and write yourself. epoll answers which fds can be touched without blocking, not please move these bytes.

This article is about that contract: interest list versus ready list, level-triggered versus edge-triggered notification, EPOLLONESHOT and EPOLLEXCLUSIVE, and why older multiplexers collapse under idle connections. It is not a second tour of the syscall ABI, TCP recovery, or io_uring rings.

Related reading: How System Calls Work, How io_uring Works, How TCP Works.

The contract

An epoll instance is a file descriptor from epoll_create1. That fd names two kernel structures: an interest list of watched fds with event masks and a 64-bit cookie, and a ready list of those with a pending matching condition.

epoll_ctl mutates the interest list. epoll_wait or epoll_pwait sleeps until the ready list is non-empty or a timeout fires, then copies events into a user array. Target sockets are not stored inside the epoll fd. Closing a socket removes it from every watching instance. Closing the epoll fd tears the instance down.

The cookie is struct epoll_event.data, a union for a pointer to connection state, an integer id, or the raw fd. The kernel copies it back unchanged so a server can map an event to a connection without scanning thousands of sockets.

Why select and poll do not scale with idle fds

select uses three bitmasks sized by the highest fd plus one, with a typical libc FD_SETSIZE cap of 1024. Every wait copies the whole mask in and a ready mask out. The kernel walks every set bit.

poll uses an array of pollfd entries. There is no 1024 cap, but the array is still copied both ways and scanned in full. An idle keep-alive socket still occupies a slot that must be visited.

epoll keeps interest in the kernel. After the add, a wait copies only fired events, up to maxevents. An idle socket costs almost nothing on the wait path until its state changes. That made C10K servers practical on Linux. A burst still costs O(ready) user-space work. epoll removes the O(n) scan of idle watchers, not the work of handling traffic.

The three calls

int efd = epoll_create1(EPOLL_CLOEXEC);
epoll_ctl(efd, EPOLL_CTL_ADD, sock, &ev);
epoll_ctl(efd, EPOLL_CTL_MOD, sock, &ev);
epoll_ctl(efd, EPOLL_CTL_DEL, sock, NULL);
n = epoll_wait(efd, events, maxevents, timeout_ms);

epoll_create(size) still exists; the size hint has been ignored for years. EPOLL_CLOEXEC prevents leaking the instance across exec. ADD fails with EEXIST if the fd is already present. MOD and DEL fail with ENOENT if it is not.

struct epoll_event ev;
ev.events = EPOLLIN | EPOLLRDHUP;
ev.data.ptr = conn;

EPOLLIN means a read-like call will not block, or the peer sent FIN. You still read until 0 or an error to distinguish those. EPOLLOUT means a write-like call will not block. EPOLLRDHUP (2.6.17) means the peer shut the write half. EPOLLERR and EPOLLHUP are always reported. Timeout 0 polls without sleeping. -1 waits forever. epoll_pwait installs a signal mask for the wait.

Level-triggered versus edge-triggered

Without EPOLLET, epoll is level-triggered. If a condition is true at epoll_wait, the fd is reported. If you return without consuming it, it is reported again. That matches poll.

Example from epoll(7): a pipe receives 2 KiB. The reader is watching EPOLLIN. After reading 1 KiB, a second level-triggered wait returns immediately because 1 KiB remains. A socket is writable most of the time, so level-triggered EPOLLOUT without a pending write spins. Register OUT only after send returns EAGAIN, then MOD it off when the buffer drains.

EPOLLET reports the transition to ready, not the continuing level. After a partial read, leftover bytes do not produce another wait until new data arrives. The safe recipe: non-blocking fd; drain read/write until EAGAIN; only then wait again. Stopping after one successful read is the classic ET stall. Bytes remain in the kernel buffer; the event loop just will not wake for them.

ONESHOT, EXCLUSIVE, and ready meaning

EPOLLONESHOT masks the fd after one delivery until EPOLL_CTL_MOD re-arms it. It does not imply ET. Use it so two threads waiting on one instance do not both receive the same socket.

EPOLLEXCLUSIVE (Linux 4.5) reduces accept stampedes when many waiters watch one listen socket: the kernel wakes one or more exclusive waiters instead of all of them. Set it at ADD time. It cannot be combined with ONESHOT. SO_REUSEPORT plus one epoll instance per thread is the other herd fix.

EPOLLIN on TCP means bytes in the receive buffer, a queued error, or a FIN. Read until you know which. EPOLLOUT means room in the send buffer, not that the peer ACKed. EPOLLRDHUP is a half-close. Regular files are typically always ready; use io_uring or threads for file I/O. Pipes, eventfd, timerfd, and signalfd work as readiness sources.

Server loop, io_uring, takeaways

Add the listener with EPOLLIN. Wait into a small stack array. On the listener, accept until EAGAIN, set each client non-blocking, ADD with EPOLLIN | EPOLLRDHUP | EPOLLET. On IN or RDHUP, read until EAGAIN or 0; close on 0 or hard error. On EAGAIN from send, MOD OUT on; when OUT fires, flush and MOD OUT off. Protocol parsing is not an epoll concern. See How WebSockets Work for the handshake above this loop.

epoll delivers readiness; you still recv/send. io_uring delivers completions for the I/O itself. IORING_OP_POLL_ADD is readiness on a ring. Fork inherits the instance; CLOEXEC only helps across exec. dup does not copy interest to the new number. maxevents caps one batch. An eventfd is the usual cross-thread wake for a blocking wait.

epoll stores interest in the kernel and returns only ready fds, which is why idle connection counts that bury select and poll stay cheap. Level-triggered repeats while the condition holds. Edge-triggered reports the edge and demands a drain to EAGAIN. Ready is not a byte count. After the wakeup you still perform the I/O.