How io_uring Works: Submission Queues, Completion Queues, and Why Batching Beats Syscalls

A tight loop of read and write pays for a privilege crossing on every operation. After Spectre-era mitigations that crossing got more expensive, not less. The kernel already knew how to move bytes. The tax was telling it, one trap at a time, which bytes to move.

io_uring is Linux's answer to that tax. The application and the kernel share two ring buffers. The application writes requests into a submission queue. The kernel writes results into a completion queue. One io_uring_enter can submit dozens of operations and, if you ask it to, wait for some of them to finish. With submission-queue polling, even that call can disappear for stretches of a hot path.

This article is about that interface: how the rings are laid out, what a submission queue entry actually contains, how completions come back, why memory ordering matters, and where SQPOLL and registered files change the cost model. It is not a second tour of the syscall ABI, of virtual-memory page faults, or of mmap. Those pieces exist so that the rings can be mapped. They are not the contract.

Related reading on this site: How System Calls Work, How mmap Works.

The contract

An io_uring instance is a file descriptor plus two shared rings created by io_uring_setup and mapped with mmap.

• The application is the only writer of submission queue entries (SQEs).
• The kernel is the only writer of completion queue entries (CQEs).
• Each side advances one index: the application advances SQ tail and CQ head; the kernel advances SQ head and CQ tail.
• A request is identified across the rings by user_data, an opaque 64-bit value the application chooses. Completions are not required to come back in submission order.

That last point is the one people miss. io_uring is not a FIFO of results. It is a pair of rings plus a matching token. If you need ordering, you encode it: link SQEs with IOSQE_IO_LINK, or wait in user space until the CQE you care about appears.

Setup: one fd, two mappings

io_uring_setup(entries, ¶ms) asks for a submission ring of at least entries slots. The kernel returns a file descriptor and fills struct io_uring_params with the actual sizes and with byte offsets of the ring fields inside the mapped pages.

Those offsets exist because the kernel and user space must agree on where head, tail, ring_mask, ring_entries, and the flags live without compiling against a single shared struct layout that would freeze forever. liburing hides the offsets. The raw interface does not.

Typical mapping:

int fd = io_uring_setup(32, &p);

void *sq_ptr = mmap(NULL, p.sq_off.array + p.sq_entries * sizeof(__u32),
                    PROT_READ | PROT_WRITE, MAP_SHARED | MAP_POPULATE,
                    fd, IORING_OFF_SQ_RING);

void *cq_ptr = mmap(NULL, p.cq_off.cqes + p.cq_entries * sizeof(struct io_uring_cqe),
                    PROT_READ | PROT_WRITE, MAP_SHARED | MAP_POPULATE,
                    fd, IORING_OFF_CQ_RING);

void *sqes = mmap(NULL, p.sq_entries * sizeof(struct io_uring_sqe),
                  PROT_READ | PROT_WRITE, MAP_SHARED | MAP_POPULATE,
                  fd, IORING_OFF_SQES);

On many kernels the SQ ring and CQ ring share one mapping when the offsets allow it. The SQE array is a separate mapping. The SQ ring itself is not an array of SQEs. It is an array of indices into the SQE array. That extra hop lets the kernel consume SQEs without copying the 64-byte structures around the ring, and it lets an application prepare SQEs out of order and then publish their indices in one tail update.

What an SQE contains

A submission queue entry is a packed description of one operation. The fields that matter for a vectored read:

sqe->opcode = IORING_OP_READV;
sqe->fd     = fd;
sqe->addr   = (unsigned long) iovecs;
sqe->len    = nr_iovecs;
sqe->off    = file_offset;
sqe->user_data = some_request_id;
sqe->flags  = 0;

opcode names the work. Early opcodes mapped closely onto existing syscalls: IORING_OP_READV, WRITEV, FSYNC, POLL_ADD, ACCEPT, CONNECT, OPENAT, CLOSE, STATX, SENDMSG, RECVMSG. Later kernels added timeouts, linked timeouts, fixed-file reads, buffer selection, and a long list of socket and filesystem ops. The important property is not the catalogue. It is that each opcode is a request the kernel can run without another user-space trap in the middle.

user_data is copied into the matching CQE. It is the only reliable way to know which request finished. Do not assume slot number equality between SQ and CQ.

Flags on the SQE change scheduling, not the opcode:

• IOSQE_ASYNC asks the kernel not to try a cheap inline completion on the enter path.
• IOSQE_IO_LINK chains this request to the next; a failure breaks the chain.
• IOSQE_IO_DRAIN waits for earlier requests on this ring to complete first.
• IOSQE_FIXED_FILE treats fd as an index into a registered file table instead of a raw descriptor.
• IOSQE_BUFFER_SELECT asks the kernel to pick a buffer from a registered group.

Publishing work and reading results

Filling an SQE is not submission. Submission is advancing the SQ tail so the kernel is allowed to observe the new entries. The application must store the SQE fields first, then update tail with release semantics. The kernel loads tail with acquire semantics, reads the SQE, and later stores a CQE and advances CQ tail.

The pairing is the same idea as any single-producer / single-consumer ring:

• Writer of a slot: write the payload, then release-store the tail.
• Reader of a slot: acquire-load the tail, then read the payload.

liburing does this with io_uring_get_sqe, prep helpers, io_uring_submit, and io_uring_wait_cqe. Underneath, io_uring_submit is usually an io_uring_enter that tells the kernel how many new SQEs exist and, optionally, how many CQEs the caller wants before the syscall returns.

A CQE is small:

struct io_uring_cqe {
    __u64 user_data;
    __s32 res;
    __u32 flags;
};

res is a syscall-style result: a non-negative byte count or file descriptor on success, a negative errno on failure. There is no parallel errno thread-local. The completion is the result.

The application then advances CQ head. Leaving head behind is how you leak ring space. A full CQ can stall further completions depending on kernel version and flags; treat reaping as part of the hot path, not as an afterthought.

Why this is cheaper than one syscall per I/O

A conventional read does three expensive things at once: it crosses into the kernel, it describes one operation, and it waits until that operation has a result (or would block). io_uring splits those jobs.

1. Description lives in shared memory. Preparing an SQE is stores to a mapped page, not a trap.
2. Submission can batch. One enter can hand over an entire burst of SQEs.
3. Waiting is optional and counted. You can submit 32 reads and sleep until 8 CQEs exist, then keep the rest of the pipeline moving.
4. With IORING_SETUP_SQPOLL, a kernel thread watches the SQ. After the rings are warm, the application can publish SQEs and reap CQEs without entering the kernel at all, until the poller goes idle and sets IORING_SQ_NEED_WAKEUP.

SQPOLL is not free. It is a dedicated kernel thread that burns a core while it is awake. On early 5.x kernels the idle behavior was poor enough that people pegged a CPU at 100% with an empty ring. Modern kernels sleep after sq_thread_idle milliseconds. Use SQPOLL when the submission rate justifies a parked thread. Do not turn it on because the flag sounds fast.

IORING_SETUP_IOPOLL is a different flag. It is for block devices that support polling completions instead of IRQs. The application must call enter to harvest those completions. Mixing it up with SQPOLL is a common reading error.

Registered files and buffers

Every request that carries a file descriptor forces the kernel to look that descriptor up in the process file table. At high IOPS that lookup and the associated lifetime rules show up in profiles. io_uring_register with IORING_REGISTER_FILES installs a snapshot of fds into the ring. Later SQEs set IOSQE_FIXED_FILE and put an index in the fd field. The kernel already holds the file.

Buffers have the same shape. IORING_REGISTER_BUFFERS pins user pages so the kernel can DMA without taking a fresh reference on every request. IORING_OP_READ_FIXED / WRITE_FIXED then name a registered buffer. Provided-buffer pools go further: the application registers a group of buffers, sets IOSQE_BUFFER_SELECT, and the completion tells it which buffer the kernel filled.

Pinning is a trade. You spend locked memory and registration syscalls to remove per-I/O accounting. That is the right trade for a long-lived server with a stable set of sockets and a recycled buffer pool. It is the wrong trade for a short command that opens a file, reads it once, and exits.

How this differs from aio and epoll

POSIX AIO and Linux libaio also queue work. Linux io_submit is limited in which file types it accepts, copies command structures on the way in, and still needs a separate harvest path. io_uring was designed as a general request ring: files, sockets, timeouts, metadata, and later a growing set of kernel operations that are not "I/O" in the block-device sense.

epoll answers a different question. It tells you that a descriptor is ready. You still issue read or recv afterwards. io_uring can replace both steps: submit the read when you want the data, and learn the result from the CQE. You can still use IORING_OP_POLL_ADD if you only want readiness. Most new servers that adopt io_uring stop treating readiness as a separate phase for the I/O they already know they will issue.

A compact walk-through

Read 4 KiB from an open file using liburing-shaped steps, written out so the rings are visible:

1. Create a ring with 8 SQ entries.
2. Get an SQE, set opcode READ, fd, buffer pointer, length 4096, offset 0, user_data = 1.
3. Store-release SQ tail += 1.
4. Call io_uring_enter(fd, 1, 1, IORING_ENTER_GETEVENTS, NULL): submit one, wait for one.
5. Acquire-load CQ tail. Read the CQE at head. Check user_data == 1 and res == 4096 (or a short count, or a negative errno).
6. Advance CQ head.

Replace step 4 with a burst: fill eight SQEs, enter once with to_submit = 8 and min_complete = 1, then drain CQEs as they appear. That is the shape databases, proxies, and NVMe-backed services actually use.

Failure modes that are part of the design

Completions can arrive out of order. Linked SQEs are the tool for "do B only if A succeeded." A drained request waits for earlier work on that ring, which can create surprising latency if you drain on a busy shared ring.

A negative res is the error. There is no extra channel. If you forget to check it, you treat -EAGAIN as a length.

The rings are shared memory. If you write tail before the SQE is fully initialized, the kernel can observe a half-built request. That is why the barrier story is not optional commentary. Use liburing, or use acquire/release on the head and tail indexes.

SQPOLL needs a wakeup when the kernel thread has gone idle. Publishing SQEs and then spinning on an empty CQ while IORING_SQ_NEED_WAKEUP is set is a self-inflicted stall. After you update tail, load the SQ flags; if the bit is set, enter with IORING_ENTER_SQ_WAKEUP.

Older kernels had a long series of io_uring CVEs around registered buffers, fixed files, and request lifetime. Treat kernel version as part of the API. A feature flag in params.features is the supported way to ask what this kernel actually implements.

Where it shows up

QEMU, SPDK-style storage paths, nginx and other proxies on recent builds, Rust runtimes such as tokio's io-uring backend, and several databases that already lived on top of Linux AIO moved or are moving the hot I/O loop here. The attraction is the same in each case: more work per privilege crossing, completions that do not require a ready-then-read dance, and optional zero-syscall submit once a poller is running.

It is still Linux-specific. Portable code keeps a blocking or epoll fallback. That is not a flaw in the ring. It is the usual cost of using a kernel interface that other operating systems did not copy verbatim.

Takeaways

io_uring turns I/O into messages on two shared rings. The application writes SQEs and advances SQ tail. The kernel writes CQEs and advances CQ tail. user_data is the correlation id. Batching and optional waiting are the reason the interface exists: one enter, many operations.

SQPOLL and registered files or buffers remove more of the remaining cost, at the price of a kernel thread and pinned state. epoll tells you a descriptor is ready. io_uring can perform the operation that readiness was standing in for.

If you already understand why a million tiny syscalls lose, this is the batch interface that sentence was pointing at.