How the Linux Network Transmit Path Works: sk_buffs, qdiscs, and Why send() Returns Before the Packet Leaves

A process that calls send() on a TCP socket expects the bytes to leave the machine eventually. It does not expect the call to wait until the NIC has finished DMA. On a quiet path the call can return while the packet is still sitting in a qdisc, or even while the driver has not yet programmed the device. The kernel has accepted the data into its own buffers; the wire has not seen it.

This article explains that contract: how userspace data becomes an sk_buff, how the protocol stack adds headers, why a large TCP write becomes a GSO super-packet, how the qdisc decides when the packet may leave, and why the hard interrupt that completes transmission is not the same event as the send() return. It is not a second tour of the receive path, of TCP congestion control, or of DMA mapping. Those live in other posts. The single search intent is how the Linux transmit path turns a send() into a packet that the driver can hand to hardware.

The contract

send() (or write(), sendmsg(), sendfile()) copies or references application data into kernel memory and returns when the kernel has taken ownership of those bytes. Ownership means the socket buffer is linked into the protocol's send queue or has been handed to the device layer. It does not mean the NIC has transmitted the frame, and it does not mean the peer has acknowledged it. TCP reliability is a separate state machine that lives above this path.

The unit that moves through the transmit path is the sk_buff. It carries the payload, the headers that higher layers have pushed, checksum offload flags, and (for GSO) the segmentation metadata that a later stage will use. Multiple sk_buffs can be chained; a single large GSO sk_buff can represent tens of kilobytes that will become many wire frames.

From userspace to the first sk_buff

For a normal TCP send(), tcp_sendmsg walks the user iovec and allocates sk_buffs (or appends to an existing one that still has room). The copy happens into the sk_buff's linear data or into page fragments. Once the bytes are in the sk_buff, the userspace buffer can be reused. The sk_buff is linked onto the socket's write queue. TCP's send buffer accounting (sk_wmem_alloc) now includes it. If the write queue is full relative to the congestion window, the advertised window, or the socket's sndbuf, the call blocks or returns EAGAIN.

UDP is simpler: one datagram usually becomes one sk_buff (or a short chain). The protocol does not retransmit, so the path is shorter, but the same device layer still sits underneath.

Protocol headers and GSO

When TCP decides a segment may be sent, tcp_write_xmit builds the TCP header by pushing it onto the sk_buff and fills sequence numbers, window, and options. If the payload is larger than one MSS and the device supports TSO (or GSO is enabled), the stack does not cut the sk_buff into MTU-sized pieces yet. It records gso_size (the intended payload per segment) and gso_type in the sk_buff's shared info. The large sk_buff continues as a single object.

The IP layer (ip_queue_xmit) performs the route lookup (often cached on the socket), pushes the IP header, and runs the OUTPUT and POSTROUTING netfilter hooks. Neighbor resolution fills the Ethernet header. The sk_buff is still one object. Only near the driver, or inside the NIC if TSO is available, does the large sk_buff become multiple frames. Keeping it large reduces the number of sk_buff allocations, qdisc operations, and header constructions the CPU must perform.

The qdisc decides when the packet may leave

dev_queue_xmit is the entry to the device layer. Every net_device has a root qdisc. On a single-queue device the default is pfifo_fast: three priority bands, packets classified by TOS or SO_PRIORITY, always dequeued from the highest non-empty band. On a multi-queue device the root is usually mq, which fans out to a per-queue qdisc (often fq_codel or fq). The qdisc's enqueue method accepts the sk_buff or drops it. A shaping qdisc (tbf, htb) may hold the packet until a token is available; a work-conserving qdisc dequeues immediately if the driver is not already busy.

If the qdisc can dequeue, the path calls the driver's ndo_start_xmit (via dev_hard_start_xmit). If the driver returns NETDEV_TX_BUSY, the sk_buff is requeued and a softirq is scheduled to try again later. Transmission completion is asynchronous: the driver maps the sk_buff pages for DMA, rings the doorbell, and returns. The actual free of the sk_buff happens later, usually from the TX completion interrupt or from the NET_TX_SOFTIRQ that drains the completion queue. That free is the moment the kernel has finished with the packet; the wire may have seen it milliseconds earlier or later, depending on the device queue depth.

A concrete send path

A process writes 64 KiB to a TCP socket whose congestion window allows it. tcp_sendmsg allocates one large sk_buff, copies the data into page fragments, and links it. tcp_write_xmit sees that the device supports TSO, sets gso_size to the MSS, and hands the sk_buff to ip_queue_xmit. The IP and Ethernet headers are added. The sk_buff reaches the qdisc, which is empty and work-conserving, so it is dequeued immediately. The driver maps the fragments, programs a TSO descriptor that tells the NIC the MSS and the header template, and rings the doorbell. send() returns. Later the NIC segments the data into ~1500-byte frames, computes checksums, and transmits them. The completion interrupt (or a polled completion) frees the original sk_buff. From the application's point of view the call finished before any of those frames left the host.

Misconceptions

• send() waits for the packet to leave the NIC. It waits for the kernel to accept the data into an sk_buff that is queued or handed to the device layer. Device transmission and completion are later events.
• Every sk_buff that reaches the qdisc is transmitted immediately. Shaping, policing, and a full driver queue can delay or drop it. pfifo_fast can drop when its bands are full.
• GSO means the CPU always segments. GSO is the metadata that allows deferred segmentation. If the NIC supports TSO, the CPU never walks the payload to cut it; the hardware does. Software GSO runs only when the device cannot.
• The transmit softirq is the same as the receive softirq. NET_TX_SOFTIRQ drains qdiscs that were stopped and completion queues. NET_RX_SOFTIRQ (and NAPI) handles incoming frames. They share the softirq mechanism but not the work.

Takeaways

The Linux transmit path turns a send() into one or more sk_buffs, adds protocol headers, optionally defers segmentation with GSO/TSO, and hands the result to a qdisc. The qdisc is the point where traffic control, priority, and device backpressure are applied. The driver's ndo_start_xmit is the moment the hardware is programmed; completion is a later asynchronous event that frees the sk_buff.

send() returns when the kernel owns the bytes. That ownership is useful: the application can reuse its buffer, and TCP can manage reliability above the device. It is not a promise that the packet has left the machine or that the peer has received it.

Related reading: How the Linux Network Receive Path Works, How TCP Works, How DMA and the IOMMU Work, and How sendfile and splice Work.