Zero-Copy I/O: From sendfile to io_uring – Evolution and Impact on Latency in Distributed Logs

Zero-copy I/O refers to techniques that transfer data without redundant CPU copying between user-space and kernel-space buffers. Over the past two decades, Linux has introduced a series of zero-copy primitives – from the early sendfile() system call to modern io_uring interfaces – dramatically reducing I/O overhead. These advancements are crucial for distributed log and streaming systems (like Kafka, Redpanda, Liftbridge, etc.), where latency budgets are tight and each extra data copy or context switch eats into those budgets. This report provides a deep technical analysis of this zero-copy journey and its implications:

• The evolution of zero-copy I/O: from sendfile to splice/vmsplice to today’s io_uring.
• Zero-copy networking vs. zero-copy file I/O: differences in how they work and interplay.
• Real-world usage in systems: how ClickHouse exploits zero-copy for analytics, Redpanda (Kafka-compatible) uses io_uring and zero-copy to beat Kafka, and how Kafka clones and log systems (Liftbridge, etc.) adopt these techniques.
• Zero-copy in storage layers: e.g. memory-mapped files and direct I/O, and their impact on performance.
• Benchmarks and diagrams: comparing traditional copy loops vs. sendfile vs. io_uring-based approaches, to visualize performance gains.
• Actionable takeaways: what system builders can learn to shrink latency in I/O-intensive workloads.

The Cost of Copying and Why Zero-Copy Matters

Every time data is copied from one memory region to another, it incurs CPU cycles and potentially triggers mode switches (user/kernel transitions). In I/O-heavy applications (web servers, databases, log brokers), copying data multiple times – e.g. reading from disk into a user buffer, then copying to a socket buffer – adds latency and CPU load. By eliminating unnecessary copies, zero-copy techniques free up CPU and shave off milliseconds (or microseconds) of latency per operation. In distributed log systems, which pipeline disk reads and network writes, even a single extra memory copy can accumulate into higher tail-latencies for end-to-end operations.

Key idea: Instead of the CPU shuffling bytes between user and kernel, have the kernel transfer data internally (or the DMA engine copy directly) while the application orchestrates at a high level. This reduces:

• CPU time per I/O – CPU doesn’t spend cycles on byte copies it doesn’t need to inspect.
• Context switches – Fewer system calls and user-kernel transitions (which also means fewer cache disruptions).
• Memory bandwidth usage – Avoiding redundant copies frees memory bandwidth for other work.

In practice, zero-copy often means reusing existing memory pages or buffers rather than copying data. For example, Linux’s pipe buffers are implemented as pointers to pages; using splice() to transfer data via a pipe just passes page references around, rather than copying the page contents.

Evolution of Zero-Copy I/O in Linux

Early Days: sendfile() – File to Socket Zero-Copy

sendfile() was introduced in Linux 2.2 (around 1999) as one of the first zero-copy mechanisms. It was designed to send data directly from a file descriptor to a socket in one kernel call, without routing the data through user-space buffers. Under the hood, sendfile() pulls data from the file (disk cache) into kernel memory and then hands those pages off to the network stack, avoiding a user-space copy. This took advantage of hardware support like NIC scatter-gather DMA, so that the NIC could directly read file pages from kernel memory and transmit them.

• Limitation: Originally, Linux sendfile only worked with a disk file as the source and a socket as the destination. It couldn’t send from socket-to-socket or file-to-file (nor socket-to-file), and it didn’t help if your data was generated on the fly (not already in a file).
• Impact: For static file serving (e.g. sending a video file over HTTP), sendfile() could dramatically improve throughput. By combining what would be a read()+write() loop into one call, it cut context switches in half and eliminated copying the file data to user space. The Linux manual notes that because the copying is done in-kernel, sendfile() is more efficient than an equivalent read/write loop.

Performance gains: Even early tests showed significant wins. A developer reported switching to sendfile doubled throughput from ~800 MB/s to ~1.5 GB/s for file transfers. Netflix engineers similarly leveraged sendfile() on FreeBSD to push 6 Gbps to 40 Gbps on 40GbE links. These improvements come from slashing CPU overhead and letting the kernel/NIC handle bulk data movement. (We’ll see concrete benchmarks shortly.)

Generalizing Zero-Copy: splice() and vmsplice()

Linux 2.6.17 (2006) expanded zero-copy capabilities with splice() and vmsplice() system calls:

• splice(fd_in, fd_out) moves data between a file descriptor and a pipe without copying to user space. It effectively allows zero-copy transfer from one FD to another by using an in-kernel pipe as the buffer. For example, one can splice a socket to a pipe, then splice the pipe to a file, to achieve socket→file zero-copy (useful for receiving network data to disk). Likewise, file→pipe→socket covers file serving like sendfile. Under the covers, splice uses the pipe’s page pointers to pass data****, copying “references” instead of copying bytes. This means data stays in kernel memory and is just linked between the source and destination.
• vmsplice(uaddr -> pipe) maps (“splices”) a user-space memory region into a pipe, essentially granting the kernel access to user pages without copying them. This can be seen as “pinning” user memory to pipe buffers. Paired with a subsequent splice(pipe -> sock), this enables sending user-space data out a socket with zero copies (similar to send(socket, ..., MSG_ZEROCOPY) which came later). However, vmsplice has caveats – it truly zero-copies only from user to pipe; the reverse direction (pipe to user) actually falls back to copying. The interface was made symmetric for future improvement, but as of now, vmsplice is mainly useful for injecting user data into the kernel zero-copy pipeline.

Together, splice/vmsplice and tee (to duplicate pipe data) gave user-space “full control of an in-kernel buffer” to route data between sources/sinks. In fact, Linux’s sendfile() was re-implemented internally as a special case of splice – it essentially splices from the file to the socket. So any performance improvements in splice benefit sendfile too. By Linux 2.6.21, splice was stable and “documented to be zero copy in most cases”, using page references to avoid actual copying.

Use cases: splice() broadened zero-copy beyond file serving. For example, HAProxy uses splice() to shovel data from incoming to outgoing sockets (TCP splicing) entirely in kernel, accelerating proxies. And as one LWN commenter noted, splice could do what MSG_ZEROCOPY does, except the latter added more explicit completion notifications (important for knowing when it’s safe to reuse a buffer).

Modern Asynchronous Zero-Copy: io_uring

The latest leap is io_uring, introduced in Linux 5.1 (2019) and rapidly evolving since. While io_uring is a general async I/O interface rather than a single “zero-copy call,” it enables new zero-copy patterns with far less overhead:

• It uses shared rings between user and kernel for submitting I/O requests and reaping completions, so batching and asynchronous execution are cheap. This cuts per-I/O system call overhead to near zero – many operations can be issued with no syscalls at all per operation (just a memory enqueue). The result: much lower CPU cost, especially under high concurrency or streaming loads (common in log systems).
• io_uring supports fixed buffers and fixed files registration. An application can register a set of data buffers up front and perform reads/writes on them without repeated alloc/free or address translation. When combined with zero-copy flags, this is powerful. For example, an app can allocate a large buffer aligned for DMA, register it, and use io_uring to read from disk and then send to network, all asynchronously, without extra copies or even buffer allocation overhead on each I/O.
• Linux has added explicit zero-copy networking ops to io_uring. Notably, io_uring_prep_send_zc() allows an async send with zero-copy semantics. This is effectively the equivalent of send(..., MSG_ZEROCOPY) but integrated into the io_uring submission queue. Using registered buffers (via io_uring_prep_send_zc_fixed), the kernel can transmit directly from user memory to the NIC without copying. The completion event tells you when it’s done (so you know when you can reuse or free the buffer). This addresses the long-standing challenge of sending application-generated data zero-copy (which sendfile couldn’t do).
• io_uring also introduced an async splice operation (as of Linux 5.13+), effectively allowing in-kernel pipe transfers to be requested asynchronously. Jens Axboe (author of both splice and io_uring) noted: “As soon as the splice stuff is integrated, you’ll have [sendfile] in io_uring as well”. In practice, one can chain an IORING_OP_SPLICE to move data from a file to a pipe, and another to move pipe to socket, all in one submission. This achieves the same zero-copy effect as sendfile, but non-blocking and batchable.

Why it matters: In distributed logs, there are typically many concurrent I/O operations (reads for consumers, writes for new events, replication traffic, etc.). io_uring shines here by handling thousands of async ops with minimal CPU, and by allowing creative combinations (like async sendfile/splice or parallel reads). Fewer context switches mean less jitter and more predictable latency.

Latency benefits: Redpanda’s team highlights io_uring as an efficient way to do asynchronous disk I/O in C++ (something very hard in the JVM-based Kafka). By using io_uring for disk reads/writes, Redpanda can serve log data faster with lower tail latencies. In one case, enabling modern Linux features (like io_uring and DMA) helped Redpanda achieve 20× lower p99.9 latency than Kafka on similar hardware. While that improvement isn’t solely due to io_uring (it’s also their threading model, etc.), the ability to efficiently saturate disk and network with minimal overhead is a big factor.

Summary of evolution: Linux went from a single-purpose zero-copy call (sendfile), to a flexible zero-copy buffer plumbing (splice/vmsplice), and now to a complete asynchronous I/O submission model (io_uring) that incorporates zero-copy capabilities. Each step aimed to remove more overhead: eliminating user-space copies, then cutting down system calls, and now enabling batch async operations with zero-copy.

Zero-Copy Networking vs. Zero-Copy File I/O

Although the goal is the same (avoid redundant data copying), zero-copy techniques in networking versus file/storage I/O have different mechanisms and considerations:

• Zero-Copy Networking: Typically refers to sending or receiving data on sockets without copying between user and kernel buffers. For sending, this includes:
  • Using calls like sendfile() or splice() to send data already in kernel memory (file pages or pipe buffers) directly out to the NIC. The NIC can use DMA gather to pull data from multiple kernel memory locations (e.g. file pages) into the wire format, so the CPU doesn’t copy those bytes.
  • Using MSG_ZEROCOPY with send(): a relatively recent addition (Linux 4.14) that lets an application send a user-space buffer without copying it into the kernel stack. Under the hood, the kernel pins the user pages and the NIC transmits from them directly. The trade-off is the app must check when the send is complete to safely reuse the buffer (via a special socket error queue message).
  • Kernel-bypass networking (DPDK, etc.) where the user process and NIC share hugepage memory buffers – effectively zero-copy by avoiding the kernel entirely. Redpanda, for example, can leverage DPDK for packet I/O, meaning it drives the NIC in user-space and eliminates kernel copy overhead. This is extreme zero-copy networking (useful when ultra-low latency is needed). For receiving data, zero-copy is trickier: normally, the kernel DMAs packets into its buffers and then copies to user. Techniques like AF_XDP or PACKET_ZEROCOPY allow the NIC to DMA directly into user-provided buffers, but those are beyond the standard socket interface. (Most streaming systems focus on send-side zero-copy because that’s where the bulk data is – e.g. brokers sending large log batches out).
• Zero-Copy File I/O: This refers to avoiding extra copies when reading from or writing to storage:
  • Using memory-mapped files (mmap): Instead of read() which copies data from kernel (page cache) to user buffer, mmap() maps the file’s pages into the process address space. Now the file data is accessible in user-space directly from the page cache, with no copy. The kernel just populates those pages via DMA from disk. This means one less data copy for reads (and potentially writes, if using MAP_SHARED). Figure 3 in the reference shows that with mmap, data goes disk -> kernel buffer (page cache) via DMA, then socket sends directly from that kernel buffer, without a separate user copy. The CPU only copies from the page cache to the socket if needed, and with modern NIC offload (DMA gather) even that copy can be skipped.
    • The downside: mapping/unmapping memory has a cost (changing page tables), and if access patterns are random, page faults can introduce latency spikes. But for sequential reads, mmap effectively gives you a zero-copy stream (especially if combined with sendfile on the fd). Many high-performance systems (e.g. certain databases, or tools like memcached for huge item storage) rely on mmap for efficient reads.
  • Using Direct I/O (O_DIRECT): This bypasses the page cache entirely. The disk controller DMA transfers data directly into user-provided buffers (which must be page-aligned). There’s no copy into the page cache, thus avoiding double-buffering. The application then holds the only copy of the data. While O_DIRECT doesn’t itself send the data to the network, it complements zero-copy: e.g. Redpanda opens log files with O_DIRECT and allocates aligned buffers, so that when data is read from disk it’s placed right into those buffers, which can then be sent out (possibly with send_zc or via DPDK) without another copy. Redpanda explicitly manages its cache in user-space (“DMA read into my buffer pool, then send from there”), rather than letting the kernel copy data around. This eliminates an extra copy through the page cache and gives more predictable latency (no kernel write-back interference).
  • Copy-on-Write cloning: In storage systems, sometimes zero-copy is achieved by sharing pages until mutation. For example, if two processes or threads need the same data, they can share an mmap’d region (or use splice between pipes) so that the memory isn’t duplicated. It’s not directly about file/network I/O, but it’s a memory-level zero-copy that storage engines use (like zero-copy cloning of buffers on write).

Interaction of both: In a pipeline like file -> network, these techniques align. For instance:

• Kafka brokers traditionally use file-to-network zero-copy by reading from log files and sending to consumers via sendfile() (FileChannel.transferTo in Java). This pulls data from the file (which is already in kernel memory via the page cache) and streams it to the socket without user-space involvement. It’s zero-copy networking (no user copy) enabled by zero-copy file access (page cache acts as the intermediary). Kafka’s use of sendfile is why consumers can fetch at high throughput with low CPU usage on the broker.
• Conversely, network -> file (producers sending data to broker, which writes to disk) can be optimized by splice(socket -> pipe) then splice(pipe -> file). This avoids copying the inbound data into user space. While not as commonly implemented in Kafka (which uses user-space compression, etc.), it’s possible. A simpler approach some systems use is to receive into an aligned buffer and write that buffer to disk with O_DIRECT (which is what Redpanda’s engine does). The buffer is filled from the NIC (one copy from kernel to user, unless using AF_XDP), then written out with DMA to disk (no second copy to kernel cache). Redpanda notes their log appends use thread-local aligned buffers and “we flush as little as we can to physical device, and do as much as possible via DMA writes”.

Networking vs File I/O focus: In summary, zero-copy networking cares about optimizing the socket layer (using kernel tricks to avoid user copies on send/recv), whereas zero-copy file I/O focuses on avoiding copies between user and the filesystem cache. In a complete system, you often use both in tandem: keep data either in kernel page cache or in pinned user buffers, and send it out to network from there. The end goal is the same: the data should ideally move from disk to NIC (and vice versa) with at most one memory-to-memory copy (or even zero CPU copies if DMA can do it all). Modern Linux achieves that: disk to page cache via DMA, then NIC does DMA from page cache to wire – the CPU mainly sets up pointers.

Zero-Copy in High-Performance Systems: Case Studies

ClickHouse: Zero-Copy for Fast Analytics

ClickHouse, a high-performance analytical database, leverages zero-copy techniques in both its storage engine and query processing to minimize latency:

• Memory-mapped files for data parts: ClickHouse stores columnar data in compressed files on disk. Rather than always reading these files into intermediate buffers, it can use memory-mapping to load data pages on demand. This means when scanning large columns, ClickHouse is often reading directly from the OS page cache memory, avoiding extra copies. Cloudflare’s engineering blog noted that memory-mapped file access and shared memory can serve millions of small messages per second with sub-microsecond latencies – demonstrating why a DB like ClickHouse would use mmap for frequently accessed data. Essentially, the OS page cache becomes an extension of ClickHouse’s process memory. This zero-copy design lets multiple queries access the same cached pages without duplicating them.
• Zero-copy data pipeline: ClickHouse’s internal I/O buffers are designed to avoid copies. It has abstractions like ReadBuffer/WriteBuffer that can work directly with existing memory regions. One example is how ClickHouse handles query output without extra copying: Instead of writing query results to a pipe or socket byte by byte (which would copy data), ClickHouse can write into a memory buffer and expose that buffer to the client. In a recent integration (the chDB project, which embeds ClickHouse in Python), the developer used ClickHouse’s WriteBufferFromVector to capture the query result in memory and then used Python’s memoryview to send it to the Python side without copying. The result was zero-copy retrieval of SQL results from the ClickHouse engine, dramatically speeding up transfer to the client. This illustrates ClickHouse’s philosophy of not copying data unless necessary – even at the interface between C++ and Python, they found a way to map memory instead of copy.
• Replication and backups: ClickHouse also introduced “zero-copy replication” for data parts – an experimental feature where replicas share the same storage (e.g. S3 or HDFS) and use file hard links instead of copying data files between nodes. While not about memory copies, it’s conceptually similar: avoid copying gigabytes of data for replication by letting multiple nodes refer to the same bytes. (This feature is still evolving and disabled by default.)

Overall, ClickHouse’s use of zero-copy is more on the storage and inter-process side (memory maps, shared buffers) than the networking side, since it typically streams results to clients over TCP in a conventional way. However, by minimizing data movement in its execution pipeline, ClickHouse achieves very high throughput. Its vectorized query execution keeps data in CPU caches as much as possible, and its storage engine avoids overhead of double-buffering. All these contribute to ClickHouse being able to scan billions of rows per second on a single node. For analytics, throughput is king, and zero-copy helps maximize throughput (and correspondingly reduce query latency for I/O-bound queries).

Redpanda (Kafka-compatible): io_uring and Zero-Copy to Outperform Kafka

Redpanda (formerly Vectorized) is a Kafka API compatible streaming log, written in C++ on a thread-per-core, fully asynchronous architecture. It was built explicitly to overcome Kafka’s performance limitations by using modern Linux I/O primitives and a zero-copy philosophy:

• Thread-Per-Core + Zero-Copy design: Redpanda’s engine (based on Seastar) runs one reactor thread per CPU core, handling disk and network I/O on that core without locks. This design encourages zero-copy messaging – data received on one core stays on that core in memory and is sent out or written to disk without bouncing to other cores or contexts. Redpanda pins memory to each thread and avoids sharing, which reduces needless copies or cache invalidation.
• Direct I/O and DMA optimizations: Redpanda bypasses the Linux page cache for its log storage. Instead, it allocates its own memory and uses O_DIRECT file access. By doing so, Redpanda controls exactly when and how data is copied to disk. They align writes to disk block boundaries and flush in large sequential chunks, relying on the disk’s DMA to do the heavy lifting. Their log writer issues direct DMA writes (via io_uring) to persist data, greatly reducing overhead of system calls and eliminating the duplication of data in page cache. As they describe: “we manually align memory [to filesystem layout] and ensure we flush as little as possible to the physical device, spending as much time as possible doing dma_write()”. This means most of the time data is just streaming from memory to disk controller without kernel copying.
• Custom network stack and zero-copy sends: On the networking side, Redpanda can use the DPDK framework for user-space networking. This allows zero-copy receives and sends – packets from the NIC are written directly into Redpanda’s memory pools, and sending is just handing a buffer to the NIC. Even when not using DPDK, Redpanda leverages Linux’s sendfile/splice or io_uring capabilities. Because Redpanda’s log data often resides in memory (in its cache) or in a readily mmappable file segment, it can send data to consumers without copying into intermediate buffers. Essentially, it tries to emulate what Kafka does with sendfile, but even for data that might still be in memory and not yet on disk (since Redpanda can serve from its write-back cache). Also, as of recent versions, Redpanda could take advantage of io_uring’s send_zc to transmit from its buffers without copy, similar to MSG_ZEROCOPY but easier to use.
• Effects on performance: The result of these optimizations is drastically lower latency and higher throughput compared to Apache Kafka. Redpanda reports show tail latencies (p99.9) under heavy load are an order of magnitude lower than Kafka’s. For example, on one test with 1 GB/s traffic, Kafka’s p99.9 latency was ~590 ms, while Redpanda’s was ~29 ms – 20× faster tail response. Redpanda’s own blog attributes this to “C++, DMA, and autotuning” among other things. By eliminating Java GC pauses, system call overheads, and using zero-copy I/O, Redpanda can guarantee that most of the time, data is either in flight via DMA or waiting in an in-memory queue, rather than being copied around or waiting on context switches.
• Intelligent batching: Another point – Redpanda and similar engines batch operations heavily (e.g., reading many log records into a buffer, writing many out in one go). Zero-copy complements this because when you send larger batches, avoiding an extra copy yields even bigger CPU savings. Kafka also batches, but because it’s constrained by the JVM, it ends up copying data between Netty, the JVM heap, and the page cache. Redpanda’s one-time copy or no-copy approach per batch is simply more efficient.

Redpanda demonstrates that using io_uring for disk, zero-copy sends for network, and avoiding the kernel cache leads to both high throughput and stable low latency. In distributed logs, this means brokers can sustain more clients and higher message rates before hitting CPU bottlenecks. It also means latency-sensitive applications (finance, IoT) see more consistent performance (e.g., p99 < 10 ms for produce/consume at high loads, according to Redpanda’s benchmarks).

Other Kafka Clones & Distributed Logs Adopting Zero-Copy

Redpanda isn’t alone – modern log systems recognize the importance of zero-copy:

• Apache Kafka itself has evolved to use zero-copy where possible (despite being Java-based). Kafka has long used the transferTo() NIO call to send file data to sockets, which under the hood invokes sendfile on Linux. This means Kafka brokers don’t copy message bytes into userland when serving consumers – the kernel sends directly from the file buffer cache. Kafka also avoids copies in its network stack by using message aggregation and sendfile for large responses. However, Kafka can’t use vmsplice or MSG_ZEROCOPY easily due to the JVM, and writing to the log (disk) still involves copying from the JVM heap to kernel (no splice on ingest). Hence, Kafka has more overhead on the produce side and more GC pressure, which is where clones like Redpanda excel by cutting those copies.
• Liftbridge (NATS-based log): Liftbridge is a lightweight Go-based log on top of NATS. Its design explicitly uses zero-copy reads for the log data. The author notes that, unlike NATS Streaming (which had an extra hop through NATS core, preventing zero-copy), Liftbridge can use sendfile to stream log files to subscribers efficiently. This means when a client reads from a Liftbridge stream, the server simply invokes sendfile on the stored log segment, sending it out the TCP socket with almost no CPU overhead. As Tyler Treat (Liftbridge’s creator) wrote, “This enables us to do zero-copy reads from a stream…which NATS Streaming is precluded from doing.”. So, even in Go, which typically involves copies with user-space buffers, by leaning on the OS zero-copy, Liftbridge serves data faster and with less CPU than if it had to read into user space (which NATS Streaming did).
• Vectorized/Redpanda we covered (Vectorized is the company name, Redpanda the product). It’s basically the poster child of zero-copy in logs.
• Apache Pulsar (another streaming platform) uses Apache BookKeeper for storage. BookKeeper is written in Java but uses mmap for its journal files and writeDirect for its logs, which likely means it also tries to minimize copies (the journal can be appended with bytes already in a direct ByteBuffer). Pulsar hasn’t highlighted zero-copy as much, but as a modern system it does use techniques like cache avoidance and direct writes for its bookies.
• LogDevice (Facebook’s C++ distributed log) and ShadowDB/L7 (by LinkedIn) – these systems also employ memory pooling, direct I/O and sendfile-like techniques. Facebook’s LogDevice, for instance, was built to deliver high throughput – it stands to reason they’d use sendfile to send log records to readers (given it’s in C++ they easily can). They also likely use batching and possibly splice for replication between storage nodes.
• Distributed databases / consensus logs: Systems like etcd or TiDB’s raft logs can use mmap or direct I/O for their WAL (write-ahead logs) and could use sendfile to ship snapshots. For example, etcd uses mmap for its bbolt storage, meaning snapshotting the data can be done by reading memory pages rather than copying.

In general, any new system that deals with logs or streams at scale tends to adopt a zero-copy mindset:

• Use the kernel as a DMA engine (sendfile, splice).
• Use page sharing (mmap, shared memory) instead of copying data between processes or components.
• Align data structures to avoid copies (e.g., use file-backed memory for queues).

The payoff is seen in better throughput per core and often better latency. For instance, a common theme is replacing a read-loop + write-loop (which can max out at some MB/s per core) with a single sendfile that can saturate the NIC or disk bandwidth with much lower CPU. As one developer quipped, “if the kernel provides a way to do something, and that way gives you better performance in practice, why not use it?”. Zero-copy interfaces are exactly that – a way for applications to utilize kernel internals for speed.

Zero-Copy in Storage Layers (Memory-Mapped Files and Friends)

At the storage layer, zero-copy typically manifests as memory mapping files, sharing buffers, and avoiding redundant caching. The impact of these techniques is huge for read-heavy or communication-heavy workflows:

• Memory-Mapped Files (mmap): Mapping a file into memory allows the file’s contents to be accessed as if it were an array in memory. The OS will load the file’s pages via disk DMA into the page cache. After that, no explicit read() copy is needed – the program is reading those pages directly. For write-heavy loads, mmap can be tricky (due to page dirties and flushes), but for reads it’s ideal. For example, Cloudflare found that for IPC between processes, memory-mapped files outperformed Unix sockets and pipes by a wide margin. Shared memory and mmap achieved under 1 µs latency for 1KB messages, versus 5–9 µs for sockets/pipes that copy data. This is essentially because with mmap, the kernel isn’t shuffling bytes between buffers – both processes are looking at the same bytes in memory (zero-copy between them). In storage engines, mmap is used similarly: e.g., a database reader process might mmap an index or a data file and then do lookups by simply reading the memory (the OS transparently handles fetching the pages). This yields near RAM-speed access for disk data when it’s cached, and when it’s not cached the penalty is just the I/O cost, not additional copying.
  • Impact on analytics: Columnar databases (like ClickHouse, as discussed) often memory-map columns or large chunks, so that analytical queries can scan through memory rather than issuing tons of read() calls. This significantly lowers CPU usage for I/O. It can also improve multi-query throughput because the OS can serve the same cached page to many queries at once without copying it for each query. The trade-off is less control over when data is paged in/out, but systems often mitigate that with madvise or by preloading data.
• Zero-Copy in Buffer Management: Many systems create layers of abstraction that allow zero-copy pass-through. For instance, the Apache Arrow project uses reference-counted memory buffers to allow zero-copy data interchange between systems (a file can be memory-mapped and represented as Arrow arrays without copying). In ClickHouse’s ecosystem, there is an integration with Arrow that uses zero-copy memory sharing – essentially letting ClickHouse and external processes share column data via memory references. In storage engines, a similar idea is the use of cache pinned buffers: e.g., a buffer cache that hands out pointers to data instead of making a copy for each consumer.
• Direct I/O and Bypassing Cache: We touched on O_DIRECT as used by Redpanda. Another example: some database systems (like MySQL with InnoDB or RocksDB in certain modes) use O_DIRECT for their file accesses. This means the DB engine itself manages a unified buffer pool. Data is read from disk into this pool via DMA (one copy), and then if it needs to be sent over the network (say a large blob), the DB can use sendfile by trick – by having a temporary file descriptor or using Linux’s copy_file_range to move data from its buffer to socket without an extra copy. Alternatively, if not using sendfile, the DB can at least avoid a second disk-to-kernel copy since O_DIRECT gave it directly the data. The reason to bypass the OS page cache is partly to avoid double buffering (saving memory and copy time), and partly to have more predictable latency by controlling eviction explicitly.
• Copy-on-write Filesystems and Snapshots: Although slightly tangential, it’s worth noting that zero-copy principles apply in filesystems too. CoW filesystems like Btrfs or ZFS can clone entire snapshots of data without copying blocks – they just copy references to blocks. This is zero-copy at a large granularity. In distributed storage, sending snapshots or doing remote replication can exploit this (e.g., send pointers and only copy when needed). It’s a higher-level use of zero-copy (not via syscalls but via FS design) that also helps latency by reducing huge bulk copies.

Effect on latency: Zero-copy storage means when a read request comes in, the data might already be “there” to be used. For instance, if an event streaming system stores messages in memory-mapped log files, when a consumer asks for the latest data, the broker can directly send those memory pages out (with sendfile). The latency is basically the network latency plus minimal overhead; the broker isn’t incurring extra CPU delay to copy from a read buffer to a send buffer. This is critical for shrinking latency in distributed logs: every millisecond saved per message read/write (by avoiding copy or waiting on I/O) adds up in pipelines.

One concrete measurement from the earlier cited experiment: using memory-mapped file I/O improved throughput by 81% compared to standard buffered I/O in a file copy test. Sendfile improved it by 91% in the same test. This was attributed to eliminating extra copies and syscalls. In microsecond terms, if a normal read+write took, say, 200 ms for some operation, sendfile might cut it to ~100 ms – meaning you handle double the requests in the same time. In high-frequency trading or real-time analytics, these savings are golden.

Benchmark Comparisons: Traditional vs Zero-Copy vs io_uring

To visualize the benefits of these techniques, let’s compare some common I/O methods:

1. File to Socket Data Transfer (1 GB file example) – Traditional read/write loop vs sendfile:

In the above test, sendfile achieved ~75 MB/s vs ~53 MB/s with read/write – purely by cutting overhead. The speedup would be even greater with larger buffers or faster NIC/DMA, as evidenced by reports of nearly doubling throughput with sendfile in real-world scenarios.

2. Relative Speed of Copy Methods (880 MB file) – Java 11 test for buffered vs mmap vs sendfile:

(Higher is better. 81% and 91% improvements were measured for mmap and sendfile respectively, confirming that eliminating user-space copy and extra syscalls nearly doubled throughput.)

3. IPC/Networking latency (1KB messages) – Memory-mapped vs sockets (Cloudflare test):

Memory sharing techniques (which avoid copying data through the kernel) are 10×–17× lower latency than traditional sockets for small messages. This underscores how zero-copy bypass of the kernel can drastically reduce latency for high-frequency small-data transfers. Distributed log systems over a network can’t share memory like that, but within a broker, using shared memory between components (or caching messages in memory ready to send) yields similar microsecond-scale savings.

4. Redpanda vs Kafka throughput and tail latency: Redpanda’s zero-copy, IO-optimized design shows up in benchmarks:

• At 1 GB/s publish rate, Kafka’s p99.99 latency was tens to hundreds of milliseconds, whereas Redpanda’s was a few milliseconds. In one test, Redpanda sustained ~1 GB/s with p99.99 = 6 ms, compared to Kafka’s p99.99 = 400 ms (this is a rough illustrative figure from various reports). Redpanda also achieved higher maximum throughput on the same hardware due to lower CPU per byte.
• In a direct comparsion, Redpanda was 20× faster at tail latencies (29 ms vs 589 ms p99.9) on one workload, and about 4× faster (24 ms vs 98 ms) on another, relative to Kafka. Both systems were durable (fsync enabled); the difference is Redpanda’s use of async I/O and zero-copy means it doesn’t stall as much waiting for data movement.

These numbers drive home that to reach low-millisecond or microsecond latencies in distributed systems, you must minimize copying and context switches. Zero-copy and io_uring are essential tools to achieve that.

Takeaways for System Builders

Designing systems with high throughput and low latency requires rethinking how data moves through the stack. Here are actionable takeaways from the evolution of zero-copy I/O:

• Use Zero-Copy syscalls for bulk data: Whenever you need to transfer large data chunks from storage to network (or vice versa), prefer sendfile() or splice() over manual read/write loops. They dramatically cut CPU usage and can often double throughput. Modern languages often expose these (e.g., Java transferTo, Go Sendfile in the net package). Ensure your application or library uses them for file transfers or network sends.
• Batch and avoid small copies: Zero-copy is most effective when you transfer data in larger blocks (to amortize setup costs). Design your system to batch messages or use gather I/O (e.g., writev or sendfile for multiple buffers) instead of many small writes. Fewer, larger transfers are easier to zero-copy and often can be done entirely with DMA.
• Minimize context switches with async I/O: Consider using io_uring for high-performance disk or network I/O. Its ability to submit many operations without a syscall per operation can significantly reduce latency variability. For example, instead of using epoll + read/write in a loop (which incurs syscalls and copies), using io_uring can handle tens of thousands of IOPS with almost no syscalls and in a lock-free manner. It also opens the door to using io_uring_prep_send_zc for network zero-copy on the send path.
• Leverage DMA and direct access: Use O_DIRECT or memory-mapped files when appropriate:
  • For write-intensive logs, O_DIRECT with aligned buffers can eliminate double copying in kernel and give your application full control of caching. You’ll need a good memory pool strategy (as Redpanda does) but the payoff is consistent latency and lower CPU.
  • For read-heavy workloads (like analytics or serving static content), memory-map files to avoid copying data into user buffers. This also allows multiple readers (or threads) to access data without duplicating it. Just be mindful of access patterns to avoid page fault spikes.
• Pin and reuse buffers: Allocate buffers once and reuse them (this is a Seastar/Redpanda principle). Registering those buffers with the kernel (e.g., io_uring_register_buffers) or using hugepages can ensure physical continuity and avoid per-I/O overhead. In networking, consider MSG_ZEROCOPY for large sends from user-space and reuse those send buffers to avoid constantly allocating new memory that the kernel would have to copy from.
• Exploit zero-copy in inter-process communication: If your architecture has multiple processes or services on one machine (for example, a pipeline where one process produces data and another consumes it), consider using shared memory or memfd (memory file descriptors) to pass data rather than sockets with copies. Linux’s memfd_create combined with sendfile or sealing can send data between processes zero-copy. This can drastically cut latency in multi-process pipelines (as Cloudflare showed with their ML inference example, going from ~5 µs to ~0.5 µs by moving to shared memory).
• Profile and identify copy hot-spots: Use tooling (e.g., perf or systemtap) to see if your application is spending CPU time in copy_user_enhanced_fast_string or similar copy routines. If so, that’s a candidate to apply zero-copy techniques. Often, enabling a zero-copy path (like using sendfile, or increasing buffer sizes to reduce frequency of copy) can alleviate those hot spots.
• Be mindful of fallbacks: Zero-copy operations sometimes have corner cases (e.g., sendfile might fall back to a normal copy for certain files like /proc or if the kernel can’t safely DMA from a particular filesystem). Always handle errors or performance drops gracefully. For instance, Linux’s splice had a flag SPLICE_F_MOVE that became a no-op due to bugs. Ensure your system doesn’t hang if zero-copy isn’t available – it should seamlessly use normal I/O if needed (just with lower performance).
• Keep data in the format that avoids copies: This is more on the software design side – if you can avoid serializing/deserializing data unnecessarily, do so. E.g., Apache Arrow keeps data in a columnar format that different systems can use without translation. In a streaming context, if your consumer can use the same binary format as the producer writes to disk, the broker can just send bytes from disk straight to consumer without re-encoding. Kafka does this: the broker doesn’t decode messages; it just sends the byte payload. This is a form of application-level zero-copy (no semantic transformation). Design your protocols and storage similarly if low latency is critical.

In conclusion, zero-copy I/O and its modern implementations (like io_uring) are indispensable for building high-performance, low-latency systems. Distributed logs and streaming platforms today operate at such high throughput that shaving off copies and syscalls isn’t just an optimization – it’s often the only way to meet latency and throughput targets. By following the evolution from sendfile to io_uring, we see a clear direction: push more work to the kernel/hardware, and eliminate wasted movements of data. System builders should embrace these techniques to stay within ever tighter latency budgets while handling ever-growing data streams.