Every distributed processing engine sits on top of something that holds bytes, and the properties of that something leak upward into every decision above it. The engine does not see disks. It sees an interface with a particular set of operations, a particular cost for each one, and a particular set of guarantees about what happens when two callers act at the same time. That interface is the storage layer, and whether it is a distributed filesystem or an object store changes how you lay out data, how you commit results, how you handle failure, and what your bill looks like.

The reason this is worth a lesson of its own is that most of the hard problems in modern data platforms are storage layer problems wearing a processing costume. A job that takes forty minutes instead of four is usually not computing slowly. It is listing a directory with two million entries, or opening eight hundred thousand small files, or renaming a result set one object at a time because the committer it inherited assumes renames are free. None of those are visible in the query, and all of them are predictable from the storage contract.

The three things an engine asks of storage

Strip away the vendor differences and a processing engine wants three capabilities. It wants to discover what exists, which means listing a prefix or a directory and getting back names and sizes. It wants to read a byte range from a named object without reading the whole thing, because columnar formats depend on seeking to a footer and then to specific column chunks. And it wants to make a set of writes visible to readers as one unit, so a half written result is never mistaken for a finished one.

Every storage layer provides all three, and the differences between them are entirely in the cost and the guarantees. Listing is a single metadata lookup in one system and a paginated series of HTTP requests in another. A byte range read is a local disk seek in one and a new request with fresh latency in the other. Making writes visible atomically is a directory rename in one, which is a metadata operation, and in the other it is either a copy of every byte or a separate transaction log that the engine has to maintain itself.

HDFS, a filesystem with a namenode

The Hadoop Distributed File System was designed for a world where the machines holding the data were the machines running the computation. It splits each file into large blocks, typically a hundred and twenty eight or two hundred and fifty six megabytes, and stores each block on several datanodes, by default three. A single namenode holds the entire directory tree and the block locations in memory, which makes metadata operations fast and also makes the namenode the limit on how many files the cluster can hold.

Because the namenode keeps a real filesystem namespace, it offers something object stores do not. Directories exist as objects in their own right. A rename is a change to the tree, not a movement of bytes, so it is atomic and effectively instantaneous regardless of how much data sits underneath. File appends are supported. Permissions follow a POSIX-like model. A reader that opens a file and holds it gets a consistent view of that file's contents.

The cost is operational. The namenode holds roughly a hundred and fifty bytes of memory per file, directory and block, so a cluster with a hundred million small files needs tens of gigabytes of namenode heap and will spend minutes in garbage collection pauses during which the whole cluster appears frozen. Storing three replicas means paying for three copies of every byte. And the datanodes are the same machines as the compute, so storage and compute scale together whether or not you want them to.

Object stores, a key value API over HTTP

An object store presents something much simpler and much less convenient. There is a flat namespace of keys within a bucket, and the operations are put an object, get an object or a byte range of it, delete an object, and list keys matching a prefix. There are no directories. The slashes in a key are a convention that listing honours by letting you ask for a prefix and a delimiter, which is why a listing looks like a directory listing even though nothing in the system models a directory.

There is no rename. Changing an object's key means copying the bytes to the new key and deleting the old one, and for a multi gigabyte object that is a real operation with real cost and real duration. There is no append in the general case. Each object is written whole, or assembled from parts through a multipart upload that becomes visible only when it is completed.

In exchange you get properties that are hard to build yourself. Durability is extremely high, because the service replicates and checksums across failure domains you never see. Capacity is effectively unbounded and you pay only for what you store. Availability is a service level agreement rather than a cluster you operate. And storage is decoupled from compute, which is the property that reshaped the industry.

Why the difference is not cosmetic

It is tempting to treat the object store as a filesystem with a different driver, and the Hadoop ecosystem encouraged that by shipping connectors that present an object store behind the same file system interface. The abstraction holds for reading and breaks for writing, because the interface promises operations that the underlying service cannot perform cheaply or atomically.

The clearest example is the output committer. The original design wrote each task's output to a temporary directory, and on success renamed that directory into place. On HDFS that rename is one metadata operation and the whole job becomes visible at once. On an object store the same logic becomes a copy of every output byte followed by a delete, performed object by object, with no atomicity anywhere in the sequence. A job that produced four hundred gigabytes in six minutes could then spend twenty minutes committing it, and a failure halfway through left a directory that was neither the old result nor the new one.

Durability is not availability, and neither is consistency

Three properties get conflated and they fail independently. Durability is the probability that a byte you wrote is still there later, and object stores are extraordinarily good at it. Availability is whether you can reach the service right now, and it is a smaller number, with regional outages that no amount of replication inside the service protects you from. Consistency is whether a read that follows a write sees the write, and whether a listing reflects every object that exists.

The major object stores now provide strong read after write consistency for individual objects, which removed a genuine class of bug that platforms spent years working around. Listing consistency is the subtler one, and the practical lesson is that a job which discovers its own input by listing a prefix is depending on a view of the world that is assembled from many requests. That is why table formats keep an explicit manifest of files instead of trusting a listing, a theme that runs through several later lessons.

What separating storage from compute actually changed

When the data lived on the compute nodes, the scheduler tried to run each task on a machine that already held its block, and reading was a local disk read. That data locality was the central performance idea of the Hadoop era. Separating the layers gave it up, and the trade was worth it for two reasons that were not obvious at the time.

The first is that network bandwidth inside a datacentre grew faster than disk bandwidth, so reading over the wire stopped being the penalty it had been. The second is economic. Clusters sized for a three hour nightly window were idle for twenty one hours a day, and every one of those hours was paid for because the storage lived on them and could not be switched off. Decoupling let the storage persist at a few cents per gigabyte per month while the compute existed only when a job was running, and that single change is responsible for most of the cost structure of modern platforms.