Every transactional guarantee a lakehouse offers is built on top of a storage system that was never designed to provide one. Object storage gives you a small, specific set of promises, and the table formats construct everything else from them. Knowing exactly where that boundary sits is what lets you reason about what can and cannot go wrong.

What you actually get

The promises are short enough to list.

A single object write is atomic. A reader either sees the whole previous object or the whole new one, never a half written mixture. This holds regardless of object size, because the service does not publish the new version until the upload is complete.

A write to a new key is immediately visible to a subsequent read. This was not always true, and a good deal of lakehouse design history is explained by the years when it was not.

A read of an existing key returns the latest version. Major services now provide this, which removed an entire category of bug that earlier designs had to defend against.

A list operation returns objects that exist, eventually and usually immediately, but ordering and completeness under concurrent writes are not something to rely on.

And on most services, a conditional write: put this object only if it does not already exist, or only if its current version matches what I last saw. This is the single primitive that makes everything else possible.

What you emphatically do not get

The absences matter more than the presences.

There is no multi object atomicity. Writing four objects is four independent operations, and any subset of them can succeed. Nothing in the storage layer can group them.

There is no rename. What looks like a rename in a tool is a copy followed by a delete, which costs the full bytes and is not atomic. Systems built on the assumption that rename is cheap and atomic behave badly here, and that assumption is exactly what the older commit protocols made.

There is no directory. Keys contain slashes and tools display them as a hierarchy, but the store holds a flat keyspace. Listing a prefix is a query over that keyspace, not a directory read, and its cost grows with the number of matching keys.

There is no locking. Nothing lets one writer exclude another for a period.

And there is no transaction across keys, no write ahead log, no recovery after a partial failure. A failed multi object write leaves exactly what it managed to write.

Why the old protocol was wrong

The output committer that batch engines inherited from distributed file systems worked like this. Each task writes into a temporary location. When a task completes, its output is renamed into a task directory. When the whole job completes, the job's output is renamed into the final location. On a file system with atomic directory rename, the final step publishes everything at once, and that is a genuine commit.

On object storage none of this holds. The renames are copies, so the data is written twice and the job pays for the bytes twice. The final rename is not atomic, so a reader during the commit sees some output files and not others. A failure partway through leaves the destination in a state that is neither the old content nor the new content. And the whole thing is slow in proportion to output size rather than to metadata size.

Teams discovered this as a correctness problem rather than a performance one. A dashboard reading a partition mid commit got a partial answer, with no error anywhere.

The primitive everything is built on

The way out is to stop trying to make the data files atomic and instead make one tiny object atomic.

Write all the data files first, wherever you like, under any names. They are referenced by nothing, so no reader can see them and no reader is confused by them. Then write a metadata object that lists them. Then publish that metadata object by a single atomic operation: either a conditional put of a pointer object, or an entry appended to a log at a numbered position that must not already exist, or a compare and swap in a catalog.

That single operation is the commit. Everything before it is preparation that costs money and no correctness. Everything after it is cleanup.

The three formats differ in which mechanism they use, and the differences are real but smaller than they look. A numbered log entry whose creation fails if the number is taken, a pointer object swapped conditionally, and a catalog row updated with compare and set are three spellings of the same primitive.

What the commit does not protect

It protects the transition. It does not protect the files.

Nothing stops a tool from deleting a data file that a live snapshot references. The format records which files belong to which version, but the storage layer does not enforce it, and anybody with write access to the bucket can remove one. This is why orphan cleanup windows matter so much and why bucket permissions are part of the correctness story rather than only a security concern.

Nothing makes the data files consistent with each other before the commit. If a writer stages four files and crashes, those four files are durable, complete, and referenced by nothing. They are not a partial transaction, they are garbage, and the only thing that distinguishes them from a live file is that no snapshot lists them.

And nothing bounds the time between the first staged file and the commit. A write can take hours, and during those hours its files look exactly like abandoned files from a crashed job.

Where the catalog enters

If the atomic operation is a conditional put on a pointer object, the storage service provides it and no catalog is strictly required. If it is a compare and set in a catalog, then the catalog is part of the transactional path, and its availability and its consistency become yours.

This is worth being precise about, because it determines what happens during a catalog outage. With a storage based pointer, reads and writes continue as long as the bucket is up. With a catalog based pointer, writes stop, and reads stop unless the engine can be told a snapshot location directly.

It also determines whether two different systems can write the same table safely. Two writers using the same catalog contend correctly on the same compare and set. Two writers where one uses the catalog and one goes directly to storage do not contend at all, and both will believe they committed.

What to verify on a new bucket

Three checks, each cheap, and each has caught a real problem.

Confirm that conditional writes behave as documented on your specific service and configuration, because this is the primitive the whole stack depends on and some compatible implementations do not provide it.

Confirm that there is exactly one write path to the table, and that no job, tool, or person writes data files into it outside the format. A single direct writer silently breaks every guarantee in this lesson.

And confirm what your lifecycle rules do. A rule that transitions objects to a colder tier or expires them by age knows nothing about snapshots, and an expiry rule on a bucket holding a table with long retention is a slow moving data loss event.

All three of these are things you can establish in an afternoon on a new bucket and cannot easily establish after a year of production traffic, which is the argument for doing them at the start. The failures they prevent are not subtle once they happen. They are simply very hard to attribute, because the symptom appears a long way from the configuration that caused it.