For about a decade, a table on a data lake was a directory. You agreed on a path, you agreed on a file format, and you agreed that everything under that path was the table. Every engine that wanted to read it listed the directory, parsed what it found, and hoped nobody was writing at the same time. That arrangement produced an enormous amount of value and it also produced an enormous amount of avoidable pain, and the table formats in this lesson exist to end it.

A directory is not a table

Start with what the directory convention cannot tell you.

It cannot tell you which files belong to the table right now. If a writer is halfway through producing two hundred files, a reader listing the directory sees some of them. There is no version, so there is nothing to read consistently.

It cannot tell you what the schema is, except by opening files and hoping they agree. Two files written six months apart may disagree about a column's type, and the only place that disagreement surfaces is in a reader, at query time, as an error nobody can attribute.

It cannot tell you what the table looked like yesterday. Yesterday's files may still be present, mixed in with today's, or they may be gone. There is no way to ask.

It cannot tell you anything about the contents without reading them. To find the rows for one customer you list every file, open every footer, and decide. On a table with four hundred thousand files, that listing is the query.

And it cannot be modified safely by two writers at once, because there is nothing to contend over. Two jobs writing to the same partition path both succeed, and the table now contains both results interleaved, with no record that this happened.

The list of files is the whole idea

Every format in this lesson solves all of that with one move: the table is no longer the directory, it is an explicit, versioned list of the files that belong to it.

That sentence is small and the consequences are not. Once a list exists, the files on storage are just bytes that may or may not be referenced. Adding data means writing new files and then publishing a new list that includes them. Removing data means publishing a new list that omits them. The old files can stay exactly where they are, because nothing that reads the new list will ever look at them.

Publishing the new list is a single operation. If it succeeds, every reader sees the whole change. If it fails, every reader sees the previous state. There is no intermediate state in which half a change is visible, because the only thing a reader consults is the list, and the list is replaced atomically.

This is the entire mechanism. Snapshots, time travel, concurrent writers, schema evolution, and partition pruning without listing are all consequences of having a list and versioning it.

Four layers, and what each one holds

All three formats arrange themselves into the same four layers, with different names and different trade-offs at each level.

At the bottom are the data files. These are ordinary Parquet, ORC, or Avro files. Nothing about them is special, and that is deliberate. Any engine that can read Parquet can read the data of an Iceberg or Delta table; what it cannot do without format support is know which files to read.

Above them is a manifest layer, which lists data files along with statistics about each one. Row counts, null counts, and the minimum and maximum value of each column. This is the layer that lets an engine skip a file without opening it.

Above that is a snapshot layer, which names the set of manifests that make up one version of the table. A snapshot is a complete, immutable description of the table at an instant.

At the top is a pointer, which says which snapshot is current. This is the only mutable thing in the entire structure, and swapping it is what a commit is.

The immutability below the pointer is what makes the whole arrangement work on object storage, which offers no rename and no lock but does offer a durable write of a new object. Write everything new, then move one pointer.

What the engine does differently once a list exists

The read path changes shape completely.

Without a format, planning a query means listing directories recursively until you have the file set, which is a sequence of requests against storage proportional to the number of directories, executed on one driver thread. On a table partitioned by day and hour for three years, that is twenty six thousand directory listings before any data is read.

With a format, planning means reading the current pointer, reading one snapshot object, reading the manifests it names, and filtering those manifest entries by the query's predicates. The statistics in the manifests mean most files are eliminated without being touched. A query for one customer against a table clustered on customer identifier can go from four hundred thousand candidate files to nine, with the elimination happening in metadata.

The second change is that the file set is fixed at the start of the query. Every task in every stage of a long job reads the same snapshot, even if a dozen writers commit while it runs. Long reads stop failing because somebody replaced a file underneath them, which on a directory convention is a routine and unexplainable occurrence.

The operations a directory cannot express

With a versioned list, several operations become expressible that previously required rewriting a partition and crossing your fingers.

Deleting rows that match a predicate. Find the files that could contain them, rewrite those files without the matching rows, and commit a snapshot that swaps the old files for the new. Readers see either all of the deletion or none of it.

Updating rows. Mechanically the same as a delete followed by an insert, committed together.

Changing the schema. Add a column, rename a column, widen a type, reorder. Because columns are tracked by identifier rather than by position or name, a rename is a metadata operation that touches no data file at all.

Changing the partitioning. New data is laid out the new way, old data keeps its old layout, and the metadata records which files follow which scheme. Queries read both correctly.

Reading the table as of a past time or version. The snapshot is still there, with its manifests, with its data files, until something explicitly expires it.

What this costs

None of this is free, and the costs land in places people do not expect.

Metadata grows. Every commit writes new metadata objects, and a table committing every minute writes one thousand four hundred and forty snapshots a day. Without expiry, planning slowly becomes the dominant cost of every query against the table, which is the same disease the format was meant to cure, arriving from the other direction.

Small commits produce small files. A streaming writer committing every thirty seconds writes files sized for thirty seconds of data. The table is correct and queries against it are slow, and the fix is compaction, which is its own ongoing operational job.

Deleted rows do not necessarily disappear from storage. Depending on the strategy, a delete may write a small file saying which rows to ignore rather than rewriting the data, and the engine reconciles at read time. That read time cost accumulates until something merges it away.

And every engine that touches the table must understand the format. A table format is a contract, and a tool that does not implement it sees either nothing or, worse, the raw directory, including files that are no longer part of any snapshot.

What to check first on a table you inherit

Four questions, answerable in minutes, tell you most of what you need to know about a table in one of these formats.

How many snapshots does it have, and when was the oldest one created. A table with forty thousand snapshots has no expiry policy, and that is a planning cost today and a storage cost already paid.

How many data files does it have, and what is their median size. Hundreds of thousands of files averaging under a megabyte means no compaction is running.

How much metadata does it have relative to its data. Metadata that is a meaningful fraction of the table is a table that will be slow to plan regardless of how fast the engine is.

And how many delete files are pending. A table where half the read cost is reconciling deletes is a table waiting for a maintenance job that nobody scheduled.