Every query begins with an act of name resolution that almost nobody thinks about, and the component that performs it is the most load bearing thing in a data platform. Before a single byte of Parquet is read, something has to turn the string analytics.marts.fct_orders into a list of object storage keys. That something is the catalog, and when it is slow the whole platform is slow, when it is unavailable the whole platform is unavailable, and when two of them disagree the platform is quietly wrong.

Engineers tend to meet the catalog twice. The first time is as an annoyance, a service that has to be running before Spark will start. The second time is as an incident, usually several years later, when a table that two systems were writing to turns out to have lost a commit. Between those two meetings sits the entire subject of this lesson.

The three part name and what each part costs

A fully qualified table name on a modern platform has three parts. The catalog, the namespace or schema, and the table. On this platform a query reads from analytics.marts.fct_orders, where analytics is the catalog, marts is the namespace and fct_orders is the table.

Those three parts are not symmetric. The table part is the only one that points at data. The namespace is a grouping with no physical meaning whatsoever, which is why renaming a namespace is cheap and renaming a table is not. The catalog part is the one that decides which service answers the question, and it is therefore the part that governs consistency, because two engines reading through two different catalogs have no mechanism by which to agree.

That last point is worth stating plainly, because it is the root of most catalog incidents. A table is not a thing in object storage. A table is an assertion made by a catalog about which files in object storage currently constitute it. Two catalogs making that assertion about the same prefix is not a redundancy. It is a race.

A table is a pointer, not a directory

The older mental model, inherited from Hive, is that a table is a directory and its partitions are subdirectories. Under that model the catalog is a convenience. You could in principle list the prefix yourself and get the same answer.

Modern table formats broke that equivalence deliberately. Under Iceberg, Delta or Hudi, the files that belong to a table are the files named by the current metadata, and files can sit in the prefix without belonging to the table at all. An orphaned file from a failed write, a compacted file that has been superseded, a file written by a job that was rolled back. Listing the prefix would find all of them, and reading them would give a wrong answer.

This is what makes the catalog unavoidable rather than merely useful. The catalog holds a single pointer, usually to one metadata file, and that pointer is the definition of the table. Everything else is derivable.

What the resolution actually touches

Trace a single query on this platform and the resolution goes through four hops before any data is read.

The engine asks the catalog for the current metadata location for analytics.marts.fct_orders. That is one round trip and it returns a pointer in about forty milliseconds. The engine reads that metadata file from object storage, which carries the schema, the partition specification, the table properties and a list of snapshots, and that read takes about eighty milliseconds. From the current snapshot the engine reads a manifest list, roughly thirty milliseconds, which names the manifest files relevant to this snapshot. Finally it reads the manifests themselves, about two hundred milliseconds for a one day predicate, and those manifests carry per file statistics that let the engine discard most of them without opening them.

Three hundred and fifty milliseconds in total, four hops, and at the end of it the engine has a list of data files and a reasonably tight bound on what each one contains.

Three hundred and fifty milliseconds against six minutes

Put that number next to the query it precedes. A one day scan of fct_orders on this platform reads four point one million rows and takes about six minutes. Planning is three hundred and fifty milliseconds of that, which is one tenth of one percent.

That ratio is the reason catalog performance is invisible until it is not. Planning cost does not scale with rows read. It scales with the number of files and manifests the engine has to consider, which is a function of how the table is maintained rather than how much data it holds. A thirty day scan plans in about seven hundred and eighty milliseconds, a ninety day scan in about one point four five seconds, and a full table scan in close to four seconds, because each widening of the predicate pulls in more manifests while reading no more rows during planning than a one day scan does.

Four seconds is still nothing against a three hour full refresh. But a badly maintained table with hundreds of thousands of small files can push planning into the tens of seconds, and an interactive dashboard that issues forty such queries has spent its entire latency budget before touching data. Planning time is where file sizing, compaction and partitioning quietly present their bill.

Technical catalogs and discovery catalogs are different products

The word catalog is used for two unrelated systems and conflating them wastes a great deal of time in design discussions.

A technical catalog is on the read path. The Hive Metastore, the Iceberg REST catalog, AWS Glue Data Catalog, Unity Catalog, Nessie and Polaris are technical catalogs. If one is down, queries fail. It answers the question of which files are in this table right now, and it is the component that accepts or rejects a commit.

A discovery catalog is not on the read path. DataHub, Amundsen, OpenMetadata, Atlan and Collibra are discovery catalogs. If one is down, nobody can find out who owns a table, but every query still runs. It answers questions about meaning, ownership, freshness and lineage, and it is typically populated by crawling the technical catalog and the query logs.

The two are frequently sold together and occasionally share a user interface, which is where the confusion comes from. Architecturally they have opposite requirements. A technical catalog needs single digit millisecond reads, strict consistency and an availability target matching the platform itself. A discovery catalog needs full text search, a graph model and tolerance for data that is a few hours stale.

One thousand one hundred and forty tables, eighty six of them public

This platform registers one thousand one hundred and forty tables in its technical catalog. Four hundred and twelve of those are produced by transformation models, and of those, eighty six are published for other teams to read. The remaining seven hundred and twenty eight are raw landing tables, staging tables and the intermediate output of multi step models.

The shape matters because it tells you what the catalog is mostly doing. The overwhelming majority of catalog traffic concerns tables that no human has ever looked at. Optimising the catalog for human browsing is optimising for about seven percent of the entries and essentially zero percent of the requests.

Why the catalog is the hardest thing to replace

Storage can be migrated file by file. Compute can be swapped by pointing a new engine at the same tables. The catalog cannot be done incrementally in the same way, because during the migration there are two services that each believe they define the same tables, and a commit that goes to one is invisible to the other.

This is why catalog migrations are planned as cutovers with a freeze rather than as gradual rollouts, and why the pointer swap mechanics covered later in this lesson are worth understanding before you need them.

The failure mode nobody rehearses

Ask a platform team what happens if the catalog is unavailable for ten minutes and you will usually get a confident answer about retries. Ask what happens if the catalog is available but returns a stale pointer and the answers get much less confident.

Staleness is the harder failure because nothing errors. Queries succeed, return rows and look exactly like correct results. On this platform a caching layer in front of the metastore once held table metadata for five minutes, which meant a dashboard could read a snapshot that had already been superseded by a corrected one. Nobody noticed for two weeks, and when somebody did, the evidence was a screenshot rather than a log line. A catalog that fails loudly is a manageable dependency. A catalog that lies quietly is not.