Every lesson in this course has been about how one piece of machinery behaves. This one is about the decision that puts the pieces together, which is a different skill and is graded differently. Nobody is ever asked whether they understand how a shuffle works in a stack decision meeting. They are asked which one to build, with incomplete information, in front of people who will remember the answer for years.

The decision is almost always presented as a single question with a product name as the answer. Spark or Trino. Snowflake or a lakehouse. Flink or a scheduled batch job. Framed that way it has no good answer, because the products are not substitutes for each other and because the real decision is six decisions that have been collapsed into one sentence.

The company this lesson is about

To keep this concrete, every number in this lesson comes from one real situation. A retail analytics company runs its reporting on a single large Postgres instance. It holds four point two terabytes and grows by about one hundred and eighty gigabytes a month. A nightly dump starts at two in the morning and takes three hours and forty minutes, which means the morning dashboards are sometimes not ready when the first analyst arrives.

There are forty one analysts, eleven data engineers and three machine learning engineers. Over the last thirty days the platform served one hundred and twenty eight thousand queries. Ninety two thousand of those were dashboard queries, thirty one thousand were ad hoc exploration, four thousand two hundred were scheduled transforms and eight hundred were feature reads from the machine learning side.

Dashboard queries have a median of one point four seconds and a ninety fifth percentile of nine point two seconds. The business has asked for a ninety fifth percentile under two seconds. Ad hoc queries have a median of eleven seconds and a ninety fifth percentile of two hundred and forty. The peak concurrency observed in thirty days was forty six queries at a quarter past nine in the morning.

Somebody has written a document proposing Spark. Somebody else has written one proposing a warehouse. Both documents are well argued and neither contains any of the numbers in the previous three paragraphs.

Six choices, not one

The thing being chosen has six parts, and conflating them is the single most common mistake in this kind of decision.

The first is the storage layer, meaning where the bytes physically live. Object storage, a distributed file system, the local disks of a database cluster, or the storage a managed warehouse owns on your behalf.

The second is the table format, meaning how a set of files is described as a table with a schema, snapshots and statistics. Iceberg, Delta, Hudi, plain directories of Parquet, or the proprietary internal format of a warehouse.

The third is the execution engine, meaning the thing that turns a query into work across machines. Spark, Trino, Flink, a warehouse engine, or a single node engine like DuckDB or ClickHouse.

The fourth is the serving layer, meaning what a dashboard actually connects to. This is often the same as the execution engine and often should not be.

The fifth is orchestration and transformation, meaning what runs the four thousand two hundred scheduled jobs and in what language they are expressed.

The sixth is the catalog and governance layer, meaning what knows which tables exist, who may read them and what they mean.

Four of the six are cheap to reverse

These six are not equally expensive to change later, and the difference is large enough to dominate the decision.

From this company's own history and from three peers who were willing to share numbers, the measured times to replace a layer already in production were as follows. Replacing the catalog took five weeks. Replacing the execution engine took six weeks. Replacing the serving layer took nine weeks. Replacing orchestration took thirteen weeks. Replacing the storage layer took thirty one weeks. Replacing the table format took thirty eight weeks.

The gap between six weeks and thirty eight weeks is the whole lesson. An engine choice is an experiment. A format choice is a commitment. Treating them with the same seriousness wastes months on the first and causes years of regret on the second.

Why the table format is the load bearing choice

The reason the format costs thirty eight weeks to change is not technical difficulty. It is that everything else has been written against it. Every transform reads it, every dashboard query plans against it, every retention job manipulates its snapshots, and every assumption about how a late arriving row gets corrected is encoded in how that format handles an update.

More importantly, the format determines how many engines you are still allowed to choose. A table stored as Iceberg on your own object storage can be read by Spark, Trino, Flink, DuckDB, ClickHouse and several warehouse engines, today, without copying the bytes. A table stored inside a warehouse's proprietary format can be read by that warehouse, and by anything else only after an export.

That is not an argument that open formats are always right. It is an argument that the format choice is where optionality is created or destroyed, and that optionality has a price you can estimate.

What an open format actually buys

It is worth being precise about what the open format buys, because the usual framing is vague and therefore unpersuasive to anybody who has to approve the extra work.

It buys the ability to add an engine without moving data. Concretely, this company later wanted interactive federated queries over data older than its working set. With Iceberg tables on their own bucket, adding Trino took five weeks and moved no bytes. Had the same tables been inside a warehouse, the same capability would have required either an export of the full history or paying the warehouse to do the work, and the estimate for the export path was four months.

It does not buy performance, it does not buy simplicity, and it usually costs both. An open stack has more moving parts, needs more maintenance, and in this company's own pilot was three to seven times slower than the warehouse on the same dashboard query until somebody spent two weeks on file sizing and clustering, which is exactly the work covered earlier in this course.

The argument for choosing the engine first

There is a serious counter argument and it deserves a fair hearing rather than a straw man.

Most companies do not survive long enough for the format decision to matter. A team of eleven engineers that spends five months building a lakehouse to preserve optionality it will exercise in year four has spent five months of its only scarce resource on an option it may never use. The warehouse that ships in six weeks produces value for three and a half months while the lakehouse is still being built, and three and a half months of working dashboards is worth a great deal to a company that is still proving the analytics function pays for itself.

The honest version of the trade is not open against closed. It is time to value against cost to exit, and both sides have a number.

Coupling is the thing to look for

When evaluating any proposal, the useful question is not whether each component is good. It is which components are welded to which others.

A proposal where the orchestrator only works with one transformation framework, which only emits one table format, which is only read well by one engine, is a single decision wearing six hats. It may still be the right decision, but it should be evaluated as one commitment with one exit cost, not as six independently reversible choices.

What to write down before anybody names a product

The output of this first stage is not a decision. It is a one page document containing the working set size, the query shape distribution, the concurrency peak, the freshness requirement expressed per dashboard, the decision horizon in years, and the exit cost of each candidate. Every one of those is a number that can be measured in a week.

A proposal that names a product before those six numbers exist is an opinion. A proposal that names a product after them is an argument, and arguments can be checked.