Every distributed data system is an argument about the same three costs: reading bytes, moving bytes between machines, and keeping state while you do it. Spark, Trino, Flink, Iceberg, Snowflake and ClickHouse are different answers to that argument, and once you can see the argument, the systems stop being a list of tools to memorise and become a small number of design decisions you can reason about. This course teaches the decisions, then the systems that implement them, with the arithmetic written out at every step.

It starts with scale out economics, which is the question of when one large machine stops being the right answer, and with the storage layer underneath everything else: what an object store actually guarantees, why listing a prefix is slow, why there is no rename, and what that does to every engine built on top. Then columnar formats in detail, because the single largest performance lever in this field is not reading data you do not need. Parquet row groups, column chunks, pages, dictionary and run length encoding, statistics and the predicate pushdown they enable, with the arithmetic of how much a scan actually skips.

The execution section builds the engine. Why MapReduce gave way to DAG engines, how a logical plan becomes stages and tasks, and then the shuffle in full: the map side write, the sort and spill, the fetch, and why the shuffle dominates the cost of almost every nontrivial job. Partitioning and skew follow, with salting and adaptive execution as the two real remedies and the arithmetic that shows when each one helps.

Spark gets four lessons of its own. The driver and executor architecture with the memory model and its regions, Catalyst and Tungsten with whole stage code generation, the join strategies with the size thresholds that select between them, and a tuning lesson that reads a real slow job from the stage view down to the task distribution and fixes it.

Table formats are the modern centre of the field. Iceberg, Delta and Hudi compared by their metadata designs rather than their marketing, ACID on object storage explained through the commit protocols that make it work, and compaction, clustering and file sizing as the maintenance work that decides whether a table stays fast. Then the query engines: Trino and federated query with its pushdown limits, the warehouse engines Snowflake and BigQuery with their separation of storage and compute, and the OLAP stores ClickHouse and Druid with the merge tree and segment designs that make sub second aggregation possible.

Streaming is treated as a distinct discipline rather than as fast batch. Flink's architecture and state backends, event time with watermarks and windows and the late data problem, and exactly once as a property produced by checkpoints plus transactional sinks rather than as a configuration flag. The last section is production: orchestration and DAG design, incremental models and transformation tools, data quality gates and backfills, catalogs and metadata services, cost control and resource isolation, and a closing lesson on choosing a stack from requirements instead of from fashion.

There are fourteen judged coding problems, because much of this subject can be implemented in miniature: a partitioner, a skew detector, a shuffle merge, a watermark generator, a manifest pruner and a file compaction planner among them. By the end you will be able to read a Spark stage view and say what the job is doing physically, predict how much of a table a query will skip before you run it, choose a table format from how the data arrives, and tell the difference between a job that is slow and a job that is badly shaped.