When you write a DataFrame expression, nothing runs. You are building a description, and the description then goes through four distinct transformations before any executor sees a task. Knowing those four phases, and knowing which one is responsible for which kind of surprise, is what turns a query plan from a wall of text into a diagnosis.

Nothing you wrote is what executes

A DataFrame operation returns another DataFrame. No data is read, no filter is evaluated, and no join is performed. What you have built is a tree of operations, and that tree is the only thing the engine knows about your intent.

This matters more than it sounds. Because the engine holds a description rather than a sequence of executed steps, it is free to rearrange them. It can move a filter you wrote last to the very beginning. It can drop a column you selected and then never used. It can replace your join algorithm with a different one. It can decide that two of your operations can be fused into a single loop. None of this is possible for code that has already run.

It is also the reason the order you write operations in has much less effect than people expect, and the reason the few cases where order does matter are so confusing. Writing a filter before a projection or after it produces the same plan. Writing a filter before a shuffle or after it also usually produces the same plan, because the optimiser moves it. But writing a filter after a user defined function that the optimiser cannot see through produces a genuinely different plan, because the barrier is real.

Phase one, analysis

The tree you built is unresolved. When you write a column by name, the engine does not yet know whether that column exists, what type it is, or which of two joined tables it came from. Analysis resolves every one of those references against the catalog.

This phase produces the errors you see immediately, before any cluster work happens at all. A column name that does not exist, a column name that is ambiguous because both sides of a join have it, a function called with the wrong number of arguments, a comparison between two types with no sensible coercion. All of these are analysis failures, and all of them arrive in under a second because nothing has been scheduled.

Analysis also inserts things you did not write. Implicit casts appear here, which is why a comparison between a string column and an integer literal silently becomes a cast rather than an error. Aliases are generated for unnamed expressions. Nested field accesses are expanded. By the end of analysis every expression has a known type and every reference points at exactly one source.

Phase two, logical optimisation

Now the engine has a valid tree and it starts rewriting it. This phase is a loop: a set of rules is applied repeatedly until the tree stops changing or an iteration limit is reached.

The rules are individually unremarkable. Push a filter below a projection. Push a filter below a join when it only references one side. Combine two adjacent filters into one. Fold an expression made entirely of literals into its result. Remove a projection that selects everything. Replace a join whose condition is always false with an empty relation. Collapse two adjacent projections. Prune columns nothing downstream reads.

What makes the phase powerful is the iteration. Pruning a column may make a projection redundant, which may expose a filter to a pushdown that was previously blocked, which may make a join condition constant, which may allow a whole branch to be removed. One pass would catch almost none of this. The loop catches most of it.

Everything in this phase is a rewrite of the logical tree into another logical tree. There is still no decision about how anything will be executed. A join in the logical plan is just a join, with no algorithm attached.

Phase three, physical planning

This is where the plan acquires a strategy. For each logical operator the planner generates one or more physical candidates, and then chooses between them using whatever statistics it has.

A logical join becomes a broadcast hash join, a shuffled hash join, a sort merge join or a broadcast nested loop join. A logical aggregate becomes a hash aggregate or a sort aggregate. A logical scan becomes a file scan with a specific set of pushed filters and a specific set of projected columns.

Physical planning also inserts operators you never asked for and which dominate the runtime. Exchanges appear wherever the data needs to be redistributed, and an exchange is a shuffle. Sorts appear wherever a chosen algorithm requires ordered input. These inserted operators are usually where the time goes, and they are visible in the physical plan as named nodes, which is why reading the physical plan is the single most useful planning skill.

Phase four, code generation

The physical plan is a tree of operators, and a naive execution of it would call a method per operator per row. Instead, runs of adjacent operators are fused and compiled into a single generated Java method that loops over rows once.

This is the Tungsten part of the system and it changes the shape of the work rather than the amount. One generated loop with the filter, the projection and the aggregation inlined into it avoids a virtual call per operator per row, avoids materialising intermediate rows, and keeps values in registers rather than reading them from objects. For simple operations over many rows the difference is several fold.

The generated code is real, compiled Java, and the plan tells you where it was produced: the operators that were fused appear inside a node marked with a stage identifier and an asterisk. Operators that could not be fused sit outside it. Learning to see where fusion stopped is how you find the operator that fell back to the slow path.

Where each phase fails, and how to tell

The four phases have four distinct signatures and matching them is most of the diagnostic work.

An analysis failure arrives instantly and names a column or a function. There is nothing to tune and nothing to measure.

A logical optimisation failure is silent. The plan is correct and it is slower than it should be, because a rewrite you expected did not happen. A filter sits above a join rather than below it. A column is read that nothing uses. You find these by reading the optimised logical plan and comparing it to the plan you expected.

A physical planning failure is a wrong strategy chosen with confidence. A sort merge join where a broadcast would have been free, because the size estimate for one side was wrong. A sort inserted because the planner did not know the input was already ordered. You find these by comparing the chosen operator to the real data sizes.

A code generation failure is a fused loop that is not fused. An operator in the middle of a chain that cannot be compiled splits the chain in two and the rows have to be materialised between them. You find it by looking for where the stage markers stop.

Reading the four plans

The explain command can print all four forms of the plan, and the habit worth building is to read them in pairs rather than individually.

The parsed plan and the analysed plan differ only by resolution, so the useful thing to look for between them is a cast you did not write. A string column compared to a number, a decimal widened to match another decimal, a timestamp coerced from a string literal. Each of those is a correctness question wearing the costume of a performance question, because a cast on a column is usually also the thing that stopped a predicate from reaching storage.

The analysed plan and the optimised logical plan differ by every rewrite the engine performed, and that difference is the most informative thing in the whole output. If the optimised plan still has your filter sitting above a join, the optimiser could not move it and there will be a reason. If the optimised plan still reads a column nothing uses, pruning was blocked. Reading this difference is how you find out what the optimiser thought of your query.

The optimised logical plan and the physical plan differ by strategy and by inserted operators. Every exchange and every sort in the physical plan was added here, and neither appears in the logical plan at all. Counting the exchanges is a reasonable first estimate of how expensive a query is going to be, because each one is a full redistribution of the data.

What this lesson covers

The rest of the lesson goes through the parts of this that you can actually influence. Which rewrites the optimiser performs and which it provably cannot. What makes a predicate reach the storage layer and what stops it. What whole stage code generation gives you and when it is abandoned. And finally adaptive execution, which is the engine admitting that the statistics it planned with were wrong and replanning in the middle of the job.