Most tuning advice is a list of settings. Settings are the last thing you should touch, and they are the first thing almost everyone reaches for, which is why so much tuning work produces no measurable change. A job that takes forty minutes takes forty minutes for a reason, and that reason is visible before you change anything. This section is about reading it.

Four levels, and only one of them burns time

An application contains jobs, a job contains stages, a stage contains tasks. Those four words are not a hierarchy of importance, they are a hierarchy of what each one tells you.

An application is a process with executors attached to it. Its duration includes waiting for resources, reading configuration, and sitting idle between queries, none of which is computation.

A job is one action. Writing a table, collecting a result, counting rows. A job is what a user waited for, so it is the unit a complaint arrives in, but it is almost never the unit a fix applies to.

A stage is a run of work with no shuffle inside it. Every stage boundary is a redistribution of data across the network, and every stage must finish entirely before the next one can begin. That last clause is the one that matters: stages do not overlap, so a job's duration is the sum of its stage durations plus the gaps between them.

A task is one partition of one stage processed by one core. Tasks are where every second of CPU time is actually spent, and they are the only level at which you can see that one piece of work behaved differently from its siblings.

Tuning happens at the task level and is verified at the job level. Reading in the other direction, from job time down, is what makes tuning feel like guesswork.

Wall clock and task time are different measurements

A stage reports two numbers that people routinely confuse. The first is its duration, the wall clock from the first task starting to the last task finishing. The second is the sum of all its task durations, which on a cluster with two hundred cores can be two hundred times larger.

Both numbers are useful and they answer different questions. Total task time is what the job costs. If you are paying for compute, that is the bill, and halving it halves the invoice whether or not the job finishes sooner. Stage duration is what the user experiences, and it only improves when the critical path gets shorter.

The ratio between them is the effective parallelism. A stage with four hundred core hours of task time that ran for two hours used two hundred cores perfectly. The same stage running for six hours used sixty seven cores on average, and the four hours of difference went somewhere that the task list will show you.

This is the first calculation to do on any slow job, because it immediately splits the problem in two. Low total task time with long wall clock is a parallelism or scheduling problem. High total task time is a work problem, and no amount of adding executors will fix it.

The average task duration tells you almost nothing

A stage finishes when its last task finishes. Not when the average task finishes, not when the ninetieth percentile finishes. The maximum is the only task duration that sets the stage duration, and the average is a number that hides exactly the case you are looking for.

Consider two stages that both report a forty second average across two thousand tasks. In the first, every task took between thirty and fifty seconds, and the stage ran for about a minute. In the second, nineteen hundred and ninety tasks took two seconds and ten took two hours, and the stage ran for two hours. The averages are identical and the jobs have nothing in common.

Read the maximum, the median, and the count. Three numbers, and together they describe the distribution well enough to tell you whether you have a skew problem before you open anything else.

Every stage boundary is a shuffle you paid for

Count the stages in a job and you have counted the shuffles, because a stage boundary is created by nothing else. Each one serialises records, writes them to local disk, exposes them over the network, fetches them, and deserialises them again.

That is a great deal of machinery to invoke, and a query that does it six times is paying six times. Some of those boundaries are unavoidable, because a join on a key and a grouping by a different key genuinely require data to move. Others exist because a query was written in a way that discards an ordering or a partitioning the engine had already established.

When a job has more stages than its logic requires, no setting will help. The fix is in the query, and recognising that early saves the afternoon that would otherwise go into memory configuration.

Four shapes, and what each one means

The task timeline inside a stage has a shape, and experienced readers classify it before reading any number at all.

A dense block that ends abruptly is a healthy stage. Every core was busy, every task was similar, and the stage ended when the work ended. There is nothing to tune here except the total amount of work.

A dense block with a long thin tail is skew. Most tasks finished together and a small number ran on long after, holding the stage open while the cluster sat idle. The tail is usually one to a few tasks, and it is almost always a data distribution problem rather than a resource problem.

A staircase means too few tasks for the available cores, run in waves. If a stage has three hundred tasks and the cluster has two hundred cores, the stage runs two waves, and the second wave uses a hundred cores while a hundred idle. Partition count is the lever.

A sparse scatter, with gaps between tasks, means the executors were not ready. Dynamic allocation still ramping, containers still being granted, or executors dying and being replaced. This looks like a slow job and is actually a cluster availability problem, and the task durations themselves may be perfectly fine.

The gaps between stages are nobody's computation

Lay the stage durations end to end and compare the total against the job duration. The difference is time the job spent doing nothing at all, and on a busy cluster it is frequently a quarter of the total.

Some of that gap is the driver. Between stages it is planning the next one, broadcasting variables, and in some cases listing files in object storage, which on a table with a hundred thousand partitions is a genuinely slow operation performed by a single thread. A driver listing files is invisible in every task metric and can be minutes long.

Some of it is scheduling. The next stage cannot start until its tasks are assigned, and under dynamic allocation that may mean waiting for executors that were released during the previous stage because they went briefly idle.

And some of it is the commit protocol at the end of a write, which renames or copies output files into their final location. On a file system this is instant. On object storage it is a per file operation, and a job writing fifty thousand small files can spend longer committing them than it spent producing them.

None of these show up as task time, so a job whose task metrics look entirely healthy can still be slow. Checking the gap first takes one subtraction and rules out a whole category of cause.

Write down six numbers before you change anything

Tuning without a baseline is indistinguishable from superstition, because you cannot tell a real improvement from a quieter cluster. Six numbers are enough, and all six are on the job page.

Wall clock duration of the job. Total task time summed across stages. Shuffle bytes written. Spill bytes to disk. The maximum task duration in the slowest stage. The median task duration in that same stage.

Record them before the first change and after every change. Most of what you try will move none of them, and knowing that quickly is the entire difference between tuning that converges and tuning that wanders.

There is a discipline here that takes about ninety seconds and saves entire afternoons. The six numbers also make a tuning change explainable to somebody else, which matters because the person who inherits the job will otherwise assume every setting in it was chosen for a reason.