An orchestrator is the piece of data infrastructure most likely to be described as a scheduler, and that description is the source of almost every bad decision made with one.

A scheduler runs things at a time, an orchestrator runs things in an order

Cron runs a command at a moment. It has no opinion about whether the data that command needs has arrived, no memory of whether yesterday's run succeeded, and no way to express that one command must finish before another starts. If you have ever written a shell script that sleeps for twenty minutes before calling the next step, you have hand rolled the thing an orchestrator exists to replace, and you have hand rolled it badly, because the twenty minutes is a guess about how long the upstream step takes and the guess is wrong on the day it matters.

An orchestrator's primary job is to hold a graph of dependencies and to run each node only when the nodes it depends on have succeeded. The schedule is a secondary concern. On the platform described throughout this lesson there are three hundred and forty scheduled graphs running about four thousand one hundred task instances a day across eighteen teams, and the overwhelming majority of production incidents in that estate trace back to a graph that encoded a schedule where it should have encoded a dependency.

The three jobs it has to do

The first job is to express the dependency graph honestly. Every edge in the graph is a claim that the downstream task reads something the upstream task produces. If the edge exists but the claim is false, the graph is slower than it needs to be. If the claim is true but the edge is missing, the graph is correct only by accident, and the accident ends the first time the upstream task gets slower.

The second job is to make a run recoverable. A run of eighty seven tasks will fail partway through, repeatedly, for years. The orchestrator has to know which tasks succeeded, make the failed one retryable without redoing the successful ones, and give a human a way to resume from the failure rather than from the start. This is the job that justifies the whole category, and it is the job that an orchestrator cannot do for you if your tasks are not safe to re run.

The third job is to make state observable. At any moment somebody needs to answer which runs are in flight, which tasks are failing, how long the critical path took today compared with last week, and whether the output a downstream team is waiting on exists yet. A graph with no answer to those questions is operated by guesswork, whatever its code looks like.

The four words you need before anything else

A graph is the definition: the set of tasks and the edges between them, living in version control and parsed into memory by the scheduler. A run is one execution of that definition for one logical slice of time. A task instance is one task inside one run, and it is the unit that succeeds, fails, retries and gets looked at by a human. A data interval is the slice of time a run is responsible for, which is almost never the same thing as the moment the run starts.

That last distinction causes more confusion than the rest of the vocabulary combined. A run scheduled at two in the morning is usually processing yesterday, because yesterday is the window that has finished arriving. So the run labelled with yesterday's date starts today, and every query inside it filters on the data interval rather than on the current clock. The moment any task in a graph calls the equivalent of now, the run stops being reproducible: re running it tomorrow produces a different answer, which destroys the backfill story this lesson closes with. Every mature orchestrator therefore hands the data interval to the task as a parameter, and every mature graph uses it rather than the system clock.

What it is not for

It is not a compute engine. A task that spends four hours transforming data inside the orchestrator's worker process has put the heavy work in the one place with the least isolation, the least elasticity and the worst failure semantics in the stack. The task should submit the work to an engine, watch it, and report the outcome. The orchestrator is a control plane, and treating it as a data plane is the single most common architectural error made with these tools.

It is also not a place to put business logic. The graph should say what runs and in what order. What the transformation does belongs in a transformation tool, a SQL model or an application, where it can be tested without starting a scheduler.

The flagship graph, in numbers

The graph this lesson keeps returning to has eighty seven tasks arranged six levels deep. Its critical path, meaning the longest chain of dependencies measured end to end, runs four hours and twelve minutes. The sum of every task's own duration is twenty six hours and forty minutes, so the graph achieves a parallelism factor of about six and a third.

It is scheduled daily at two in the morning and the downstream commitment is nine, so a path of four hours and twelve minutes finishes at twelve minutes past six and leaves two hours and forty eight minutes of slack. That slack is the single most important number in the design, because it is the budget from which every retry, every late upstream file and every unlucky scheduling decision is paid.

Over ninety consecutive days this graph completed cleanly on seventy four of them. On sixteen days at least one task failed. Nine of those were upstream data arriving late, four were transient infrastructure, two were a code defect, and one was an upstream schema change. Retries alone resolved eleven of the sixteen. Five required a person.

Where it sits relative to everything else in this course

Every engine covered so far in this course is something an orchestrator invokes rather than replaces. A Spark job is submitted by a task and watched by that task. A Trino query is issued by a task. An Iceberg compaction is a maintenance task on a schedule. A Flink job is the one genuine exception, because a streaming job is long lived and does not fit the run and finish shape a graph expects, so orchestrators usually only deploy and health check them rather than running them.

This is why orchestration lands near the end of a course about engines. The graph is the layer where the engines meet, and the questions it answers are the ones no single engine can see: whether the warehouse load should wait for the lakehouse compaction, whether yesterday's late correction has been applied before this morning's report is built, whether the three teams writing into the same table have agreed on an order. Those are relationships between systems, and nothing inside any one system is in a position to enforce them.

Why the numbers matter more than the tool

There is an enormous amount of argument about which orchestrator to use, and almost none of it changes any of the numbers above. The critical path is a property of your dependency graph. The parallelism factor is a property of how you split the work. The slack is a property of the schedule and the commitment. The failure mix is a property of your upstream suppliers and your own code quality.

A different tool changes the syntax you use to express the graph, the quality of the user interface, and how much operational work the orchestrator itself demands. Those are real considerations and the final lesson of this course weighs them. They are not what makes a pipeline reliable.