The scheduler's job is one sentence long: give every pod without a node the name of a node it can run on. Everything interesting is in how it does that at scale without ever placing a pod somewhere it cannot run.

What the scheduler actually writes

The output is smaller than people expect, which explains several behaviours.

Scheduling a pod means setting one field, the node name, by creating a binding. The scheduler does not start the container, reserve anything on the node, or communicate with the node at all. Once the field is set, the node agent on that node sees a pod assigned to it and takes over.

So a scheduling decision is a claim about the future that the node may refuse. If the node lacks capacity by the time the agent looks, the pod fails to start there and is not rescheduled, because the node name is already set and the scheduler no longer considers it its business.

The two queues

Pending pods live in one of two queues and the difference matters for latency.

The active queue holds pods ready to be scheduled, ordered by priority and then by the time they arrived. The backoff queue holds pods whose last attempt failed, released back to the active queue after an exponentially growing delay. There is also an unschedulable set for pods that failed and are waiting for a cluster change rather than a retry.

A pod moves out of the unschedulable set when an event suggests it might fit now: a node added, a pod deleted, a node's labels or taints changed. That event-driven requeue is why a pending pod becomes scheduled immediately after a node joins rather than after a polling interval.

The cycle: one pod at a time

The core loop is serial, which is the single most important thing to know about the scheduler.

It takes the highest-priority pod from the active queue, runs the whole decision for it, and writes the binding, then takes the next. There is no parallel scheduling of multiple pods, because each decision depends on the state left by the previous one, and two parallel decisions could both place a pod in the last available slot.

The throughput consequence is direct: scheduler throughput is one over the per-pod decision time. A cluster creating ten thousand pods in a burst waits for ten thousand serial decisions, which is why decision cost per pod is the quantity worth optimising.

Filter then score

Each decision has two phases with different purposes.

Filtering removes every node the pod cannot run on, producing a set of feasible nodes. It is a hard predicate: a node either can or cannot host the pod. Scoring then ranks the feasible nodes, each scoring function returning a number that is weighted and summed, with the highest total winning and ties broken at random among equals.

The division is worth internalising because it answers most questions about scheduler behaviour. Anything expressed as a requirement is a filter and a pod will stay pending forever if nothing passes. Anything expressed as a preference is a score and is silently ignored when it cannot be satisfied.

The scheduling cycle and the binding cycle

One split exists for throughput and explains a subtle class of failures.

The scheduling cycle ends when a node is chosen and its resources are optimistically marked used in the scheduler's cache. The binding cycle, which writes the binding object and may involve slower work such as waiting for a volume to be provisioned, runs asynchronously so the next pod can be scheduled immediately.

The optimistic cache update is what makes that safe: the next decision sees the node as already carrying the previous pod, even though nothing has been written yet. If the binding fails, the cache entry is rolled back and the pod returns to the queue, which is why a binding failure costs one retry rather than corrupting subsequent decisions.

Extension points, in order

The framework exposes a fixed sequence of hooks, and knowing their order explains what a plugin can and cannot do.

Queue sort decides pod order. Pre-filter computes state once per pod. Filter rejects nodes. Post-filter runs only when no node passed, which is where preemption lives. Pre-score prepares state. Score ranks nodes. Reserve claims resources. Permit can delay or reject. Pre-bind, bind and post-bind complete the placement.

Everything built into the scheduler is a plugin at one or more of those points, including resource fitting and affinity, which means the default behaviour is configurable by enabling, disabling and weighting plugins rather than by patching code.

Where gang scheduling fits

One extension point exists for a use case the default scheduler does not handle.

The permit phase can hold a pod after its node is chosen but before it is bound, waiting for a condition. That is how all-or-nothing scheduling for a group of pods is implemented: each pod in the group waits at permit until the whole group has nodes, then all are allowed through, or all are rejected and returned to the queue.

Without it, a group of pods is scheduled individually, so a training job needing eight pods may get five placed and three pending, holding capacity while making no progress. The default scheduler does exactly that, which is why batch systems with group requirements use a plugin or a separate scheduler.

Scheduling gates

One mechanism lets something else hold a pod before the scheduler sees it.

A pod can carry scheduling gates, and while any gate is present the scheduler skips the pod entirely: it is not considered unschedulable, it is not retried, it simply is not a candidate. Removing the last gate makes it eligible.

That is the clean way to implement admission-time coordination such as quota reservation, cost approval or waiting for an external resource, because the pod exists and can be inspected while being definitively not scheduled. Before gates, such coordination meant a fake unsatisfiable requirement, which polluted the unschedulable reporting and retried pointlessly.

Why a scheduled pod can still fail to run

One consequence of the write-a-field design is worth stating directly.

Between the scheduler's decision and the node agent acting, the node's actual state may have changed: another pod may have been admitted, memory may have been consumed by processes outside any pod, a device may have become unavailable. The agent performs its own admission check and rejects the pod if it does not fit.

A rejected pod is marked failed with a reason, and for a bare pod that is the end of it. Under a controller, the controller creates a replacement, which goes through scheduling again and may well choose the same node. That loop is visible as a pod repeatedly failing admission on one node, and the usual cause is capacity the scheduler cannot see.

Only one scheduler is active

One detail about availability is worth knowing because it bounds what high availability buys.

Several scheduler instances can run, but they contend for a lease and only the holder schedules; the others are idle standbys. That is required by the serial design, since two active schedulers would each hold their own optimistic cache and could both claim the same slot.

So running three instances improves recovery time after a failure, not throughput. The failover is a lease expiry away, during which nothing is scheduled, and the practical effect of a scheduler outage is that pending pods accumulate and running pods are untouched, which is a benign failure mode compared with most control plane components.