A job is the only workload controller whose desired state is a number of successful completions rather than a number of running pods, and every field on it exists to answer either how many, how fast, or when to give up.

Completions and parallelism

Two numbers describe the work and they are independent.

Completions is how many pods must exit successfully for the job to be complete. Parallelism is how many pods may run at once. A job with twelve completions and three parallelism runs three pods at a time until twelve have succeeded, which is four waves if nothing fails.

Leaving completions unset means one, so a job with only parallelism set is a work queue: it runs that many pods, and the job completes when any pod succeeds and no others are running, which suits pods that pull from an external queue until it is empty. Those two configurations are genuinely different patterns and the field that distinguishes them is easy to omit by accident.

The restart policy constraint

Jobs reject the default restart policy, and the reason is worth understanding.

A job's pod template must use a restart policy of never or on failure; always is rejected, because a container that always restarts can never be observed to have completed. With never, a failed container means a failed pod and the job creates a replacement pod. With on failure, the node agent restarts the container in place and the pod persists.

The difference matters for diagnosis more than for behaviour: with never, each attempt is a separate pod whose logs survive until the pod is cleaned up, which is usually what you want for batch work. With on failure, the attempts are restarts of one pod and only the current and previous attempt's logs are available, which loses history on a job that fails many times.

The backoff limit

One number decides when a job gives up, and it counts something less obvious than it appears.

The backoff limit, six by default, counts pod failures across the entire job rather than per completion. So a job with fifty completions and a backoff limit of six fails permanently after six individual pod failures, even if forty of the fifty completions succeeded. For a large job with any flakiness, that default is often far too low and produces a job marked failed with most of its work done.

Each retry is delayed by a backoff that starts at ten seconds and doubles to a ceiling of six minutes, applied at the job level, so a job that keeps failing slows down considerably:

The active deadline

A second bound is on wall clock time rather than on failures.

The active deadline, measured from the job's start, terminates every running pod and marks the job failed when it expires, regardless of progress. It is the only mechanism that bounds a job's total duration, and it overrides the backoff limit: a job that hits the deadline fails even with retries remaining.

This is the field to set for anything that must not run forever, such as a nightly report that is useless if it finishes after the morning. It is also the field people forget, which is how a job with an infinite loop in it occupies a node for a week. Note that it is deliberately not reset by retries, so a job retrying slowly can hit the deadline rather than the backoff limit.

Cleanup after finishing

Finished jobs and their pods persist, and one field is the correct way to deal with that.

Setting a time to live after finishing means the job and its pods are deleted that many seconds after completion or failure. Setting it to zero means immediate deletion once finished, and leaving it unset means the job stays forever.

Unset is the default and is the reason clusters accumulate thousands of completed jobs and tens of thousands of completed pods, which inflates every list operation and eventually shows up as interface server latency. The value to choose is however long someone might want the logs: an hour for a frequent job, a day or three for something whose failure needs investigating. Relying on a cleanup script is how the problem returns six months later.

Completion modes

A newer field changes what completions mean, and it unlocks the most useful parallel pattern.

In the default non-indexed mode, completions are interchangeable: any pod succeeding counts towards the total, and nothing distinguishes one from another. In indexed mode, each completion has an index from zero to completions minus one, exposed to the pod through an annotation and an environment variable, and the job is complete when every index has succeeded exactly once.

Indexed mode is how a job partitions a fixed data set without an external queue: each pod reads its index, computes its slice, and processes it. Non-indexed mode is for work drawn from somewhere else. Choosing wrongly produces either pods that all process the same data or an index nobody uses.

The job's status, and what to read

Four status fields answer the usual questions.

Active is how many pods are running now. Succeeded and failed are cumulative counts. The conditions carry a complete or failed condition with a reason, which for a failure distinguishes the backoff limit from the deadline. The start and completion times give the duration, which is the number worth graphing for anything on a schedule.

For an indexed job, the status also carries the completed indexes as a compact range expression, which is the fastest way to see which slices remain. A job appearing stuck with active pods and no progress is almost always a single slow index, and that field identifies it without inspecting pods.

Suspending a job

A job can be paused, which is more useful for batch than it sounds.

Setting a job suspended deletes its active pods and stops creating new ones, resetting the job's start time when it is resumed. Creating a job already suspended means it exists without running, which is the mechanism queueing systems use to admit jobs on their own terms rather than when they were submitted.

That is the basis of every fair-share batch scheduler built on top of the cluster: the queue creates jobs suspended, decides which may run according to its own quotas and priorities, and unsuspends them. Knowing the mechanism makes those systems much less mysterious, since the controller behaviour underneath is just this one field.

What a job does not provide

Three absences shape how batch systems are built around jobs.

It provides no ordering between pods, so anything requiring a sequence needs the application or an external coordinator. It provides no exactly-once guarantee, because a pod can succeed in its work and then fail to report it, so the work must be idempotent. And it provides no scheduling fairness or queueing beyond the cluster's own priority mechanism, which is why batch-heavy platforms add a queue controller above it.

Each of those is a deliberate boundary rather than a missing feature, and a batch platform is built by adding exactly those three things on top of the job primitive.

Where the job's pods come from

One detail about ownership explains some confusing cleanup behaviour.

A job owns its pods through an owner reference, so deleting the job deletes them by garbage collection. Deleting a job with an orphan propagation policy leaves the pods running, which is occasionally useful for inspecting a failure after the job object is gone but usually leaves pods nobody is tracking.

The related detail is that a job's pods carry labels identifying both the job and, for a cron job's jobs, the controller above it. Those labels are how logs and metrics are attributed to a workload across runs, and a template that overrides them breaks that attribution, which is a quiet way to lose the ability to compare a job's behaviour over time.