Deployments, Stateful Sets and Daemon Sets

Topics Covered

Controlling a Deployment's Rollout

The two strategies

The surge and unavailable window

Minimum ready seconds

The progress deadline

Revision history and the rollback that is not there

Pausing, and what it is for

Reading a rollout's state

What a completed rollout does not prove

The capacity question nobody asks

Scaling and rolling at the same time

A deployment's behaviour during a change is governed by six fields, and almost every complaint about rollouts being too slow, too risky or apparently stuck is one of those fields doing exactly what it says.

The two strategies

There are only two, and the choice is about whether two versions may coexist.

Recreate deletes every existing pod and then creates the new ones, which means a complete outage for the duration and exactly one version running at a time. Rolling update replaces pods incrementally within bounds, which means no outage and two versions serving simultaneously for the duration.

Recreate is correct more often than its reputation suggests: a workload holding an exclusive lock, one whose two versions cannot share a database schema, or any singleton where two instances would be incorrect. Choosing rolling update for such a workload produces a brief period of real misbehaviour that is easy to miss in testing because it only happens during a deploy.

The surge and unavailable window

Two fields bound the rollout, and reading them as a window makes the pacing predictable.

With a desired count, a maximum surge and a maximum unavailable, the controller keeps the total pod count at or below the desired plus surge, and the available count at or above the desired minus unavailable:

R≤total≤R+⌈S⋅R⌉andavailable≥R−⌊U⋅R⌋R \le \text{total} \le R + \lceil S \cdot R \rceil \quad\text{and}\quad \text{available} \ge R - \lfloor U \cdot R \rfloor

Both default to twenty five percent. With ten replicas that means up to thirteen pods and at least eight available, so the controller works in batches of three. Setting surge to zero makes the rollout take capacity away before adding, which needs no spare capacity and reduces serving capacity during the deploy. Setting unavailable to zero preserves full capacity and requires spare room for the surge.

Surge bounds the total above desired and unavailable bounds the available below it, which together set the batch size and the capacity required.

Minimum ready seconds

One field decides how much confidence a new pod must earn before it counts.

A pod becomes available only after it has been ready for the stated number of seconds, and only available pods count towards the unavailable bound. The default is zero, which means a pod that passes readiness once immediately licenses the removal of an old pod.

That default is the reason a rollout can complete successfully and leave a broken version serving: an application that passes its readiness probe at start and fails ten seconds later under real traffic satisfies a zero-second requirement at every step. Setting this to thirty or sixty seconds costs a minute of rollout time and converts that failure into a rollout that stalls with the old version still present, which is a far better outcome.

The progress deadline

A rollout that cannot proceed needs to say so, and one field makes it say so.

If no progress is made within the deadline, six hundred seconds by default, the deployment's progressing condition becomes false with a reason naming the deadline. Progress means any advance in the rollout, not completion, so a rollout making slow headway never trips it.

The important property is that tripping the deadline does not roll anything back; it only reports. The deployment stays part-rolled with both versions present until someone acts. So the deadline is a signal to automate on, and a pipeline that applies a change and then waits on the rollout status is relying on exactly this condition to stop waiting.

Revision history and the rollback that is not there

Old replica sets are the rollback mechanism, and one setting can remove it.

Each pod template change creates a replica set, kept at zero replicas with a revision annotation, up to the history limit of ten by default. A rollback sets the template back to a retained revision and rolls forward to it normally.

Setting the limit to zero deletes old replica sets immediately, which looks like housekeeping and removes the ability to roll back at all. The failure arrives at the worst moment, during an incident, with a message that no revision is available. Two or three is a reasonable reduction if object count matters; zero is not a reduction, it is a removal of a capability.

Pausing, and what it is for

A deployment can be paused, which stops the controller acting on it without stopping anything running.

A paused deployment accepts template changes without rolling them out, so several edits can be batched into one rollout, which matters when a change involves both a new image and a new configuration reference. It is also the mechanism for a manual canary: pause after the first new pod appears, observe, then resume or roll back.

The hazard is forgetting. A paused deployment looks normal in most listings, accepts changes silently, and does nothing, so a team can spend an afternoon wondering why their change has not deployed. The condition is visible on the object, and checking it is worth doing early when a deployment appears inert.

Reading a rollout's state

A short sequence covers the whole question of what a rollout is doing.

bash
1kubectl rollout status deploy/web --timeout=120s        # blocks until complete or deadline
2kubectl get deploy web -o jsonpath='{range .status.conditions[*]}{.type}={.status}:{.reason} {end}{"\n"}'
3kubectl get rs -l app=web --sort-by=.metadata.creationTimestamp -o wide
4kubectl rollout history deploy/web                      # retained revisions
5kubectl rollout undo deploy/web --to-revision=7

The conditions are the fastest read: progressing false with a deadline reason means it gave up waiting, available false means capacity is below the minimum, and both true with the newest replica set fully ready means it finished. The replica set listing is the second read, because it shows which revisions exist and how far each has advanced.

What a completed rollout does not prove

One habit is worth correcting because it hides real failures.

A completed rollout means the new pods became ready and stayed ready for the minimum period, which is a statement about the readiness probe and nothing more. It does not prove the application works, that its dependencies are reachable, that its performance is acceptable, or that error rates are unchanged.

So a pipeline that treats rollout completion as deployment success is only as good as the readiness probe behind it. The practices that close the gap are a readiness probe that checks something meaningful, a minimum ready period long enough to catch early failures, and a post-rollout check of actual service metrics rather than only the cluster's own view.

The capacity question nobody asks

One arithmetic check prevents a class of rollout failure in a full cluster.

A rollout with a surge needs room for the surge pods somewhere, and in a cluster near capacity that room may not exist. The new pods go pending, no progress is made, and the rollout eventually trips its deadline with the old version still serving, which is the safe outcome but is reported as a mysterious stall.

The check is whether the cluster can fit the surge for every workload that might deploy at once, which in a shared cluster is not a question any one team can answer. The mitigations are a surge expressed as a small absolute number rather than a percentage for large deployments, a cluster autoscaler that can add a node promptly, and accepting a surge of zero for workloads that can tolerate a capacity dip.

Scaling and rolling at the same time

One interaction produces behaviour that looks like a bug and is not.

A scale change during a rollout is applied to both replica sets proportionally, so a deployment scaled from ten to twenty mid-rollout grows both the old and the new set rather than putting all the new capacity on the new version. The rollout then continues from the new counts.

That is the right behaviour, since the bounds are expressed relative to the desired count and the controller is maintaining them throughout, but it means the rollout takes longer and the version mix at any moment is harder to predict. The practical advice is the same as for stateful sets: scale or roll, not both, and let an autoscaler be the one exception you reason about deliberately.

Level Expectations

A strong answer treats a rollout as a window rather than a sequence: surge bounds the total above desired, unavailable bounds the available below it, and the two together determine both the batch size and the spare capacity required. It then names minimum ready seconds as the field that decides whether a rollout can complete while shipping a broken version, and the progress deadline as a report rather than a rollback, which is why automation must act on the condition itself.

Loading problem...