An upgrade is not one operation. It is a sequence of component upgrades constrained by which versions are allowed to talk to each other, and the whole procedure follows from those constraints plus one rule about which direction is recoverable.

The components and what they talk to

One topology explains the constraints.

The interface server is the hub: every other component talks to it and nothing talks to anything else. The controller manager, the scheduler and the cloud controller manager are clients of it. Node agents and proxies are clients of it. Command-line tools and controllers are clients of it. The datastore sits behind it and is spoken to only by it.

Because everything is a client of one component, the skew rules are all expressed relative to that component's version, and the upgrade order falls out: the hub goes first, because clients tolerate a newer hub and a newer client against an older hub is the unsupported direction.

The skew rules that matter

Four rules cover the planning.

The interface server may be at most one minor version ahead of the node agents, and in recent versions three. The controller manager and scheduler must not be newer than the interface server, and may be one minor behind. Node agents must not be newer than the interface server. And command-line clients may be one minor on either side.

The practical consequence is that one minor version per upgrade cycle is the only supported path, so a cluster three versions behind needs three sequential upgrades with validation between them. Teams hoping to jump from an old version to a current one in a single operation find that there is no such operation, and the realistic planning figure is weeks rather than a day.

Why the control plane goes first

One asymmetry decides the order.

A newer interface server serving older node agents is supported and tested. An older interface server serving newer node agents is not, because a new agent may use interface versions or fields the older server does not have.

So the sequence is the interface server, then the other control plane components, then the node agents, then everything else. And the corollary matters more: the control plane cannot be rolled back after the nodes have been upgraded, because that would put newer agents against an older server. Which is why the control plane upgrade must be validated before touching the nodes.

The datastore and the point of no return

One step is the real risk in the whole procedure.

The interface server upgrade may migrate how objects are stored, and a downgrade after that is not generally supported. The datastore itself has its own version and its own upgrade, which is the operation with genuine data risk.

So a verified, restorable backup taken immediately before the upgrade is the one non-negotiable prerequisite, and verified means a restore was actually performed somewhere, not that a file exists. The teams that discover their backup is unusable discover it at the worst possible moment, and the check costs an hour.

Highly available control planes

One arrangement changes the failure mode.

With several interface servers behind a load balancer, they are upgraded one at a time, so during the upgrade clients reach a mixture of versions. That is supported and it has a visible consequence: a client may see an object through a new server and then write it through an old one.

The practical implications are that the window should be short, that a long window with mixed versions is where odd behaviour appears, and that the leader-elected components, the controller manager and scheduler, will fail over during their own upgrade, which means a brief pause in reconciliation and scheduling. Neither is harmful, and both are worth announcing so that a slow rollout during the window is not mistaken for an incident.

Reading the release notes properly

One task is boring and unavoidable.

Each release has a set of removed interface versions, changed defaults, deprecated features and behaviour changes. The removed versions are the ones that break manifests, and the changed defaults are the ones that break workloads quietly.

The discipline is to read the notes for every intermediate version, not just the target, because a cluster going up three versions inherits all three sets of changes. A default that changed two versions ago and was only an advisory then is a behaviour change now, and the symptom is a workload that behaves differently after an upgrade for no reason anyone can find.

Feature gates and their lifecycle

One mechanism is where behaviour changes come from.

Features move through stages: off by default, on by default, then permanently enabled with the gate removed. An upgrade can therefore enable something new without any configuration change, and can remove a gate that a cluster was explicitly setting.

Two failures follow. A control plane that sets a gate removed in the new version fails to start, with an error naming the unknown gate, which is a short outage if the control plane is upgraded without checking. And a feature becoming enabled by default changes behaviour nobody asked for, which is why the gate list in the release notes deserves as much attention as the removed interface versions.

Validating the control plane before the nodes

One checklist gates the irreversible part.

After the control plane is upgraded and before any node is touched: every control plane component reporting healthy, the datastore healthy with all members at the expected version, objects of every kind listable, a test object created and deleted, controllers reconciling as shown by a deliberate scale change, the scheduler placing a new pod, and no errors in the components' logs.

Doing this before the nodes is what preserves the ability to roll back, since the control plane can be reverted while the nodes are still older. After the nodes move, that option is gone, which makes this checklist the most valuable fifteen minutes in the procedure.

Managed clusters and what is still yours

One division of labour is worth stating.

With a managed control plane, the provider performs the control plane upgrade and the skew rules are enforced for you, often by refusing to upgrade nodes beyond the allowed gap.

What remains yours is everything in this lesson from the nodes down: the node upgrade strategy, the drains, the disruption budgets, the workloads' tolerance of being moved, and the removed interface versions in your own manifests and controllers. The provider's upgrade going smoothly tells you nothing about whether your workloads survive a node replacement, which is the part that actually causes incidents.

Planning the cadence

One decision avoids the worst position.

Releases are supported for about a year, and three or four appear each year, so a cluster upgraded once a year is permanently near the end of support and any delay means running something unsupported.

The comfortable cadence is every release or every other release, treating it as routine rather than a project. The alternative, which many organisations reach, is a cluster several versions behind where the upgrade is a quarter of work because every intermediate version's removals have accumulated. The cheapest moment to upgrade is always now, and that argument is easier to make before the gap opens than after.

Certificates, which expire whether you upgrade or not

One maintenance item rides along with upgrades.

The control plane's internal certificates have a validity period, often a year, and an upgrade typically renews them as a side effect. A cluster that is never upgraded is therefore a cluster whose certificates will expire.

The failure is abrupt and total: components cannot authenticate to the interface server, so nothing reconciles, and the fix requires reissuing certificates on a cluster that is no longer cooperating. This is a second, independent argument for a regular upgrade cadence, and the mitigation for clusters that cannot upgrade often is a monitored expiry date rather than hoping.

Addons are part of the upgrade

One category is routinely forgotten.

The networking plugin, the storage drivers, the metrics server, the ingress controller, the certificate manager and every operator have their own version compatibility with the cluster, and some will not work on the new version.

So the inventory for an upgrade includes each of these with its supported range, and the order matters: a networking plugin too old for the new node agents breaks pod networking on every upgraded node at once. Checking the addon matrix before starting is unglamorous and it is where most upgrade surprises actually come from, rather than from the cluster components themselves.