Cluster failures look varied and are not. Almost all of them live in one of five layers, each layer depends only on the ones below it, and a few minutes spent establishing which layer is broken saves hours of looking in the wrong place.

The five layers

One ordering explains the dependency structure.

The datastore, which holds all state. The interface server, which is the only thing that reads it. The controllers and scheduler, which read and write through the interface server. The node agents, which are also clients of it. And the data path, meaning name resolution, service routing and pod networking, which depends on the agents and the networking components but not on the control plane continuing to run.

Each layer fails in its own way and each produces a characteristic symptom at the layer above, which means the symptom narrows the search immediately if you know the mapping. Working from the top down wastes time because a broken node agent and a broken datastore both present as pods not starting.

Four questions that locate the failure

One sequence takes about three minutes.

Can the interface server be reached and does it return objects? Is the datastore healthy, with all its members present and at the same revision? Are the nodes reporting ready, and when did they last report? And are the controllers acting, which a deliberate scale change answers directly.

Those four, in that order, partition almost every failure. An interface server that answers and a datastore that is healthy but nodes that stopped reporting twenty minutes ago is a different incident from the same nodes reporting ready while nothing reconciles. The discipline is asking all four before forming a theory, because the first plausible theory is usually built on the first symptom seen rather than the lowest broken layer.

Running traffic survives a dead control plane

One property is widely misunderstood and it changes incident priority.

A pod already running keeps running with no control plane at all. Its container keeps serving. The proxy rules already programmed on each node keep routing. Name resolution keeps working while its pods are up.

So a control plane outage is an inability to change anything, not an outage of the applications: nothing can be deployed, scaled, rescheduled or recovered, and a pod that dies is not replaced. That distinction decides whether this is a wake-everyone incident or an urgent-but-not-critical one, and getting it right prevents a panicked restore that causes the actual outage.

Events, and why they disappear

One source is the most useful and the most perishable.

Events explain why a decision was made: why a pod could not be scheduled, why a volume did not attach, why an image pull failed. They are stored in the datastore with a short retention, typically an hour.

So the first action in a slow-moving incident is to capture the events before they expire, because an hour later the explanation is gone and only the state remains. This is also why an incident reported the next morning is much harder than the same incident reported immediately, and why shipping events to longer-term storage is worth doing before the incident that needs it.

Reading state against reading intent

One distinction keeps the search honest.

An object's spec is what was asked for and its status is what the responsible component believes. Comparing them tells you whether anything is acting on the intent at all.

A deployment asking for ten replicas with a status showing three, updated two minutes ago, is a controller working slowly or blocked. The same deployment with a status last updated an hour ago is a controller that has stopped. The timestamp on the status is the signal that separates slow from dead, and it is the single most informative field in a stuck-reconciliation incident.

The component logs and where they live

One practical point wastes time when forgotten.

The control plane components run either as containers managed directly by the node agent from manifests on disk, or as processes on the host. In the first case their logs are reachable through the interface server, which is useless when the interface server is the broken thing.

So the fallback is logging onto the machine and reading the container runtime's logs or the system journal directly. Knowing which arrangement a given cluster uses, and having the access to do it, is a prerequisite rather than something to work out during the incident. For a managed control plane the equivalent is knowing where the provider surfaces those logs.

Changing one thing at a time

One rule is the hardest to follow and the most valuable.

Under pressure the temptation is to restart several components, apply two fixes and bounce the nodes. If the symptom then clears, nothing has been learned, and if it does not, the state is now worse and less understood.

So each action is taken alone, its effect observed, and the result recorded before the next. The exception is a genuine emergency where restoring service dominates understanding, and even then the actions should be written down as they happen. The reason this rule is worth stating explicitly is that every incident review that concludes "we are not sure what fixed it" is an incident where it was broken.

Keeping a record while it happens

One habit pays twice.

A running log of the time, the action and the observation, written as the incident proceeds rather than reconstructed afterwards, is what makes the review useful and what lets a second responder join without a long handover.

It also prevents the commonest error in a long incident, which is repeating an action already taken because nobody remembers whether it was. A shared document with timestamps costs nothing and is the difference between a review that produces fixes and one that produces a vague narrative.

What the cluster looks like when it is healthy

One baseline makes abnormality visible.

Normal node count and readiness, normal pod count by namespace, the usual handful of pods that are always restarting for benign reasons, normal interface server request latency, normal datastore size and revision rate.

Without those numbers, every observation during an incident needs interpretation: a datastore of eight hundred megabytes is either fine or alarming depending on what it usually is. Capturing the baseline is a half-hour task done once, and its absence is why incidents start with ten minutes of arguing about whether something is unusual.

Access before the incident

One preparation is frequently missing.

Debugging a broken control plane needs access that does not depend on the control plane: credentials that work when the interface server is refusing requests, machine access to the control plane hosts, and the datastore client tooling with its certificates.

The failure mode is an authentication outage where the only administrative access is through the mechanism that is broken, which turns a twenty-minute fix into a long escalation. A break-glass credential, stored outside the cluster and tested quarterly, is the answer, and the test is the part that gets skipped.

Asking what changed

One question resolves more incidents than any diagnostic.

Most cluster failures follow a change: a deployment, a webhook, a policy, an addon upgrade, a certificate rotation, a node image, a configuration applied by a pipeline nobody was watching.

So the second thing to do, in parallel with the four questions, is to establish what changed in the last few hours. The audit log answers it for anything written through the interface server, and the delivery pipeline's history answers it for everything applied automatically. An incident whose cause is a change made forty minutes ago is the common case, and reasoning from first principles about a cluster that was fine an hour ago is usually the slower route.

Reproducing at a smaller scope

One technique turns a vague report into a specific one.

Reports arrive as broad statements: the cluster is broken, nothing works, deployments are failing. The useful move is to narrow until the smallest failing case is known.

Does it fail in one namespace or all? For one kind of object or every kind? For one user or everyone? With a trivial object or only a real one? Each answer halves the search space, and the narrowing usually identifies the cause before the diagnosis is formally done: a failure confined to one namespace is a quota, a policy or a webhook with a selector, and that is a different investigation from one affecting everything.