Introduction

Recovering a Kubernetes cluster is less about one magic command and more about restoring state in a controlled order. The recovery path depends on what failed, including control plane loss, worker node loss, or accidental resource deletion. A solid runbook combines etcd snapshots, manifest backups, and clear validation checkpoints.

Define the Recovery Scope First

Before touching the cluster, identify blast radius. Ask whether only workloads are affected, or whether control plane components are unavailable.

Quick triage commands:

1kubectl get nodes
2kubectl get pods -A
3kubectl get events -A --sort-by=.metadata.creationTimestamp | tail -n 50
4kubectl cluster-info

If API access is down, connect directly to a control plane node and inspect systemd services and logs.

sudo systemctl status kubelet
sudo crictl ps | grep kube-apiserver
sudo journalctl -u kubelet -n 100 --no-pager

Start with facts. Recovery is faster when you avoid simultaneous changes across multiple layers.

Restore etcd Safely

For kubeadm based control planes, etcd is the source of truth for Kubernetes objects. If etcd data is corrupted or lost, restore from a known good snapshot.

1# create a snapshot during normal operations
2ETCDCTL_API=3 etcdctl   --endpoints=https://127.0.0.1:2379   --cacert=/etc/kubernetes/pki/etcd/ca.crt   --cert=/etc/kubernetes/pki/etcd/server.crt   --key=/etc/kubernetes/pki/etcd/server.key   snapshot save /var/backups/etcd-snapshot.db
3
4# inspect snapshot metadata
5ETCDCTL_API=3 etcdctl snapshot status /var/backups/etcd-snapshot.db -w table

Restore on the control plane node:

1sudo systemctl stop kubelet
2sudo mv /var/lib/etcd /var/lib/etcd.broken.$(date +%s)
3
4ETCDCTL_API=3 etcdctl snapshot restore /var/backups/etcd-snapshot.db   --data-dir=/var/lib/etcd-restored
5
6sudo mv /var/lib/etcd-restored /var/lib/etcd
7sudo systemctl start kubelet

After etcd recovery, verify API server health before restoring workloads.

Reconcile Control Plane and Node Membership

If control plane components fail to come up, regenerate configs only when necessary and verify certificates are valid.

kubectl get componentstatuses
kubectl get --raw='/readyz?verbose'

For failed worker nodes, drain and rejoin carefully to avoid data loss for stateful services.

1kubectl cordon worker-1
2kubectl drain worker-1 --ignore-daemonsets --delete-emptydir-data
3
4# on worker node
5sudo kubeadm reset -f
6sudo kubeadm join <api-server>:6443 --token <token>   --discovery-token-ca-cert-hash sha256:<hash>

Replace the placeholders with values from kubeadm token create --print-join-command on a healthy control plane node.

Restore Workloads and Persistent Data

Cluster state recovery does not guarantee application data recovery. Restore namespaces, manifests, and persistent volumes from your backup tooling.

1# example using Velero
2velero restore create --from-backup nightly-backup-2026-03-01
3velero restore describe --details <restore-name>
4
5# validate workloads
6kubectl get deploy,sts,svc,ing -A
7kubectl get pvc -A

Prioritize critical namespaces first. Run smoke checks against core APIs, message queues, and databases before declaring incident closure.

Common Pitfalls

A common mistake is restoring etcd from an old snapshot without communicating expected data loss. Always publish snapshot timestamp and potential object drift to stakeholders.

Another issue is restoring control plane state and forgetting admission webhooks or custom resource definitions managed outside core backups. Missing platform components can break workload startup silently.

A third issue is rejoining nodes before verifying network plugin and CNI health. Nodes may appear Ready while pod networking still fails.

Finally, teams often skip recovery rehearsals. Disaster recovery plans decay quickly when not practiced. Run game day drills and record exact command sequences that worked. Keep a post incident timeline so future responders can see which checkpoints reduced recovery time most effectively.

Summary

• Identify whether failure scope is workloads, control plane, or both before acting
• Keep frequent etcd snapshots and validate they can be restored
• Restore control plane first, then reconcile node membership and workloads
• Recover application data separately from Kubernetes object state
• Rehearse disaster recovery regularly so incident execution is predictable