kubectl
Kubernetes
troubleshooting
nodes
NotReady

kubectl get nodes shows NotReady

System Design practice on Codemia

Work through 120+ system design problems with detailed solutions, from rate limiters to multi-region storage.

Practice system design

Introduction

When kubectl get nodes shows NotReady, Kubernetes has stopped treating that node as healthy enough for normal scheduling. The fix is rarely a single magic command. You need to determine whether the problem is kubelet health, container runtime issues, node pressure, or broken networking between the node and the control plane.

Start With the Node Conditions

The NotReady label is only the summary. The useful detail lives in the node conditions and recent events.

bash
kubectl get nodes
kubectl describe node worker-1
kubectl get events --sort-by=.lastTimestamp -A

In kubectl describe node, check the Conditions section for flags such as:

  • 'Ready=False'
  • 'MemoryPressure=True'
  • 'DiskPressure=True'
  • 'PIDPressure=True'
  • 'NetworkUnavailable=True'

Those conditions narrow the investigation quickly. If you see DiskPressure, there is no point starting with CNI logs. If NetworkUnavailable is true, look at the network plugin before touching workload manifests.

Check the Kubelet and Container Runtime

A node becomes NotReady very quickly when kubelet stops posting status updates. On the node itself, inspect the two services that matter most: kubelet and the container runtime.

bash
1sudo systemctl status kubelet
2sudo journalctl -u kubelet -n 100 --no-pager
3sudo systemctl status containerd
4sudo journalctl -u containerd -n 100 --no-pager

Typical kubelet problems include:

  • certificate or authentication failures when talking to the API server
  • inability to start pods because the runtime is down
  • repeated CNI setup errors
  • swap still enabled on Linux nodes that expect it off

If kubelet is down, fix that first and wait a minute before chasing anything else. Kubernetes cannot mark the node healthy when the agent responsible for reporting health is not functioning.

Rule Out Resource Pressure

Nodes under heavy resource pressure often show NotReady even though the operating system is technically up. Disk pressure is especially common on small worker nodes because images, logs, and writable layers accumulate faster than expected.

bash
1kubectl describe node worker-1 | sed -n '/Allocated resources:/,/Events:/p'
2df -h
3free -m
4sudo crictl images

Useful corrective actions include cleaning unused images, rotating large logs, or resizing the node.

bash
sudo crictl rmi --prune
sudo journalctl --vacuum-time=3d

Do not blindly delete running containers or kubelet directories. If the node hosts stateful workloads, careless cleanup can turn a health issue into data loss.

Inspect the CNI Plugin and Network Path

If kubelet logs mention sandbox creation failures or NetworkUnavailable, inspect the CNI components. Most Kubernetes distributions run the CNI agents as DaemonSet pods in kube-system.

bash
kubectl get pods -n kube-system -o wide
kubectl get daemonsets -n kube-system
kubectl logs -n kube-system ds/calico-node --tail=50

Replace calico-node with the correct DaemonSet name for your environment, such as Flannel, Cilium, or Weave.

Also confirm that the node can still reach the API server on the expected port and resolve cluster DNS if your bootstrap depends on it.

bash
curl -k https://YOUR_API_SERVER:6443/healthz
nslookup kubernetes.default.svc.cluster.local

A firewall change, expired route entry, or broken VPN path can leave the node powered on but effectively isolated from the cluster.

Example Troubleshooting Flow

A practical sequence helps avoid random guessing:

bash
1kubectl describe node worker-1
2ssh worker-1
3sudo systemctl status kubelet
4sudo journalctl -u kubelet -n 50 --no-pager
5sudo systemctl status containerd
6sudo df -h

Suppose the kubelet log contains repeated lines about image garbage collection failing because the disk is full. That points to DiskPressure. After cleanup, restart kubelet if necessary:

bash
sudo crictl rmi --prune
sudo systemctl restart kubelet

Then verify recovery from the control plane side:

bash
kubectl get nodes -w

If the node returns to Ready, the remediation was correct. If it remains NotReady, go back to the node conditions and events instead of trying unrelated changes.

Common Pitfalls

  • Treating NotReady as a single error instead of checking the underlying node conditions.
  • Restarting workloads before checking whether kubelet or the container runtime is actually down.
  • Ignoring disk pressure on nodes that build up large image caches.
  • Looking only at application pods and forgetting that the CNI DaemonSet may be broken.
  • Draining or deleting a node before confirming whether the problem is a small, reversible configuration issue.

Summary

  • Start with kubectl describe node and recent events, not guesswork.
  • Verify kubelet and the container runtime on the affected node.
  • Check for memory, disk, and PID pressure before changing network settings.
  • Inspect the CNI plugin when node conditions or kubelet logs point to networking.
  • Confirm recovery with kubectl get nodes -w after each fix.

Related reading
Course
Beginner
27 lessons
10 hours
System Design Fundamentals

Build a strong foundation in designing scalable, reliable distributed systems.

View the course
Track what you have practised

A free account saves your progress, solutions and study plan across every problem on Codemia.

System Design practice on Codemia

Work through 120+ system design problems with detailed solutions, from rate limiters to multi-region storage.

Practice system design

All Rights Reserved.