kubelet won't start after kuberntes/manifest update
System Design practice on Codemia
Work through 120+ system design problems with detailed solutions, from rate limiters to multi-region storage.
Introduction
If kubelet stops starting after a manifest-related change, the first thing to verify is what actually failed. A bad static pod manifest can break control-plane components, but kubelet itself usually fails to start only when its own configuration, runtime dependency, certificates, or host environment is invalid.
Separate "kubelet is down" from "pods are down"
This distinction matters. Static pod manifests under /etc/kubernetes/manifests are watched by kubelet. If you edit one of those files incorrectly, kubelet may still start, but the affected static pod may crash-loop or never come up.
So begin with:
If the service will not stay up, the logs are the source of truth. Guessing from the manifest alone wastes time.
Common Root Causes
Typical reasons include:
- malformed kubelet config
- wrong
staticPodPath - expired or missing certificates
- container runtime not reachable
- cgroup driver mismatch
- swap or disk pressure issues
- invalid static pod manifest causing repeated startup errors around control-plane dependencies
On kubeadm-based systems, control-plane components are often defined as static pod manifests in /etc/kubernetes/manifests. Kubelet watches that directory and starts those pods.
A Focused Troubleshooting Flow
1. Validate the service logs
Look for clear messages such as "failed to load kubelet config file", "container runtime is down", or certificate-related failures.
2. Check kubelet configuration
Many systems store it here:
You are looking for settings such as:
- '
staticPodPath' - '
containerRuntimeEndpoint' - TLS certificate paths
- authentication and authorization sections
If the manifest update accidentally included kubelet config changes, that is often the real breakage.
3. Inspect static pod manifests
A YAML syntax error, bad image reference, invalid host path, or unsupported field can prevent the static pod from starting. That may look like "the node is dead" when the actual problem is only one control-plane component.
4. Confirm container runtime health
If kubelet cannot talk to the runtime, it may exit or log repeated runtime connection errors.
Safe Recovery Pattern
If the problem started immediately after editing a manifest, the safest rollback is usually to restore the previous known-good file, then restart kubelet:
On a production control-plane node, do not improvise with multiple simultaneous changes. Revert first, confirm recovery, then reapply one controlled edit at a time.
Why This Happens So Often
Static pod manifests feel like ordinary YAML files, but they are operating-system-level cluster control inputs. A small mistake in a command argument, volume mount, certificate path, or image tag can take down the component kubelet is trying to launch.
That is why backups and diffs matter:
Common Pitfalls
The biggest mistake is assuming the manifest file alone controls whether kubelet starts. Kubelet may be failing for a completely different reason, such as containerd or certificate state.
Another mistake is editing files under /etc/kubernetes/manifests without saving a backup first. On control-plane nodes, rollback speed matters.
A third issue is changing several files at once. If you update manifests, runtime config, and certificates together, you lose the ability to isolate the real failure quickly.
Summary
- First determine whether kubelet is down or only the static pods are failing.
- Read
journalctl -u kubeletbefore changing anything else. - Check kubelet config, manifest syntax, certificate paths, and container runtime health.
- Restore the last known-good manifest if the failure started right after an edit.
- Make control-plane manifest changes one at a time so rollback and diagnosis stay manageable.
Related reading
- Kubenetes Is it possible to hit multiple pods with a single request in Kubernetes cluster
- Kubernetes-Helm Charts pointing to a local docker image
- Kubernetes-services load balancing
- Kubernetes - can a Deployment have multiple ReplicaSets?
- Kubernetes - Can't connect to a service IP from the service's pod
- Kubernetes - pod has unbound immediate PersistentVolumeClaims
- Kubernetes - Container image already present on machine
- Kubernetes - delete all jobs in bulk

System Design Fundamentals
Build a strong foundation in designing scalable, reliable distributed systems.
View the courseTrack what you have practised
A free account saves your progress, solutions and study plan across every problem on Codemia.
System Design practice on Codemia
Work through 120+ system design problems with detailed solutions, from rate limiters to multi-region storage.