Introduction

When a Kubernetes node reports "no free space," the problem is usually local node storage exhaustion, not a failure of the API-level garbage collector. In practice, the pressure comes from container images, writable layers, logs, dead containers, or emptyDir data filling the node's filesystem. The fix is to identify which storage path is full and whether kubelet image garbage collection or node eviction is failing to reclaim enough space.

Know Which Kind of Garbage Collection Is Involved

Kubernetes uses the phrase garbage collection in more than one place. For this error, the relevant mechanisms are usually on the node:

• image garbage collection by kubelet
• dead container cleanup by the container runtime
• pod eviction when disk pressure thresholds are crossed

That is different from owner-reference cleanup in the control plane. If a node cannot pull images or start pods because disk is full, you need node-level debugging.

Inspect the Node First

Start by checking node conditions and recent events.

kubectl describe node worker-1
kubectl get events -A --sort-by=.lastTimestamp

Look for conditions such as:

• 'DiskPressure=True'
• image garbage collection failures
• eviction messages
• pod sandbox creation failures

Then inspect the node directly.

1ssh worker-1
2df -h
3sudo du -sh /var/lib/containerd 2>/dev/null
4sudo du -sh /var/log 2>/dev/null

On modern Kubernetes nodes using containerd, /var/lib/containerd is a common source of space usage. On older Docker-based nodes, the heavy path may be under /var/lib/docker.

Check Kubelet and Runtime Cleanup

If kubelet is supposed to reclaim space but does not, look at its logs.

sudo journalctl -u kubelet -n 100 --no-pager
sudo crictl images
sudo crictl ps -a

The goal is to answer two questions:

1. is kubelet trying image garbage collection
2. is the container runtime holding onto images, dead containers, or logs

If unused images have accumulated, runtime-level cleanup may help immediately.

sudo crictl rmi --prune

That is a tactical fix, not a long-term policy. If the node fills up again tomorrow, you still need to understand why normal reclamation was not enough.

Understand Disk Pressure Behavior

Kubelet monitors node filesystems and can start evicting pods or deleting images when thresholds are crossed. If those thresholds are too lenient for the workload, the node can hit an operational cliff before cleanup frees enough space.

Common contributors include:

• very large container images
• chatty applications producing huge logs
• 'emptyDir volumes growing without limits'
• completed jobs and dead containers piling up
• image pull churn from frequent deployments

A node with small root storage is especially vulnerable. In many clusters, the real fix is to increase node disk size or reduce image and log footprint rather than repeatedly pruning by hand.

Example Cleanup Workflow

A reasonable on-call workflow looks like this:

1kubectl describe node worker-1
2ssh worker-1
3df -h
4sudo journalctl -u kubelet -n 100 --no-pager
5sudo crictl images
6sudo crictl rmi --prune
7sudo journalctl --vacuum-time=3d

After cleanup, watch the node condition from the control plane:

kubectl get nodes -w

If DiskPressure clears and new pods schedule normally, the immediate incident is resolved. Then follow up with a permanent fix such as smaller images, shorter log retention, or larger node disks.

Common Pitfalls

• Debugging API-object garbage collection when the real issue is node-local disk exhaustion.
• Looking only at pod status and never checking the node filesystem directly.
• Cleaning images manually without investigating why kubelet GC or eviction did not protect the node.
• Ignoring large log files or emptyDir usage and blaming only container images.
• Assuming every cluster still uses Docker paths instead of checking the active container runtime.

Summary

• "No free space" in Kubernetes usually points to node-local storage pressure.
• Check node conditions, kubelet logs, and the container runtime storage paths first.
• 'DiskPressure, image buildup, logs, and emptyDir growth are common causes.'
• 'crictl rmi --prune can help tactically, but it is not the whole solution.'
• Permanent fixes usually involve smaller images, log control, or larger node storage.