Introduction

kubectl drain does not directly "create replacement Pods" at all. Its job is to mark the node unschedulable and evict or delete evictable Pods from that node. Replacement Pods appear only because a higher-level controller such as a Deployment, StatefulSet, or ReplicaSet notices that a replica is leaving and reconciles the desired state somewhere else.

What kubectl drain Actually Does

When you drain a node, Kubernetes follows a rough sequence like this:

1. Mark the node unschedulable
2. Evict or delete the evictable Pods on that node
3. Let controllers and the scheduler place replacements on other schedulable nodes

A typical command looks like this:

kubectl drain node-1 --ignore-daemonsets --delete-emptydir-data

The crucial point is that kubectl drain initiates removal. It does not issue a separate "create new Pod now" command.

So Which Happens First?

From a control-flow perspective, drain starts by evicting the existing Pod. But from an observed cluster-state perspective, replacement creation can overlap with old-Pod termination.

For example, if a Deployment manages three replicas and one Pod is evicted:

• The old Pod enters termination on the drained node
• The ReplicaSet notices the desired count is no longer satisfied
• A replacement Pod can be scheduled on another node while the old one is still terminating

That means the practical answer is:

• Drain initiates removal first
• Replacement is controller-driven
• The visible lifecycle can overlap

So the question is not purely "remove first or create first." The better answer is "drain starts eviction, and the controller may create the replacement before the old Pod fully disappears."

PodDisruptionBudgets and Graceful Shutdown Matter

kubectl drain respects PodDisruptionBudgets and graceful termination periods. If a workload says at least some number of replicas must stay available, drain may pause or fail until that condition can be satisfied.

This behavior is why drain is safer than just deleting the node or force-deleting Pods. It coordinates with Kubernetes' disruption and reconciliation model instead of bypassing it.

Example Mental Model with a Deployment

Suppose you have a Deployment with two replicas on a two-node cluster:

before drain:
  node-a -> web-1
  node-b -> web-2

You drain node-a:

kubectl drain node-a --ignore-daemonsets

Possible sequence:

1step 1: node-a becomes unschedulable
2step 2: web-1 begins eviction and termination
3step 3: ReplicaSet wants another healthy replica
4step 4: scheduler places replacement on node-b or another node

Depending on timing, you may briefly see the terminating Pod and the new Pod both existing in the API at the same time. That does not mean drain created the new one directly.

Common Pitfalls

The biggest mistake is assuming kubectl drain performs rolling-update logic by itself. It does not. Rolling behavior comes from workload controllers plus scheduler decisions.

Another issue is ignoring PodDisruptionBudgets and then being surprised when drain blocks or fails. Drain is designed to be conservative when evictions would violate availability rules.

Developers also sometimes forget that DaemonSet Pods are handled differently. Drain normally requires --ignore-daemonsets because those Pods are expected to exist on nodes by design and are not managed like ordinary replica-based application Pods.

Finally, watch out for workloads without controllers. If a standalone Pod is drained and nothing manages its desired state, no replacement appears anywhere else.

Summary

• 'kubectl drain marks a node unschedulable and evicts Pods from it.'
• Replacement Pods are created by controllers such as Deployments or StatefulSets, not by drain itself.
• Eviction begins first, but replacement creation can overlap with old-Pod termination.
• PodDisruptionBudgets and graceful shutdown rules affect the timing and success of drain.
• If a Pod has no controller, draining it does not automatically create a replacement.