AWS EKS NodeGroup Create failed Instances failed to join the kubernetes cluster
System Design practice on Codemia
Work through 120+ system design problems with detailed solutions, from rate limiters to multi-region storage.
Introduction
The EKS error NodeCreationFailure or Instances failed to join the kubernetes cluster means the EC2 instances launched successfully, but they never completed the bootstrap path that turns them into Kubernetes worker nodes. The failure is usually not in Auto Scaling itself; it is in networking, IAM, DNS, or node bootstrap configuration.
The fastest way to debug it is to follow the join path in order: can the instance reach the cluster endpoint, can it retrieve cluster details, can kubelet start, and can the control plane recognize the node identity.
What Has to Happen for a Node to Join
An EKS worker node must do several things during startup:
- Launch with the correct AMI or compatible bootstrap logic.
- Reach AWS APIs and the EKS cluster endpoint.
- Use an IAM role that allows node bootstrap operations.
- Start kubelet with the correct cluster name and CA data.
- Authenticate to the cluster and register itself.
If any step fails, the node group may remain stuck in CREATE_FAILED.
The Most Common Root Causes
The first category is networking. Nodes in private subnets still need outbound access to the services required during bootstrap. That often means a NAT gateway or the right VPC endpoints, plus route tables, security groups, and network ACLs that allow the traffic.
The second category is IAM. The node instance role must include the standard worker policies. If the role is missing permissions, the node cannot discover cluster details or pull container images correctly.
The third category is bootstrap customization. This shows up when teams use a custom AMI or launch template and accidentally override the EKS bootstrap user data. Managed node groups can handle this well, but only if the bootstrap step is still present and correct.
The fourth category is DNS. If the VPC DHCP options or private DNS setup are broken, the instance may be unable to resolve the cluster endpoint or AWS service endpoints.
Check IAM First
For standard Linux worker nodes, the instance role usually needs at least these managed policies attached:
- '
AmazonEKSWorkerNodePolicy' - '
AmazonEC2ContainerRegistryPullOnlyor the current ECR pull policy used by your environment' - '
AmazonEKS_CNI_Policyfor IPv4 setups, unless your networking design attaches it elsewhere'
A quick CLI check:
If you are using a custom launch template, also confirm that the node role attached to the EC2 instances is the same role you expect.
Check Network Reachability
A node cannot join if it cannot reach the control plane or required AWS APIs. In practice, inspect:
- Subnet route tables
- Security groups on the node and cluster
- Private or public endpoint settings on the cluster
- NAT gateway or VPC endpoint coverage
- DNS resolution inside the VPC
If the cluster endpoint is private-only, the node subnets must have direct network reachability to that private endpoint. If the nodes are in isolated subnets with no NAT and no required VPC endpoints, bootstrap often fails silently until you inspect instance logs.
Read the Instance Logs
Do not guess. SSH into a failed node if possible, or use SSM, then inspect the startup logs:
These logs usually reveal whether the bootstrap script failed, kubelet could not resolve the endpoint, or authentication never succeeded.
If you use a custom AMI, verify that your user data still runs the EKS bootstrap logic. A common mistake is replacing the default script with unrelated initialization commands and never starting kubelet correctly.
Verify Cluster and Node Group Configuration
A few configuration mismatches also cause join failures:
- Wrong cluster name passed to the bootstrap script
- Kubernetes version mismatch between the node AMI and the cluster
- Wrong subnet selection for the node group
- Missing outbound rules for HTTPS and DNS
- Custom launch template settings that override working defaults
For managed node groups using the EKS-optimized AMI, staying close to the default bootstrap path is usually safest. The more customization you add, the more carefully you need to verify it.
A Practical Triage Flow
Use a short, repeatable checklist:
Then answer these questions in order:
- Did the EC2 instances launch?
- Do they have the expected IAM role?
- Can they resolve and reach the cluster endpoint?
- Did the bootstrap script run successfully?
- Is kubelet running and trying to register?
That sequence is much faster than changing several settings at once and hoping one helps.
Common Pitfalls
The most common mistake is focusing only on EKS and ignoring plain EC2 networking. Most node join failures are standard infrastructure failures wearing a Kubernetes error message.
Another common issue is custom launch templates. Teams add their own AMI, user data, or security groups and unintentionally remove a required bootstrap assumption.
People also overlook DNS. The instance may have route-table access and still fail because it cannot resolve the endpoint names it needs during startup.
Finally, do not assume a managed node group means zero bootstrap responsibility. Managed means EKS manages the node group lifecycle, not that every custom instance setting becomes automatically valid.
Summary
- '
Instances failed to join the kubernetes clustermeans the EC2 node launched but did not complete bootstrap and registration.' - Check IAM, networking, DNS, and bootstrap logs before changing cluster settings.
- Custom launch templates and AMIs are a common source of broken node joins.
- Inspect
/var/log/cloud-init-output.logand kubelet logs on a failed instance. - Debug the join path step by step: launch, reachability, bootstrap, kubelet, registration.
Related reading
- AWS Nginx Ingress creating Classic Load Balancer instead of Application Load Balancer
- Azure Functions Kubernetes cannot find local.settings.json
- Azure Kubernetes Service Setup an Internal load balancer with static IP address
- Azure Kubernetes TLS handshake timeout
- AWS elastic-search. FORBIDDEN/8/index write api. Unable to write to index
- AWS Elastic Beanstalk - Increase Instance Disk Capacity
- AWS error - sudo unable to resolve host ip-10-0-xx-xx
- AWS error downloading object from S3, profile file cannot be null

System Design Fundamentals
Build a strong foundation in designing scalable, reliable distributed systems.
View the courseTrack what you have practised
A free account saves your progress, solutions and study plan across every problem on Codemia.
System Design practice on Codemia
Work through 120+ system design problems with detailed solutions, from rate limiters to multi-region storage.