AWS
EKS
NodeGroup
Kubernetes
Troubleshooting

AWS EKS NodeGroup Create failed Instances failed to join the kubernetes cluster

System Design practice on Codemia

Work through 120+ system design problems with detailed solutions, from rate limiters to multi-region storage.

Practice system design

Introduction

The EKS error NodeCreationFailure or Instances failed to join the kubernetes cluster means the EC2 instances launched successfully, but they never completed the bootstrap path that turns them into Kubernetes worker nodes. The failure is usually not in Auto Scaling itself; it is in networking, IAM, DNS, or node bootstrap configuration.

The fastest way to debug it is to follow the join path in order: can the instance reach the cluster endpoint, can it retrieve cluster details, can kubelet start, and can the control plane recognize the node identity.

What Has to Happen for a Node to Join

An EKS worker node must do several things during startup:

  1. Launch with the correct AMI or compatible bootstrap logic.
  2. Reach AWS APIs and the EKS cluster endpoint.
  3. Use an IAM role that allows node bootstrap operations.
  4. Start kubelet with the correct cluster name and CA data.
  5. Authenticate to the cluster and register itself.

If any step fails, the node group may remain stuck in CREATE_FAILED.

The Most Common Root Causes

The first category is networking. Nodes in private subnets still need outbound access to the services required during bootstrap. That often means a NAT gateway or the right VPC endpoints, plus route tables, security groups, and network ACLs that allow the traffic.

The second category is IAM. The node instance role must include the standard worker policies. If the role is missing permissions, the node cannot discover cluster details or pull container images correctly.

The third category is bootstrap customization. This shows up when teams use a custom AMI or launch template and accidentally override the EKS bootstrap user data. Managed node groups can handle this well, but only if the bootstrap step is still present and correct.

The fourth category is DNS. If the VPC DHCP options or private DNS setup are broken, the instance may be unable to resolve the cluster endpoint or AWS service endpoints.

Check IAM First

For standard Linux worker nodes, the instance role usually needs at least these managed policies attached:

  • 'AmazonEKSWorkerNodePolicy'
  • 'AmazonEC2ContainerRegistryPullOnly or the current ECR pull policy used by your environment'
  • 'AmazonEKS_CNI_Policy for IPv4 setups, unless your networking design attaches it elsewhere'

A quick CLI check:

bash
1aws eks describe-nodegroup \
2  --cluster-name my-cluster \
3  --nodegroup-name my-nodes \
4  --query 'nodegroup.health.issues'
5
6aws iam list-attached-role-policies \
7  --role-name MyEksNodeRole

If you are using a custom launch template, also confirm that the node role attached to the EC2 instances is the same role you expect.

Check Network Reachability

A node cannot join if it cannot reach the control plane or required AWS APIs. In practice, inspect:

  • Subnet route tables
  • Security groups on the node and cluster
  • Private or public endpoint settings on the cluster
  • NAT gateway or VPC endpoint coverage
  • DNS resolution inside the VPC

If the cluster endpoint is private-only, the node subnets must have direct network reachability to that private endpoint. If the nodes are in isolated subnets with no NAT and no required VPC endpoints, bootstrap often fails silently until you inspect instance logs.

Read the Instance Logs

Do not guess. SSH into a failed node if possible, or use SSM, then inspect the startup logs:

bash
sudo cat /var/log/cloud-init-output.log
sudo journalctl -u kubelet -xe
sudo systemctl status kubelet

These logs usually reveal whether the bootstrap script failed, kubelet could not resolve the endpoint, or authentication never succeeded.

If you use a custom AMI, verify that your user data still runs the EKS bootstrap logic. A common mistake is replacing the default script with unrelated initialization commands and never starting kubelet correctly.

Verify Cluster and Node Group Configuration

A few configuration mismatches also cause join failures:

  • Wrong cluster name passed to the bootstrap script
  • Kubernetes version mismatch between the node AMI and the cluster
  • Wrong subnet selection for the node group
  • Missing outbound rules for HTTPS and DNS
  • Custom launch template settings that override working defaults

For managed node groups using the EKS-optimized AMI, staying close to the default bootstrap path is usually safest. The more customization you add, the more carefully you need to verify it.

A Practical Triage Flow

Use a short, repeatable checklist:

bash
aws eks describe-cluster --name my-cluster --query 'cluster.resourcesVpcConfig'
aws eks describe-nodegroup --cluster-name my-cluster --nodegroup-name my-nodes
kubectl get nodes

Then answer these questions in order:

  1. Did the EC2 instances launch?
  2. Do they have the expected IAM role?
  3. Can they resolve and reach the cluster endpoint?
  4. Did the bootstrap script run successfully?
  5. Is kubelet running and trying to register?

That sequence is much faster than changing several settings at once and hoping one helps.

Common Pitfalls

The most common mistake is focusing only on EKS and ignoring plain EC2 networking. Most node join failures are standard infrastructure failures wearing a Kubernetes error message.

Another common issue is custom launch templates. Teams add their own AMI, user data, or security groups and unintentionally remove a required bootstrap assumption.

People also overlook DNS. The instance may have route-table access and still fail because it cannot resolve the endpoint names it needs during startup.

Finally, do not assume a managed node group means zero bootstrap responsibility. Managed means EKS manages the node group lifecycle, not that every custom instance setting becomes automatically valid.

Summary

  • 'Instances failed to join the kubernetes cluster means the EC2 node launched but did not complete bootstrap and registration.'
  • Check IAM, networking, DNS, and bootstrap logs before changing cluster settings.
  • Custom launch templates and AMIs are a common source of broken node joins.
  • Inspect /var/log/cloud-init-output.log and kubelet logs on a failed instance.
  • Debug the join path step by step: launch, reachability, bootstrap, kubelet, registration.

Related reading
Course
Beginner
27 lessons
10 hours
System Design Fundamentals

Build a strong foundation in designing scalable, reliable distributed systems.

View the course
Track what you have practised

A free account saves your progress, solutions and study plan across every problem on Codemia.

System Design practice on Codemia

Work through 120+ system design problems with detailed solutions, from rate limiters to multi-region storage.

Practice system design