When deploying an Amazon EKS (Elastic Kubernetes Service) cluster, encountering a "Create failed" error due to instances not joining the Kubernetes cluster can be a frustrating experience. This issue arises during the creation of an EKS Managed Node Group, where the EC2 instances that should comprise the nodes of the Kubernetes cluster fail to successfully connect to the Master Control Plane. In this article, we delve into potential causes and solutions for this problem, providing technical insights and examples to guide you in resolving the issue.

Understanding the EKS Node Group

Amazon EKS automates much of the setup required to run a Kubernetes cluster in AWS, including tasks such as launching and managing instances as nodes that join the cluster. When creating a Node Group, AWS spins up EC2 instances that should automatically register with the EKS Control Plane as nodes.

Key Components Involved:

1. Amazon EKS Control Plane: The managed Kubernetes control plane, consisting of multiple components like API server, etcd, etc.
2. EC2 Instances: Physical or virtual servers running the node agent (kubelet).
3. Node Group: A group of EC2 instances within EKS that conforms to certain configurations and scaling properties.

Common Causes of "Create Failed"

1. Networking and Security Issues

One of the primary reasons that can lead to the failure of instances joining the Kubernetes cluster is networking misconfigurations. Here are some potential issues:

• VPC and Subnet Misconfiguration: Ensure that the VPC and subnets are correctly set up. The subnets should have proper routing to allow the EC2 nodes to communicate with EKS control plane securely.
• Security Group Rules: Check the security groups attached to the instances. They should permit inbound traffic to the required ports for the Kubernetes API (typically 443) and allow outbound traffic for node traffic.

2. IAM Role Permissions

Proper IAM role and policy configurations are crucial. Without adequate permissions, nodes might not join due to authentication failures.

• Node Role: The EC2 instances require an IAM role with permissions to register with the EKS control plane. Ensure `AmazonEKSWorkerNodePolicy`, `AmazonEC2ContainerRegistryReadOnly`, and `AmazonEKS_CNI_Policy` are attached to the IAM role.
• Policy Misattachment: Verify that the appropriate instance profile associated with a role is attached to the worker nodes.

3. Bootstrap Script Errors

When spinning up instances, EKS utilizes a bootstrap script that helps configure and authenticate the node with the cluster.

• Bootstrap Arguments: The script may require specific parameters such as `--apiserver-endpoint` and `--b64-cluster-ca`, which are critical for successful cluster connection.
• Script Execution: Errors during execution, such as incorrect syntax or missing variables, can prevent the node from joining.

4. DNS Configuration

DNS resolution is vital for node registration and Kubernetes service discovery.

• EC2 Instance Resolvability: Make sure the DNS server is correctly set in the VPC and instances. Enable DNS support for the VPC.

Troubleshooting Steps

1. Check EKS Console and Logs
   Navigate to the EKS console to review logs and events associated with the node group creation process. CloudTrail and CloudWatch logs provide insights into possible reasons for failure.
2. Review Network Configurations
   • Examine the subnet association with the EKS cluster for proper CIDR ranges and availability zones.
   • Modify security group configurations to ensure necessary traffic is allowed.
3. Validate IAM Roles
   • Confirm that the IAM roles have necessary permissions.
   • Re-evaluate role attachment for misconfigurations.
4. Examine Bootstrap Script
   • Verify the EKS bootstrap script parameters and execution logs on EC2 instances.
   • Check for errors related to API server endpoint URLs and CA certificates.
5. Modify DNS and Instance Settings
   • Ensure that VPC-level DNS settings allow instances to resolve external and cluster domain names.

Summary Table

By methodically addressing each of these components, you can effectively diagnose and resolve the "Create failed" error when instances fail to join the EKS Kubernetes cluster. With proper configuration and diligent troubleshooting, leveraging Amazon EKS to manage your Kubernetes workloads becomes a seamless process.