Understanding Pod Limits on an AWS EKS Node

Amazon Elastic Kubernetes Service (EKS) is a popular choice for orchestrating containerized applications using Kubernetes on AWS. One key aspect of managing EKS clusters is understanding the limit on the number of Pods that can be scheduled on a Kubernetes Node. This article delves into the technical aspects of Pod limits on EKS Nodes, explores various factors affecting these limits, and provides insights on optimizing your workloads.

Pod Limit Per Node

In Kubernetes, a cluster consists of Nodes, and Pods are the smallest deployable units that can be scheduled on these Nodes. AWS EKS, like other Kubernetes environments, has a specified limit for the number of Pods that can be run on each Node. This limit is not arbitrary but is determined by a combination of factors, including instance type, network interfaces, and the number of IP addresses available from those interfaces.

Understanding the Limitation Factors

1. Instance Type: • Each AWS EC2 instance type supports a different number of elastic network interfaces (ENIs) and IP addresses per ENI, which directly impacts the Pod limit. • For example, a `t3.medium` EC2 instance supports 3 ENIs with a maximum of 6 IP addresses per ENI. Thus, it can handle up to 17 Pods: `1` (for the Node itself) + `16` (from the IP addresses provisioned for ENIs).
2. Elastic Network Interfaces (ENIs) and IP Addresses: • ENIs provide the network interfaces within VPC, and each ENI can have multiple secondary IP addresses. Each secondary IP address can be used to allocate an IP to a Pod. • The Pod limit for each instance type is determined by the following formula:

$$\text{Pod Limit} = (\text{ENI Count} \times \text{IP per ENI}) - 1$$

• The `-1` is to reserve an IP address for the primary network interface of the Node.

1. VPC CNI Plugin: • AWS uses the VPC CNI (Container Network Interface) plugin for networking in EKS. It's the component responsible for configuring network interfaces for pods. It's essential to configure this plugin properly as it governs the allocation of IP addresses to Pods.

Customization and Optimization

• Custom Pod Networking: • In scenarios demanding more Pods than a single Node's capacity, custom Pod networking can be employed. Assigning multiple IPv6 CIDR blocks or dividing large IPv4 blocks among ENIs can be a feasible solution.

• Cluster Scaling: • EKS supports automatic scaling of Nodes, enabling clusters to accommodate more Pods by adding new Nodes when existing ones reach their limit.

• Checking Node and Pod Limits: • You can check the number of Pods a particular Node can handle using the command:

• Node Selector and Affinity: Efficiently allocate Pods across Nodes by utilizing node selectors and affinity rules to enhance scheduling policies. • Networking Best Practices: Keep your networking architecture simple and adopt IP address management strategies to avoid IP exhaustion. • Cluster Autoscaler: Leverage Cluster Autoscaler to dynamically adjust the size of your cluster and ensure efficient resource utilization.