Kubernetes is a powerful open-source container orchestration platform that automates the deployment, scaling, and management of containerized applications. When running Kubernetes on AWS, one of the common integration patterns is the use of AWS Network Load Balancers (NLBs) to distribute traffic across pods and provide external access to services. However, there is a critical issue that many users encounter: the creation of an excessive number of security group rules. This article explores this problem in detail, providing technical explanations, examples, and potential solutions.

Understanding AWS NLB and Kubernetes Integration

AWS Network Load Balancer (NLB)

AWS Network Load Balancer is designed to handle millions of requests per second and can provide low-latency performance with optimal scaling. It operates at the connection level (Layer 4) to route traffic based on IP protocol data. NLBs are commonly used with Kubernetes for their ability to handle high throughput and support for TCP/UDP traffic.

Kubernetes Service Types

In a Kubernetes cluster, a Service is an abstraction that defines a set of pods and a policy to access them. Types of services include:

• ClusterIP: Exposes the service on a cluster-internal IP.
• NodePort: Exposes the service on each Node's IP at a static port.
• LoadBalancer: Creates an external load balancer (or integrates with one) to expose the service.

When a LoadBalancer type service is used, Kubernetes can automatically create an AWS NLB to handle traffic to the Kubernetes pods.

Excessive Security Group Rules

The Problem

When provisioning AWS NLB with Kubernetes, it's common to encounter a rapid increase in security group rules. This is due to the way NLB integrates with Kubernetes, specifically how it handles pod IPs for services.

How It Happens

1. Pod IP Changes: Kubernetes dynamically assigns IPs to pods. When these pods are exposed via an NLB, the corresponding security groups on AWS need to be updated to allow traffic from/to these dynamic pod IPs.
2. Rule Proliferation: For each Service of type LoadBalancer, AWS creates security group rules that allow traffic between NLB and the Kubernetes nodes. Since pod IPs frequently change, this can lead to a large number of rules being created.
3. IP Target Mode: By default, NLB operates in "IP" target mode, which associates security group rules with the pod IP addresses directly.

Impacts

• AWS Security Group Limits: AWS imposes limits on the number of security group rules per group. Excessive creation of rules can quickly exhaust these limits.
• Cluster Performance: An increased number of security group rules can degrade the AWS API performance, impacting the creation and management of resources on AWS.
• Operational Complexity: Managing a large number of security group rules can become operationally challenging, leading to a cluttered security configuration that is difficult to audit or troubleshoot.

Technical Solutions and Best Practices

Disable Source/Destination Checks

One proposed solution to mitigate the excessive security group rules is to disable source/destination checks on the EC2 instances in the Kubernetes cluster. This permits more generic rules and reduces the specific rules required for pod IPs.

Use Helm and Kube2IAM

Modernize IAM management using tools like Kube2IAM and Helm charts. This can minimize unnecessary AWS resource creations and optimize existing configurations.

Alternative Load Balancer Options

Considering other options like using the AWS Application Load Balancer (ALB) or custom Ingress controllers that may manage security group rules more efficiently.

Roll Your Own Security Groups

Custom security groups managed through automation or infrastructure-as-code tools (like Terraform or AWS CloudFormation) might provide more tailor-made solutions that reduce the barrage of rules created by default.

Example Resolution

Consider a scenario where you have an NLB serving a service with dynamically scaling pods:

• protocol: TCP