DNS problem on AWS EKS when running in private subnets
System Design practice on Codemia
Work through 120+ system design problems with detailed solutions, from rate limiters to multi-region storage.
Understanding DNS Issues in AWS EKS with Private Subnets
Amazon Elastic Kubernetes Service (EKS) is a managed container service that allows you to run and scale Kubernetes applications in the cloud. When deploying EKS in AWS, particularly in a private subnet environment, you may encounter DNS resolution problems. This article delves into the potential DNS issues experienced when running EKS in private subnets, presents technical explanations, examples, and offers solutions.
Introduction to AWS EKS and Private Subnets
AWS EKS abstracts the Kubernetes control plane setup and management, allowing users to focus on running their containers. When deploying an EKS Cluster into a Virtual Private Cloud (VPC), resources can be launched in public, private, or hybrid subnets. Private subnets are commonly used for security reasons, but they have their unique challenges, especially related to network connectivity and DNS resolution.
DNS Resolution in AWS EKS
In Kubernetes clusters, CoreDNS is the primary DNS server, handling domain name resolution for service discovery. When running EKS in private subnets, DNS resolution must be appropriately configured to ensure that pods can reach the necessary endpoints, including external APIs, S3, ECR, and other AWS services.
Common DNS Issues
- CoreDNS FQDN Configuration:
- The default CoreDNS setup in EKS may not resolve Fully Qualified Domain Names (FQDN) correctly, especially for services hosted outside the cluster.
- DNS Traffic Not Routed:
- Traffic from nodes in private subnets to Amazon-provided DNS servers might not be routed because of misconfigured route tables or missing NAT Gateway configurations.
- VPC DHCP Option Set Misconfiguration:
- Incorrect settings in the VPC's DHCP option set can lead to DNS resolution failures, where nodes and pods cannot resolve needed external URLs.
- Network ACLs:
- Network Access Control Lists (ACLs) might block UDP packets, which are primarily used for DNS queries.
Technical Insights and Solutions
Verifying CoreDNS Configuration
Check the CoreDNS configuration in your EKS cluster using:
Related reading
- Do cross-partition queries break infinite CosmosDB horizontal scalability?
- Do deleted Amazon S3 Bucket names become available for re-use
- Do I have to explicitly create Persistent Volume when I am using Persistent Volume Claim?
- do we really need port for a headless service?
- DNS resolve problem in kubernetes cluster
- DNS Server Load Balancing
- Do you get charged for a 'stopped' instance on EC2?
- Do you recommend any good book about distributed systems and cloud computing?
System Design Fundamentals
Build a strong foundation in designing scalable, reliable distributed systems.
View the courseTrack what you have practised
A free account saves your progress, solutions and study plan across every problem on Codemia.
System Design practice on Codemia
Work through 120+ system design problems with detailed solutions, from rate limiters to multi-region storage.