Node Taints
Terraform
Amazon EKS
Kubernetes
Cloud Infrastructure

How to set node taints using Terraform for Amazon EKS

System Design practice on Codemia

Work through 120+ system design problems with detailed solutions, from rate limiters to multi-region storage.

Practice system design

Introduction

In EKS, node taints are the cluster-side mechanism for reserving certain nodes for specific workloads. Terraform can manage those taints declaratively, which is much safer than adding them by hand with kubectl and hoping they stay aligned with the node group definition. The key is to attach the taints at the node-group layer and then add matching tolerations to the pods that are allowed onto those nodes.

What a Taint Does

A taint tells the scheduler that pods should stay off a node unless they explicitly tolerate the taint. A taint has three parts:

  • key
  • value
  • effect

The common effects are:

  • 'NO_SCHEDULE'
  • 'PREFER_NO_SCHEDULE'
  • 'NO_EXECUTE'

The effect determines how strongly Kubernetes enforces the rule. For most reservation use cases in EKS managed node groups, NO_SCHEDULE is the normal choice.

Terraform on an EKS Managed Node Group

With the AWS Terraform provider, an aws_eks_node_group resource can declare taints directly. A minimal example looks like this:

hcl
1resource "aws_eks_node_group" "gpu" {
2  cluster_name    = aws_eks_cluster.main.name
3  node_group_name = "gpu-workers"
4  node_role_arn   = aws_iam_role.eks_nodes.arn
5  subnet_ids      = aws_subnet.private[*].id
6
7  scaling_config {
8    desired_size = 2
9    min_size     = 1
10    max_size     = 4
11  }
12
13  taint {
14    key    = "workload"
15    value  = "gpu"
16    effect = "NO_SCHEDULE"
17  }
18}

This declares that nodes in the managed node group should reject pods unless those pods tolerate workload=gpu with the NO_SCHEDULE effect.

Match the Pod With a Toleration

A taint does nothing useful until the workload side is configured correctly. A pod that is allowed onto the tainted nodes needs a matching toleration.

yaml
1apiVersion: v1
2kind: Pod
3metadata:
4  name: gpu-job
5spec:
6  tolerations:
7    - key: workload
8      operator: Equal
9      value: gpu
10      effect: NoSchedule
11  containers:
12    - name: app
13      image: public.ecr.aws/docker/library/busybox:latest
14      command: ["sh", "-c", "sleep 3600"]

Notice the effect spelling difference: Terraform follows the EKS API style with NO_SCHEDULE, while Kubernetes YAML uses NoSchedule.

Taints Are Only Half of Placement

If you want a workload to prefer those nodes rather than merely tolerate them, combine the toleration with a selector or affinity rule. Otherwise, the pod may still land on any other node that also satisfies its requirements.

A simple deployment example uses both:

yaml
1apiVersion: apps/v1
2kind: Deployment
3metadata:
4  name: gpu-workers
5spec:
6  replicas: 1
7  selector:
8    matchLabels:
9      app: gpu-workers
10  template:
11    metadata:
12      labels:
13        app: gpu-workers
14    spec:
15      nodeSelector:
16        eks.amazonaws.com/nodegroup: gpu-workers
17      tolerations:
18        - key: workload
19          operator: Equal
20          value: gpu
21          effect: NoSchedule
22      containers:
23        - name: app
24          image: public.ecr.aws/docker/library/nginx:stable

The toleration allows placement on the tainted node. The selector makes that placement intentional.

Managed Versus Self-Managed Nodes

This article is easiest when the cluster uses EKS managed node groups. If you use self-managed nodes or a custom bootstrap process, you can still apply taints, but the mechanism is different. In those setups, kubectl taint nodes ... is not enough by itself because new nodes may join without the desired taint unless the bootstrap logic applies it consistently.

So for Terraform-managed infrastructure, the cleanest answer is:

  • managed node group: define taints in the node-group resource
  • self-managed nodes: bake the taint into bootstrap or node configuration

Operational Advice

A small number of intentional taints is usually enough. Common examples are:

  • GPU nodes reserved for GPU workloads
  • system nodes reserved for cluster add-ons
  • spot nodes reserved for disruption-tolerant jobs

If every node group gets multiple taints, scheduling becomes harder to reason about. Keep the rules narrow and aligned with real placement constraints.

Common Pitfalls

The biggest mistake is adding a taint in Terraform and forgetting the matching pod toleration. The nodes become available, but the workloads never schedule.

Another mistake is relying only on tolerations. A toleration permits a pod to use the node, but it does not force that placement. Add nodeSelector or affinity if the workload should actually target that node group.

Teams also sometimes mix effect names between APIs. In Terraform for EKS managed node groups, use values such as NO_SCHEDULE; in Kubernetes manifests, use NoSchedule.

Finally, avoid manual kubectl taint changes on nodes that are managed by Terraform. They drift away from your declared infrastructure state and can disappear when nodes are replaced.

Summary

  • Terraform can define taints directly on EKS managed node groups.
  • Pods need matching tolerations before they can land on tainted nodes.
  • Use selectors or affinity in addition to tolerations when placement should be explicit.
  • Keep taint usage narrow and tied to real workload separation needs.
  • Avoid one-off manual taint changes that Terraform does not manage.

Related reading
Course
Beginner
27 lessons
10 hours
System Design Fundamentals

Build a strong foundation in designing scalable, reliable distributed systems.

View the course
Track what you have practised

A free account saves your progress, solutions and study plan across every problem on Codemia.

System Design practice on Codemia

Work through 120+ system design problems with detailed solutions, from rate limiters to multi-region storage.

Practice system design

All Rights Reserved.