Introduction

Spark on Kubernetes often needs scheduling control so driver and executor pods land on the right nodes. Taints and tolerations are the standard way to isolate workloads such as batch jobs, GPU tasks, or cost-optimized pools. This guide shows how to apply tolerations through Spark pod templates and verify that scheduler behavior is correct.

Core Topic Sections

Why tolerations matter for Spark workloads

In many clusters, nodes are segmented by taints:

1. Dedicated analytics nodes.
2. Spot or preemptible node pools.
3. GPU nodes.

Without matching tolerations, Spark pods can stay pending or run on unintended nodes. Tolerations are necessary but not always sufficient, since affinity and resource requests also influence placement.

Spark pod template basics

Spark supports custom pod specs through template files. You can provide separate templates for:

1. Driver pod.
2. Executor pods.

Templates let you inject Kubernetes-level fields not directly exposed by Spark flags.

Driver template example:

1apiVersion: v1
2kind: Pod
3metadata:
4  labels:
5    app: spark-driver
6spec:
7  tolerations:
8    - key: "workload"
9      operator: "Equal"
10      value: "spark"
11      effect: "NoSchedule"

Executor template example:

1apiVersion: v1
2kind: Pod
3metadata:
4  labels:
5    app: spark-executor
6spec:
7  tolerations:
8    - key: "workload"
9      operator: "Equal"
10      value: "spark"
11      effect: "NoSchedule"

Submit Spark job with template references

Use Spark submit configuration keys to point at templates.

1spark-submit \
2  --master k8s://https://kubernetes.default.svc \
3  --deploy-mode cluster \
4  --name spark-etl \
5  --class com.example.Main \
6  --conf spark.kubernetes.namespace=data \
7  --conf spark.kubernetes.container.image=example/spark:3.5.1 \
8  --conf spark.kubernetes.driver.podTemplateFile=/opt/spark/driver-template.yaml \
9  --conf spark.kubernetes.executor.podTemplateFile=/opt/spark/executor-template.yaml \
10  local:///opt/spark/app.jar

Ensure template paths are accessible in submission environment.

Combine tolerations with node selection

Tolerations only allow scheduling onto tainted nodes. They do not force it. To prefer specific pools, combine with node selectors or affinity.

1spec:
2  nodeSelector:
3    nodepool: spark-workers
4  tolerations:
5    - key: "workload"
6      operator: "Equal"
7      value: "spark"
8      effect: "NoSchedule"

This produces predictable placement and reduces noisy-neighbor effects.

Handle preemptible node pools deliberately

If you use preemptible nodes for executors, keep driver on stable nodes in many workloads.

Practical pattern:

1. Driver template without preemptible toleration.
2. Executor template with preemptible toleration.
3. Spark dynamic allocation enabled with retry strategy.

This balances cost and reliability.

Validate scheduling behavior after deployment

Check pod placement and events, not only Spark logs.

kubectl get pods -n data -l spark-role=driver -o wide
kubectl get pods -n data -l spark-role=executor -o wide
kubectl describe pod <executor-pod> -n data

Look for scheduler event messages about taints, tolerations, and node fit failures.

Troubleshooting common pending states

If pods stay pending:

1. Toleration key, value, or effect mismatch.
2. Missing resources on tolerated nodes.
3. Affinity conflicts with available nodes.
4. Template not applied due to wrong path or key.

Start with kubectl describe pod scheduler events, they usually identify the exact mismatch.

Governance and maintainability

Keep templates in version control and review them like code. For multi-team clusters, standardize toleration keys and values to avoid conflicting conventions.

A shared convention document for taints and tolerations prevents random scheduling drift across Spark jobs.

Common Pitfalls

• Adding tolerations and assuming Spark pods will prefer those nodes automatically.
• Applying only driver tolerations and forgetting executor template configuration.
• Using wrong effect string or mismatched key and value against node taints.
• Relying on templates without verifying they are actually loaded by submission config.
• Ignoring scheduler event logs and debugging only Spark application logs.

Summary

• Pod templates are the practical way to add tolerations to Spark driver and executors.
• Tolerations allow scheduling on tainted nodes but do not guarantee node preference.
• Combine tolerations with selectors or affinity for deterministic placement.
• Validate behavior through Kubernetes scheduling events and pod placement checks.
• Treat template configuration as versioned infrastructure policy, not one-off flags.