Introduction

The best way to run Airflow tasks on Kubernetes depends on what you mean by "run Airflow on Kubernetes." There are really two decisions: how to deploy the Airflow control plane itself, and how each task should execute. For most teams, the practical answer is to deploy Airflow with the official Helm chart and choose either KubernetesExecutor or KubernetesPodOperator based on how much per-task isolation and image customization you need.

Deploy the Airflow Control Plane Cleanly

For the Airflow webserver, scheduler, triggerer, and metadata database integration, the official Helm chart is usually the simplest path. It gives you a structured way to manage configuration, secrets, ingress, and resource requests.

A minimal values snippet might look like this:

1executor: KubernetesExecutor
2
3webserver:
4  replicas: 1
5
6scheduler:
7  replicas: 1
8
9workers:
10  enabled: false

This is not a full production file, but it shows the direction: let Kubernetes manage the Airflow components, and let Airflow manage task launching.

Use KubernetesExecutor for Per-Task Pods

KubernetesExecutor is a strong choice when you want each task instance to run in its own Pod. That gives you:

• Strong isolation between tasks
• Per-task resource requests and limits
• Natural cleanup after task completion
• Good alignment with Kubernetes-native operations

It works especially well when your tasks are already container-friendly and do not require a long-lived worker fleet.

Use KubernetesPodOperator for Custom One-Off Task Images

Even if your Airflow deployment uses another executor, KubernetesPodOperator is useful when a specific task should run in a custom container image with explicit Kubernetes settings.

Example DAG task:

1from airflow import DAG
2from airflow.providers.cncf.kubernetes.operators.pod import KubernetesPodOperator
3from pendulum import datetime
4
5with DAG(
6    dag_id="example_kpo",
7    start_date=datetime(2024, 1, 1),
8    schedule=None,
9    catchup=False,
10) as dag:
11    run_job = KubernetesPodOperator(
12        task_id="run_job",
13        name="run-job",
14        namespace="airflow",
15        image="python:3.12-slim",
16        cmds=["python", "-c"],
17        arguments=["print('hello from KubernetesPodOperator')"],
18        is_delete_operator_pod=True,
19        get_logs=True,
20    )

This is great when different tasks need different runtimes, such as one task needing Python data libraries and another needing a Java or shell-based tool image.

Choose Based on Task Shape

A practical rule is:

• Use KubernetesExecutor when most tasks should run as isolated Pods automatically
• Use KubernetesPodOperator when specific tasks need explicit Pod-level control
• Use CeleryExecutor only if you specifically want persistent workers and already understand the tradeoffs

For many Kubernetes-first teams, KubernetesExecutor plus KubernetesPodOperator for special cases is the cleanest model.

Operate It Like a Platform, Not a Laptop App

Airflow on Kubernetes still needs:

• External metadata database
• Secret management
• Log persistence or aggregation
• Proper resource requests and limits
• Node-level capacity planning

The common anti-pattern is getting Airflow to start and then treating every task as if the cluster has infinite room. Airflow can queue work much faster than the cluster can actually schedule and execute it.

Common Pitfalls

The biggest mistake is choosing an executor before understanding the workload. If every task needs a distinct image and runtime, forcing everything through generic long-lived workers creates friction.

Another issue is underestimating startup overhead. Pod-per-task execution gives excellent isolation, but it also introduces Pod scheduling latency. For very tiny short-lived tasks, that overhead may matter.

Teams also often forget shared dependencies such as secrets, network access, and logging. A task Pod that starts successfully but cannot reach its database or object store is still operationally broken.

Finally, do not overload one cluster blindly. Airflow parallelism settings, DAG design, and Kubernetes resource quotas must agree with each other or you will create backlogs and noisy failure patterns.

Summary

• Deploy Airflow itself cleanly on Kubernetes, usually with the official Helm chart.
• 'KubernetesExecutor is a strong default for Pod-per-task execution.'
• 'KubernetesPodOperator is ideal for tasks that need custom images or explicit Pod settings.'
• Choose executor strategy based on workload shape, not fashion.
• Success depends on databases, secrets, logs, and cluster capacity as much as on the executor choice.