Introduction

Spark 2.3 was the first Spark release with Kubernetes support, and that support was explicitly early-stage. Because of that, many submission failures are not one mysterious bug but one of a small set of environment mismatches: wrong cluster URL, missing container image, bad RBAC, inaccessible application jars, or driver pods that never become healthy. The right way to debug it is to treat spark-submit as a Kubernetes job launcher and inspect the driver pod first.

Start With the Submission Model

For Spark 2.3 on Kubernetes, the driver runs as a pod. Executors are then created by that driver.

A typical submission command looks like this:

1./bin/spark-submit \
2  --master k8s://https://my-k8s-api:6443 \
3  --deploy-mode cluster \
4  --name spark-pi \
5  --class org.apache.spark.examples.SparkPi \
6  --conf spark.executor.instances=2 \
7  --conf spark.kubernetes.container.image=my-registry/spark:2.3.0 \
8  local:///opt/spark/examples/jars/spark-examples_2.11-2.3.0.jar

That command already hints at common failure points:

• '--master must point to the Kubernetes API correctly'
• the image must exist and be pullable
• the application jar path must make sense for the driver container
• cluster mode must have permissions to create executor pods

Inspect the Driver Pod First

If submission fails, find the driver pod and inspect it.

kubectl get pods -n default
kubectl describe pod <driver-pod-name> -n default
kubectl logs <driver-pod-name> -n default

This is the fastest way to distinguish among:

• scheduling failure
• image pull failure
• permissions failure
• application startup failure
• Spark configuration error

Without the driver pod logs, spark-submit output is often too indirect.

Common Failure: Image Pull Problems

One of the most common 2.3-era failures is that the driver or executor image cannot be pulled.

Typical causes:

• wrong image name or tag
• private registry credentials not configured
• image built without Spark or your app jar inside it

If the app uses a local:///... jar path, that jar must exist inside the container image. local:/// refers to the container filesystem, not your laptop.

Common Failure: RBAC and Service Accounts

The Spark driver needs permission to create executor pods and related Kubernetes resources. If RBAC is missing, driver logs often show authorization failures.

A service account example:

1apiVersion: v1
2kind: ServiceAccount
3metadata:
4  name: spark

And in the submission command:

--conf spark.kubernetes.authenticate.driver.serviceAccountName=spark

If Spark cannot create executors, check RBAC before changing Spark-level settings randomly.

Common Failure: Wrong Resource Paths

Spark-on-Kubernetes setup often fails because the application resource path is wrong for the chosen deployment model.

For example:

• 'local:///path/to.jar means the jar is already inside the image'
• remote storage paths need to be reachable from the driver
• a local workstation path is not visible inside the pod

This mistake is especially common when a submission works in local mode and then fails on Kubernetes.

Networking and DNS Issues

Once the driver starts, executors must connect back correctly. Early Spark-on-Kubernetes setups were sensitive to cluster DNS, service discovery, and networking assumptions.

If the driver launches but executors fail repeatedly, inspect:

kubectl get pods
kubectl logs <executor-pod-name>

Look for:

• DNS resolution errors
• connection refusals
• pod-to-pod network restrictions
• missing service exposure for driver communication

A Reality Check About Spark 2.3

Spark 2.3 Kubernetes support was the beginning, not the polished end state. If you are debugging a real production issue on 2.3 today, one honest answer is that upgrading Spark may remove entire classes of early integration pain.

That does not help if you are locked to 2.3, but it matters for planning. Some failures are easier to work around than to perfect on that vintage of the stack.

Common Pitfalls

The most common mistake is debugging spark-submit output without checking the driver pod logs. The real failure is usually inside the pod.

Another mistake is using local:/// for a jar that is not actually inside the container image.

Teams also often forget RBAC and assume Spark itself is broken when the driver simply lacks permission to create executor pods.

Finally, remember that Spark 2.3 Kubernetes support was early and somewhat brittle. Be careful about assuming later-version examples apply unchanged.

Summary

• Treat Spark 2.3 on Kubernetes as driver-pod-first debugging.
• Verify the Kubernetes API URL, image, service account, and application jar path first.
• Use kubectl describe and kubectl logs on the driver pod to find the real error.
• 'local:/// paths refer to files inside the container image, not local disk.'
• If possible, remember that some 2.3-era issues are limitations of the old integration itself.