Unable to create spark session

Spark

PySpark

SparkSession

Big Data

Troubleshooting

Unable to create spark session

Master System Design with Codemia

Enhance your system design skills with over 120 practice problems, detailed solutions, and hands-on exercises.

Start Practicing Learn More

Introduction

The "Unable to create SparkSession" error in PySpark typically comes from missing Java/JDK installation, incorrect JAVA_HOME or SPARK_HOME environment variables, version mismatches between PySpark and Java, or port conflicts from previous Spark instances. The fix depends on the specific error message — the most common resolution is installing Java 8 or 11, setting JAVA_HOME, and ensuring PySpark can find the Spark binaries.

Basic SparkSession Creation

python

1from pyspark.sql import SparkSession
2
3spark = SparkSession.builder \
4    .appName("MyApp") \
5    .master("local[*]") \
6    .getOrCreate()
7
8print(spark.version)
9spark.stop()

If this fails, check the error message to identify which fix applies.

Fix 1: Install Java and Set JAVA_HOME

Spark requires Java 8, 11, or 17 (depending on Spark version). The most common error is:

RuntimeError: Java gateway process exited before sending its port number

bash

1# Check if Java is installed
2java -version
3
4# macOS — install via Homebrew
5brew install openjdk@11
6export JAVA_HOME=$(/usr/libexec/java_home -v 11)
7
8# Ubuntu/Debian
9sudo apt install openjdk-11-jdk
10export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64
11
12# Verify
13echo $JAVA_HOME
14java -version

Add to your shell profile (~/.zshrc or ~/.bashrc):

bash

export JAVA_HOME=$(/usr/libexec/java_home -v 11)  # macOS
# export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64  # Linux
export PATH=$JAVA_HOME/bin:$PATH

Fix 2: Set SPARK_HOME

If you installed Spark manually (not via pip):

bash

export SPARK_HOME=/opt/spark
export PATH=$SPARK_HOME/bin:$PATH
export PYSPARK_PYTHON=python3

For PySpark installed via pip, SPARK_HOME is usually set automatically:

bash

# Find where PySpark installed Spark
python3 -c "import pyspark; print(pyspark.__path__)"

Fix 3: Version Compatibility

Spark, Java, and Python versions must be compatible:

Spark Version	Java Versions	Python Versions
3.5.x	8, 11, 17	3.8 - 3.12
3.4.x	8, 11, 17	3.7 - 3.11
3.3.x	8, 11, 17	3.7 - 3.10
3.2.x	8, 11	3.6 - 3.9

bash

1# Check versions
2java -version
3python3 --version
4pip show pyspark

Common mismatch error:

Exception: Java gateway process exited before sending its port number
# Often caused by Java 21+ with older Spark versions

Fix 4: Kill Existing Spark Processes

Previous Spark sessions may hold ports:

bash

1# Find running Spark/Java processes
2ps aux | grep spark
3ps aux | grep java
4
5# Kill lingering processes
6kill -9 $(lsof -t -i:4040)  # Spark UI port
7kill -9 $(lsof -t -i:4041)  # Secondary Spark UI port
8
9# In PySpark, always stop sessions properly
10spark.stop()

Fix 5: Memory Configuration

Insufficient memory allocation causes startup failures:

python

1spark = SparkSession.builder \
2    .appName("MyApp") \
3    .master("local[*]") \
4    .config("spark.driver.memory", "4g") \
5    .config("spark.executor.memory", "4g") \
6    .config("spark.driver.maxResultSize", "2g") \
7    .getOrCreate()

On systems with limited RAM:

python

1spark = SparkSession.builder \
2    .appName("MyApp") \
3    .master("local[2]") \
4    .config("spark.driver.memory", "1g") \
5    .getOrCreate()

Fix 6: Hadoop/WinUtils on Windows

On Windows, Spark requires winutils.exe:

ERROR: Could not find or load main class org.apache.spark.launcher.Main

python

1import os
2os.environ['HADOOP_HOME'] = 'C:\\hadoop'
3# Place winutils.exe in C:\hadoop\bin\winutils.exe
4
5spark = SparkSession.builder \
6    .appName("MyApp") \
7    .master("local[*]") \
8    .config("spark.sql.warehouse.dir", "file:///C:/tmp/spark-warehouse") \
9    .getOrCreate()

Download winutils.exe from the Hadoop winutils repository matching your Hadoop version.

Fix 7: Jupyter Notebook Configuration

SparkSession in Jupyter needs findspark or proper environment:

python

1# Option 1: Using findspark
2import findspark
3findspark.init()
4
5from pyspark.sql import SparkSession
6spark = SparkSession.builder.appName("Jupyter").getOrCreate()
7
8# Option 2: Set environment before importing
9import os
10os.environ['JAVA_HOME'] = '/usr/lib/jvm/java-11-openjdk-amd64'
11os.environ['SPARK_HOME'] = '/opt/spark'
12
13from pyspark.sql import SparkSession
14spark = SparkSession.builder.getOrCreate()

Diagnostic Script

Run this to identify the issue:

python

1import os
2import sys
3
4print(f"Python: {sys.executable} ({sys.version})")
5print(f"JAVA_HOME: {os.environ.get('JAVA_HOME', 'NOT SET')}")
6print(f"SPARK_HOME: {os.environ.get('SPARK_HOME', 'NOT SET')}")
7
8# Check Java
9import subprocess
10try:
11    result = subprocess.run(['java', '-version'], capture_output=True, text=True)
12    print(f"Java: {result.stderr.splitlines()[0]}")
13except FileNotFoundError:
14    print("Java: NOT FOUND")
15
16# Check PySpark
17try:
18    import pyspark
19    print(f"PySpark: {pyspark.__version__}")
20    print(f"PySpark path: {pyspark.__path__}")
21except ImportError:
22    print("PySpark: NOT INSTALLED")

Common Pitfalls

Java not installed or wrong version: Spark 3.x requires Java 8, 11, or 17. Java 21+ may not be compatible with older Spark versions. Check with java -version and install a supported JDK.
JAVA_HOME pointing to JRE instead of JDK: Spark needs the full JDK, not just the JRE. JAVA_HOME should point to the JDK directory containing bin/javac, not just bin/java.
Calling SparkSession.builder.getOrCreate() multiple times with different configs: getOrCreate() returns the existing session if one exists, ignoring new configuration. To change configs, call spark.stop() first, then create a new session with the new configuration.
Port 4040 already in use from a previous crash: If a previous Spark session did not shut down cleanly, port 4040 is still bound. Either kill the orphaned Java process or configure a different port with .config("spark.ui.port", "4041").
Missing winutils.exe on Windows: Spark uses Hadoop libraries that expect Unix-like file operations. On Windows, download winutils.exe and set HADOOP_HOME to the directory containing bin/winutils.exe.

Summary

Install Java 8, 11, or 17 and set JAVA_HOME — this resolves most SparkSession creation errors
Use pip install pyspark for the simplest installation (includes bundled Spark)
Check version compatibility between Spark, Java, and Python
Always call spark.stop() when done to release ports and resources
On Windows, install winutils.exe and set HADOOP_HOME
Run the diagnostic script to quickly identify missing dependencies