Introduction

The error NameError: name 'spark' is not defined appears when your code uses spark before creating a SparkSession variable in the current scope. In notebooks this variable may be auto created, but in scripts and jobs it usually is not. The fix is to initialize SparkSession explicitly and manage scope carefully.

Create SparkSession Explicitly

In standalone scripts, always create SparkSession before any DataFrame operation.

1from pyspark.sql import SparkSession
2
3spark = (
4    SparkSession.builder
5    .appName("example-job")
6    .master("local[*]")
7    .getOrCreate()
8)
9
10df = spark.createDataFrame([(1, "Ana"), (2, "Ben")], ["id", "name"])
11df.show()
12
13spark.stop()

This removes ambiguity and makes script behavior consistent across local and cluster runs.

Notebook Versus Script Behavior

Some managed notebook environments expose spark automatically, which can hide missing initialization in copied code. The same notebook cell copied into a plain Python file will fail with NameError.

To keep code portable, initialize session explicitly even in notebooks unless platform conventions require otherwise. Consistent setup makes migration to scheduled jobs easier.

Scope and Import Issues

Another common cause is defining spark inside one function and using it elsewhere without passing it. Keep session creation at module level entry points, or pass spark as an argument.

1from pyspark.sql import SparkSession
2
3
4def read_customers(spark):
5    return spark.read.option("header", True).csv("customers.csv")
6
7
8def main():
9    spark = SparkSession.builder.appName("customer-job").getOrCreate()
10    try:
11        df = read_customers(spark)
12        df.show(5)
13    finally:
14        spark.stop()
15
16
17if __name__ == "__main__":
18    main()

This pattern makes dependencies explicit and improves testability.

Production Setup Recommendations

For production jobs, set app name, log level, and relevant configs in one setup function. That centralizes Spark lifecycle rules and prevents duplicated initialization logic.

You can also validate session creation early with a lightweight command such as spark.range(1).count() to fail fast on cluster connectivity issues.

In team codebases, avoid global mutable session variables spread across modules. Either pass session objects explicitly or create a dedicated context module with clear ownership.

During deployment, verify how the job is launched. A script that works with spark-submit may fail when run with plain python because Spark dependencies and environment variables are not initialized the same way. Document your standard launch command and keep it in CI pipelines so local and production behavior match.

Add a startup check section in logs that prints Spark version, app name, and master URL. These details make incident triage much faster when jobs run in multiple environments.

A quick troubleshooting checklist helps junior engineers resolve this error fast. Confirm that SparkSession creation code runs before any DataFrame call, check that the variable name is exactly spark, and verify that setup code was not skipped by conditional flags. These small checks solve most incidents without deep debugging. Keep this checklist near your runbook so on call engineers can respond quickly during job failures.

Common Pitfalls

A common pitfall is running one notebook cell that defines spark, then restarting kernel and running a later cell that assumes it still exists. Re run setup cells after kernel reset.

Another issue is naming collisions. If a local variable or import shadows spark, later DataFrame calls can fail in confusing ways.

Developers also forget to stop sessions in long running scripts, which can leave resources allocated unnecessarily.

Finally, code copied from Databricks style notebooks may rely on environment magic not present in plain PySpark jobs. Always verify startup assumptions when moving code between platforms.

Summary

• NameError for spark means no active SparkSession variable in scope.
• Initialize SparkSession explicitly in scripts and portable notebooks.
• Pass spark into functions rather than relying on hidden globals.
• Standardize setup and teardown to keep jobs reliable.
• Recheck environment assumptions when moving code across Spark platforms in every release for production and staging.