Introduction

Repartitioning a PySpark DataFrame changes how rows are distributed across Spark partitions. You usually do it to increase parallelism, reduce tiny output files, or distribute data by a key before a join or aggregation.

Check the Current Partition Count

Before changing anything, inspect the current number of partitions:

1from pyspark.sql import SparkSession
2
3spark = SparkSession.builder.getOrCreate()
4
5df = spark.range(0, 1000)
6print(df.rdd.getNumPartitions())

That tells you how many partitions back the DataFrame right now. It does not tell you whether the distribution is balanced, but it is the first thing to check.

Use repartition() When You Need a Shuffle

repartition() creates a new DataFrame with a new partition layout. It performs a shuffle, so rows can move across the cluster.

To set an explicit number of partitions:

df2 = df.repartition(8)
print(df2.rdd.getNumPartitions())

This is useful when the current partition count is too low for the cluster or when an upstream operation left you with a poor layout.

You can also repartition by a column:

1sales = spark.createDataFrame(
2    [
3        ("east", 100),
4        ("east", 120),
5        ("west", 90),
6        ("north", 75),
7        ("west", 110),
8    ],
9    ["region", "amount"],
10)
11
12by_region = sales.repartition(4, "region")
13print(by_region.rdd.getNumPartitions())

That tells Spark to hash-partition the rows using region. It is often helpful before joins or aggregations on the same key because related rows end up together more predictably.

Use coalesce() When Reducing Partitions

If your goal is only to reduce the number of partitions, coalesce() is often cheaper:

smaller = df.coalesce(2)
print(smaller.rdd.getNumPartitions())

Unlike repartition(), coalesce() tries to avoid a full shuffle. That makes it a better option when you filtered the data down and now want fewer output files.

A common write pattern looks like this:

filtered = df.filter("id < 100")
filtered.coalesce(1).write.mode("overwrite").csv("/tmp/output")

That can be useful for tiny datasets or demos, but be careful with coalesce(1) in production. For large data, forcing everything into one partition creates a bottleneck.

Repartition for Joins and Aggregations

Partitioning strategy matters most around shuffle-heavy operations.

Suppose two large DataFrames are joined on customer_id. Repartitioning both by that key can improve distribution:

1orders = spark.createDataFrame(
2    [(1, "A"), (2, "B"), (1, "C")],
3    ["customer_id", "order_name"],
4)
5
6customers = spark.createDataFrame(
7    [(1, "Ada"), (2, "Ben")],
8    ["customer_id", "customer_name"],
9)
10
11orders_p = orders.repartition(4, "customer_id")
12customers_p = customers.repartition(4, "customer_id")
13
14joined = orders_p.join(customers_p, on="customer_id")
15joined.show()

This is not a universal rule. Spark's optimizer may choose a broadcast join or another plan that makes manual repartitioning unnecessary. Still, when you are working with large, similarly sized tables and a clear join key, repartitioning by that key is a standard tool.

Repartition Before Writing Output

Output layout is another common reason to repartition.

If a DataFrame has 500 partitions and you write it to disk, you may create roughly 500 part files. Sometimes that is fine. Sometimes it is a maintenance problem.

To reduce file count:

df.repartition(20).write.mode("overwrite").parquet("/tmp/parquet-output")

Choose the number based on data size, downstream expectations, and cluster resources. There is no magic number that fits every workload.

Common Pitfalls

The biggest mistake is using repartition() casually. It triggers a shuffle, and shuffles are expensive. If you only need fewer partitions, prefer coalesce().

Another common issue is picking a partition key with heavy skew. If one value appears far more often than others, one partition may become much larger and slower than the rest.

People also overuse coalesce(1) to get a single output file. That works for small data, but it destroys parallelism and can make large jobs painfully slow.

Finally, checking only the partition count is not enough. A DataFrame can have the "right" number of partitions and still be poorly balanced. Inspect the execution plan and job behavior when performance matters.

Summary

• Use df.rdd.getNumPartitions() to inspect the current partition count.
• Use repartition() when you need a new distribution or partitioning by key.
• Use coalesce() when reducing partitions without a full shuffle.
• Repartitioning is especially relevant before joins, aggregations, and writes.
• Avoid unnecessary shuffles and be careful with skewed keys or coalesce(1).