Introduction

Slow writes from EMR and Spark to S3 are usually not caused by raw S3 bandwidth alone. More often, the bottleneck is the write pattern: too many small output files, too many partitions, expensive commit behavior, or a job topology that forces heavy shuffle work before the final write.

Start by checking the output pattern

Spark writes one file per output task. If your job produces thousands of tiny partitions, S3 receives thousands of small objects and commit requests instead of a smaller number of large sequential uploads.

That is a common reason a write feels "mysteriously slow" even when the cluster and bucket are in the same region.

A simple first fix is to reduce the number of output partitions:

df.coalesce(50).write.mode("overwrite").parquet("s3://my-bucket/output/")

or, if you need a shuffle for better balance:

df.repartition(50).write.mode("overwrite").parquet("s3://my-bucket/output/")

Use coalesce when you want fewer partitions without a full reshuffle. Use repartition when data skew makes a balanced redistribution necessary.

Understand the S3 commit path

S3 is an object store, not HDFS. Spark and Hadoop commit logic matters because jobs often stage files and then finalize them. Older output commit approaches can be especially expensive on object storage.

On EMR, you should inspect which filesystem and committer path the job is using. Modern EMR and Spark distributions are much better than older ones here, but misconfiguration can still cause slow finalize stages.

Relevant settings often include:

spark.conf.set("mapreduce.fileoutputcommitter.algorithm.version", "2")
spark.conf.set("spark.sql.sources.commitProtocolClass",
               "org.apache.spark.sql.execution.datasources.SQLHadoopMapReduceCommitProtocol")

The exact best settings depend on the EMR release and write format, but the main idea is stable: object-store-aware committers matter.

Small files and skew are the usual killers

Two patterns show up repeatedly:

• thousands of tiny files
• one or two oversized partitions because of skew

Tiny files create too much request overhead. Skew causes a few executors to do most of the work while the rest sit idle. In both cases, the S3 write stage looks slow even though the real issue started earlier in the Spark plan.

Inspect the Spark UI:

• how many tasks are writing
• whether task durations are balanced
• whether the final stage is waiting on a few stragglers

If the last stage has a long tail, fix partitioning before blaming S3.

Keep the bucket and cluster close

Region mismatch still matters. Writing from an EMR cluster in one region to an S3 bucket in another increases latency and can reduce throughput noticeably. The cluster and target bucket should normally be in the same region unless cross-region design is intentional.

You should also confirm you are not traversing unexpected network boundaries such as private routing patterns or restrictive endpoint setups that throttle transfer behavior.

A practical write pattern

For a typical parquet write, a clean baseline looks like this:

1output = (
2    df
3    .repartition(100)
4    .write
5    .mode("overwrite")
6    .option("compression", "snappy")
7    .parquet("s3://analytics-bucket/events/date=2026-03-11/")
8)

The important part is not the exact number 100. It is choosing a partition count that matches data volume and cluster size rather than accepting whatever fragmented layout an upstream shuffle produced.

Common Pitfalls

The most common mistake is focusing on S3 itself while ignoring that Spark is writing an extreme number of small files. That pattern is expensive regardless of cloud bandwidth.

Another mistake is assuming the final write stage is isolated from the rest of the plan. Skew created in joins or aggregations often shows up as "slow S3 writes" later.

Teams also overlook commit behavior and EMR release differences. Object-store-aware committers and sane output settings matter a lot more on S3 than they do on HDFS.

Finally, do not ignore region placement. Cross-region writes can be slower and more expensive even when everything else looks fine.

Summary

• Slow EMR and Spark writes to S3 are often caused by partitioning and commit behavior, not just raw storage speed.
• Too many small files are one of the most common performance killers.
• Use coalesce or repartition intentionally before writing.
• Check the Spark UI for skew and long-tail tasks before blaming S3 alone.
• Keep the cluster and bucket in the same region and use object-store-friendly commit settings.