Introduction

maxOffsetsPerTrigger is a Kafka source option in Spark Structured Streaming that limits how many offsets are read in each micro-batch. It is mainly a throughput-control setting: you use it when a stream should consume data steadily instead of reading a large backlog all at once.

What the Option Actually Does

When Spark reads from Kafka in micro-batch mode, each trigger asks Kafka for a range of offsets. By default, Spark can read aggressively if there is a lot of available data. maxOffsetsPerTrigger places an upper bound on that read for each trigger.

A minimal PySpark example looks like this:

1from pyspark.sql import SparkSession
2
3spark = SparkSession.builder.appName("kafka-rate-control").getOrCreate()
4
5kafka_df = (
6    spark.readStream
7    .format("kafka")
8    .option("kafka.bootstrap.servers", "localhost:9092")
9    .option("subscribe", "orders")
10    .option("startingOffsets", "earliest")
11    .option("maxOffsetsPerTrigger", 5000)
12    .load()
13)

In that configuration, each trigger will read at most 5000 Kafka offsets across the subscribed topic partitions.

Why This Setting Is Useful

Without a cap, a newly started job might attempt to process a very large backlog immediately. That can cause:

• long micro-batches,
• executor memory pressure,
• unstable latency,
• slow recovery after restarts,
• downstream sink overload.

maxOffsetsPerTrigger is useful when the job needs predictable batch sizes. Instead of letting the stream consume everything available, you tell Spark to move forward in smaller, more controlled steps.

That is especially helpful when the consumer is catching up after downtime.

It Works Together With the Trigger Interval

This option only makes sense when you consider it together with the trigger schedule. Reading 5000 offsets every second is very different from reading 5000 offsets every thirty seconds.

1query = (
2    kafka_df.writeStream
3    .format("console")
4    .trigger(processingTime="10 seconds")
5    .start()
6)
7
8query.awaitTermination()

With maxOffsetsPerTrigger=5000 and a ten-second trigger, you are effectively capping the stream near 500 offsets per second on average, assuming the job keeps up. That is a rough mental model, not a hard SLA, but it helps during tuning.

Partitioning Still Matters

Kafka data is partitioned, and Spark distributes the read across partitions. The maxOffsetsPerTrigger value is shared across the subscribed partitions rather than applied independently to each one.

That means topic shape matters. A stream with many partitions and uneven traffic may behave differently from a stream with only one or two balanced partitions. Do not choose a number in isolation from the actual Kafka topic layout.

Choosing a Good Value

There is no universal best setting. A safe number depends on:

• average message size,
• partition count,
• transformation cost,
• available executor memory and CPU,
• sink throughput,
• acceptable end-to-end latency.

A practical strategy is:

1. start with a conservative value,
2. measure batch duration and lag,
3. raise the cap gradually if the job is underutilized,
4. lower it if batches become unstable or the sink struggles.

In other words, treat the option as a control knob, not a one-time magic number.

What It Does Not Solve

A common misunderstanding is that maxOffsetsPerTrigger solves all streaming performance issues. It does not. If the transformations are expensive, the cluster is too small, or the sink is slow, lag can still grow forever.

This setting limits intake per trigger. It does not guarantee that processing finishes fast enough to keep up with production traffic.

That distinction matters. Rate limiting consumption is sometimes the right operational decision, but it is still different from increasing actual processing capacity.

Monitoring After Configuration

After setting the option, watch the streaming query carefully. Useful signals include:

• micro-batch duration,
• input rows per second,
• processed rows per second,
• Kafka lag,
• sink-side latency.

If lag keeps rising even with stable batches, the system is underprovisioned or the cap is too low for the incoming rate.

Common Pitfalls

One common mistake is setting maxOffsetsPerTrigger and then ignoring the trigger interval. The batch cap only makes sense relative to how often the query runs.

Another issue is assuming the setting applies to every source type in Structured Streaming. It is specifically a Kafka source option based on offsets.

Teams also sometimes choose a number based only on record count, forgetting that message size and downstream processing cost matter just as much.

Summary

• 'maxOffsetsPerTrigger limits how many Kafka offsets Spark reads per micro-batch.'
• It is mainly a rate-control and stability setting.
• Tune it together with trigger interval, partition layout, and cluster capacity.
• It improves predictability, but it does not fix an underpowered pipeline.
• Measure lag and batch duration after changes instead of guessing once and leaving it alone.