Introduction

The optimal pattern in Spark Structured Streaming is usually to read the relevant Kafka topics through one Kafka source, parse once, and then branch the resulting stream into separate output queries or route inside foreachBatch. The right choice depends on whether the sinks are independent or need coordinated batch-level routing logic.

Read Multiple Topics Through One Kafka Source

Spark can subscribe to several topics in one source by listing them or using a pattern. That is usually cleaner than starting one Spark application per topic unless the pipelines are completely unrelated.

1from pyspark.sql import SparkSession
2from pyspark.sql.functions import col
3
4spark = SparkSession.builder.appName("multi-topic-stream").getOrCreate()
5
6raw = (
7    spark.readStream
8    .format("kafka")
9    .option("kafka.bootstrap.servers", "broker1:9092,broker2:9092")
10    .option("subscribe", "orders,payments,shipments")
11    .load()
12)
13
14events = raw.select(
15    col("topic"),
16    col("timestamp"),
17    col("key").cast("string").alias("key"),
18    col("value").cast("string").alias("value"),
19)

That gives you one ingestion point, one offset-tracking story, and one place to handle shared parsing and enrichment.

Split the Stream by Topic or Record Type

After parsing, branch into filtered DataFrames for each output path.

orders = events.filter(col("topic") == "orders")
payments = events.filter(col("topic") == "payments")
shipments = events.filter(col("topic") == "shipments")

If the same topic carries multiple event types, filter on a decoded field instead of the Kafka topic name.

Spark does not let one writeStream send to multiple sinks by itself, so each independent sink becomes its own streaming query:

1orders_query = (
2    orders.writeStream
3    .format("parquet")
4    .option("path", "/data/orders")
5    .option("checkpointLocation", "/chk/orders")
6    .start()
7)
8
9payments_query = (
10    payments.writeStream
11    .format("kafka")
12    .option("kafka.bootstrap.servers", "broker1:9092,broker2:9092")
13    .option("topic", "payments_clean")
14    .option("checkpointLocation", "/chk/payments")
15    .start()
16)

Use a different checkpoint location per sink query.

Use foreachBatch When Routing Logic Is More Complex

If several sinks need batch-aware logic or one sink depends on another branch's computation, foreachBatch can be cleaner.

1def write_batch(batch_df, batch_id):
2    batch_df.cache()
3
4    batch_df.filter(col("topic") == "orders") \
5        .write.mode("append").parquet("/data/orders")
6
7    batch_df.filter(col("topic") == "payments") \
8        .write.mode("append").json("/data/payments")
9
10    batch_df.unpersist()
11
12
13query = (
14    events.writeStream
15    .foreachBatch(write_batch)
16    .option("checkpointLocation", "/chk/router")
17    .start()
18)

This approach gives you full batch DataFrame control, but you are now responsible for sink-side idempotency and failure handling more explicitly.

Choose the Pattern Based on Sink Independence

A useful rule is:

• use multiple streaming queries when each sink is operationally independent
• use foreachBatch when one batch needs coordinated routing logic or shared batch computation

Independent queries are often easier to monitor because each sink has its own streaming query status and checkpoint. foreachBatch is more flexible, but it concentrates more responsibility in your code.

Common Pitfalls

The biggest mistake is thinking one writeStream can fan out to many sinks automatically. In practice, you either start multiple output queries from shared transformations or use foreachBatch to route manually.

Another common issue is reparsing the same Kafka payload in several branches. Parse once near the source, then reuse the structured columns.

People also reuse the same checkpoint path for multiple queries. That breaks correctness because each query needs its own state and progress tracking.

Finally, do not overstate guarantees. Spark checkpointing and Kafka offsets help with fault tolerance, but end-to-end delivery behavior still depends on the semantics of each sink.

Summary

• Read multiple Kafka topics through one Kafka source when the pipelines share parsing or infrastructure.
• Parse once, then branch by topic or event type.
• Use separate writeStream queries for independent sinks.
• Use foreachBatch when you need custom routing or batch-level control.
• Keep checkpoint locations separate and think carefully about sink-specific delivery guarantees.