Introduction

Reading JSON from Kafka in Spark usually means building a Structured Streaming pipeline: Kafka provides raw bytes, Spark reads them as a streaming DataFrame, and then you parse the value column into typed fields. The important part is that Spark does not automatically treat Kafka payloads as JSON. You have to cast and parse them yourself.

Read the Kafka topic as a stream

Spark's Kafka source exposes several columns such as key, value, topic, partition, and timestamp. The value column arrives as binary data.

1import org.apache.spark.sql.SparkSession
2
3val spark = SparkSession.builder()
4  .appName("KafkaJsonReader")
5  .master("local[*]")
6  .getOrCreate()
7
8spark.sparkContext.setLogLevel("WARN")
9
10val raw = spark.readStream
11  .format("kafka")
12  .option("kafka.bootstrap.servers", "localhost:9092")
13  .option("subscribe", "orders")
14  .option("startingOffsets", "latest")
15  .load()

At this point, raw contains Kafka records, but not yet structured JSON fields.

Cast the Kafka value and parse with a schema

The usual pattern is:

1. cast value to STRING
2. define a schema
3. call from_json

1import org.apache.spark.sql.functions.{col, from_json}
2import org.apache.spark.sql.types._
3
4val orderSchema = new StructType()
5  .add("orderId", StringType)
6  .add("customerId", StringType)
7  .add("amount", DoubleType)
8  .add("createdAt", StringType)
9
10val parsed = raw
11  .selectExpr("CAST(value AS STRING) AS json_text")
12  .select(from_json(col("json_text"), orderSchema).as("data"))
13  .select("data.*")
14
15parsed.printSchema()

Once parsed, parsed behaves like a normal streaming DataFrame. You can filter, group, and write it with the rest of the Spark SQL API.

Transform the structured fields

After parsing, downstream logic becomes ordinary Spark code:

val validOrders = parsed
  .filter(col("amount") > 0.0)
  .select("orderId", "customerId", "amount", "createdAt")

If you want aggregates, you can add them after parsing:

val totals = validOrders
  .groupBy("customerId")
  .sum("amount")

This is the main reason to parse JSON early. Once the payload is typed, Spark can optimize and validate operations much better than it can on raw strings.

Write the stream somewhere useful

For debugging, the console sink is enough:

1val query = parsed.writeStream
2  .format("console")
3  .outputMode("append")
4  .option("truncate", "false")
5  .start()
6
7query.awaitTermination()

In production, you might write to Delta, Parquet, another Kafka topic, or a database sink. The parsing logic stays the same.

If you write aggregates such as totals, make sure the output mode matches the query. Aggregate queries often require update or complete rather than plain append.

Prefer an explicit schema

For streaming jobs, an explicit schema is usually the right choice. It gives you:

• predictable field types
• better documentation of the event contract
• earlier detection of malformed payloads
• less surprise than runtime inference

If a field fails to parse, Spark usually produces null for that field or for the parsed struct. That is a useful signal, but only if you actually inspect or filter invalid rows.

Think about event shape over time

If your Kafka topic carries several unrelated JSON shapes, parsing becomes harder fast. In practice, a stable topic contract is much easier to maintain than a topic full of unrelated payloads.

When event versions evolve, keep the schema changes deliberate. Adding an optional field is easy to handle. Completely changing the event shape is not.

Common Pitfalls

The most common mistake is forgetting that Kafka value is binary and trying to read nested JSON fields before casting and parsing it.

Another common issue is relying on schema inference instead of declaring a stable schema. That usually makes production pipelines more fragile.

People also ignore rows where parsing produced null, which means bad messages silently slip through the pipeline.

Finally, aggregate queries need the right output mode. A correct parse step does not help if the sink configuration is wrong for the query shape.

Summary

• Spark reads Kafka messages as binary columns, not as parsed JSON objects.
• Cast value to STRING and parse it with from_json and an explicit schema.
• After parsing, use normal Spark SQL transformations on typed fields.
• Prefer explicit schemas in streaming jobs.
• Inspect malformed or null-parsed records instead of assuming every Kafka message matches the schema.