Introduction

KafkaUtils not found in a Spark Streaming job almost always means the Kafka connector jar is missing or mismatched. The fix is usually not in your Scala code at all. It is in dependency coordinates, Scala binary version alignment, or how the job is launched on the cluster.

Add the Correct Kafka Connector

Spark Streaming does not ship every Kafka integration class in the core Spark jar. You need the separate connector module that matches your Spark and Scala versions.

For an sbt build:

1ThisBuild / scalaVersion := "2.12.18"
2
3val sparkVersion = "3.5.1"
4
5libraryDependencies ++= Seq(
6  "org.apache.spark" %% "spark-streaming" % sparkVersion % "provided",
7  "org.apache.spark" %% "spark-streaming-kafka-0-10" % sparkVersion
8)

For Maven:

1<dependencies>
2  <dependency>
3    <groupId>org.apache.spark</groupId>
4    <artifactId>spark-streaming_2.12</artifactId>
5    <version>3.5.1</version>
6    <scope>provided</scope>
7  </dependency>
8  <dependency>
9    <groupId>org.apache.spark</groupId>
10    <artifactId>spark-streaming-kafka-0-10_2.12</artifactId>
11    <version>3.5.1</version>
12  </dependency>
13</dependencies>

The important detail is the suffix such as _2.12. It must match the Scala version used by your Spark distribution. If your cluster runs Spark built for Scala 2.12, depending on a _2.13 artifact will compile badly or fail at runtime.

Import the Right Package and Ship the Jar

Once the dependency is correct, the code should import the kafka010 package:

1import org.apache.kafka.common.serialization.StringDeserializer
2import org.apache.spark.SparkConf
3import org.apache.spark.streaming.Seconds
4import org.apache.spark.streaming.StreamingContext
5import org.apache.spark.streaming.kafka010.ConsumerStrategies
6import org.apache.spark.streaming.kafka010.KafkaUtils
7import org.apache.spark.streaming.kafka010.LocationStrategies
8
9object KafkaStreamingApp {
10  def main(args: Array[String]): Unit = {
11    val conf = new SparkConf()
12      .setAppName("KafkaStreamingApp")
13      .setMaster("local[2]")
14
15    val ssc = new StreamingContext(conf, Seconds(5))
16
17    val kafkaParams = Map[String, Object](
18      "bootstrap.servers" -> "localhost:9092",
19      "key.deserializer" -> classOf[StringDeserializer],
20      "value.deserializer" -> classOf[StringDeserializer],
21      "group.id" -> "demo-group",
22      "auto.offset.reset" -> "latest",
23      "enable.auto.commit" -> java.lang.Boolean.FALSE
24    )
25
26    val topics = Array("events")
27
28    val stream = KafkaUtils.createDirectStream[String, String](
29      ssc,
30      LocationStrategies.PreferConsistent,
31      ConsumerStrategies.Subscribe[String, String](topics, kafkaParams)
32    )
33
34    stream.map(_.value()).print()
35
36    ssc.start()
37    ssc.awaitTermination()
38  }
39}

If this compiles locally but fails when submitted, the cluster probably does not have the connector jar. In that case, include it at submit time:

1spark-submit \
2  --class KafkaStreamingApp \
3  --packages org.apache.spark:spark-streaming-kafka-0-10_2.12:3.5.1 \
4  app.jar

That solves many "works on my machine" cases because executors also receive the dependency.

Distinguish Old and New Kafka Integrations

A lot of confusing search results come from mixing Spark's older Kafka integration with the newer 0-10 connector. If you copy imports from an old tutorial, you may end up looking for classes in the wrong package or using an artifact that no longer matches your Spark version.

As a rule:

• use org.apache.spark.streaming.kafka010.KafkaUtils for the direct stream connector
• keep Spark, Scala, and connector versions aligned
• make sure the runtime classpath on the cluster matches your compile classpath

If you are starting a new project, also consider whether DStreams are the right tool. Spark Structured Streaming is the newer API for many Kafka workloads.

Common Pitfalls

The most common issue is a Scala binary mismatch. A dependency ending in _2.13 will not work with a Spark installation built for _2.12, even if the Spark version number looks correct.

Another frequent mistake is relying on provided dependencies locally and forgetting that the Kafka connector is not actually present on the cluster. Core Spark jars may exist there, but the Kafka integration module often does not.

People also run into problems by importing org.apache.spark.streaming.kafka.KafkaUtils from older examples. Modern Spark Streaming Kafka integration uses the kafka010 package instead.

Finally, if the error appears only at runtime, inspect the full stack trace. A true ClassNotFoundException points to missing jars, while a NoSuchMethodError often means incompatible versions are both present.

Summary

• 'KafkaUtils errors usually come from missing or mismatched Spark Kafka connector jars.'
• The connector artifact must match both the Spark version and the Scala binary version.
• Use imports from org.apache.spark.streaming.kafka010.
• If the job fails only after submission, pass the connector through --packages or ship the jar explicitly.
• For new systems, evaluate Spark Structured Streaming before committing to DStreams.