Spark 2.3.0 Failed to find data source kafka
Master System Design with Codemia
Enhance your system design skills with over 120 practice problems, detailed solutions, and hands-on exercises.
Apache Spark is widely used for large scale data processing. One common use case is processing real-time data streams, for which Kafka is often used as the source. However, those new to using Spark with Kafka may encounter the error: "Failed to find data source: kafka". This error is typically due to a misconfiguration or missing components needed for integration.
Understanding the Error
The "Failed to find data source: kafka" error in Spark 2.3.0 occurs when Spark cannot locate the Kafka data source package. This usually means that the Kafka connector for Spark is not properly included in the project or classpath.
Spark's ability to interact with Kafka is facilitated through the Spark-SQL Kafka 0-10 connector, which is an external package. For Spark 2.3.0, this package needs to be explicitly added because it is not included in the standard Spark distribution.
Setting Up Spark with Kafka
To resolve the error and set up Spark to read data from Kafka, follow these steps:
1. Include the Kafka Connector
When launching your Spark job, you need to include the spark-sql-kafka-0-10_2.11 library if using Scala 2.11 or the corresponding library for Scala 2.12:
2. Use the Correct Format in Your Code
Ensure your Spark session is using "kafka" format when reading from Kafka:
3. Check Scala and Spark Versions
Verify that your project is using a compatible version of Scala for the Spark version. Spark 2.3.0 is compatible with Scala 2.11 by default.
4. Build Dependency Management
If you are building a project with SBT or Maven, include the Kafka connector in your build file (build.sbt for SBT):
Common Troubleshooting Tips
- Dependency Issues: Even with correct dependencies, conflicts with other libraries might cause issues. Ensuring minimal dependency exclusions can help.
- Environmental Variables: Sometimes, environmental settings or configuration files such as
spark-defaults.confmight override your application settings. Ensure these settings are not conflicting with your Kafka configuration. - Resource Allocation: Insufficient resources (like memory) might cause the application not to start, often obscured by a generic error message.
Summary of Key Points
| Topic | Detail |
| Error Description | "Failed to find data source: kafka" indicates missing Kafka data source connector. |
| Required Spark Package | spark-sql-kafka-0-10_2.11:2.3.0 (or corresponding version for Scala 2.12) |
| Spark Submission Example | Include --packages org.apache.spark:spark-sql-kafka-0-10_2.11:2.3.0 in the spark-submit command. |
| Code Example | Use .format("kafka") to specify Kafka as the data source in Spark. |
| Dependency Management in SBT | Add "org.apache.spark" %% "spark-sql-kafka-0-10" % "2.3.0" to libraryDependencies. |
| Troubleshooting | Check Scala version compatibility, manage dependencies without exclusions, ensure correct environmental settings. |
Understanding these aspects and ensuring proper setup usually resolves the issue, enabling smooth operation of Spark streaming applications with Kafka as a source.

