Apache Kafka is a distributed messaging system that is widely used for streaming large volumes of data efficiently. Apache Avro, on the other hand, is a binary serialization format. It is compact, fast, and integrates seamlessly with a variety of programming languages, which makes it advantageous for streaming data scenarios. Apache Spark, particularly in version 2.0.2, introduces the concept of structured streaming which is an extension to the DataFrame and Dataset API. This combination offers a robust solution for processing streams of data in real time.

Integration of Apache Kafka with Spark Structured Streaming

With Spark 2.0.2, structured streaming APIs allow you to easily consume records from a Kafka topic. Structured streaming treats streaming data as an unbounded table where new rows get appended to the table as new messages arrive from the Kafka topic. In combination with Avro, this can be used to decode the binary data records into a structured format which can then be easily manipulated and processed using Spark.

Reading Avro Messages from Kafka

To begin reading Avro messages from Kafka using Spark 2.0.2, you will need to set up a Kafka source for structured streaming. Below are the steps and code necessary to create a DataFrame that reads from Kafka and parses the Avro messages.

Step 1: Include Spark-Avro and Kafka Packages

Make sure to include the necessary dependencies in your Spark session. These would be the Spark-Avro and Kafka integration packages.

1val spark = SparkSession
2  .builder
3  .appName("AvroKafkaExample")
4  .config("spark.master", "local")
5  .config("spark.jars.packages", "org.apache.spark:spark-avro_2.11:2.0.2,org.apache.spark:spark-sql-kafka-0-10_2.11:2.0.2")
6  .getOrCreate()

Step 2: Define Kafka Source for Streaming Query

Create the streaming query by defining the Kafka source. You need to specify the Kafka server details and the topic to subscribe to.

1val df = spark
2  .readStream
3  .format("kafka")
4  .option("kafka.bootstrap.servers", "localhost:9092") // Change it to your Kafka server config
5  .option("subscribe", "avro_topic") // Change to your topic name
6  .load()

Step 3: Deserialize Avro Data

Assuming the Kafka messages are serialized in Avro format, the data in the DataFrame df will be contained in a column named "value" as a binary blob. You can deserialize this using Spark-Avro.

val avroDF = df.select(from_avro(col("value"), "your_avro_schema_string_here").as("data"))

You should replace "your_avro_schema_string_here" with the actual Avro schema string that was used to encode the Kafka messages.

Transformations and Outputs

Once you have the data deserialized, you can perform a variety of transformations using the DataFrame API. Hereafter, you can output the processed data into various sinks such as databases, files, or even back into Kafka.

1val query = avroDF.writeStream
2  .outputMode("append")
3  .format("console")
4  .start()
5
6query.awaitTermination()

Summary

The following table summarizes the key components and their purposes when reading Avro messages from Kafka using Spark 2.0.2 structured streaming:

Conclusion

Using Apache Kafka, Avro, and Spark together provides a powerful infrastructure for streaming data analytics. Avro's compact format ensures efficient data serialization, which, when combined with Kafka's robust streaming capabilities and Spark's real-time data processing power, can significantly enhance any real-time data-driven application.