Kafka Stream from JSON to Avro
System Design practice on Codemia
Work through 120+ system design problems with detailed solutions, from rate limiters to multi-region storage.
Apache Kafka is a popular distributed streaming platform that enables building real-time data pipelines and streaming applications. Kafka Streams is a client library for building applications and microservices, where the input and output data are stored in Kafka clusters. In this discussion, we focus on a common data transformation scenario: streaming data from JSON format to Avro format using Kafka Streams.
Key Concepts and Technologies
JSON (JavaScript Object Notation) is a lightweight data interchange format that is easy for humans to read and write and easy for machines to parse and generate. It is commonly used in various data exchange scenarios in web applications.
Avro is a data serialization system that enables efficient and compact binary format data exchange. It is often used in Apache Kafka to ensure schema management and compatibility across distributed data streams.
Kafka Streams is the stream processing library provided by Apache Kafka. It allows for stateful and stateless processing, windowing, and complex event processing by transforming input Kafka topics into output Kafka topics.
Schema Registry is a service provided by Confluent (and compatible with other platforms) that manages Avro schemas and their versions, ensuring that only valid Avro data is published to Kafka topics.
Processing Flow: From JSON to Avro
The typical processing flow involves reading messages from a Kafka topic with data in JSON format, transforming these messages into Avro format, and then writing the transformed messages back to a different Kafka topic. To accomplish this, developers need to use Kafka Streams for the transformation process, which can be broken down into a few steps:
- Read from Kafka: Stream messages from a Kafka topic with JSON values.
- Deserialize JSON: Convert JSON string messages into Java objects.
- Serialize to Avro: Convert Java objects into Avro format.
- Write to Kafka: Stream the Avro messages back to a Kafka topic.
Example Workflow with Code Snippets
Assuming you have Kafka and Schema Registry running, you can implement the Kafka Streams application using Java:
Table of Key Points
| Feature | JSON | Avro |
| Format | Text based | Binary |
| Readability | Human-readable | Not directly human-readable |
| Compression | Generally less efficient | More efficient due to binary format |
| Schema Management | No native support | Supports schema evolution |
| Use Case | Web APIs, configurations | Large-scale data storage, inter-service communication |
Additional Considerations
- Schema Evolution: Avro supports schema evolution, allowing you to modify schemas without breaking existing applications. Schema Registry helps manage this by enforcing compatibility rules.
- Performance: Avro's binary format generally offers better performance and compression compared to JSON. This is crucial in high-throughput environments such as Kafka.
- Tooling and Ecosystem: Avro is deeply integrated with the Kafka ecosystem, and tools like Schema Registry provide essential services that enhance the robustness of Kafka's data handling capabilities.
Conclusion
Converting JSON to Avro within a Kafka Streams application involves deserializing JSON into Java objects, transforming these objects according to an Avro schema, and serializing them into Avro format. By leveraging Kafka Streams along with Schema Registry, developers can build robust and efficient streaming applications that capitalize on Kafka's powerful data streaming capabilities.
Related reading
- Kafka Stream offset reset to zero for consumer group
- Kafka Stream output to a topic first or persist directly?
- Kafka stream PolicyViolationException Topic replication factor must be 3
- Kafka Stream Scala API slow performance
- Kafka Stream Suppress session-windowed-aggregation
- Kafka Stream to sort messages based on timestamp key in json message
- Kafka stream TopicAuthorizationException Not authorized to access topics for an internal state store
- Kafka stream vs kafka consumer how to make decision on what to use

System Design Fundamentals
Build a strong foundation in designing scalable, reliable distributed systems.
View the courseTrack what you have practised
A free account saves your progress, solutions and study plan across every problem on Codemia.
System Design practice on Codemia
Work through 120+ system design problems with detailed solutions, from rate limiters to multi-region storage.