Apache Kafka is a widely-used platform designed to build real-time data pipelines and streaming applications. Kafka Connect, a component of Apache Kafka, simplifies integrating Kafka with other systems like databases, key-value stores, and search indexes. Apache Avro is a popular serialization framework used within Kafka ecosystems, primarily because of its compact binary data format and rich data structures. Typically, Avro is used in conjunction with Confluent Schema Registry which serves as a repository for Avro schemas and provides schema evolution capabilities. However, it's entirely possible to use Avro in Kafka Connect without relying on Confluent's Schema Registry. Below, we explore how to implement this and handle schema management manually.

Understanding Avro Serialization

Avro data is always serialized with its schema, making it possible to precisely reconstruct the data upon deserialization. This characteristic ensures that the data structure is not misplaced or misinterpreted across different systems. Avro schemas are defined in JSON format and specify the data structures in a language-neutral way.

Schema Management without Schema Registry

When not using the Confluent Schema Registry, you need to manually manage your Avro schemas. One common approach is to store the Avro schema alongside the data, either directly embedded in messages (enabling self-describing messages) or by storing it in a separate system location that is accessible by both producers and consumers.

Configuring Kafka Connect with Avro without Schema Registry

Here's an overview of how you can configure Kafka Connect to use Avro serialization without the Schema Registry:

1. Avro Converter Configuration: Configure the Kafka Connect worker to use Avro Converters. Within the Kafka Connect configuration file, set the following properties:

    key.converter=org.apache.kafka.connect.storage.StringConverter
    value.converter=io.confluent.connect.avro.AvroConverter
    value.converter.schema.registry.url=file:///path/to/schema/folder

Note: Although we're not using a Schema Registry service, the Avro converter can still be used by providing a file path to schemas stored in the local filesystem.

1. Schema Storage: You must manually manage the storage and retrieval of schema files. Generally, schemas are versioned and stored in a directory whose path is provided to each Kafka Connect worker.
2. Producer and Consumer Configuration: Both producers and consumers need to handle Avro data using the same schema. Make sure they point to the correct version of the schema files during data serialization and deserialization.

Example Scenario: Kafka Connect with Local File System for Schemas

Consider a scenario where Kafka Connect needs to write data to a Kafka topic from a database and then another Connect instance consumes this data to sink to another database.

• Producers serialize records into Avro format using the relevant schema file.
• Kafka Connect Source Connector reads data from the source database, serializes records using Avro, and publishes them to a Kafka topic.
• Consumers deserialize these Avro records using the same Avro schema to ensure the structure is maintained.

Sample Avro Schema

1{
2  "type": "record",
3  "name": "User",
4  "fields" : [
5    {"name": "id", "type": "int"},
6    {"name": "name", "type": "string"}
7  ]
8}

Pros and Cons of Not Using Schema Registry

Conclusion

Using Avro in Kafka Connect without a Schema Registry is feasible and may be desirable under certain conditions, like when operational simplicity and control are prioritized over automated schema management. However, it requires careful handling of schema versions and strict coordination between teams to maintain compatibility across all components in the data pipeline.

Ultimately, this approach welcomes an in-depth understanding of Kafka's internal mechanisms for data serialization and schema management, opening up possibilities for customized system designs tailored to specific needs.