Apache Kafka Connect is a versatile tool used to integrate Apache Kafka with external systems such as databases, key-value stores, search indexes, and file systems. Using Kafka Connect, you can import data into Kafka as well as stream data out of Kafka. When deploying Kafka Connect, you have two options: standalone or distributed mode. This article focuses on the distributed mode, discussing its benefits, architecture, configuration, and management.

1. Distributed Mode Overview

In distributed mode, the Kafka Connect service runs as a cluster of one or more workers that share the job of executing connectors. This mode used primarily in a production setting offers several advantages over the standalone mode, including scalability, fault tolerance, and dynamic balancing of work across available workers.

2. Architecture of Kafka Connect in Distributed Mode

Kafka Connect in distributed mode uses a group of worker nodes. Each node in the cluster can execute one or more connector instances or tasks. Connectors and tasks are distributed among the cluster members, ensuring load balancing.

The key components of Kafka Connect’s architecture in distributed mode include:

• Connectors: Responsible for managing integration between Kafka and external systems.
• Tasks: The operational processes that perform data ingestion or egress. A connector could split the load into multiple tasks, which can be distributed across several workers.
• Config Storage: A Kafka topic where the configurations of all connectors and tasks are stored.
• Offset Storage: A Kafka topic that logs the offsets, a way to track the progress of data collection or delivery for each task.
• Status Storage: Another Kafka topic used for storing the states (running, paused, stopped) of connectors and their tasks.

Here is an example cluster of Kafka Connect in distributed mode:

3. Configuration

Configuring Kafka Connect in distributed mode involves setting up various properties in the Connect worker configuration file. Below is an example of essential configurations:

1# Bootstrap Kafka servers
2bootstrap.servers=localhost:9092
3
4# Unique string that identifies the Connect cluster group
5group.id=connect-cluster
6
7# Converter settings for keys and values
8key.converter=org.apache.kafka.connect.json.JsonConverter
9value.converter=org.apache.kafka.connect.json.JsonConverter
10
11# Configuration, offset and status storage topics
12config.storage.topic=connect-configs
13offset.storage.topic=connect-offsets
14status.storage.topic=connect-statuses
15
16# Flush intervals for offset and config storage
17offset.flush.interval.ms=10000

4. Running and Managing Connectors

To deploy a new connector in distributed mode, you post a JSON configuration to the REST interface that all workers can access. Here’s an example of how to configure and launch a simple file source connector via CURL command:

1curl -X POST -H "Content-Type: application/json" --data '{
2  "name": "local-file-source",
3  "config": {
4    "connector.class":"org.apache.kafka.connect.file.FileStreamSourceConnector",
5    "tasks.max":"2",
6    "file": "/tmp/test.txt",
7    "topic": "test-topic"
8  }
9}' http://localhost:8083/connectors

5. Scalability and Fault Tolerance

In distributed mode, Kafka Connect automatically handles failures and rebalances connectors and tasks. If a worker node fails, its tasks are redistributed among existing active nodes. This dynamic rerouting is critical for maintaining consistent throughput and is a significant advantage over standalone mode.

6. Monitoring and Performance

Monitoring Kafka Connect involves checking various metrics like throughput, connector status, and task distribution. Use tools like JMX, Prometheus, and Grafana for real-time monitoring.

Conclusion

Running Kafka Connect in distributed mode provides essential features suitable for scalable, large-scale production environments. It ensures that the data flow is not only manageable and elastic but also fault-tolerant. As Kafka continues to evolve, Kafka Connect in distributed mode stands as a robust tool in the ever-growing Kafka ecosystem.