Apache Kafka is a distributed streaming platform that is used for building real-time data pipelines and streaming applications. It frequently serves as a backbone for data ingestion and dissemination across large-scale, data-driven environments. Spring Boot, a widely used framework for building Java applications, offers seamless integration with Kafka through the Spring Kafka project.

When working with microservices or systems that require interactions with multiple Kafka clusters, configuring a Spring Boot application to accommodate this can be somewhat tricky but entirely feasible. In this article, we’ll explore how to configure and consume messages from multiple Kafka clusters within a single Spring Boot application.

Basic Configuration

Let’s start with the fundamentals. By default, Spring Boot allows configuring a Kafka producer and consumer through application properties or a YAML file. However, when dealing with multiple Kafka clusters, you need to explicitly define each cluster's configuration programmatically.

Here's the basic concept: each Kafka cluster should have its own set of configuration beans such as ConsumerFactory, ProducerFactory, and KafkaTemplate. These beans should be defined as separate Configuration classes within the Spring Boot application.

Step-by-step Configuration Approach

1. Define Kafka Properties

Firstly, you need to define properties for each Kafka cluster you intend to interact with. Here’s an example of how you might define these properties in application.yml:

1kafka:
2  cluster1:
3    bootstrap-servers: server1:9092
4    group-id: group1
5    auto-offset-reset: earliest
6  cluster2:
7    bootstrap-servers: server2:9092
8    group-id: group2
9    auto-offset-reset: latest

2. Create Configuration Classes

For each cluster, create a Java configuration class that specifies the producer and consumer factory beans.

1@Configuration
2public class KafkaCluster1Config {
3
4    @Value("${kafka.cluster1.bootstrap-servers}")
5    private String bootstrapServers;
6
7    @Bean
8    public ProducerFactory<String, String> producerFactoryCluster1() {
9        Map<String, Object> configProps = new HashMap<>();
10        configProps.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);
11        configProps.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
12        configProps.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
13        return new DefaultKafkaProducerFactory<>(configProps);
14    }
15
16    @Bean
17    public KafkaTemplate<String, String> kafkaTemplateCluster1() {
18        return new KafkaTemplate<>(producerFactoryCluster1());
19    }
20    
21    // Similarly define ConsumerFactory and other necessary beans
22}

Repeat this for all the clusters you need to configure, changing the class and bean names as necessary.

3. Using the Kafka Templates

Now that you have configured your clusters, you can inject the respective KafkaTemplate into your services or controllers to produce messages to the specific Kafka clusters:

1@Service
2public class MyService {
3    
4    @Autowired
5    private KafkaTemplate<String, String> kafkaTemplateCluster1;
6    
7    @Autowired
8    private KafkaTemplate<String, String> kafkaTemplateCluster2;
9    
10    public void sendToKafkaCluster1(String msg) {
11        kafkaTemplateCluster1.send("someTopic", msg);
12    }
13    
14    public void sendToKafkaCluster2(String msg) {
15        kafkaTemplateCluster2.send("anotherTopic", msg);
16    }
17}

Summary Table

Here’s a quick overview of the key configurations and their purposes:

Conclusion

Configuring multiple Kafka clusters in a single Spring Boot application involves defining separate configuration settings for each cluster. By isolating the configuration for each Kafka instance, your application can interact with multiple clusters efficiently, making it flexible to scale and maintain.

This setup is particularly useful in complex environments where services need to interact with different isolated Kafka clusters due to reasons like geographical distribution, data segregation, or organizational policies.