KafkaListener
Concurrency
Multiple Topics
Message Queuing
Software Development

KafkaListener concurrency multiple topics

Master System Design with Codemia

Enhance your system design skills with over 120 practice problems, detailed solutions, and hands-on exercises.

Introduction

A Spring @KafkaListener can subscribe to multiple topics, and it can process records concurrently, but those two capabilities are related only indirectly. Concurrency comes from consumer threads and partition assignments, so the real limits are set by Kafka partitions and consumer-group membership, not by how many topic names you put into the annotation.

One Listener Can Subscribe to Many Topics

A single listener method can consume from more than one topic:

java
1import org.springframework.kafka.annotation.KafkaListener;
2import org.springframework.stereotype.Component;
3
4@Component
5public class OrderListener {
6
7    @KafkaListener(
8        topics = {"orders.created", "orders.cancelled"},
9        groupId = "order-service",
10        concurrency = "3"
11    )
12    public void listen(String payload) {
13        System.out.println(payload);
14    }
15}

This tells Spring Kafka to create one listener container for the method and up to three consumer threads inside that container.

What concurrency Actually Means

The concurrency setting is not “number of topics processed in parallel.” It is “number of Kafka consumer instances in the listener container.”

Those consumers then get partitions assigned by Kafka. This means actual parallelism depends on partition count.

Examples:

  • if the subscribed topics expose only two partitions total, concurrency 5 leaves idle consumers
  • if the subscribed topics expose ten partitions total, concurrency 5 can keep five consumers busy
  • ordering is still guaranteed only inside one partition

So concurrency scales with partitions, not with topic labels.

Why Multiple Topics Can Be Fine

Using one listener for multiple topics is reasonable when the topics share:

  • the same deserialization format
  • similar processing logic
  • similar throughput needs
  • the same retry and error-handling policy

For example, orders.created and orders.cancelled may both be JSON messages handled by the same domain service, so one listener can be a good operational fit.

But that is a design choice, not a requirement.

When Separate Listeners Are Better

Separate listeners are cleaner when the topics need different behavior:

java
1@KafkaListener(topics = "orders.created", groupId = "order-service", concurrency = "4")
2public void onCreated(String payload) {
3    System.out.println("created: " + payload);
4}
5
6@KafkaListener(topics = "orders.cancelled", groupId = "order-service", concurrency = "1")
7public void onCancelled(String payload) {
8    System.out.println("cancelled: " + payload);
9}

This is better when topics differ in:

  • expected traffic volume
  • deserializer type
  • SLA or retry policy
  • downstream processing cost

One noisy topic can otherwise dominate a shared listener and make tuning harder.

Container Factory Configuration

For larger applications, concurrency is often configured at the container-factory level rather than repeated on every annotation.

java
1import org.springframework.context.annotation.Bean;
2import org.springframework.context.annotation.Configuration;
3import org.springframework.kafka.config.ConcurrentKafkaListenerContainerFactory;
4import org.springframework.kafka.core.ConsumerFactory;
5
6@Configuration
7public class KafkaConfig {
8
9    @Bean
10    public ConcurrentKafkaListenerContainerFactory<String, String>
11    kafkaListenerContainerFactory(ConsumerFactory<String, String> consumerFactory) {
12        ConcurrentKafkaListenerContainerFactory<String, String> factory =
13            new ConcurrentKafkaListenerContainerFactory<>();
14        factory.setConsumerFactory(consumerFactory);
15        factory.setConcurrency(3);
16        return factory;
17    }
18}

That keeps concurrency policy in one place and reduces annotation noise.

Partition Count Is the Real Ceiling

This is the most important operational fact: Kafka assigns partitions, not arbitrary records, to consumers. If there are fewer partitions than consumer threads across the consumer group, some consumers will sit idle.

So before increasing concurrency, check:

  • how many partitions exist across the subscribed topics
  • how many application instances are already in the same group
  • whether downstream systems can handle more parallel processing

Increasing concurrency in Spring without enough partitions does not create new parallel work.

Ordering and Back Pressure

A higher concurrency setting also changes operational behavior:

  • more records can be processed simultaneously
  • ordering is only preserved per partition
  • downstream systems may see higher parallel load

If your consumer writes to a database or calls an external service, higher concurrency can expose locking, race conditions, or throughput bottlenecks outside Kafka itself.

Common Pitfalls

  • Assuming concurrency means one thread per topic instead of one consumer thread per listener container.
  • Setting concurrency higher than the total useful partition count and expecting better throughput.
  • Combining unrelated topics in one listener just because the annotation allows multiple topic names.
  • Expecting ordering across topics or across partitions when Kafka only guarantees order within a partition.
  • Tuning the listener container without checking whether the real bottleneck is a downstream database or external service.

Summary

  • '@KafkaListener can subscribe to multiple topics in a single method.'
  • Spring Kafka concurrency creates multiple consumer threads, not “one thread per topic.”
  • Effective parallelism is limited by Kafka partition assignments.
  • Shared listeners are fine for similar topics, but separate listeners are better when topics need different scaling or processing rules.
  • Tune concurrency with partitions, consumer-group layout, and downstream capacity in mind.

Course illustration
Course illustration

All Rights Reserved.