Introduction

If you want a Kafka consumer to pick up new topics automatically, the usual solution is not to poll listTopics() and resubscribe manually. The built-in Kafka approach is to subscribe with a topic pattern, keep polling, and let the consumer group rebalance when newly created topics match that pattern.

Prefer Pattern Subscription Over Manual Topic Discovery

The Kafka consumer API supports subscribing to a regular-expression pattern instead of a fixed list of topic names.

1import java.time.Duration;
2import java.util.Properties;
3import java.util.regex.Pattern;
4import org.apache.kafka.clients.consumer.ConsumerRecord;
5import org.apache.kafka.clients.consumer.ConsumerRecords;
6import org.apache.kafka.clients.consumer.KafkaConsumer;
7
8public class PatternConsumerExample {
9    public static void main(String[] args) {
10        Properties props = new Properties();
11        props.put("bootstrap.servers", "localhost:9092");
12        props.put("group.id", "orders-consumer");
13        props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
14        props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
15        props.put("auto.offset.reset", "earliest");
16
17        try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props)) {
18            consumer.subscribe(Pattern.compile("orders-.*"));
19
20            while (true) {
21                ConsumerRecords<String, String> records = consumer.poll(Duration.ofSeconds(1));
22                for (ConsumerRecord<String, String> record : records) {
23                    System.out.println(record.topic() + " -> " + record.value());
24                }
25            }
26        }
27    }
28}

If a new topic such as orders-eu is created and it matches the pattern, the consumer can be assigned partitions from that topic during the normal subscription and rebalance flow.

Why Manual Resubscribe Loops Are Usually the Wrong Tool

A common idea is to call listTopics(), compare the result with a saved set, and then call subscribe() again with an updated list. That seems reasonable, but it adds extra complexity and can interfere with stable consumer behavior.

Problems with manual discovery include:

• needless resubscription churn
• extra metadata requests
• more complicated state tracking
• harder-to-reason-about rebalances

Kafka already knows how to do dynamic pattern-based discovery. If the topic naming scheme allows it, let the consumer API handle the job directly.

Keep Polling So Rebalances Can Happen

Pattern subscription is not magic by itself. Group rebalances and assignment updates happen during active polling, so a consumer that subscribes and then stops polling will not behave the way you want.

1while (running) {
2    ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500));
3    // process records
4}

This is one of the reasons background scheduler tricks are often unnecessary. The main consumer loop is already the mechanism that drives membership, metadata refresh, and partition assignment updates.

Naming Conventions Matter

Dynamic discovery works best when the topic names follow a predictable convention. For example:

• 'orders-us'
• 'orders-eu'
• 'orders-apac'

Then a pattern such as orders-.* is easy to reason about. If the topics are unrelated and there is no stable naming scheme, dynamic pattern subscription may not be a good fit. In that case, an external source of truth or explicit topic configuration may be cleaner than trying to guess intent from broker metadata.

When a Manual Approach Is Still Reasonable

There are cases where pattern subscription is not enough. For example:

• topics must be selected from a registry or database, not from name shape alone
• only a subset of matching topics should be consumed
• tenant permissions determine eligibility dynamically

In those cases, a management layer outside the consumer may maintain the approved topic list and call subscribe() with that curated list. The key point is that this should be driven by business rules, not by reinventing Kafka's built-in pattern feature.

A Safer Manual Example

If you truly need manual topic selection, keep it simple and avoid changing the subscription unless the desired set actually changed.

1import java.time.Duration;
2import java.util.HashSet;
3import java.util.Set;
4
5Set<String> subscribed = new HashSet<>();
6
7while (true) {
8    Set<String> desired = fetchApprovedTopicsFromConfigStore();
9
10    if (!desired.equals(subscribed)) {
11        consumer.subscribe(desired);
12        subscribed = new HashSet<>(desired);
13    }
14
15    consumer.poll(Duration.ofSeconds(1));
16}

That is still more fragile than pattern subscription, but at least the subscription changes are deliberate and minimal.

Common Pitfalls

The biggest mistake is manually polling broker topic metadata when a regex subscription would have solved the problem with less code and less churn. Another is forgetting that group rebalances happen during active poll calls, so newly matching topics are not assigned if the consumer is not polling normally. Teams also underestimate how important topic naming conventions are. Without a predictable naming scheme, "dynamic detection" becomes vague and often turns into custom control-plane logic rather than consumer configuration.

Summary

• Use consumer.subscribe(Pattern.compile(...)) when new topics should be detected automatically by name.
• Keep the consumer polling so rebalances and new assignments can occur.
• Prefer naming conventions that make topic-pattern subscription clear and predictable.
• Avoid manual listTopics() plus resubscribe loops unless business rules require explicit control.
• If manual topic selection is necessary, drive it from a clear source of truth instead of ad hoc metadata polling.