Apache Kafka is a highly robust and scalable event streaming platform, widely used for building real-time data pipelines and streaming applications. One of its most fundamental components is a messaging system, through which consumers can read and process data. There are several methods consumers can utilize to consume messages from Kafka, each with its specific usages and characteristics.

1. Single Consumer

The most straightforward method is using a single consumer in a consumer group or as an independent entity. The consumer subscribes to one or more topics and reads messages in the order they are stored in the partitions. The simplicity of this approach makes it useful for scenarios with minimal processing needs or low-throughput requirements.

Example:

1Properties props = new Properties();
2props.setProperty("bootstrap.servers", "localhost:9092");
3props.setProperty("group.id", "test-group");
4props.setProperty("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
5props.setProperty("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
6
7KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
8consumer.subscribe(Arrays.asList("topic1", "topic2"));
9while (true) {
10    ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
11    for (ConsumerRecord<String, String> record : records) {
12        System.out.printf("offset = %d, key = %s, value = %s%n", record.offset(), record.key(), record.value());
13    }
14}

2. Group of Consumers

For higher throughput and better fault tolerance, multiple consumers can form a consumer group to read from the same topic(s). Kafka distributes the partitions of a topic across the group members, allowing them to process messages concurrently. This approach provides load balancing and allows horizontal scaling.

Example: Suppose a topic has four partitions. In a consumer group of two consumers, each consumer might read from two partitions.

3. Manual Partition Assignment

Unlike subscribing to topics and leaving partition assignment to Kafka, consumers can manually assign themselves specific partitions. This approach provides more control over what data each consumer processes but requires more management effort and careful design to avoid imbalances in processing load.

Example:

1Properties props = new Properties();
2props.setProperty("bootstrap.servers", "localhost:9092");
3props.setProperty("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
4props.setProperty("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
5
6KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
7consumer.assign(Arrays.asList(new TopicPartition("topic1", 0), new TopicPartition("topic1", 1)));
8while (true) {
9    ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
10    for (ConsumerRecord<String, String> record : records) {
11        System.out.println("Received message: " + record.value());
12    }
13}

4. Replay Messages

Consumers can also replay messages from a specific offset or timestamp. This feature is useful for recovering from failures or re-processing data under certain conditions.

Example:

consumer.seek(new TopicPartition("topic1", 0), 10);  // Start reading from offset 10

5. Custom Consumer Logic

Kafka allows for custom consumer logic where you can define exactly how messages should be processed, what actions are to be taken on errors, how to handle offsets, or even how to integrate with external systems or databases.

Summary Table:

Conclusion

Choosing the right message consumption method in Kafka depends on specific project requirements, such as throughput, data processing needs, and system robustness. By understanding and utilizing these different methods effectively, developers can build highly efficient and scalable streaming applications using Apache Kafka.