In the context of Apache Kafka, the term 'commit' refers primarily to the process of marking a specific message or a set of messages as processed within a consumer group, designating the consumer's position in a Kafka topic log as up-to-date. This ensures that each message is processed only once, even in the event of a failure, and is fundamental to Kafka’s ability to provide fault-tolerant message consumption.

Understanding Kafka's Message Committing

Kafka maintains a distributed log consisting of records (messages) that are appended sequentially. Consumers read from this log at their own pace. The position from which a consumer will begin reading after a restart or failure is controlled through offsets. An offset is a unique identifier for each record within a partition and signifies a specific location in the partition.

When a consumer reads messages from a partition, it needs a way to track which messages have been processed. This is managed by committing offsets. When a consumer processes messages from a partition, it can commit the offsets of messages to Kafka, which essentially means it is informing Kafka that it has successfully processed all preceding messages. As a result, in the event of a consumer restart or failure, Kafka will only deliver messages from the last committed offset, ensuring no message is processed more than once.

How Committing Works

Kafka consumers can commit offsets in two modes:

1. Automatic Offset Committing: This uses the enable.auto.commit setting in the consumer configuration, which is set to true by default. The interval at which offsets are committed can be controlled using the auto.commit.interval.ms setting.
2. Manual Offset Committing: This gives the programmer explicit control over when offsets are committed, typically after the consumer has finished processing messages. There are two common ways to manually commit offsets:
   • Synchronously: where commit operations will block the thread until the broker acknowledges it, ensuring more precise control over offset commits but introducing delays in consumption.
   • Asynchronously: which does not block the consumer while offsets are committed. If a commit fails, there's a callback mechanism to handle this situation.

Example

Consider a consumer application subscribed to a Kafka topic with partitions. The application is set up with automatic committing of offsets:

1Properties props = new Properties();
2props.put("bootstrap.servers", "localhost:9092");
3props.put("group.id", "test-consumer-group");
4props.put("enable.auto.commit", "true");
5props.put("auto.commit.interval.ms", "1000");
6props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
7props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
8
9KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
10consumer.subscribe(Arrays.asList("example-topic"));
11try {
12    while (true) {
13        ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
14        for (ConsumerRecord<String, String> record : records) {
15            System.out.printf("offset = %d, key = %s, value = %s%n", record.offset(), record.key(), record.value());
16        }
17    }
18} finally {
19    consumer.close();
20}

Key Points Table

Conclusion

In summary, committing in the context of a Kafka broker is crucial for ensuring message processing is reliable and fault-tolerant across consumer restarts and failures. By effectively managing offsets through committing, Kafka supports exactly-once and at-least-once processing semantics, which are vital for building robust streaming applications. Whether manually or automatically managed, understanding and implementing proper offset commits is fundamental for any application using Kafka for message consumption.