Introduction

To read all messages from a Kafka topic, the consumer must start at the beginning of every partition rather than resuming from previously committed offsets. The exact method depends on whether you are using the command-line tools for a one-time inspection or writing a consumer application that explicitly seeks to the earliest offsets.

Understand What "All Messages" Means In Kafka

Kafka stores records by topic and partition. There is no single flat list of messages behind a topic. Reading "everything" means:

• enumerate the topic partitions
• begin at the earliest offset for each partition
• consume until the current end offset

That also means retention matters. If older records have already expired due to retention policy, they are gone and cannot be read back.

Quick CLI Method

For ad hoc inspection, the console consumer is the easiest tool.

1kafka-console-consumer \
2  --bootstrap-server localhost:9092 \
3  --topic my-topic \
4  --from-beginning

--from-beginning tells Kafka to start at the earliest available offset instead of the latest.

For large topics, this may print a lot of data, so it is often used with output redirection or a filter.

Important Consumer Group Detail

If you run a regular consumer with a group that already has committed offsets, Kafka will typically resume from those offsets instead of replaying the whole topic.

That is why a one-time full read often uses:

• a brand-new group ID
• or explicit seekToBeginning
• or a CLI command designed for replay from the start

If you reuse an existing consumer group, you may get only new messages rather than the full topic history.

Java Consumer Example

A programmatic consumer can subscribe, wait for partition assignment, and then explicitly seek to the beginning.

1import java.time.Duration;
2import java.util.Collections;
3import java.util.Properties;
4import org.apache.kafka.clients.consumer.ConsumerConfig;
5import org.apache.kafka.clients.consumer.ConsumerRecord;
6import org.apache.kafka.clients.consumer.ConsumerRecords;
7import org.apache.kafka.clients.consumer.KafkaConsumer;
8
9Properties props = new Properties();
10props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
11props.put(ConsumerConfig.GROUP_ID_CONFIG, "read-all-example");
12props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringDeserializer");
13props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringDeserializer");
14props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
15
16KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
17consumer.subscribe(Collections.singletonList("my-topic"));
18consumer.poll(Duration.ofSeconds(1));
19consumer.seekToBeginning(consumer.assignment());
20
21while (true) {
22    ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500));
23    for (ConsumerRecord<String, String> record : records) {
24        System.out.println(record.offset() + " => " + record.value());
25    }
26}

The explicit seekToBeginning is the important step when you want the replay behavior to be unambiguous.

Reading Until The End, Then Stopping

If you want a bounded read rather than an infinite consumer loop, compare current offsets to end offsets and stop when caught up. That is useful for export jobs or debugging tools.

The key idea is that "read all messages" usually means "read all messages currently retained," not "stay subscribed forever."

For shell-based inspection, many teams also use kcat because it is convenient for quick replay and filtering. The underlying concept is the same: start at the earliest retained offset and read until the current end.

Common Pitfalls

The most common mistake is setting auto.offset.reset=earliest and assuming that guarantees a full replay even for an existing consumer group. It usually affects only partitions with no committed offset yet.

Another mistake is forgetting that topics are partitioned. To read everything, you need to cover every partition’s earliest-to-latest offset range.

A third issue is expecting expired messages to still exist. Kafka retention can delete old records even though the topic still exists.

Summary

• To read all topic messages, start from the earliest retained offsets for every partition.
• For quick inspection, use kafka-console-consumer --from-beginning.
• In code, use a fresh group or call seekToBeginning explicitly.
• Existing committed offsets can prevent a full replay if you reuse a consumer group.
• Kafka can only return messages still retained under the topic’s retention policy.