Introduction

A kafka-python consumer does not start from one universal "default offset." The starting position depends first on whether the consumer group already has a committed offset and only then, if no valid committed offset exists, on the auto_offset_reset policy.

The Two Main Cases

When a consumer joins Kafka with a group_id, Kafka first looks for committed offsets for that group and partition.

• If a committed offset exists, the consumer resumes from there.
• If no valid committed offset exists, auto_offset_reset decides the fallback behavior.

That is the key mental model. Many offset bugs come from assuming auto_offset_reset always applies, when in reality it is only a fallback.

Basic Group-Managed Consumer

Here is a normal group-managed consumer:

1from kafka import KafkaConsumer
2
3consumer = KafkaConsumer(
4    "orders",
5    bootstrap_servers="localhost:9092",
6    group_id="orders-reader",
7    auto_offset_reset="earliest",
8    enable_auto_commit=True,
9)
10
11for message in consumer:
12    print(message.offset, message.value.decode())

In this setup, earliest matters only when the group has no valid stored offset for a partition. Once offsets are committed, the consumer usually resumes from those commits instead.

What auto_offset_reset Really Means

The two common choices are:

• 'earliest, start from the oldest available record,'
• 'latest, start from the end and consume only new records.'

This choice applies only when Kafka cannot resume from a valid committed position. That may happen the first time a group reads a topic, or later if the stored offset is no longer usable because retention has removed the old data.

Start From an Exact Offset With seek

If you need deterministic replay from a specific offset, do not rely on reset policies. Assign the partition manually and seek explicitly.

1from kafka import KafkaConsumer, TopicPartition
2
3consumer = KafkaConsumer(
4    bootstrap_servers="localhost:9092",
5    enable_auto_commit=False,
6    auto_offset_reset="none",
7)
8
9tp = TopicPartition("orders", 0)
10consumer.assign([tp])
11consumer.seek(tp, 42)
12
13for message in consumer:
14    print(message.offset, message.value.decode())
15    if message.offset >= 45:
16        break

This bypasses normal group-offset resume behavior and gives you exact control over the starting position.

Reset the Position of a Group-Managed Consumer

Sometimes you still want a group-managed consumer, but you want to move it programmatically before processing. In that case, wait for partition assignment and then seek.

1from kafka import KafkaConsumer
2
3consumer = KafkaConsumer(
4    "orders",
5    bootstrap_servers="localhost:9092",
6    group_id="orders-replay",
7    enable_auto_commit=False,
8    auto_offset_reset="earliest",
9)
10
11consumer.poll(timeout_ms=1000)
12
13for tp in consumer.assignment():
14    consumer.seek_to_beginning(tp)
15
16for message in consumer:
17    print(message.offset, message.value.decode())

The initial poll matters because assignment usually happens during polling. Without assigned partitions, there is nothing to seek.

Use a New Group for Experiments

A very practical debugging trick is to use a fresh throwaway group_id when you want to see auto_offset_reset in action. If you reuse an old group, the previously committed offsets will often override the reset policy and make the behavior look confusing.

That is why replay and debugging often work better in a temporary consumer group than in a long-lived production one.

Be Careful With Auto-Commit

During replay or experiments, automatic commits can silently store progress you did not intend to keep. If you are investigating offsets or replaying history, it is often safer to disable auto-commit until you know exactly what should be persisted.

That way the consumer does not accidentally rewrite the group's position just because you ran a debugging script once.

Common Pitfalls

A common mistake is expecting auto_offset_reset to override an existing committed offset. It does not.

Another issue is calling seek before the consumer has partitions assigned. In group-managed mode, that usually means polling first.

Teams also often leave enable_auto_commit=True during replay experiments and then wonder why the group's position changed afterward.

Summary

• A kafka-python consumer resumes from committed offsets when they exist.
• 'auto_offset_reset applies only when no valid committed offset is available.'
• Use seek for exact-offset replay instead of relying on reset policy.
• In group-managed mode, wait for assignment before seeking.
• Disable auto-commit during replay or debugging when you do not want to persist the new position.