Introduction

Stopping a Kafka consumer cleanly is mostly about control flow, not a special Kafka shutdown command. The program needs a stop signal, a poll loop that checks it, a final offset decision if you are managing commits manually, and a guaranteed close() call.

That is the core pattern whether you use kafka-python or confluent_kafka. The consumer should finish its current work, stop polling for more records, and release its broker resources in a predictable way.

Use a Stop Flag Around the Poll Loop

With kafka-python, the usual structure is a loop controlled by a flag or event.

1from kafka import KafkaConsumer
2import threading
3
4stop_event = threading.Event()
5
6consumer = KafkaConsumer(
7    "orders",
8    bootstrap_servers=["localhost:9092"],
9    group_id="orders-group",
10    enable_auto_commit=False,
11)
12
13try:
14    while not stop_event.is_set():
15        records = consumer.poll(timeout_ms=1000)
16
17        for _, batch in records.items():
18            for message in batch:
19                print(message.value)
20
21        consumer.commit()
22finally:
23    consumer.close()

The timeout matters. If the consumer is blocked forever waiting for new data, it cannot notice your stop signal quickly.

Handle Signals in Long-Running Programs

For a service process, connect the stop flag to SIGINT or SIGTERM.

1import signal
2
3def request_shutdown(signum, frame):
4    stop_event.set()
5
6signal.signal(signal.SIGINT, request_shutdown)
7signal.signal(signal.SIGTERM, request_shutdown)

This gives the consumer a chance to finish the current loop, commit any final offsets you intend to keep, and close normally instead of being killed mid-processing.

The same idea works with confluent_kafka, except you poll one message at a time:

1from confluent_kafka import Consumer
2
3consumer = Consumer({
4    "bootstrap.servers": "localhost:9092",
5    "group.id": "orders-group",
6    "auto.offset.reset": "earliest",
7})
8
9consumer.subscribe(["orders"])
10
11try:
12    while not stop_event.is_set():
13        msg = consumer.poll(1.0)
14        if msg is None:
15            continue
16        if msg.error():
17            continue
18
19        print(msg.value().decode("utf-8"))
20finally:
21    consumer.close()

Decide When to Commit Offsets

Stopping the consumer and committing offsets are related, but they are not the same decision. If you commit before processing is safe, you can skip work after restart. If you never commit, you can replay work unnecessarily.

A reasonable rule is:

• auto-commit for simple low-risk consumers
• manual commit after successful processing for important workloads

That is why graceful shutdown often includes one final commit only if the current batch has really been handled.

In worker-style programs, it is also worth making the shutdown path idempotent. If the stop signal is delivered twice, the code should still just set the same flag and let the existing loop exit naturally.

That small detail prevents shutdown code from becoming its own source of bugs during deployments, container restarts, or repeated signal delivery from supervisors.

Common Pitfalls

The biggest mistake is breaking out of the loop without closing the consumer. That leaves sockets and group membership cleanup to timeouts instead of ending cleanly.

Another common issue is using an infinite loop with a blocking poll and no timeout. The program cannot react quickly to a shutdown request if it never returns to the loop body.

It is also easy to commit offsets blindly during shutdown even when the current messages have not finished processing. That can create message loss after restart.

Finally, do not rely on KeyboardInterrupt alone in production services. Use signal handling or another explicit shutdown path so the consumer lifecycle is under program control.

Summary

• Use a stop flag or event to control the consumer loop.
• Poll with a timeout so shutdown can be noticed promptly.
• Close the consumer in a finally block.
• Commit offsets according to processing success, not just because the program is exiting.
• Wire graceful shutdown to process signals in long-running services.