Introduction

If you want Python log records to go directly to Kafka, the standard approach is to write a custom logging.Handler that serializes each log record and hands it to a Kafka producer. The important design choice is not just how to send the message, but how to do it without blocking the application or flushing the producer on every single log call.

Build A Custom Logging Handler

1import json
2import logging
3from kafka import KafkaProducer
4
5
6class KafkaLoggingHandler(logging.Handler):
7    def __init__(self, bootstrap_servers: str, topic: str):
8        super().__init__()
9        self.topic = topic
10        self.producer = KafkaProducer(
11            bootstrap_servers=bootstrap_servers,
12            value_serializer=lambda value: json.dumps(value).encode("utf-8"),
13        )
14
15    def emit(self, record: logging.LogRecord) -> None:
16        message = {
17            "name": record.name,
18            "level": record.levelname,
19            "message": self.format(record),
20            "timestamp": record.created,
21        }
22        self.producer.send(self.topic, message)
23
24    def close(self) -> None:
25        try:
26            self.producer.flush()
27            self.producer.close()
28        finally:
29            super().close()

This integrates with Python's logging framework while keeping Kafka concerns inside the handler.

Attach The Handler To A Logger

1logger = logging.getLogger("app")
2logger.setLevel(logging.INFO)
3
4handler = KafkaLoggingHandler("localhost:9092", "application-logs")
5handler.setFormatter(logging.Formatter("%(message)s"))
6
7logger.addHandler(handler)
8logger.info("service started")

Now log messages go through the normal logging system and are forwarded to Kafka.

Do Not Flush On Every Emit

A common beginner implementation calls flush() inside emit(). That is simple, but it destroys throughput because every log call waits for producer delivery work.

Bad idea inside emit:

self.producer.send(self.topic, message)
self.producer.flush()

It is usually better to let the Kafka producer batch naturally and flush on shutdown or at controlled checkpoints.

Include Structured Fields

Kafka becomes more useful when logs are structured rather than raw strings.

1message = {
2    "logger": record.name,
3    "level": record.levelname,
4    "message": record.getMessage(),
5    "module": record.module,
6    "line": record.lineno,
7}

This makes downstream indexing, filtering, and alerting easier.

Error Handling In A Logging Handler

A logging handler should not crash the whole process if Kafka is temporarily unavailable. Override handleError behavior only when you have a clear policy, and think carefully about whether dropped logs, fallback logging, or retries are appropriate for your application.

Remember that logging infrastructure should usually fail more gracefully than the business path it observes.

Shutdown Cleanly

Because the Kafka producer holds network resources and buffers, close it during application shutdown.

import atexit

atexit.register(handler.close)

Without an orderly flush and close, buffered log messages may never reach Kafka.

Choice Of Kafka Client

The example above uses kafka-python, which is easy to demonstrate. Some systems prefer confluent-kafka for higher throughput or operational reasons. The logging-handler pattern stays the same even if the producer library changes.

The key abstraction is the logging.Handler, not the exact Kafka client package.

Queue-Based Logging Is Often Better

If Kafka publishing should never slow down the code path producing the log, combine a Kafka handler with QueueHandler and QueueListener so application threads enqueue records and a background thread performs the Kafka I/O. That keeps business logic less exposed to broker latency spikes.

Common Pitfalls

The biggest mistake is flushing on every emit, which turns logging into a latency bottleneck. Another is sending only formatted strings and then regretting the loss of structured fields downstream. Developers also sometimes let Kafka producer exceptions bubble out of emit, which can interfere with normal application execution. Finally, forgetting to close the producer on shutdown can silently drop buffered log records.

Summary

• Send Python logs to Kafka by implementing a custom logging.Handler.
• Serialize log records into structured payloads before sending.
• Avoid flushing on every log event so batching can work.
• Close the producer cleanly to avoid losing buffered messages.
• The handler pattern is the main idea, regardless of which Kafka client library you use.