Introduction

When you use Python multiprocessing, each child process has its own memory space, so logging cannot be treated the same way as threading. The most reliable pattern is to centralize output in one listener process or thread and let worker processes send log records through a queue.

Why Direct File Logging Is Risky

If several processes all write to the same log file through independent handlers, the output can become interleaved, reordered, or partially corrupted depending on buffering and operating system behavior.

That is why the safe default is:

• worker processes create log records
• a central listener handles formatting and writing

This keeps one place responsible for the real output destination.

Use QueueHandler and QueueListener

Python's logging module already supports this pattern. Workers use QueueHandler, and the main process owns a QueueListener.

1import logging
2import multiprocessing
3from logging.handlers import QueueHandler, QueueListener
4
5
6def worker(log_queue, number):
7    logger = logging.getLogger(f"worker-{number}")
8    logger.setLevel(logging.INFO)
9    logger.addHandler(QueueHandler(log_queue))
10    logger.info("starting work")
11    logger.info("finished work")
12
13
14if __name__ == "__main__":
15    log_queue = multiprocessing.Queue()
16
17    stream_handler = logging.StreamHandler()
18    stream_handler.setFormatter(
19        logging.Formatter("%(processName)s %(name)s %(levelname)s: %(message)s")
20    )
21
22    listener = QueueListener(log_queue, stream_handler)
23    listener.start()
24
25    processes = [
26        multiprocessing.Process(target=worker, args=(log_queue, i))
27        for i in range(3)
28    ]
29
30    for p in processes:
31        p.start()
32
33    for p in processes:
34        p.join()
35
36    listener.stop()

This is the most practical answer for robust multiprocessing logging.

Why This Pattern Works

Each worker just enqueues LogRecord objects. The listener is the only component that formats and writes them, so:

• handlers are configured once
• output stays consistent
• file writes happen in one place

It is much easier to reason about than sharing file handlers across multiple processes.

Include Process Context in the Format

Multiprocessing logs are much easier to read when the formatter includes process name or process ID.

logging.Formatter(
    "%(asctime)s %(processName)s %(levelname)s %(message)s"
)

Without that information, logs from several workers quickly become hard to interpret.

Initialize Workers Explicitly

If your workers need a logger, configure the queue handler inside the worker entry point or through a process initializer. That avoids relying on implicit logger state copied from the parent.

This is especially important on platforms that use the spawn start method, where child processes begin with a fresh interpreter.

Separate Logging From Business Output

Avoid mixing print() calls and structured logging across worker processes. print() may be acceptable for quick experiments, but once you care about ordering, formatting, or severity, central logging through the queue becomes much easier to debug and much easier to ship to a file or log collector.

What Not to Do

Do not assume a logger configured in the parent will automatically behave exactly the same way in all children. Child processes may inherit some state on some platforms, but explicit setup is safer and clearer.

Also avoid printing from some workers and using structured logging from others. Mixed output channels make debugging far harder.

Common Pitfalls

• Writing directly to one file from many processes is the most common mistake.
• Child processes should configure a queue-based handler explicitly instead of relying on accidental inheritance.
• If the listener is not started before workers log, early messages may be lost or mishandled.
• Include process metadata in the log format or the output becomes difficult to trace.

Summary

• Treat multiprocessing logging as a centralized logging problem.
• Use QueueHandler in workers and QueueListener in the main process.
• Let one place own formatting and output handlers.
• Include process identity in the log format so concurrent activity remains understandable.