Introduction

Asynchronously logging stdout and stderr usually means one of two things: reading another process's output without blocking your event loop, or decoupling log writes from the main execution path. The most common real-world need is the first one, especially when you launch subprocesses with asyncio. The right pattern is to read each stream concurrently and forward lines into the normal Python logging system.

Reading stdout and stderr with asyncio

asyncio.create_subprocess_exec exposes stdout and stderr as asynchronous streams when you pass asyncio.subprocess.PIPE.

1import asyncio
2import logging
3
4logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")
5logger = logging.getLogger(__name__)
6
7
8async def log_stream(stream, level):
9    while True:
10        line = await stream.readline()
11        if not line:
12            break
13        logger.log(level, line.decode().rstrip())
14
15
16async def main():
17    process = await asyncio.create_subprocess_exec(
18        "python",
19        "-c",
20        "import sys; print('hello'); print('oops', file=sys.stderr)",
21        stdout=asyncio.subprocess.PIPE,
22        stderr=asyncio.subprocess.PIPE,
23    )
24
25    await asyncio.gather(
26        log_stream(process.stdout, logging.INFO),
27        log_stream(process.stderr, logging.ERROR),
28    )
29
30    return_code = await process.wait()
31    logger.info("Process exited with code %s", return_code)
32
33
34asyncio.run(main())

This pattern keeps both streams draining concurrently, so one stream cannot block the other by filling its pipe buffer.

Why Separate Tasks Matter

If you read all of stdout first and only then read stderr, a noisy child process can deadlock if the unread stream fills up. That is why concurrent consumption matters.

Using asyncio.gather with one coroutine per stream is the simple, correct pattern in most cases. It also lets you tag each line with a different log level so errors show up distinctly.

Logging Your Own App Output Asynchronously

If you mean asynchronous logging for your own application rather than a subprocess, the built-in logging module is still the usual destination. The async part is often about putting messages onto a queue or keeping file I/O off the hot path.

For many apps, though, normal logging calls from async code are enough. The real bottleneck is usually external I/O, not the logging function itself. It is better to start simple and only add queue-based handlers when profiling shows logging overhead is real.

If you do need more separation, the next step is usually not a custom async logger but a queue-backed logging setup. That lets subprocess-reading coroutines stay lean while a dedicated consumer formats or ships log records elsewhere.

Handling Partial Lines and Binary Output

The readline() pattern assumes line-oriented text output. If the child process writes binary data or very long chunks without newlines, you may need read(n) instead and your own buffering logic.

You should also decide how to decode bytes. UTF-8 is common, but tools can emit different encodings. A more defensive decode looks like this:

text = line.decode("utf-8", errors="replace").rstrip()

That prevents decoding failures from crashing the logging task.

Common Pitfalls

• Reading stdout and stderr sequentially instead of concurrently.
• Forgetting that subprocess pipe buffers can fill and block the child process.
• Assuming all output is clean UTF-8 text.
• Logging from async code with blocking custom handlers that defeat the point of the design.
• Overengineering queue-based logging before measuring whether plain logging is actually a bottleneck.

Summary

• The standard async pattern is to read stdout and stderr concurrently with asyncio.
• One coroutine per stream avoids deadlocks and keeps the event loop responsive.
• Forward the decoded lines into Python's normal logging system with appropriate levels.
• Use line-based reading only when the child output is actually line-oriented. For command runners, CI tools, and async process supervisors, this line-by-line approach is usually exactly what you want because it preserves timing and severity information while the process is still running.
• Keep the solution simple until real profiling shows logging overhead is the problem.