Introduction

Reading from subprocess.PIPE is blocking by default, which becomes a problem when your main program needs to stay responsive while a child process is still running. The safest non-blocking pattern depends on your environment: threads are the most portable option, while selectors or asyncio are good fits when the rest of the program is already event-driven.

Why Pipe Reads Block

If you call read, readline, or iterate over a pipe directly, Python waits until data is available or the stream closes.

1import subprocess
2
3proc = subprocess.Popen(
4    ["python", "-u", "child.py"],
5    stdout=subprocess.PIPE,
6    text=True,
7)
8
9line = proc.stdout.readline()
10print(line)

That readline() call blocks until a full line arrives. If the child is slow or writing partial output, your program waits too.

The Most Portable Pattern: Thread Plus Queue

A background reader thread is often the simplest cross-platform answer. The blocking read happens in the thread, while the main program polls a queue without freezing.

1import queue
2import subprocess
3import threading
4import time
5
6
7def enqueue_output(pipe, output_queue):
8    for line in iter(pipe.readline, ""):
9        output_queue.put(line)
10    pipe.close()
11
12
13proc = subprocess.Popen(
14    ["python", "-u", "child.py"],
15    stdout=subprocess.PIPE,
16    stderr=subprocess.STDOUT,
17    text=True,
18    bufsize=1,
19)
20
21q = queue.Queue()
22thread = threading.Thread(target=enqueue_output, args=(proc.stdout, q), daemon=True)
23thread.start()
24
25while proc.poll() is None or not q.empty():
26    try:
27        line = q.get_nowait()
28        print("received:", line.strip())
29    except queue.Empty:
30        time.sleep(0.05)

This pattern is popular because it works on Windows and Unix-like systems without relying on pipe readiness behavior that varies by platform.

selectors for POSIX-Style Readiness

If your program runs on Unix-like systems and you want readiness-based polling, selectors can be a good fit.

1import selectors
2import subprocess
3
4proc = subprocess.Popen(
5    ["python", "-u", "child.py"],
6    stdout=subprocess.PIPE,
7    text=True,
8    bufsize=1,
9)
10
11selector = selectors.DefaultSelector()
12selector.register(proc.stdout, selectors.EVENT_READ)
13
14while proc.poll() is None:
15    for key, _ in selector.select(timeout=0.1):
16        line = key.fileobj.readline()
17        if line:
18            print("received:", line.strip())

This is clean on POSIX systems, but it is less portable for subprocess pipes on Windows than the thread approach.

asyncio for Async Programs

If the surrounding application already uses asyncio, the async subprocess APIs are usually the best match.

1import asyncio
2
3
4async def main():
5    proc = await asyncio.create_subprocess_exec(
6        "python", "-u", "child.py",
7        stdout=asyncio.subprocess.PIPE,
8        stderr=asyncio.subprocess.STDOUT,
9    )
10
11    while True:
12        line = await proc.stdout.readline()
13        if not line:
14            break
15        print("received:", line.decode().strip())
16
17    await proc.wait()
18
19
20asyncio.run(main())

This still waits for child output, but it does not block the entire event loop the way a normal pipe read would.

Buffering Is Part of the Problem Too

Sometimes the parent code is fine and the child process is what is buffering output. If you expect line-by-line updates, the child must flush appropriately.

For Python child processes, the -u flag helps:

python -u child.py

Without line buffering or unbuffered output, the parent may appear to "miss" data even though the child simply has not flushed it yet.

Avoid Deadlocks from Unread Streams

If the child writes enough data to stdout or stderr and the parent never drains it, the child can block because the pipe buffer fills up. That is why it is important either to consume both streams or to redirect one into the other.

If you only need all output after the process exits, subprocess.communicate() is often simpler than building a streaming non-blocking reader at all.

Common Pitfalls

Trying to make the parent non-blocking while the child still buffers heavily is a common reason streaming output appears inconsistent.

Using readiness-based APIs and assuming subprocess pipe behavior is identical on Unix and Windows creates portability bugs.

Reading only stdout and ignoring stderr can still deadlock the child if the error stream fills its buffer.

Building a complex streaming solution when communicate() would have solved the actual use case is unnecessary complexity.

Treating one blocking readline() call as harmless can freeze the rest of the program if the child stalls.

Summary

• Reads from subprocess.PIPE block by default.
• A reader thread plus queue.Queue is the most portable non-blocking pattern.
• 'selectors is useful on Unix-like systems for readiness-based reading.'
• 'asyncio is the best fit when the rest of the application is already asynchronous.'
• Always account for child-process buffering and unread stderr when diagnosing hangs.