Introduction

If you need live output from a child process, reading stdout line by line is usually better than waiting for the process to finish and collecting everything at once. Python's subprocess module supports this directly, but you need the right combination of text mode, buffering, and process cleanup to avoid confusing behavior.

The Simple Streaming Pattern

For line-based text output, the most straightforward approach is to start the process with Popen, enable text mode, and iterate over process.stdout.

1import subprocess
2
3command = ["python", "-u", "worker.py"]
4
5with subprocess.Popen(
6    command,
7    stdout=subprocess.PIPE,
8    stderr=subprocess.STDOUT,
9    text=True,
10    bufsize=1,
11) as process:
12    assert process.stdout is not None
13
14    for line in process.stdout:
15        print(f"child: {line.rstrip()}")
16
17    return_code = process.wait()
18    print(f"process exited with {return_code}")

This works well for logs, progress messages, and long-running commands. Using stderr=subprocess.STDOUT also avoids a common deadlock pattern where stdout is drained but stderr fills its pipe and blocks the child.

Why -u or Flushing Matters

Many people get stuck because their loop is correct but the child process buffers its own output. In the example above, python -u tells the child Python process to run unbuffered, which makes lines appear as they are printed.

The same issue exists with non-Python programs. If the child buffers heavily, your parent process cannot read lines that have not been flushed yet. In that case, look for an unbuffered mode, a flush option, or a logging configuration change in the child program.

Binary Mode and Manual Decoding

If you need raw bytes instead of decoded strings, keep text mode off and read with readline.

1import subprocess
2
3process = subprocess.Popen(
4    ["ping", "-c", "3", "localhost"],
5    stdout=subprocess.PIPE,
6    stderr=subprocess.STDOUT,
7)
8
9assert process.stdout is not None
10
11for raw_line in iter(process.stdout.readline, b""):
12    print(raw_line.decode("utf-8", errors="replace").rstrip())
13
14process.wait()

This pattern is useful when you need explicit encoding control or you are dealing with mixed binary and text data.

When communicate() Is Better

Streaming is not always the right tool. If you only need the final output after the child exits, process.communicate() is simpler and handles both pipes cleanly. Use line-by-line reading when timing matters, not just because it feels lower level.

That distinction is important in tests and short-lived scripts. Sometimes the simplest correct solution is to wait for completion and inspect the full captured output afterwards.

If you also need timeout control, wrap the wait step or the whole operation in your own timeout handling. Streaming output does not remove the need to decide what should happen when a child process stalls forever.

Common Pitfalls

The biggest trap is forgetting that the child may buffer output. When nothing appears, the parent loop is often innocent.

Another mistake is reading stdout line by line while leaving stderr unread in a separate pipe. If the child writes enough stderr data, it can block. Merge the streams or consume both.

Be careful to wait for the process and inspect its return code. Reaching end-of-file on stdout does not automatically mean the command succeeded.

Finally, remember that line iteration works best for text output that is actually line-oriented. If the child prints progress updates without newline characters, you need a different reading strategy.

Summary

• Use subprocess.Popen with stdout=PIPE and iterate over process.stdout for live line-based output.
• Enable text mode for normal string processing, or read bytes and decode manually when needed.
• Make sure the child process flushes output, or use an unbuffered mode if available.
• Avoid deadlocks by handling stderr deliberately.
• Use communicate() instead when you do not need real-time streaming.