Introduction

If you want to print subprocess output continuously while the process is still running, use subprocess.Popen and read from the pipe incrementally. The important details are line buffering, child-process flushing, and making sure you do not block forever waiting for output that the child has not flushed yet.

Why subprocess.run() Is Not Enough

subprocess.run() collects output and returns only after the process exits. That is fine for simple scripts, but it does not give you live progress.

For streaming output, you need a long-lived Popen object:

1import subprocess
2
3process = subprocess.Popen(
4    ["python", "-u", "child.py"],
5    stdout=subprocess.PIPE,
6    stderr=subprocess.STDOUT,
7    text=True,
8    bufsize=1,
9)

Now the parent can read output while the child is still running.

Read Line by Line

A common pattern is to iterate over stdout.readline until the stream closes.

1import subprocess
2
3process = subprocess.Popen(
4    ["python", "-u", "child.py"],
5    stdout=subprocess.PIPE,
6    stderr=subprocess.STDOUT,
7    text=True,
8    bufsize=1,
9)
10
11for line in iter(process.stdout.readline, ""):
12    print(line, end="")
13
14process.stdout.close()
15return_code = process.wait()
16print("exit code:", return_code)

This works well when the child writes newline-terminated output progressively.

Child Buffering Still Matters

If the child process buffers its own output heavily, the parent may not see lines immediately even with correct Python code. That is why the example uses python -u for the child process. The -u flag forces unbuffered output for Python children.

Without that, developers often think the parent is broken when the real problem is that the child has not flushed its output yet.

Merge stderr When Appropriate

Long-running tools often write progress or warnings to stderr. If you want one combined live stream, redirect stderr into stdout as shown above with:

stderr=subprocess.STDOUT

If you do not read stderr at all and the child writes enough to fill that pipe, the child can block even while you are reading stdout.

When You Need More Control

If the main program must keep doing other work while also consuming subprocess output, you may need a background thread, selectors, or asyncio rather than a simple foreground loop.

The foreground line-reading pattern is ideal when your main job is to mirror the child output live. If the parent has its own UI or event loop, then a more structured non-blocking design is often better.

Example Child Script

For testing, it helps to use a child program that prints slowly:

1# child.py
2import time
3
4for i in range(5):
5    print(f"step {i}", flush=True)
6    time.sleep(1)

With that child, the parent loop prints one line per second instead of waiting until the child exits.

Common Pitfalls

Using subprocess.run() when you actually need streaming output is the first design mistake.

Forgetting that the child process may buffer its own output can make the parent appear broken even when the reading code is correct.

Ignoring stderr can deadlock the child if the error pipe fills up while the parent reads only standard output.

Expecting a foreground line-reading loop to be "non-blocking" for the parent itself is incorrect because it still dedicates the current thread to output consumption.

Assuming every tool prints newline-terminated output can lead to apparent stalls when the child writes partial lines or progress bars.

Summary

• Use subprocess.Popen, not subprocess.run(), for live subprocess output.
• Read from stdout incrementally, usually line by line.
• Make sure the child process flushes its output if you expect real-time printing.
• Merge or otherwise consume stderr so the child does not block on an unread pipe.
• If the parent must stay responsive too, move output reading into a thread, selectors, or asyncio.