Practice Now
Python
subprocess
callback
real-time output
programming

python subprocess callback when updated output

Interview Questions practice on Codemia

Over 8,000 real interview questions from top companies, searchable by company and role.

Browse interview questions
Python

Introduction

To react to subprocess output in real time, open the process with subprocess.Popen, set stdout=subprocess.PIPE, and read lines in a loop. Each line can trigger a callback function for logging, progress tracking, or parsing. For asynchronous handling, use asyncio.create_subprocess_exec with process.stdout.readline() in an async loop. The key is avoiding communicate() or run() when you need incremental output — both buffer everything until the process exits.

Basic Real-Time Line Reading

python
1import subprocess
2
3def on_output(line):
4    print(f"[CALLBACK] {line}")
5
6process = subprocess.Popen(
7    ["ping", "-c", "4", "google.com"],
8    stdout=subprocess.PIPE,
9    stderr=subprocess.PIPE,
10    text=True,
11    bufsize=1  # Line-buffered
12)
13
14for line in iter(process.stdout.readline, ""):
15    on_output(line.strip())
16
17process.wait()
18print(f"Exit code: {process.returncode}")

The iter(readline, "") pattern reads lines until the pipe returns an empty string (process exits). Each line fires the callback immediately.

Callback with Progress Parsing

python
1import subprocess
2import re
3
4def progress_callback(percent, message):
5    bar = "=" * (percent // 2) + " " * (50 - percent // 2)
6    print(f"\r[{bar}] {percent}% {message}", end="", flush=True)
7
8def run_with_progress(cmd, callback):
9    process = subprocess.Popen(
10        cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
11        text=True, bufsize=1
12    )
13
14    for line in iter(process.stdout.readline, ""):
15        line = line.strip()
16        # Parse progress from output like "Progress: 45%"
17        match = re.search(r"(\d+)%", line)
18        if match:
19            callback(int(match.group(1)), line)
20
21    process.wait()
22    if process.returncode != 0:
23        raise subprocess.CalledProcessError(process.returncode, cmd)
24
25run_with_progress(["my-build-tool", "--verbose"], progress_callback)

Handling Both stdout and stderr

python
1import subprocess
2import threading
3
4def run_with_callbacks(cmd, on_stdout, on_stderr):
5    process = subprocess.Popen(
6        cmd,
7        stdout=subprocess.PIPE,
8        stderr=subprocess.PIPE,
9        text=True,
10        bufsize=1
11    )
12
13    def read_stream(stream, callback):
14        for line in iter(stream.readline, ""):
15            callback(line.strip())
16        stream.close()
17
18    # Read stdout and stderr in separate threads to avoid deadlock
19    stdout_thread = threading.Thread(
20        target=read_stream, args=(process.stdout, on_stdout)
21    )
22    stderr_thread = threading.Thread(
23        target=read_stream, args=(process.stderr, on_stderr)
24    )
25
26    stdout_thread.start()
27    stderr_thread.start()
28    stdout_thread.join()
29    stderr_thread.join()
30
31    return process.wait()
32
33exit_code = run_with_callbacks(
34    ["make", "build"],
35    on_stdout=lambda line: print(f"[OUT] {line}"),
36    on_stderr=lambda line: print(f"[ERR] {line}")
37)

Using threads prevents deadlock — if both stdout and stderr buffers fill up and you only read one, the process blocks waiting for the other to drain.

Async Subprocess with asyncio

python
1import asyncio
2
3async def run_async(cmd, callback):
4    process = await asyncio.create_subprocess_exec(
5        *cmd,
6        stdout=asyncio.subprocess.PIPE,
7        stderr=asyncio.subprocess.PIPE
8    )
9
10    async for line in process.stdout:
11        callback(line.decode().strip())
12
13    await process.wait()
14    return process.returncode
15
16async def main():
17    code = await run_async(
18        ["python", "-u", "-c", "import time\nfor i in range(5):\n print(f'Step {i}')\n time.sleep(1)"],
19        callback=lambda line: print(f"[ASYNC] {line}")
20    )
21    print(f"Done with code {code}")
22
23asyncio.run(main())

Timeout with Real-Time Output

python
1import subprocess
2import threading
3
4def run_with_timeout(cmd, callback, timeout=30):
5    process = subprocess.Popen(
6        cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
7        text=True, bufsize=1
8    )
9
10    output_lines = []
11
12    def read_output():
13        for line in iter(process.stdout.readline, ""):
14            stripped = line.strip()
15            output_lines.append(stripped)
16            callback(stripped)
17
18    reader = threading.Thread(target=read_output)
19    reader.start()
20    reader.join(timeout=timeout)
21
22    if process.poll() is None:
23        process.kill()
24        process.wait()
25        raise TimeoutError(
26            f"Process timed out after {timeout}s. Last output: {output_lines[-1] if output_lines else 'none'}"
27        )
28
29    return process.returncode

Buffering Fix for Child Processes

python
1import subprocess
2import os
3
4# Problem: child process buffers stdout, no real-time output
5# Solution 1: Python child — use -u flag for unbuffered output
6process = subprocess.Popen(
7    ["python", "-u", "worker.py"],
8    stdout=subprocess.PIPE, text=True, bufsize=1
9)
10
11# Solution 2: Set PYTHONUNBUFFERED environment variable
12env = os.environ.copy()
13env["PYTHONUNBUFFERED"] = "1"
14process = subprocess.Popen(
15    ["python", "worker.py"],
16    stdout=subprocess.PIPE, text=True, bufsize=1,
17    env=env
18)
19
20# Solution 3: Use stdbuf for non-Python programs
21process = subprocess.Popen(
22    ["stdbuf", "-oL", "./my_program"],  # Line-buffer stdout
23    stdout=subprocess.PIPE, text=True, bufsize=1
24)

Common Pitfalls

  • Using process.communicate() for real-time output: communicate() waits for the process to finish and returns all output at once. It is designed for batch processing, not streaming. Use readline() in a loop or async iteration for incremental output.
  • Deadlock when reading both stdout and stderr sequentially: If you read all of stdout first, stderr may fill its buffer, blocking the child process. The child never finishes writing to stdout because it is stuck on stderr. Use threads or stderr=subprocess.STDOUT to merge both streams.
  • Child process buffering delays output: Many programs buffer stdout when it is a pipe (not a TTY). Python buffers by default. Use python -u, set PYTHONUNBUFFERED=1, or use stdbuf -oL for C programs. Without this, output arrives in large chunks instead of line by line.
  • Forgetting bufsize=1 on the parent side: Even if the child sends unbuffered output, the parent Popen may buffer reads. Set bufsize=1 (line-buffered) and text=True to ensure readline() returns as soon as a newline arrives.
  • Not closing the process on callback exceptions: If the callback raises an exception, the loop exits but the subprocess may keep running. Wrap the read loop in try/finally with process.kill() and process.wait() to ensure cleanup on errors.

Summary

  • Use Popen with stdout=PIPE and iter(stdout.readline, "") for real-time line-by-line output
  • Read stdout and stderr in separate threads to prevent deadlock
  • Use asyncio.create_subprocess_exec for async subprocess handling
  • Fix child buffering with python -u, PYTHONUNBUFFERED=1, or stdbuf -oL
  • Always set bufsize=1 and text=True on the parent for line-buffered reads
Related reading
Free course
Beginner
7 lessons
2 hours
Tackling System Design Interview Problems

A short course that equips you with the skills to approach system design interviews methodically.

Start the free course
Track what you have practised

A free account saves your progress, solutions and study plan across every problem on Codemia.

Interview Questions practice on Codemia

Over 8,000 real interview questions from top companies, searchable by company and role.

Browse interview questions
Codemia
© 2026 Codemia
Resources
Legal
Social
All Rights Reserved.