Introduction

Running an external command asynchronously in Python means starting a child process without blocking the rest of your application. The right API depends on the rest of your program: subprocess.Popen fits normal synchronous code, while asyncio.create_subprocess_exec is the right choice when your application already uses an event loop.

The important distinction is not only how you start the process, but how you handle output, timeouts, and cleanup. A child process that starts in the background but is never monitored can still hang, fill buffers, or become an orphaned process.

Use subprocess.Popen in Synchronous Programs

If your program is otherwise synchronous, subprocess.Popen is the standard tool. It starts the child process immediately and returns a process handle that you can poll, wait on, or terminate later.

1import subprocess
2import time
3
4process = subprocess.Popen(
5    ["python3", "-c", "import time; time.sleep(2); print('done')"]
6)
7
8print("child started")
9
10while process.poll() is None:
11    print("main program still running")
12    time.sleep(0.5)
13
14print("exit code:", process.returncode)

This is asynchronous in the sense that the parent keeps running while the child is still working. The child is not tied to an event loop. It is just another operating-system process.

Capture Output Safely

If you need stdout or stderr, attach pipes and use communicate(). That method reads the streams safely and waits for the child to finish:

1import subprocess
2
3process = subprocess.Popen(
4    ["python3", "-c", "print('hello'); print('warning', file=__import__('sys').stderr)"],
5    stdout=subprocess.PIPE,
6    stderr=subprocess.PIPE,
7    text=True,
8)
9
10stdout, stderr = process.communicate()
11
12print("stdout:", stdout.strip())
13print("stderr:", stderr.strip())
14print("code:", process.returncode)

The main reason to prefer communicate() over ad hoc reads is deadlock avoidance. If the child writes enough output to fill a pipe and the parent is not draining it correctly, both sides can stall.

Use asyncio in Async Applications

If your application already uses async and await, do not mix in blocking subprocess waits. Use the asyncio subprocess API so the event loop can continue serving other tasks:

1import asyncio
2
3
4async def run_command():
5    process = await asyncio.create_subprocess_exec(
6        "python3",
7        "-c",
8        "print('async child')",
9        stdout=asyncio.subprocess.PIPE,
10        stderr=asyncio.subprocess.PIPE,
11    )
12
13    stdout, stderr = await process.communicate()
14
15    print("stdout:", stdout.decode().strip())
16    print("stderr:", stderr.decode().strip())
17    print("code:", process.returncode)
18
19
20asyncio.run(run_command())

This is the cleanest approach in web servers, async task runners, or CLI tools that already depend on asyncio.

Run Multiple Commands Concurrently

Once you are in an async design, running several external commands in parallel becomes straightforward:

1import asyncio
2
3
4async def run_one(name, seconds):
5    process = await asyncio.create_subprocess_exec(
6        "python3",
7        "-c",
8        f"import time; time.sleep({seconds}); print('{name}')",
9        stdout=asyncio.subprocess.PIPE,
10    )
11    stdout, _ = await process.communicate()
12    return stdout.decode().strip()
13
14
15async def main():
16    results = await asyncio.gather(
17        run_one("first", 2),
18        run_one("second", 1),
19    )
20    print(results)
21
22
23asyncio.run(main())

This does not turn the external commands into Python coroutines. It simply lets the parent Python process manage multiple child processes without blocking on one at a time.

Handle Timeouts and Cleanup

External commands can hang, so you should treat timeout handling as part of the design, not an optional add-on.

With subprocess, a timeout looks like this:

1import subprocess
2
3process = subprocess.Popen(["python3", "-c", "import time; time.sleep(10)"])
4
5try:
6    process.communicate(timeout=2)
7except subprocess.TimeoutExpired:
8    process.kill()
9    process.communicate()
10    print("timed out")

With asyncio, use asyncio.wait_for:

1import asyncio
2
3
4async def run_with_timeout():
5    process = await asyncio.create_subprocess_exec(
6        "python3",
7        "-c",
8        "import time; time.sleep(10)",
9    )
10    try:
11        await asyncio.wait_for(process.wait(), timeout=2)
12    except asyncio.TimeoutError:
13        process.kill()
14        await process.wait()
15        print("timed out")
16
17
18asyncio.run(run_with_timeout())

Killing and waiting is important. It ensures the child really exits and prevents process leaks.

Common Pitfalls

The biggest mistake is calling a blocking method immediately after starting the child and still describing the design as asynchronous. If you call .wait() right away, you are blocking again.

Another common issue is using shell=True for convenience even when you already have clean argument values. Prefer argument lists unless you truly need shell syntax such as pipes or wildcard expansion.

Developers also mix blocking subprocess code into asyncio applications and then wonder why the event loop freezes. If the application is async, the subprocess handling needs to be async as well.

Finally, never ignore the child after starting it. Check the return code, read the streams, and clean up on timeouts or cancellation.

Summary

• Use subprocess.Popen for background processes in synchronous Python code.
• Use asyncio.create_subprocess_exec when the rest of the application is async.
• Prefer communicate() for safe stdout and stderr handling.
• Add timeout and cleanup logic so hung processes do not accumulate.
• Avoid shell=True unless shell features are actually required.