Introduction

When you launch a subprocess with shell=True, Python starts a shell process (/bin/sh on Unix, cmd.exe on Windows), which then spawns your actual command as a child. Calling process.terminate() or process.kill() only kills the shell, not the child command. To kill both, you need to kill the entire process group using os.killpg(), or avoid shell=True altogether and pass the command as a list.

The Problem

1import subprocess
2import time
3
4# This spawns: /bin/sh -c "sleep 60"
5proc = subprocess.Popen("sleep 60", shell=True)
6
7time.sleep(2)
8proc.terminate()  # Kills /bin/sh, but "sleep 60" continues running!
9proc.wait()
10
11# "sleep 60" is now an orphan process still running

proc.pid is the PID of the shell, not the sleep command. Terminating the shell leaves the child as an orphan process.

Fix 1: Kill the Process Group (Recommended)

Start the subprocess as a process group leader and kill the entire group:

1import subprocess
2import os
3import signal
4
5proc = subprocess.Popen(
6    "sleep 60",
7    shell=True,
8    preexec_fn=os.setsid  # Start a new process group
9)
10
11# Later: kill the entire process group
12os.killpg(os.getpgid(proc.pid), signal.SIGTERM)
13proc.wait()

os.setsid makes the shell process the leader of a new session/process group. os.killpg() sends the signal to every process in that group, including the child command.

For SIGKILL (force kill):

os.killpg(os.getpgid(proc.pid), signal.SIGKILL)

Fix 2: Avoid shell=True

The simplest and safest approach — pass the command as a list:

1import subprocess
2
3# No shell=True — Python launches "sleep" directly
4proc = subprocess.Popen(["sleep", "60"])
5
6proc.terminate()  # Kills "sleep" directly — no orphan
7proc.wait()

Without shell=True, proc.pid is the PID of the actual command, and terminate() works as expected.

When you need shell features (pipes, wildcards, variable expansion), use Python equivalents:

1# Instead of: shell=True, "cat file.txt | grep error"
2import subprocess
3
4cat = subprocess.Popen(["cat", "file.txt"], stdout=subprocess.PIPE)
5grep = subprocess.Popen(["grep", "error"], stdin=cat.stdout, stdout=subprocess.PIPE)
6cat.stdout.close()
7output = grep.communicate()[0]

Fix 3: Using subprocess.run with timeout

subprocess.run() with timeout handles cleanup automatically:

1import subprocess
2
3try:
4    result = subprocess.run(
5        "sleep 60",
6        shell=True,
7        timeout=5,  # Kill after 5 seconds
8        capture_output=True,
9        text=True
10    )
11except subprocess.TimeoutExpired:
12    print("Process timed out and was killed")

However, subprocess.run with shell=True and timeout may still leave orphan children. Combine with preexec_fn=os.setsid and a custom handler for reliable cleanup.

Fix 4: Using psutil for Cross-Platform Process Tree Kill

The psutil library can kill an entire process tree:

1import subprocess
2import psutil
3
4proc = subprocess.Popen("sleep 60", shell=True)
5
6# Kill the process and all its children
7parent = psutil.Process(proc.pid)
8children = parent.children(recursive=True)
9
10for child in children:
11    child.terminate()
12parent.terminate()
13
14# Wait for all to finish
15gone, alive = psutil.wait_procs(children + [parent], timeout=5)
16for p in alive:
17    p.kill()  # Force kill any survivors

This works on both Unix and Windows.

Fix 5: Using start_new_session (Python 3.2+)

Instead of preexec_fn=os.setsid, use the start_new_session parameter:

1import subprocess
2import os
3import signal
4
5proc = subprocess.Popen(
6    "sleep 60",
7    shell=True,
8    start_new_session=True  # Cleaner than preexec_fn=os.setsid
9)
10
11# Kill the entire session
12os.killpg(os.getpgid(proc.pid), signal.SIGTERM)
13proc.wait()

start_new_session=True is the recommended replacement for preexec_fn=os.setsid — it is safer because preexec_fn is not compatible with threads.

Windows Considerations

On Windows, process groups work differently:

1import subprocess
2import signal
3
4# Windows: CREATE_NEW_PROCESS_GROUP flag
5proc = subprocess.Popen(
6    "ping -n 60 localhost",
7    shell=True,
8    creationflags=subprocess.CREATE_NEW_PROCESS_GROUP
9)
10
11# Send CTRL_BREAK to the process group
12proc.send_signal(signal.CTRL_BREAK_EVENT)
13proc.wait()

On Windows, os.killpg and os.setsid are not available. Use CREATE_NEW_PROCESS_GROUP and CTRL_BREAK_EVENT instead.

Common Pitfalls

• Calling terminate() on a shell=True process: This only kills the shell (/bin/sh), leaving the actual command running as an orphan. Always use os.killpg() with start_new_session=True, or avoid shell=True.
• Using preexec_fn=os.setsid in threaded code: preexec_fn runs between fork() and exec(), which is not safe in multi-threaded programs. Use start_new_session=True instead (Python 3.2+).
• Forgetting to call proc.wait() after kill: Without wait(), the process becomes a zombie on Unix. Always call proc.wait() (or proc.communicate()) after sending a kill signal.
• Security risks of shell=True: Passing user input in a shell=True command enables shell injection attacks. Popen(f"grep {user_input} file.txt", shell=True) allows arbitrary command execution. Use list form: Popen(["grep", user_input, "file.txt"]).
• SIGTERM vs SIGKILL: SIGTERM allows the process to clean up (close files, release resources). SIGKILL terminates immediately with no cleanup. Try SIGTERM first, then SIGKILL after a timeout if the process does not exit.

Summary

• shell=True creates a shell parent process — terminate() kills the shell but not its child
• Use start_new_session=True and os.killpg() to kill the entire process group
• Best approach: avoid shell=True and pass commands as a list (["cmd", "arg1", "arg2"])
• Use psutil for cross-platform process tree management
• On Windows, use CREATE_NEW_PROCESS_GROUP and CTRL_BREAK_EVENT
• Always call proc.wait() after killing a process to prevent zombies