Introduction

Launching a new multiprocessing.Process for every job can overwhelm a machine long before the work is finished. CPU time, memory usage, open file descriptors, and process startup overhead all become bottlenecks if concurrency is unbounded. The practical solution is to cap the number of simultaneous processes and choose a coordination pattern that matches the workload.

The Simplest Limit Is a Process Pool

If your goal is just “run these jobs with at most N workers,” a process pool is usually better than hand-managing Process objects.

1from concurrent.futures import ProcessPoolExecutor, as_completed
2import time
3
4
5def square(x: int) -> int:
6    time.sleep(0.2)
7    return x * x
8
9
10def main() -> None:
11    jobs = range(10)
12    with ProcessPoolExecutor(max_workers=4) as pool:
13        futures = [pool.submit(square, job) for job in jobs]
14        for future in as_completed(futures):
15            print(future.result())
16
17
18if __name__ == "__main__":
19    main()

Here, max_workers=4 is the limit. The executor reuses worker processes instead of spawning a fresh one for each task, which reduces overhead.

Manual Control With multiprocessing.Process

Sometimes you need lower-level control. In that case, you can keep a list of running processes and wait when the cap is reached.

1from multiprocessing import Process
2import time
3
4
5def worker(task_id: int) -> None:
6    print(f"starting {task_id}")
7    time.sleep(0.5)
8    print(f"finished {task_id}")
9
10
11def run_limited(total_tasks: int, max_parallel: int) -> None:
12    running: list[Process] = []
13
14    for task_id in range(total_tasks):
15        process = Process(target=worker, args=(task_id,))
16        process.start()
17        running.append(process)
18
19        while len(running) >= max_parallel:
20            running[0].join()
21            running = [p for p in running if p.is_alive()]
22
23    for process in running:
24        process.join()
25
26
27if __name__ == "__main__":
28    run_limited(total_tasks=8, max_parallel=3)

This approach works, but it is more fragile than using a pool. Choose it only when you genuinely need direct process lifecycle control.

Use a Semaphore When Only Part of the Work Must Be Limited

A semaphore is useful when a process can do some preparation freely, but must enter a limited critical section to protect a scarce resource.

1from multiprocessing import Process, Semaphore
2import time
3
4
5def guarded_worker(task_id: int, sem: Semaphore) -> None:
6    print(f"preparing {task_id}")
7    with sem:
8        print(f"running limited section {task_id}")
9        time.sleep(0.4)
10
11
12def main() -> None:
13    sem = Semaphore(2)
14    processes = [Process(target=guarded_worker, args=(i, sem)) for i in range(6)]
15
16    for process in processes:
17        process.start()
18
19    for process in processes:
20        process.join()
21
22
23if __name__ == "__main__":
24    main()

In that example, more than two processes may exist at once, but only two can execute the protected section concurrently.

Collect Results Without Shared State Confusion

When manually creating processes, result collection often becomes the next challenge. A Queue is usually the safest way to bring results back to the parent process.

1from multiprocessing import Process, Queue
2
3
4def worker(task_id: int, out: Queue) -> None:
5    out.put((task_id, task_id * task_id))
6
7
8def main() -> None:
9    output = Queue()
10    processes = [Process(target=worker, args=(i, output)) for i in range(4)]
11
12    for process in processes:
13        process.start()
14
15    for process in processes:
16        process.join()
17
18    results = [output.get() for _ in range(4)]
19    print(sorted(results))
20
21
22if __name__ == "__main__":
23    main()

This avoids accidental races on shared lists or dictionaries.

Choosing the Right Concurrency Limit

For CPU-bound work, start near the number of available CPU cores and measure. More processes do not automatically mean more throughput. Once the CPU is saturated, extra workers mostly add context-switching overhead.

For I/O-heavy workloads, a higher process count can make sense, but multiprocessing may not be the best tool at all. Threads or async I/O may be simpler if the main wait is network or disk latency rather than CPU time.

The right limit is not a rule of thumb you memorize once. It is something you benchmark on the hardware and data that matter to your application.

Common Pitfalls

The first pitfall is skipping the if __name__ == "__main__": guard. On platforms that use spawn-based process startup, forgetting that guard can recursively re-import the module and create broken behavior.

Another mistake is starting unbounded processes in a loop. The code looks fine with ten jobs and then collapses with ten thousand.

Developers also underestimate serialization cost. Every argument and result passed between processes must be pickled, so large payloads can erase the benefit of parallel execution.

Finally, do not use multiprocessing for work that is mostly waiting on external I/O unless you have measured that it helps. It is heavier than threads and often unnecessary for that case.

Summary

• Limit process concurrency to protect CPU, memory, and system stability.
• Prefer ProcessPoolExecutor or a pool when you only need capped parallel job execution.
• Use manual Process management only when you need explicit lifecycle control.
• Semaphores are useful when only a limited section of work must be restricted.
• Benchmark the worker count instead of assuming more processes always means faster code.