Introduction

Running pytest tests in separate processes is useful for isolation, speed, and stability, especially when tests mutate global state or use CPU-heavy workloads. The most common approach is pytest-xdist, but subprocess execution can also be useful for targeted process boundaries. This guide shows practical options and when to use each.

Core Topic Sections

Why run tests in separate processes

Process-level separation helps with:

1. Isolation from shared module state.
2. Better CPU utilization on multi-core machines.
3. Containment of crashes in one worker process.

It does not automatically solve flaky tests caused by timing or external dependencies, so test design still matters.

Option 1: use pytest-xdist for parallel workers

Install plugin:

pip install pytest pytest-xdist

Run with automatic worker count:

pytest -n auto

Or fixed count:

pytest -n 4

This is the standard way to distribute test cases across multiple processes.

Control test distribution strategy

For uneven test durations, distribution strategy affects total runtime.

Examples:

pytest -n 4 --dist=loadscope
pytest -n 4 --dist=loadfile

loadscope keeps tests from same module or class together, which can reduce fixture setup duplication.

Option 2: run one test in a dedicated subprocess

Sometimes you need explicit isolation for one problematic test or fixture chain.

1import subprocess
2import sys
3
4
5def run_test_isolated(nodeid: str) -> int:
6    cmd = [sys.executable, "-m", "pytest", nodeid, "-q"]
7    completed = subprocess.run(cmd, check=False)
8    return completed.returncode
9
10
11if __name__ == "__main__":
12    code = run_test_isolated("tests/test_api.py::test_rate_limit")
13    print("exit code", code)

This pattern is useful in custom CI orchestration or debugging harnesses.

Use markers to separate process-sensitive tests

Some tests should run serially or with dedicated environment due to shared resources. Use markers and split commands.

pytest.ini example:

[pytest]
markers =
    isolated: requires dedicated process or serial execution

Run isolated group first or last:

pytest -m isolated -n 1
pytest -m "not isolated" -n auto

This keeps parallelism high while containing risky tests.

Avoid resource collisions in parallel mode

Parallel processes can collide on:

1. Temp files with fixed names.
2. Shared databases.
3. Network ports.

Use per-worker resources:

1import os
2import tempfile
3
4
5def worker_temp_dir() -> str:
6    worker = os.getenv("PYTEST_XDIST_WORKER", "main")
7    path = os.path.join(tempfile.gettempdir(), f"tests-{worker}")
8    os.makedirs(path, exist_ok=True)
9    return path

Per-worker partitioning removes many nondeterministic failures.

Coverage with multiple processes

Coverage works with parallel test execution when configured properly.

pip install pytest-cov
pytest -n auto --cov=my_package --cov-report=term-missing

If you run separate subprocess commands, ensure coverage combine is done in CI workflow.

Debugging failures in parallel runs

A practical debugging approach:

1. Re-run failing test serially.
2. Re-run with same seed or deterministic env.
3. Compare behavior under -n 1 and -n auto.

This helps distinguish race issues from functional failures.

CI recommendations

In CI, choose worker count relative to available cores and memory. Too many workers can increase context-switch overhead or cause resource starvation.

Start conservative, measure runtime and flake rate, then tune gradually.

Common Pitfalls

• Enabling parallel workers without isolating shared filesystem or port resources.
• Assuming process isolation eliminates all test flakiness automatically.
• Running too many workers and degrading performance due to overhead.
• Mixing serial-only fixtures with parallel execution without markers.
• Forgetting coverage aggregation when tests run in separate subprocesses.

Summary

• Running pytest in different processes improves isolation and often speed.
• 'pytest-xdist is the best default for parallel process execution.'
• Use dedicated subprocess runs for targeted isolation scenarios.
• Separate serial-only tests with markers and resource partitioning.
• Tune worker count and validate reliability, not only raw execution speed.