Introduction

When a Python program performs the same I/O-bound operation across many items in a loop, multithreading can significantly reduce total execution time. Instead of processing each item sequentially and waiting for I/O to complete before moving to the next, threads allow overlapping work so that one thread can process while another waits on a network response or file read. Python's threading module and concurrent.futures.ThreadPoolExecutor make this straightforward to implement.

This article shows how to convert a sequential loop into a multithreaded one, covers both low-level threading and the higher-level executor API, and discusses the trade-offs you need to consider.

When Multithreading Helps (and When It Does Not)

Python has a Global Interpreter Lock (GIL) that prevents multiple threads from executing Python bytecode simultaneously. This means threads do not provide true parallelism for CPU-bound work like number crunching or image processing. For CPU-bound tasks, use multiprocessing instead.

However, the GIL is released during I/O operations such as network requests, file reads, database queries, and subprocess calls. Multithreading is effective for these operations because threads can overlap waiting time.

Approach 1: Manual Threading with threading.Thread

The most basic approach creates a thread for each item in the loop.

1import threading
2import time
3import requests
4
5def fetch_url(url):
6    """Simulate fetching a URL."""
7    response = requests.get(url)
8    print(f"{url}: {response.status_code}")
9
10urls = [
11    "https://httpbin.org/delay/1",
12    "https://httpbin.org/delay/1",
13    "https://httpbin.org/delay/1",
14    "https://httpbin.org/delay/1",
15]
16
17# Sequential: takes about 4 seconds
18start = time.time()
19for url in urls:
20    fetch_url(url)
21print(f"Sequential: {time.time() - start:.1f}s")
22
23# Threaded: takes about 1 second
24start = time.time()
25threads = []
26for url in urls:
27    t = threading.Thread(target=fetch_url, args=(url,))
28    threads.append(t)
29    t.start()
30
31# Wait for all threads to complete
32for t in threads:
33    t.join()
34print(f"Threaded: {time.time() - start:.1f}s")

The join() call blocks the main thread until each worker thread finishes. Without it, the main thread could exit before the workers complete their work.

Approach 2: ThreadPoolExecutor (Recommended)

Creating a raw thread per item is fine for small lists, but it does not scale. If you have 10,000 URLs, spawning 10,000 threads wastes resources and can crash the program. ThreadPoolExecutor solves this by maintaining a fixed pool of worker threads and distributing tasks among them.

1from concurrent.futures import ThreadPoolExecutor, as_completed
2import requests
3import time
4
5def fetch_url(url):
6    response = requests.get(url, timeout=10)
7    return url, response.status_code
8
9urls = [f"https://httpbin.org/delay/1" for _ in range(20)]
10
11start = time.time()
12results = []
13
14with ThreadPoolExecutor(max_workers=5) as executor:
15    futures = {executor.submit(fetch_url, url): url for url in urls}
16
17    for future in as_completed(futures):
18        url, status = future.result()
19        results.append((url, status))
20        print(f"{url}: {status}")
21
22print(f"Completed {len(results)} requests in {time.time() - start:.1f}s")

With max_workers=5, at most 5 threads run concurrently. The 20 URLs are processed in groups of 5, taking about 4 seconds total instead of 20 seconds sequentially.

Approach 3: Using executor.map for Simple Cases

When you want to apply the same function to every item and collect results in order, executor.map is the most concise option.

1from concurrent.futures import ThreadPoolExecutor
2import requests
3
4def fetch_status(url):
5    response = requests.get(url, timeout=10)
6    return response.status_code
7
8urls = [
9    "https://httpbin.org/status/200",
10    "https://httpbin.org/status/404",
11    "https://httpbin.org/status/500",
12    "https://httpbin.org/status/200",
13]
14
15with ThreadPoolExecutor(max_workers=4) as executor:
16    statuses = list(executor.map(fetch_status, urls))
17
18print(statuses)  # [200, 404, 500, 200]

Unlike as_completed, map returns results in the same order as the input iterable. It also re-raises exceptions from worker threads when you iterate over the results.

Collecting Results Safely with a Lock

When multiple threads write to a shared data structure, you need synchronization to prevent race conditions.

1import threading
2from concurrent.futures import ThreadPoolExecutor
3
4results = []
5lock = threading.Lock()
6
7def process_item(item):
8    computed = item * item  # Some computation
9    with lock:
10        results.append(computed)
11    return computed
12
13items = list(range(100))
14
15with ThreadPoolExecutor(max_workers=8) as executor:
16    executor.map(process_item, items)
17
18print(f"Processed {len(results)} items")

The with lock block ensures that only one thread appends to results at a time. Without the lock, concurrent appends could corrupt the list or lose items on some Python implementations.

However, if you are using executor.map or as_completed and collecting results from the returned futures, you do not need a lock because each future's result is isolated.

Handling Exceptions in Threads

Exceptions in worker threads are silent by default with manual threading.Thread. They do not propagate to the main thread.

1from concurrent.futures import ThreadPoolExecutor, as_completed
2
3def risky_operation(n):
4    if n == 3:
5        raise ValueError(f"Bad input: {n}")
6    return n * 10
7
8with ThreadPoolExecutor(max_workers=4) as executor:
9    futures = {executor.submit(risky_operation, i): i for i in range(5)}
10
11    for future in as_completed(futures):
12        try:
13            result = future.result()
14            print(f"Result: {result}")
15        except ValueError as e:
16            print(f"Error: {e}")

With ThreadPoolExecutor, exceptions are captured and re-raised when you call future.result(). This makes error handling predictable and prevents silent failures.

Choosing the Right Number of Workers

The optimal max_workers value depends on the workload.

For I/O-bound tasks (network requests, file I/O), a good starting point is between 5 and 20 workers. More workers help when each task spends most of its time waiting. Too many workers can overwhelm the target server or exhaust file descriptors.

For CPU-bound tasks (where you should use ProcessPoolExecutor instead), the number of workers should match the CPU core count.

1import os
2from concurrent.futures import ThreadPoolExecutor
3
4# For I/O-bound work
5io_workers = min(32, os.cpu_count() + 4)  # Python 3.8+ default formula
6
7# For CPU-bound work (use ProcessPoolExecutor)
8cpu_workers = os.cpu_count()

Common Pitfalls

Using threads for CPU-bound work. Due to the GIL, threading does not speed up pure computation in Python. CPU-bound loops should use multiprocessing.Pool or concurrent.futures.ProcessPoolExecutor.

Creating too many threads. Each thread consumes memory for its stack (typically 8 MB on Linux). Creating thousands of threads can exhaust memory. Always use a thread pool with a bounded max_workers.

Ignoring thread safety. Shared mutable state accessed from multiple threads without a lock can produce corrupted data. Use threading.Lock, queue.Queue, or avoid shared state entirely by collecting results through futures.

Not calling join() or using a context manager. If you create threads manually and do not call join(), the main thread may exit while workers are still running. With ThreadPoolExecutor, using the with statement ensures all workers finish before the block exits.

Swallowing exceptions. With raw threading.Thread, exceptions in worker threads are printed to stderr but do not stop the main program. Use ThreadPoolExecutor and check future.result() to catch and handle errors properly.

Summary

To multithread an operation within a loop in Python, use concurrent.futures.ThreadPoolExecutor for most cases. It manages thread lifecycle, limits concurrency, and provides clean error handling through futures. Use executor.map for simple apply-and-collect patterns, and as_completed when you need to process results as they arrive. Reserve manual threading.Thread for cases where you need fine-grained control over thread behavior. Always remember that Python threading is effective for I/O-bound work but does not improve CPU-bound performance due to the GIL.