Introduction

When you use Python's requests library to fetch many URLs, the bottleneck is usually network waiting time, not CPU. That distinction matters because it tells you whether threads, processes, or asynchronous I/O will actually improve throughput.

requests Is Blocking but Often Fine With Threads

requests performs synchronous network I/O. While one call waits for a server response, that worker is mostly idle. For I/O-bound workloads like HTTP fetching, threads are often the simplest performance win because blocked threads release the GIL while the socket wait happens.

1from concurrent.futures import ThreadPoolExecutor, as_completed
2import requests
3
4URLS = [
5    "https://example.com",
6    "https://httpbin.org/get",
7    "https://www.python.org",
8]
9
10def fetch(url):
11    response = requests.get(url, timeout=10)
12    response.raise_for_status()
13    return url, len(response.text)
14
15with ThreadPoolExecutor(max_workers=8) as executor:
16    futures = [executor.submit(fetch, url) for url in URLS]
17    for future in as_completed(futures):
18        print(future.result())

For a modest number of concurrent requests, this approach is usually good enough and requires minimal code changes.

When Multiprocessing Helps and When It Does Not

Processes are useful for CPU-bound work, not for ordinary HTTP waiting. If each request is followed by heavy HTML parsing, image processing, or data compression, moving the CPU-intensive part into processes can help you use multiple cores.

If the job is mostly "send request, wait, read response," multiprocessing is usually wasteful. Each process has higher startup and memory cost, and you lose the easy connection reuse that threads can get from a shared session strategy.

1from concurrent.futures import ProcessPoolExecutor
2
3def cpu_heavy_parse(text):
4    return sum(ord(ch) for ch in text)
5
6with ProcessPoolExecutor() as executor:
7    results = list(executor.map(cpu_heavy_parse, ["abc", "def", "ghi"]))
8    print(results)

This pattern makes sense only after the network step if parsing dominates total runtime.

Async I/O Is a Different Model

If you want to scale to a large number of concurrent HTTP operations, asynchronous I/O is often the best model, but requests itself is not async. The usual move is to switch to an async-capable client such as httpx or aiohttp.

1import asyncio
2import httpx
3
4URLS = [
5    "https://example.com",
6    "https://httpbin.org/get",
7    "https://www.python.org",
8]
9
10async def fetch(client, url):
11    response = await client.get(url)
12    response.raise_for_status()
13    return url, len(response.text)
14
15async def main():
16    async with httpx.AsyncClient(timeout=10) as client:
17        tasks = [fetch(client, url) for url in URLS]
18        for result in await asyncio.gather(*tasks):
19            print(result)
20
21asyncio.run(main())

Async shines when you have many in-flight requests and you are willing to adopt the async programming model across the relevant code path.

Reuse Connections Before Chasing More Concurrency

Many requests programs are slow because they open a new connection for every call. Before changing concurrency strategy, use requests.Session() so HTTP connections can be pooled and reused.

1import requests
2
3with requests.Session() as session:
4    for url in URLS:
5        response = session.get(url, timeout=10)
6        print(url, response.status_code)

In a threaded program, it is often cleaner to create one session per thread or to benchmark carefully before sharing sessions broadly. The key point is that transport-level reuse can matter as much as concurrency choice.

Choosing the Right Approach

Use threads when:

• you already have synchronous requests code,
• the workload is mostly network waiting,
• and the number of simultaneous requests is moderate.

Use processes when:

• post-request computation is CPU-heavy,
• and multi-core parallelism matters more than shared state.

Use async I/O when:

• you need high connection counts,
• you can switch to an async HTTP client,
• and the surrounding architecture can tolerate async code.

Common Pitfalls

The biggest mistake is using multiprocessing to speed up a purely I/O-bound requests workload. It usually adds complexity without matching the real bottleneck.

Another mistake is assuming async means "faster" regardless of context. For a small script hitting a few URLs, threads may be simpler and entirely sufficient.

Developers also overlook timeouts, retries, and connection reuse. Those operational details often affect performance and reliability more than the thread-versus-async debate.

Finally, if responses are large and parsing is expensive, split the problem mentally into two stages: network I/O and CPU work. The best concurrency model may differ for each stage.

Summary

• For ordinary requests workloads, threads are usually the first concurrency tool to try.
• Multiprocessing is mainly for CPU-bound work, not socket waiting.
• Async I/O requires an async client such as httpx or aiohttp, not plain requests.
• Reusing connections with Session is an important optimization on its own.
• Pick the model that matches the actual bottleneck instead of optimizing by habit.