Introduction

Parallel URL fetching is a classic example of an I/O-bound workload. Most of the time is spent waiting on network responses, so a small amount of multithreading can reduce total runtime even in CPython.

If the goal is "simple" and "without queue," the cleanest answer is usually a thread pool. It gives you parallel requests without forcing you to build and coordinate your own queue.Queue.

Why Threads Help For Network Requests

Python's GIL limits true parallel execution for CPU-heavy work, but a blocked network call releases the interpreter to let other threads run. That means one thread can wait on example.com while another waits on python.org, and the total wall-clock time can be much better than a purely sequential loop.

For a small script, the core pattern is just a worker function plus a bounded pool:

1from concurrent.futures import ThreadPoolExecutor, as_completed
2import requests
3
4
5def fetch(url: str) -> tuple[str, int]:
6    response = requests.get(url, timeout=10)
7    response.raise_for_status()
8    return url, len(response.text)
9
10
11urls = [
12    "https://example.com",
13    "https://httpbin.org/get",
14    "https://www.python.org",
15]
16
17with ThreadPoolExecutor(max_workers=5) as executor:
18    futures = [executor.submit(fetch, url) for url in urls]
19    for future in as_completed(futures):
20        url, size = future.result()
21        print(f"{url} -> {size} bytes")

This keeps the concurrency model simple and uses only the standard library plus requests.

Why A Thread Pool Is Better Than One Raw Thread Per URL

You can start one threading.Thread per URL, but that becomes sloppy as soon as the list grows. A raw thread per task has no natural backpressure, no easy concurrency cap, and more overhead than necessary. A thread pool gives you a maximum number of active workers and a much clearer shutdown story.

That matters because "parallel" does not mean "start hundreds of requests instantly." Too much concurrency can overload your own machine, hit remote rate limits, or produce worse results than a small, steady worker count.

A Minimal Manual Thread Example

If you really want explicit threads and no helper pool abstraction, it can still be done:

1import threading
2import requests
3
4results = []
5lock = threading.Lock()
6
7
8def fetch(url: str) -> None:
9    try:
10        response = requests.get(url, timeout=10)
11        response.raise_for_status()
12        item = {"url": url, "status": response.status_code, "error": None}
13    except Exception as exc:
14        item = {"url": url, "status": None, "error": str(exc)}
15
16    with lock:
17        results.append(item)
18
19
20threads = [threading.Thread(target=fetch, args=(url,)) for url in urls]
21
22for thread in threads:
23    thread.start()
24
25for thread in threads:
26    thread.join()
27
28print(results)

This is readable for a handful of URLs. Once you need throttling or many inputs, ThreadPoolExecutor is usually the better "simple" solution.

Handle Failures Per URL

Network work fails regularly. Timeouts, DNS errors, TLS issues, and 500 responses are normal possibilities, not edge cases. A good worker function should isolate failure to one URL and return structured information instead of crashing the whole batch.

That is why examples often catch exceptions inside the worker. The main thread can then print or record success and failure together, which makes the overall job easier to reason about.

If you prefer to preserve input order instead of processing results as they finish, executor.map() is useful:

with ThreadPoolExecutor(max_workers=5) as executor:
    for result in executor.map(fetch, urls):
        print(result)

Use as_completed() when completion order matters more than original order.

Choose A Sensible Worker Count

The best max_workers value depends on network latency, server policy, and how heavy each response is to process. For many scripts, values like 5, 10, or 20 are more reasonable than something extreme. The right number is the smallest value that gives good throughput without creating instability.

If you are talking to a third-party API, be especially careful. A high worker count may trigger throttling or abuse defenses. Parallel fetching is still your responsibility to use politely.

Common Pitfalls

One common mistake is forgetting timeouts, which lets a slow server stall a worker indefinitely. Another is creating one thread per URL for large inputs, which looks simple at first but scales poorly. Developers also sometimes assume the GIL makes threads useless, which is false for network-heavy tasks. Finally, error handling often gets ignored in the first version, and one failed request ends up hiding all the successful ones.

Summary

• URL fetching is I/O-bound, so multithreading can reduce total runtime in Python.
• 'ThreadPoolExecutor is the simplest practical way to fetch many URLs in parallel without managing your own queue.'
• A bounded pool is usually safer than one raw thread per URL.
• Always add timeouts and per-request exception handling.
• Tune concurrency conservatively so you do not overload your machine or the remote service.