Introduction

Asynchronous HTTP calls let a Python program overlap many network waits without blocking one thread per request. This is especially useful for APIs, crawlers, batch data fetchers, and integration workers that spend most of their time waiting on remote servers. In modern Python, the usual approach is asyncio plus an async HTTP client such as aiohttp.

Why Async HTTP Helps

HTTP requests are usually I/O-bound, not CPU-bound. The program sends a request and then waits for the server, the network, and the response body.

If you perform those waits synchronously one by one, total runtime grows with the sum of all request times. With async I/O, the event loop can switch to other pending requests while one request is waiting on the network.

That means async shines when:

• many requests are in flight
• each request spends time waiting
• CPU work per response is relatively small

It is not a universal speed button, but it is a very good fit for network concurrency.

Use aiohttp With One Reused Session

The most important practice is to reuse a ClientSession instead of creating a new one for every request.

1import asyncio
2import aiohttp
3
4
5async def fetch(session: aiohttp.ClientSession, url: str) -> str:
6    async with session.get(url) as response:
7        response.raise_for_status()
8        return await response.text()
9
10
11async def main() -> None:
12    urls = [
13        "https://httpbin.org/get",
14        "https://httpbin.org/uuid",
15    ]
16
17    async with aiohttp.ClientSession() as session:
18        results = await asyncio.gather(*(fetch(session, url) for url in urls))
19        for item in results:
20            print(item[:60])
21
22
23asyncio.run(main())

The shared session keeps connection management efficient and avoids unnecessary setup overhead.

asyncio.gather Runs Requests Concurrently

asyncio.gather is a convenient way to await multiple coroutines together.

1results = await asyncio.gather(
2    fetch(session, "https://httpbin.org/get"),
3    fetch(session, "https://httpbin.org/uuid"),
4    fetch(session, "https://httpbin.org/ip"),
5)

This does not make the network itself faster. It lets your program wait on several network operations at the same time instead of serially.

Limit Concurrency When Needed

Sending hundreds of requests at once can overload the remote service or your own machine. A semaphore is a simple way to cap concurrency.

1import asyncio
2import aiohttp
3
4
5async def fetch_limited(session, url, semaphore):
6    async with semaphore:
7        async with session.get(url) as response:
8            response.raise_for_status()
9            return await response.text()
10
11
12async def main():
13    urls = ["https://httpbin.org/get"] * 20
14    semaphore = asyncio.Semaphore(5)
15
16    async with aiohttp.ClientSession() as session:
17        results = await asyncio.gather(
18            *(fetch_limited(session, url, semaphore) for url in urls)
19        )
20        print(len(results))
21
22
23asyncio.run(main())

This pattern is often better than “fire every request immediately.”

Add Timeouts and Error Handling

Async code still needs the same operational discipline as sync code. Use timeouts and handle failures explicitly.

1import asyncio
2import aiohttp
3
4
5async def fetch_json(session, url):
6    try:
7        async with session.get(url, timeout=aiohttp.ClientTimeout(total=5)) as response:
8            response.raise_for_status()
9            return await response.json()
10    except asyncio.TimeoutError:
11        return {"error": "timeout"}
12    except aiohttp.ClientError as exc:
13        return {"error": str(exc)}

Without timeouts, a few bad requests can stall the workflow much longer than expected.

Know When Async Is the Wrong Tool

Async HTTP is ideal for concurrent I/O, but it does not automatically help with CPU-heavy work such as parsing huge payloads, running ML inference, or compressing large files. If the program spends most of its time on CPU, threads or processes may be the more relevant optimization.

Also, if you only make one or two requests in a simple script, async may add complexity without much benefit. The payoff is biggest when concurrency is real.

Common Pitfalls

The first pitfall is creating a new ClientSession per request. That defeats connection reuse and wastes resources.

Another issue is launching huge numbers of requests with no concurrency limit. The program may then hit rate limits, resource exhaustion, or unstable throughput.

Developers also forget that async code still needs explicit timeout handling. Non-blocking code can still wait forever if nothing cancels it.

Finally, avoid mixing blocking libraries inside async workflows unless you deliberately isolate them. A blocking call inside an async coroutine can stall the event loop.

Summary

• Async HTTP in Python is usually built with asyncio and an async client such as aiohttp.
• Reuse one ClientSession instead of creating a session per request.
• Use asyncio.gather to overlap multiple HTTP waits.
• Limit concurrency with a semaphore when the request volume is high.
• Add timeouts and error handling so async code stays operational under failure conditions.