asyncio
requests
Python
asynchronous programming
HTTP requests

How could I use requests in asyncio?

Master System Design with Codemia

Enhance your system design skills with over 120 practice problems, detailed solutions, and hands-on exercises.

Introduction

The requests library is synchronous, so it blocks the thread while waiting for network I/O. asyncio expects cooperative, non-blocking tasks, which means requests does not become asynchronous just because you call it inside an async def function.

The Core Constraint

This is the mistake people usually make:

python
1import requests
2
3async def fetch():
4    response = requests.get("https://example.com")
5    return response.text

That function is syntactically asynchronous, but the HTTP call still blocks the event loop. While requests.get(...) is running, other coroutines cannot make progress.

If You Must Keep requests, Use a Thread

If an existing codebase already depends heavily on requests, the practical bridge is to run the blocking call in a worker thread. In modern Python, asyncio.to_thread(...) is the cleanest option.

python
1import asyncio
2import requests
3
4
5def fetch_sync(url: str) -> str:
6    response = requests.get(url, timeout=10)
7    response.raise_for_status()
8    return response.text
9
10
11async def fetch(url: str) -> str:
12    return await asyncio.to_thread(fetch_sync, url)
13
14
15async def main():
16    urls = [
17        "https://example.com",
18        "https://httpbin.org/get"
19    ]
20    pages = await asyncio.gather(*(fetch(url) for url in urls))
21    print([len(page) for page in pages])
22
23
24asyncio.run(main())

This does not make requests itself non-blocking. It just moves the blocking work off the event-loop thread so the rest of your asynchronous program can continue.

Why Native Async Clients Are Better

If you are designing new code rather than adapting old code, use an async HTTP client such as aiohttp or httpx.AsyncClient. Those libraries integrate with the event loop directly and avoid the thread handoff.

python
1import asyncio
2import aiohttp
3
4
5async def fetch(session: aiohttp.ClientSession, url: str) -> str:
6    async with session.get(url, timeout=10) as response:
7        response.raise_for_status()
8        return await response.text()
9
10
11async def main():
12    urls = [
13        "https://example.com",
14        "https://httpbin.org/get"
15    ]
16    async with aiohttp.ClientSession() as session:
17        pages = await asyncio.gather(*(fetch(session, url) for url in urls))
18        print([len(page) for page in pages])
19
20
21asyncio.run(main())

This is the model to prefer when you expect high concurrency, connection pooling, or many in-flight requests.

Choosing Between the Two Approaches

Use requests plus asyncio.to_thread(...) when:

  • you are incrementally modernizing a synchronous codebase
  • you only have a modest number of outbound requests
  • you want to reuse existing requests middleware or authentication code

Use a native async client when:

  • the application is primarily asynchronous already
  • throughput and connection reuse matter
  • you want cleaner cancellation and backpressure behavior

Error Handling Still Matters

Whether you use threads or a native async client, keep timeouts and exception handling explicit. Networking code that omits timeouts tends to fail in production in the least convenient way.

A reasonable baseline with requests is timeout=10 and response.raise_for_status(). The async equivalent should do the same conceptually.

Common Pitfalls

  • Calling requests.get(...) directly inside async def and assuming it is non-blocking.
  • Wrapping blocking code in async def without moving it to a thread or process.
  • Rewriting a whole codebase to asyncio while keeping a synchronous HTTP layer at the center.
  • Forgetting timeouts, which can stall both synchronous and asynchronous systems.
  • Using too many thread-backed requests at once and expecting the same scaling profile as a native async client.

Summary

  • 'requests is synchronous and blocks the event loop if called directly.'
  • The practical compatibility option is await asyncio.to_thread(...) around a synchronous helper.
  • For new asynchronous systems, prefer aiohttp or another native async HTTP client.
  • Timeouts and explicit error handling matter in both models.
  • The right tool depends on whether you are adapting legacy code or building an async-first service.

Course illustration
Course illustration

All Rights Reserved.