boto3
async programming
Python
AWS SDK
asynchronous functions

I want to use boto3 in async function, python

Master System Design with Codemia

Enhance your system design skills with over 120 practice problems, detailed solutions, and hands-on exercises.

Introduction

boto3 is a synchronous SDK, so you cannot make it magically non-blocking just by calling it inside an async def. If you use boto3 directly on the event loop, the AWS call blocks that loop until the request finishes.

The Core Constraint

This is the important conceptual rule:

  • 'async def does not make blocking code asynchronous'
  • 'boto3 performs blocking network I/O'
  • therefore, raw boto3 calls must be offloaded if you are using asyncio

That means the practical choices are:

  • run boto3 in a worker thread
  • use a third-party async wrapper such as aioboto3 or aiobotocore

If you want to stay close to the official AWS SDK, thread offloading is the simplest answer.

Use asyncio.to_thread In Modern Python

A clean approach in current Python is asyncio.to_thread.

python
1import asyncio
2import boto3
3
4s3 = boto3.client("s3")
5
6
7def list_buckets_sync():
8    return s3.list_buckets()
9
10async def main():
11    response = await asyncio.to_thread(list_buckets_sync)
12    names = [bucket["Name"] for bucket in response.get("Buckets", [])]
13    print(names)
14
15asyncio.run(main())

This keeps the blocking SDK call off the event loop while still letting your async application remain responsive.

run_in_executor Works Too

If you need compatibility with older async patterns or more control over the executor, use loop.run_in_executor.

python
1import asyncio
2import boto3
3
4s3 = boto3.client("s3")
5
6
7def list_buckets_sync():
8    return s3.list_buckets()
9
10async def main():
11    loop = asyncio.get_running_loop()
12    response = await loop.run_in_executor(None, list_buckets_sync)
13    print(response)
14
15asyncio.run(main())

This is the older explicit form of the same idea.

Parameterized AWS Calls

You will often need to pass arguments into the sync function you offload.

python
1import asyncio
2import boto3
3
4s3 = boto3.client("s3")
5
6
7def get_object_sync(bucket, key):
8    return s3.get_object(Bucket=bucket, Key=key)
9
10async def read_object(bucket, key):
11    response = await asyncio.to_thread(get_object_sync, bucket, key)
12    body = response["Body"].read().decode("utf-8")
13    return body
14
15async def main():
16    text = await read_object("my-bucket", "notes.txt")
17    print(text)
18
19asyncio.run(main())

The key point is that the synchronous AWS SDK call happens in a worker thread, not on the event loop.

Be Careful With Large Numbers Of Concurrent Calls

Offloading one or two calls is easy. Offloading hundreds of calls blindly can overwhelm the default thread pool or your process.

Use concurrency limits when you fan out many requests.

python
1import asyncio
2import boto3
3
4s3 = boto3.client("s3")
5semaphore = asyncio.Semaphore(10)
6
7
8def head_object_sync(bucket, key):
9    return s3.head_object(Bucket=bucket, Key=key)
10
11async def safe_head(bucket, key):
12    async with semaphore:
13        return await asyncio.to_thread(head_object_sync, bucket, key)

This keeps async orchestration while preventing uncontrolled thread growth.

Should You Use aioboto3 Instead

If your codebase is heavily asynchronous and AWS I/O dominates the workload, a third-party async wrapper can be convenient. But the important design fact remains: boto3 itself is not natively async.

So the question is not whether await boto3_call() exists in standard boto3. It does not. The question is whether you want:

  • official boto3 plus thread offloading
  • a third-party async abstraction layer

For many applications, thread offloading is simpler and easier to reason about.

Connection Reuse And Clients

Create clients once when possible instead of rebuilding them inside every coroutine.

Good:

python
s3 = boto3.client("s3")

Less good:

python
# creating a new client inside every async call adds overhead

This matters because the async wrapper does not change the fact that you still want efficient client reuse and predictable request behavior.

Common Pitfalls

  • Calling boto3 directly inside async def and assuming the event loop stays non-blocking.
  • Spawning too many thread-offloaded AWS calls at once without a concurrency limit.
  • Recreating clients for every request instead of reusing them.
  • Treating boto3 as if it provided native async methods.
  • Forgetting that reading response bodies can also be blocking work if done carelessly.

Summary

  • 'boto3 is synchronous and blocks if called directly on the event loop.'
  • Use asyncio.to_thread or run_in_executor to integrate it into async code.
  • Limit concurrency when you offload many AWS requests.
  • Reuse clients instead of recreating them in every coroutine.
  • If you need native-feeling async APIs, use a third-party wrapper knowingly rather than expecting that behavior from boto3 itself.

Course illustration
Course illustration

All Rights Reserved.