How can I easily determine if a Boto 3 S3 bucket resource exists?

Boto3

AWS

Python

Cloud Storage

How can I easily determine if a Boto 3 S3 bucket resource exists?

Master System Design with Codemia

Enhance your system design skills with over 120 practice problems, detailed solutions, and hands-on exercises.

Start Practicing Learn More

Introduction

To check if an S3 bucket exists using Boto3, call head_bucket() on the S3 client and catch ClientError. A 404 response means the bucket does not exist, while a 403 means it exists but you lack access. Alternatives include listing all buckets with list_buckets() or using the S3 resource interface. The head_bucket() approach is the most efficient because it makes a single lightweight API call without downloading any bucket contents.

Method 1: head_bucket (Recommended)

head_bucket() sends a HEAD request to S3. It returns metadata if the bucket exists and you have permission, or raises ClientError with a specific HTTP status code:

python

1import boto3
2from botocore.exceptions import ClientError
3
4def bucket_exists(bucket_name):
5    s3 = boto3.client('s3')
6    try:
7        s3.head_bucket(Bucket=bucket_name)
8        return True
9    except ClientError as e:
10        error_code = int(e.response['Error']['Code'])
11        if error_code == 404:
12            return False  # Bucket does not exist
13        elif error_code == 403:
14            # Bucket exists but you don't have access
15            print(f"Bucket '{bucket_name}' exists but access is denied")
16            return True
17        else:
18            raise  # Unexpected error
19
20# Usage
21print(bucket_exists('my-existing-bucket'))  # True
22print(bucket_exists('nonexistent-bucket'))  # False

Status Codes

Code	Meaning
200	Bucket exists, you have access
301	Bucket exists in a different region
403	Bucket exists, access denied
404	Bucket does not exist

Method 2: S3 Resource Interface

The S3 resource API provides a higher-level interface:

python

1import boto3
2from botocore.exceptions import ClientError
3
4def bucket_exists_resource(bucket_name):
5    s3 = boto3.resource('s3')
6    bucket = s3.Bucket(bucket_name)
7    try:
8        bucket.meta.client.head_bucket(Bucket=bucket_name)
9        return True
10    except ClientError as e:
11        error_code = int(e.response['Error']['Code'])
12        if error_code == 404:
13            return False
14        raise
15
16print(bucket_exists_resource('my-bucket'))

You can also check the creation_date attribute — it is None for non-existent buckets when accessed through s3.Bucket():

python

1import boto3
2
3def bucket_exists_via_creation_date(bucket_name):
4    s3 = boto3.resource('s3')
5    bucket = s3.Bucket(bucket_name)
6    # creation_date is only set for buckets you own and that exist
7    return bucket.creation_date is not None
8
9print(bucket_exists_via_creation_date('my-bucket'))

Note: creation_date is only available for buckets owned by the authenticated account. For buckets owned by others, use head_bucket().

Method 3: list_buckets

List all buckets and check if the target is in the list:

python

1import boto3
2
3def bucket_exists_list(bucket_name):
4    s3 = boto3.client('s3')
5    response = s3.list_buckets()
6    bucket_names = [b['Name'] for b in response['Buckets']]
7    return bucket_name in bucket_names
8
9print(bucket_exists_list('my-bucket'))

This method only finds buckets owned by the authenticated account. It is less efficient than head_bucket() because it retrieves the entire bucket list, which can be slow if you own hundreds of buckets.

Checking if an Object Exists

The same pattern works for checking individual objects within a bucket:

python

1import boto3
2from botocore.exceptions import ClientError
3
4def object_exists(bucket_name, key):
5    s3 = boto3.client('s3')
6    try:
7        s3.head_object(Bucket=bucket_name, Key=key)
8        return True
9    except ClientError as e:
10        if int(e.response['Error']['Code']) == 404:
11            return False
12        raise
13
14print(object_exists('my-bucket', 'data/report.csv'))

Using Waiters

Boto3 provides waiters that poll until a condition is met:

python

1import boto3
2
3def wait_for_bucket(bucket_name, timeout=60):
4    s3 = boto3.client('s3')
5
6    # Wait until bucket exists (polls every 5 seconds)
7    waiter = s3.get_waiter('bucket_exists')
8    try:
9        waiter.wait(
10            Bucket=bucket_name,
11            WaiterConfig={'Delay': 5, 'MaxAttempts': timeout // 5}
12        )
13        print(f"Bucket '{bucket_name}' exists")
14        return True
15    except Exception:
16        print(f"Bucket '{bucket_name}' not found within {timeout}s")
17        return False

Waiters are useful when you create a bucket and need to wait for it to be available before uploading objects.

Create Bucket If Not Exists

A common pattern combining existence check with creation:

python

1import boto3
2from botocore.exceptions import ClientError
3
4def ensure_bucket_exists(bucket_name, region='us-east-1'):
5    s3 = boto3.client('s3', region_name=region)
6
7    try:
8        s3.head_bucket(Bucket=bucket_name)
9        print(f"Bucket '{bucket_name}' already exists")
10    except ClientError as e:
11        error_code = int(e.response['Error']['Code'])
12        if error_code == 404:
13            if region == 'us-east-1':
14                s3.create_bucket(Bucket=bucket_name)
15            else:
16                s3.create_bucket(
17                    Bucket=bucket_name,
18                    CreateBucketConfiguration={
19                        'LocationConstraint': region
20                    }
21                )
22            print(f"Created bucket '{bucket_name}'")
23        else:
24            raise
25
26ensure_bucket_exists('my-app-data', region='us-west-2')

Common Pitfalls

Not distinguishing 403 from 404: A 403 error means the bucket exists but you lack permission. Treating 403 as "not found" leads to incorrect logic or failed attempts to create a bucket that already exists.
Using list_buckets() for existence checks: list_buckets() only returns buckets you own and fetches the entire list. head_bucket() is faster and works for buckets in other accounts (returning 403 for existing but inaccessible buckets).
Ignoring region-specific behavior: head_bucket() returns 301 for buckets in a different region. Configure the client with the correct region or handle 301 responses.
Creating buckets without LocationConstraint: For any region other than us-east-1, you must pass CreateBucketConfiguration={'LocationConstraint': region}. Omitting it defaults to us-east-1.
Not handling eventual consistency: S3 bucket creation is eventually consistent. Immediately after creating a bucket, head_bucket() may still return 404 briefly. Use a waiter or add a short delay.

Summary

Use head_bucket() to check if a bucket exists — it is the fastest single-call method
Catch ClientError and check the HTTP status code: 404 means not found, 403 means exists but access denied
Use creation_date on the S3 resource for a quick ownership check on your own buckets
Avoid list_buckets() for existence checks — it retrieves all buckets and only finds ones you own
Use waiters when checking for buckets immediately after creation to handle eventual consistency