Introduction

Uploading a directory to S3 is really a loop over files plus a rule for turning local paths into S3 object keys. S3 does not store directories in the filesystem sense, so the job is to walk the local tree, preserve relative paths, and upload each file under a predictable key.

With boto3, the most practical method is upload_file, which handles multipart upload behavior for larger files and is easier to use than lower-level request APIs.

Walk the Local Directory Tree

The first step is to visit every file under the base directory:

1from pathlib import Path
2
3base_dir = Path("site-assets").resolve()
4
5for path in base_dir.rglob("*"):
6    if path.is_file():
7        print(path)

This gives you every file, including nested ones. The important next step is to convert each file into a relative path so the S3 key does not accidentally include machine-specific absolute path segments.

Build Stable S3 Keys

Relative paths are the cleanest source for S3 keys:

1from pathlib import Path
2
3def build_key(base_dir: Path, file_path: Path, prefix: str = "") -> str:
4    relative = file_path.relative_to(base_dir).as_posix()
5    prefix = prefix.strip("/")
6    return f"{prefix}/{relative}" if prefix else relative

If file_path is /tmp/site-assets/css/app.css and the base directory is /tmp/site-assets, the key becomes css/app.css. Adding a prefix such as deployments/current produces deployments/current/css/app.css.

Upload Each File with boto3

Now combine the directory walk and key builder:

1from pathlib import Path
2import boto3
3
4def upload_directory(local_dir: str, bucket: str, prefix: str = "") -> None:
5    s3 = boto3.client("s3")
6    base_dir = Path(local_dir).resolve()
7
8    if not base_dir.is_dir():
9        raise ValueError(f"Not a directory: {base_dir}")
10
11    for file_path in base_dir.rglob("*"):
12        if not file_path.is_file():
13            continue
14
15        key = build_key(base_dir, file_path, prefix)
16        s3.upload_file(str(file_path), bucket, key)
17        print(f"uploaded {file_path} -> s3://{bucket}/{key}")
18
19upload_directory("site-assets", "my-bucket", "releases/current")

That is enough for many backup, deployment, and asset-publishing workflows.

Add Metadata When Needed

If the uploaded files will be served directly from S3 or through CloudFront, content types matter. You can pass them through ExtraArgs:

1import mimetypes
2
3def upload_file_with_metadata(s3, local_file: str, bucket: str, key: str) -> None:
4    content_type, _ = mimetypes.guess_type(local_file)
5    extra_args = {}
6
7    if content_type:
8        extra_args["ContentType"] = content_type
9
10    s3.upload_file(local_file, bucket, key, ExtraArgs=extra_args)

Without the correct content type, browsers may download files instead of rendering them as HTML, CSS, JavaScript, or images.

Use the Standard Credential Chain

Do not hard-code AWS credentials in the script. boto3 already knows how to load credentials from:

• environment variables
• AWS shared credentials files
• IAM roles on AWS infrastructure

That means the same upload script can run locally and in CI without changing the code, as long as credentials are provided through standard AWS mechanisms.

Know What This Code Does Not Do

An upload loop only sends files to S3. It does not delete old objects that were removed locally. If you need true synchronization behavior, deletion has to be handled separately and very carefully.

That distinction matters because "upload directory" and "mirror directory" are not the same task.

Common Pitfalls

• Treating S3 keys as if they were real directories instead of object names with prefixes.
• Building keys from absolute paths and uploading unwanted machine-specific segments.
• Forgetting content types for web assets.
• Hard-coding AWS credentials in the script.
• Assuming upload-only code also deletes obsolete remote objects.

Summary

• Uploading a directory to S3 means walking files and mapping relative paths to object keys.
• Use boto3.client("s3").upload_file(...) as the default upload primitive.
• Preserve relative paths so the bucket layout stays predictable.
• Add metadata such as ContentType when files will be served directly.
• Keep upload logic separate from deletion or sync logic.