Introduction

Writing to Amazon S3 with boto3 usually means one of three things: uploading a file from disk, uploading a file-like object, or sending bytes that were generated in memory. Boto3 supports all three, and the best API depends on where the data comes from and how much control you need over metadata and transfer behavior.

Create an S3 Client the Normal Way

Most upload code starts with an S3 client:

import boto3

s3 = boto3.client("s3", region_name="us-east-1")

In production, do not hardcode credentials in the script. Let boto3 resolve them from:

• an IAM role on EC2, ECS, or Lambda
• environment variables
• AWS CLI credentials
• a named profile

Using roles is usually the safest option because the code stays free of long-lived secrets.

Upload a Real File from Disk

If the content already exists as a local file, upload_file is the simplest high-level method.

1import boto3
2
3s3 = boto3.client("s3")
4
5s3.upload_file(
6    Filename="report.csv",
7    Bucket="my-app-bucket",
8    Key="exports/report.csv",
9)

This is the right default for local files because boto3 handles multipart transfer behavior for you when needed. It is usually better than reading the file yourself and pushing the bytes through put_object.

Upload Generated Data from Memory

If your program has already built the content in memory, put_object is often the most direct choice.

1import boto3
2
3s3 = boto3.client("s3")
4
5csv_data = "id,name\n1,Ada\n2,Grace\n"
6
7s3.put_object(
8    Bucket="my-app-bucket",
9    Key="exports/users.csv",
10    Body=csv_data.encode("utf-8"),
11    ContentType="text/csv",
12)

The Body value can be bytes or a file-like stream. Setting ContentType is a good habit because downstream tools and browsers use that metadata to interpret the object correctly.

Upload a File-Like Object

For in-memory buffers, zipped output, or generated content that behaves like a file, use upload_fileobj.

1import boto3
2from io import BytesIO
3
4s3 = boto3.client("s3")
5
6buffer = BytesIO()
7buffer.write(b"hello from memory\n")
8buffer.seek(0)
9
10s3.upload_fileobj(buffer, "my-app-bucket", "logs/hello.txt")

The call to seek(0) matters. If you leave the pointer at the end of the stream, boto3 uploads zero bytes.

Add Metadata and Storage Options

Uploads often need more than raw bytes. You may want metadata, cache headers, or encryption settings. With client uploads, that usually goes through ExtraArgs or direct put_object parameters.

1import boto3
2
3s3 = boto3.client("s3")
4
5s3.upload_file(
6    Filename="logo.png",
7    Bucket="my-app-bucket",
8    Key="images/logo.png",
9    ExtraArgs={
10        "ContentType": "image/png",
11        "Metadata": {"source": "branding-pipeline"},
12        "ServerSideEncryption": "AES256",
13    },
14)

For small in-memory writes, the equivalent put_object call is often more explicit:

1s3.put_object(
2    Bucket="my-app-bucket",
3    Key="data/test.txt",
4    Body=b"ok",
5    ServerSideEncryption="AES256",
6)

Choose the form that matches how the data is already represented in your program.

Handle Errors Explicitly

S3 writes can fail for reasons that matter operationally: missing permissions, wrong bucket names, invalid regions, or network issues. Catch boto3 exceptions explicitly so you keep the AWS error details.

1import boto3
2from botocore.exceptions import ClientError, BotoCoreError
3
4s3 = boto3.client("s3")
5
6try:
7    s3.put_object(Bucket="my-app-bucket", Key="data/test.txt", Body=b"ok")
8except ClientError as exc:
9    print(f"AWS client error: {exc.response['Error']['Code']}")
10except BotoCoreError as exc:
11    print(f"Boto core error: {exc}")

That is much more useful than catching a generic exception and losing the real reason the upload failed.

Verify Important Uploads

If the uploaded object matters to a workflow, verify it instead of assuming success means the right key and metadata were written.

response = s3.head_object(Bucket="my-app-bucket", Key="exports/users.csv")
print(response["ContentLength"])
print(response["ContentType"])

That check catches mistakes such as:

• wrong key names
• missing content type
• empty uploads from an unrewound stream

For business-critical pipelines, validating the result is often worth the extra request.

Choosing the Right Method

A practical rule is:

• 'upload_file for an existing local file'
• 'upload_fileobj for a readable binary stream'
• 'put_object for smaller in-memory payloads and explicit object metadata'

That keeps the code aligned with the actual data source instead of forcing everything through one API.

Common Pitfalls

• Hardcoding AWS keys in source code instead of using IAM roles or external credential configuration.
• Using put_object for large local files when upload_file is the more suitable upload primitive.
• Forgetting seek(0) before uploading a BytesIO object.
• Writing text without encoding it intentionally or without setting ContentType.
• Swallowing boto3 exceptions and losing the specific AWS error code that explains the failure.

Summary

• Use upload_file for files on disk, upload_fileobj for streams, and put_object for direct in-memory writes.
• Let boto3 obtain credentials from roles, profiles, or environment configuration instead of hardcoding them.
• Add metadata such as ContentType and encryption settings deliberately.
• Handle AWS and boto3 exceptions explicitly so failures are diagnosable.
• Verify important uploads with head_object when correctness matters.