Introduction

Amazon S3 (Simple Storage Service) is a highly scalable, reliable, and low-latency data storage infrastructure. One common use case is storing large datasets that can be updated or extended over time. This article focuses on appending data to an existing S3 object—a task that's not natively supported by S3, which treats objects as immutable.

How S3 Objects Work

Before delving into appending data, it's essential to understand the immutability principle of S3. In Amazon S3, objects are considered immutable after they're created. This means you can't alter or append data directly to an existing object without creating a new version.

Immutability and Versioning

• Immutable Objects: Once an object is uploaded, it cannot be changed. To modify an object, you need to overwrite it with a new version.
• Versioning: Enables keeping all versions of an object in the same bucket, allowing for object restoration or reverting to previous versions.

Strategies for Appending Data

Due to the immutability of S3 objects, appending data requires creative strategies:

Option 1: Client-Side Concatenation

1. Download and Concatenate: Download the current object, append new data client-side, and upload it as the same object name to overwrite it.

1   import boto3
2   
3   s3 = boto3.client('s3')
4   bucket_name = 'my-bucket'
5   object_key = 'my-object.txt'
6   
7   # Download existing object
8   existing_data = s3.get_object(Bucket=bucket_name, Key=object_key)['Body'].read()
9   
10   # New data to append
11   new_data = b"\nThis is the appended data."
12   
13   # Concatenate existing and new data
14   concatenated_data = existing_data + new_data
15   
16   # Upload concatenated object
17   s3.put_object(Bucket=bucket_name, Key=object_key, Body=concatenated_data)

Option 2: Multipart Upload

1. Multipart Upload: Upload large objects using a multipart transfer. This approach allows you to append new parts to an existing upload, without re-uploading the entire object each time:
   • Initiate Multipart Upload: Start with an upload ID.
   • Upload Parts: Upload data in parts; parts can be appended sequentially.
   • Complete Upload: Finalize the upload with a call that includes all the parts' etags.

1   import boto3
2   
3   s3 = boto3.client('s3')
4   bucket_name = 'my-bucket'
5   object_key = 'large-object.dat'
6   
7   # Initiate multipart upload
8   response = s3.create_multipart_upload(Bucket=bucket_name, Key=object_key)
9   upload_id = response['UploadId']
10   
11   # Upload the first part
12   first_part_data = b'first part of the file'
13   part1 = s3.upload_part(Bucket=bucket_name, Key=object_key, PartNumber=1, UploadId=upload_id, Body=first_part_data)
14   
15   # Append by uploading additional parts
16   new_data_to_append = b'new data to append'
17   part2 = s3.upload_part(Bucket=bucket_name, Key=object_key, PartNumber=2, UploadId=upload_id, Body=new_data_to_append)
18   
19   # Complete the multipart upload
20   s3.complete_multipart_upload(
21       Bucket=bucket_name,
22       Key=object_key,
23       MultipartUpload={
24           'Parts': [
25               {'ETag': part1['ETag'], 'PartNumber': 1},
26               {'ETag': part2['ETag'], 'PartNumber': 2}
27           ]
28       },
29       UploadId=upload_id
30   )

Option 3: Append Using Lambda and Event Notifications

To make the append operation more responsive and automated, consider using AWS Lambda triggered by S3 events:

• Configure Event Notification: Trigger a Lambda function when a new file is uploaded.
• Lambda Function: Download the existing file, append data, and upload the combined file.

Option 4: Preprocessing Data Before Upload

If suitable, preprocess data before initially uploading to S3 such that future appends can be avoided or are minimized.

Considerations

• Cost: Appending through client-side requires downloading and re-uploading the object, which incurs costs.
• Performance: Multipart uploads are efficient and reduce re-upload overhead, particularly for large files.
• Consistency: Ensure atomic operations during the append process to prevent data loss or corruption.

Summary Table

Conclusion

Appending data to an S3 object requires innovative approaches due to its immutability. Strategies like client-side concatenation and multipart uploads can facilitate the process, each with distinct strengths and drawbacks. By understanding these techniques, you can effectively manage data life cycles and optimize S3 storage usage.