Introduction

Reading a gzip-compressed file from S3 in Python usually means combining boto3 with the standard gzip module. The main decision is whether you want the whole decompressed file in memory at once or whether you want to stream it line by line. For large objects, streaming is usually the better choice because it avoids loading both the compressed and decompressed content into memory.

Basic S3 Download and Decompression

The get_object call returns a response whose Body is a streaming object. You can pass that directly to gzip.GzipFile.

1import boto3
2import gzip
3import io
4
5s3 = boto3.client("s3")
6
7response = s3.get_object(Bucket="my-bucket", Key="logs/app.log.gz")
8
9with gzip.GzipFile(fileobj=response["Body"]) as gz:
10    text = gz.read().decode("utf-8")
11
12print(text[:200])

This is fine for smaller files where reading the entire object into memory is acceptable.

Stream Text Line by Line

For log files or large datasets, wrap the gzip stream in io.TextIOWrapper and iterate.

1import boto3
2import gzip
3import io
4
5s3 = boto3.client("s3")
6response = s3.get_object(Bucket="my-bucket", Key="logs/app.log.gz")
7
8with gzip.GzipFile(fileobj=response["Body"]) as gz:
9    with io.TextIOWrapper(gz, encoding="utf-8") as reader:
10        for line in reader:
11            print(line.rstrip())

This pattern avoids building one huge string and is usually the right answer when processing large files.

Parse Structured Content

If the gzip file contains JSON lines, CSV rows, or other structured text, parse each line as it streams.

1import boto3
2import gzip
3import io
4import json
5
6s3 = boto3.client("s3")
7response = s3.get_object(Bucket="my-bucket", Key="events/data.jsonl.gz")
8
9with gzip.GzipFile(fileobj=response["Body"]) as gz:
10    with io.TextIOWrapper(gz, encoding="utf-8") as reader:
11        for line in reader:
12            event = json.loads(line)
13            print(event["event_type"])

This approach scales much better than decompressing the entire file first and splitting later.

If You Need Raw Bytes

Sometimes the gzip payload is binary, not text. In that case, do not decode it.

1import boto3
2import gzip
3
4s3 = boto3.client("s3")
5response = s3.get_object(Bucket="my-bucket", Key="backup/data.bin.gz")
6
7with gzip.GzipFile(fileobj=response["Body"]) as gz:
8    raw = gz.read()
9
10print(len(raw))

Choose text or bytes handling based on the actual file content, not by default habit.

Credentials and Region Handling

The examples assume boto3 can already find AWS credentials through the normal provider chain, such as environment variables, an IAM role, or a configured profile. If S3 access fails, that is a separate problem from gzip handling.

For example:

1import boto3
2
3session = boto3.Session(profile_name="dev")
4s3 = session.client("s3", region_name="us-east-1")

Get the S3 access working first, then deal with decompression.

Handling Errors

There are two main failure categories:

• S3 access errors, such as missing object or permission denied
• decompression errors, such as corrupted gzip content

1import boto3
2import botocore
3import gzip
4
5s3 = boto3.client("s3")
6
7try:
8    response = s3.get_object(Bucket="my-bucket", Key="logs/app.log.gz")
9    with gzip.GzipFile(fileobj=response["Body"]) as gz:
10        print(gz.read(100))
11except botocore.exceptions.ClientError as exc:
12    print("S3 error:", exc)
13except gzip.BadGzipFile as exc:
14    print("gzip error:", exc)

Separating those cases makes debugging much faster.

When BytesIO Is Still Useful

You will sometimes see code that reads the S3 body into io.BytesIO first. That is valid, but it forces the full compressed object into memory.

1import io
2
3body = response["Body"].read()
4buffer = io.BytesIO(body)

Use that only if another API requires a seekable in-memory file. For straightforward read-once processing, the streaming body is simpler and more memory efficient.

Practical Guidance

For small files, reading the full decompressed content into one string is fine. For logs, JSON lines, or large exports, stream through TextIOWrapper and process incrementally. If performance matters, measure both network and decompression costs before optimizing; most correctness issues happen because people choose the wrong memory model, not because gzip is too slow.

Common Pitfalls

The main mistake is reading the entire S3 object into memory when streaming would be enough. Another is decoding binary content as UTF-8 without confirming the file is actually text. Developers also sometimes treat S3 permission errors as gzip problems because both show up in the same workflow. Finally, if the object key ends with .gz but the content is not valid gzip, decompression will fail regardless of the filename.

Summary

• Use boto3.get_object to fetch the S3 object and gzip.GzipFile to decompress it.
• Stream line by line for large text files instead of reading everything into memory.
• Wrap the gzip stream in io.TextIOWrapper when you want decoded text.
• Handle S3 access errors separately from gzip format errors.
• Use BytesIO only when you truly need an in-memory buffer.