Introduction

When an S3 object is large, reading it line by line is much safer than loading the whole file into memory. With boto3, the normal pattern is to stream the object body returned by get_object and iterate over its lines as bytes, decoding them only as needed.

Use iter_lines on the Response Body

The simplest streaming approach is Body.iter_lines().

1import boto3
2
3s3 = boto3.client("s3")
4
5response = s3.get_object(Bucket="my-bucket", Key="logs/app.log")
6body = response["Body"]
7
8for raw_line in body.iter_lines():
9    line = raw_line.decode("utf-8")
10    print(line)

This avoids reading the entire object into memory first, which is exactly what you want for log files, exports, and large text datasets.

Wrap It in a Reusable Generator

Application code is usually cleaner if the S3 streaming logic lives in one helper.

1from typing import Iterator
2import boto3
3
4def read_s3_lines(bucket: str, key: str, encoding: str = "utf-8") -> Iterator[str]:
5    s3 = boto3.client("s3")
6    response = s3.get_object(Bucket=bucket, Key=key)
7
8    for raw_line in response["Body"].iter_lines():
9        yield raw_line.decode(encoding)
10
11for line in read_s3_lines("my-bucket", "data/events.txt"):
12    if "ERROR" in line:
13        print(line)

This keeps downstream code focused on processing rather than S3 plumbing.

Handle Gzip Files Without Losing Streaming

Many S3 text files are compressed. You can still stream them line by line by wrapping the response body.

1import boto3
2import gzip
3import io
4
5s3 = boto3.client("s3")
6response = s3.get_object(Bucket="my-bucket", Key="logs/app.log.gz")
7
8with gzip.GzipFile(fileobj=response["Body"]) as gz:
9    with io.TextIOWrapper(gz, encoding="utf-8") as reader:
10        for line in reader:
11            print(line.rstrip("\n"))

This preserves the streaming behavior while correctly handling compressed content.

Decode Bytes Deliberately

S3 gives you bytes, not text. If the file encoding is not guaranteed, decode carefully.

for raw_line in body.iter_lines():
    line = raw_line.decode("utf-8", errors="replace")
    print(line)

Using errors="replace" can keep a long-running job alive even when a few malformed lines contain unexpected bytes.

Parse Structured Line Formats Incrementally

A common real-world case is JSON Lines. You can decode each line independently and continue past malformed rows if the workload allows it.

1import json
2
3for line in read_s3_lines("my-bucket", "events.jsonl"):
4    try:
5        event = json.loads(line)
6        print(event.get("event_type"))
7    except json.JSONDecodeError:
8        continue

That keeps memory stable and isolates bad records to the single line that failed.

Add Retries and IAM Correctness

Production use is rarely just about the loop. Make sure the caller has s3:GetObject permission for the target key, and configure retries if the workload runs over unstable networks or long-lived sessions.

1from botocore.config import Config
2import boto3
3
4cfg = Config(retries={"max_attempts": 10, "mode": "standard"})
5s3 = boto3.client("s3", config=cfg)

If the code runs inside AWS, IAM roles are preferable to hard-coded credentials. For local development, named profiles are usually cleaner.

Avoid Full Reads Unless You Actually Need Them

This is the main anti-pattern:

data = body.read().decode("utf-8")
lines = data.splitlines()

That works, but it loads the full object into memory and defeats the point of line-by-line processing. It is fine for small files. It is the wrong default for unknown or large file sizes.

Common Pitfalls

The most common mistake is calling read() first and only then splitting lines. That loads the whole object into memory and removes the streaming advantage.

Another issue is forgetting that the S3 response body yields bytes, not strings. Decode explicitly with the encoding you expect.

Developers also often ignore compression and try to parse gzip content as plain text. If the object is compressed, wrap it accordingly.

Finally, make sure permission and retry behavior are part of the design. A streaming loop is only useful if it is allowed to read the object and can recover from ordinary transient failures.

Summary

• Use get_object plus Body.iter_lines() to stream S3 text objects line by line.
• Decode bytes explicitly instead of assuming you already have text.
• Wrap the logic in a generator when several parts of the codebase need it.
• Handle gzip objects with streaming decompression wrappers.
• Avoid full-body reads unless the file is known to be small and memory cost is irrelevant.