Introduction

Streaming a response with requests is the right approach when the payload is large or when the server keeps sending data over time. Instead of buffering the entire body into memory, you consume bytes or lines incrementally. The crucial rule is to request the response with stream=True and then iterate using the right API for the data shape you expect.

Start the Request in Streaming Mode

Without stream=True, requests downloads the full body before you begin processing it. For large files or long-lived responses, that defeats the purpose.

Basic chunked download:

1import requests
2
3url = "https://example.com/large-file.bin"
4
5with requests.get(url, stream=True, timeout=30) as response:
6    response.raise_for_status()
7
8    with open("large-file.bin", "wb") as target:
9        for chunk in response.iter_content(chunk_size=8192):
10            if chunk:
11                target.write(chunk)

This is the standard file-download pattern. iter_content yields bytes incrementally as they arrive.

Use iter_lines for Line-Oriented Streams

If the server emits newline-delimited text, iter_lines is usually a better choice than iter_content.

1import requests
2
3with requests.get("https://example.com/events", stream=True, timeout=30) as response:
4    response.raise_for_status()
5
6    for line in response.iter_lines(decode_unicode=True):
7        if line:
8            print("received:", line)

This works well for log streams, newline-delimited JSON, and similar protocols.

Processing NDJSON Safely

Many streaming APIs send one JSON object per line. You can parse each line as it arrives:

1import json
2import requests
3
4with requests.get("https://example.com/stream.ndjson", stream=True, timeout=30) as response:
5    response.raise_for_status()
6
7    for line in response.iter_lines(decode_unicode=True):
8        if not line:
9            continue
10        item = json.loads(line)
11        print(item)

This keeps memory usage low and lets you act on each record immediately.

Why response.text and response.json() Are the Wrong Tools Here

These convenience methods assume you want the full response body. They buffer the response and only then decode or parse it. That is fine for small responses, but it is the opposite of streaming behavior.

If you call response.text on a large or endless stream, you lose the main benefit of incremental processing.

Timeouts and Connection Hygiene

Streaming code should always define a timeout. Otherwise, a stalled connection can hang the process indefinitely.

Also use a context manager or explicitly close the response so the connection returns to the pool:

1import requests
2
3response = requests.get("https://example.com/data", stream=True, timeout=30)
4try:
5    response.raise_for_status()
6    for chunk in response.iter_content(chunk_size=4096):
7        if chunk:
8            print(len(chunk))
9finally:
10    response.close()

The with form is usually cleaner, but both patterns are correct.

Choosing the Right Chunk Size

There is no universally correct chunk size. Smaller chunks improve responsiveness and reduce latency to first processing, while larger chunks reduce loop overhead.

Practical defaults:

• '8192 bytes for general file downloads'
• smaller sizes for highly interactive streaming
• line-based iteration when the stream is naturally line-delimited

Do not tune this prematurely unless profiling shows it matters.

Long-Lived Streams Need Failure Handling

Network streams break. Production code should decide what to do when the server disconnects:

• stop and report the failure
• reconnect with backoff
• resume from a checkpoint if the protocol supports it

The requests iteration helpers make the read loop easy, but they do not solve retry strategy for you.

Common Pitfalls

• Forgetting stream=True and buffering the whole response anyway.
• Using response.text or response.json() on data that should be consumed incrementally.
• Ignoring chunk emptiness checks when writing streamed file data.
• Omitting timeouts and letting the process hang on slow or broken connections.
• Using iter_content when the protocol is actually line-oriented and iter_lines would be simpler.

Summary

• Use stream=True to keep response bodies incremental.
• Use iter_content for binary or chunk-based processing.
• Use iter_lines for text streams and newline-delimited protocols.
• Avoid full-body helpers like response.text for genuinely streamed workloads.
• Add timeouts and connection cleanup so the streaming loop behaves predictably.