Introduction

Large downloads fail for different reasons than small ones. The goal is not only to fetch the bytes, but to do it without loading the whole response into memory, while still handling timeouts, partial files, and HTTP errors cleanly.

Stream the Response Instead of Loading It All

The most important setting in requests is stream=True. Without it, requests may download the full body before your code starts writing to disk.

1import requests
2
3url = "https://example.com/big-file.zip"
4output_path = "big-file.zip"
5
6with requests.get(url, stream=True, timeout=(5, 30)) as response:
7    response.raise_for_status()
8
9    with open(output_path, "wb") as file:
10        for chunk in response.iter_content(chunk_size=1024 * 1024):
11            if chunk:
12                file.write(chunk)

This writes the file in one-megabyte chunks and keeps memory usage low even for very large responses.

Write to a Temporary File First

A robust downloader should not leave a half-finished target file behind if the request fails. Writing to a temporary file and renaming it only after success is a safer pattern.

1from pathlib import Path
2import requests
3
4url = "https://example.com/big-file.zip"
5target = Path("big-file.zip")
6temp = target.with_suffix(target.suffix + ".part")
7
8with requests.get(url, stream=True, timeout=(5, 30)) as response:
9    response.raise_for_status()
10
11    with temp.open("wb") as file:
12        for chunk in response.iter_content(chunk_size=1024 * 1024):
13            if chunk:
14                file.write(chunk)
15
16temp.replace(target)

That way, the final path contains either a complete file or no file at all.

Add Basic Progress Reporting

If the server sends Content-Length, you can show progress while writing.

1import requests
2
3url = "https://example.com/big-file.zip"
4
5with requests.get(url, stream=True, timeout=(5, 30)) as response:
6    response.raise_for_status()
7    total = int(response.headers.get("content-length", 0))
8    downloaded = 0
9
10    with open("big-file.zip", "wb") as file:
11        for chunk in response.iter_content(chunk_size=1024 * 1024):
12            if not chunk:
13                continue
14
15            file.write(chunk)
16            downloaded += len(chunk)
17            if total:
18                percent = downloaded / total * 100
19                print(f"{percent:.1f}%")

This is optional, but it is very useful for long transfers.

Resuming an Interrupted Download

If the server supports range requests, you can resume from the last written byte instead of starting over.

1from pathlib import Path
2import requests
3
4url = "https://example.com/big-file.zip"
5path = Path("big-file.zip")
6existing_size = path.stat().st_size if path.exists() else 0
7headers = {"Range": f"bytes={existing_size}-"} if existing_size else {}
8mode = "ab" if existing_size else "wb"
9
10with requests.get(url, headers=headers, stream=True, timeout=(5, 30)) as response:
11    response.raise_for_status()
12
13    with path.open(mode) as file:
14        for chunk in response.iter_content(chunk_size=1024 * 1024):
15            if chunk:
16                file.write(chunk)

If the server ignores the Range header and returns the full file with status 200, you should discard the partial file and restart instead of appending blindly.

Handle Errors Deliberately

Large downloads are more exposed to network interruptions, permission problems, and short server outages. Use raise_for_status() for HTTP errors and wrap the request in try and except when the calling code needs retry or cleanup behavior.

It also helps to set both connect and read timeouts, as shown above with timeout=(5, 30). That avoids hanging forever on a stalled connection.

Common Pitfalls

• Omitting stream=True can cause unnecessary memory use on large files.
• Writing directly to the final filename risks leaving behind a corrupted partial file.
• Appending to an existing file without checking whether the server honored the Range header can corrupt the download.
• Using very tiny chunk sizes increases overhead, while extremely large chunks can reduce responsiveness.
• Ignoring timeouts and status checks makes failures harder to diagnose.

Summary

• Use stream=True and iter_content(...) for large downloads.
• Write to disk incrementally instead of loading the full response into memory.
• Prefer a temporary file and rename-on-success pattern.
• Support resuming only when the server actually honors range requests.
• Combine chunked writing with timeouts and status checks for a reliable downloader.