Introduction

If you need to download a file from the internet into S3, the main design choice is whether you want to stream it directly or save it locally first. In most server-side Python workflows, streaming the HTTP response into S3 is the better pattern because it avoids unnecessary disk writes and scales better for large objects. The other important point is that your application, not S3 itself, is usually the component performing the internet download unless you are using a separate AWS transfer service.

A Streaming Python Approach With requests And boto3

A practical approach is to open the remote file as a streamed HTTP response and pass that stream into S3.

1import boto3
2import requests
3
4s3 = boto3.client("s3")
5
6url = "https://example.com/report.csv"
7bucket = "my-bucket"
8key = "imports/report.csv"
9
10with requests.get(url, stream=True, timeout=30) as response:
11    response.raise_for_status()
12    s3.upload_fileobj(response.raw, bucket, key)
13
14print("uploaded to s3://%s/%s" % (bucket, key))

This avoids saving the file to local disk first. It is efficient and usually the cleanest option for a backend service.

Why Streaming Is Often Better

A local temp-file workflow can be fine, but direct streaming is usually better because it:

• reduces local disk usage,
• avoids extra cleanup logic,
• handles large files more gracefully,
• keeps the data path simpler.

The more your application behaves like a transfer pipe, the less local state you need to manage.

Add Metadata And Content Type When Needed

Sometimes you want the uploaded S3 object to carry metadata or a known content type.

1with requests.get(url, stream=True, timeout=30) as response:
2    response.raise_for_status()
3    s3.upload_fileobj(
4        response.raw,
5        bucket,
6        key,
7        ExtraArgs={"ContentType": "text/csv"},
8    )

This is useful when the object will later be served directly from S3 or through CloudFront.

A Temp-File Workflow Is Still Valid

If you need to validate, scan, or transform the content before upload, downloading to a local file first can be the right choice.

1import boto3
2import requests
3
4url = "https://example.com/report.csv"
5local_path = "/tmp/report.csv"
6
7with requests.get(url, timeout=30) as response:
8    response.raise_for_status()
9    with open(local_path, "wb") as f:
10        f.write(response.content)
11
12boto3.client("s3").upload_file(local_path, "my-bucket", "imports/report.csv")

This is less elegant for straight transfer, but it is useful when local processing is part of the workflow.

Handle Errors At Both Ends

There are two failure domains here:

• the internet download can fail,
• the S3 upload can fail.

So robust code should treat them separately.

1from botocore.exceptions import BotoCoreError, ClientError
2import requests
3
4try:
5    with requests.get(url, stream=True, timeout=30) as response:
6        response.raise_for_status()
7        s3.upload_fileobj(response.raw, bucket, key)
8except requests.RequestException as err:
9    print("download failed:", err)
10    raise
11except (BotoCoreError, ClientError) as err:
12    print("s3 upload failed:", err)
13    raise

This makes troubleshooting much easier than one broad exception block.

IAM And Bucket Access Still Matter

Even perfect Python code will fail if the AWS identity running it lacks permissions. The caller usually needs permission such as s3:PutObject on the target key path.

Also remember that S3 is not “pulling from the internet” in this pattern. Your application is downloading the content and then writing it to S3 using AWS credentials.

Security Considerations

A few practical rules matter here:

• prefer HTTPS sources,
• validate or restrict allowed source domains when possible,
• avoid blindly mirroring arbitrary URLs into your bucket,
• consider antivirus or content validation for untrusted inputs,
• keep IAM permissions scoped to the intended bucket and prefix.

This is especially important if the source URL comes from a user rather than from trusted internal configuration.

Common Pitfalls

• Downloading the full response into memory when streaming would work better.
• Assuming S3 itself fetches the internet URL automatically.
• Forgetting to handle download errors and upload errors separately.
• Uploading without setting useful metadata when the downstream workflow depends on it.
• Ignoring IAM permissions and then debugging the Python code instead of the AWS policy.

Summary

• The usual way to get an internet file into S3 is to download it in your application and upload it with boto3.
• Streaming the HTTP response directly into upload_fileobj is often the best pattern.
• Local temp files are still useful when validation or transformation is required.
• Handle HTTP and S3 failures as separate error cases.
• Keep IAM, source validation, and transport security in mind for production workflows.