Amazon S3
multiple object retrieval
cloud storage
API request optimization
data transfer efficiency

Possible to get multiple object from Amazon S3 in single request?

Master System Design with Codemia

Enhance your system design skills with over 120 practice problems, detailed solutions, and hands-on exercises.

Introduction

In standard Amazon S3, a GetObject request fetches one object by key. There is no general-purpose API that takes an arbitrary list of object keys and returns all of those objects in one normal download response.

The Short Answer

If the question is "can I fetch three unrelated objects from S3 with one ordinary read request," the answer is no.

S3's object API is built around one object per key. A normal GetObject targets a single bucket and key pair.

python
1import boto3
2
3s3 = boto3.client("s3")
4response = s3.get_object(Bucket="my-bucket", Key="reports/january.csv")
5print(response["ContentLength"])

To fetch multiple keys, clients normally issue multiple GetObject requests.

What You Can Do Instead

Even though S3 does not provide a general multi-key get API, there are several practical alternatives.

The first is client-side parallelism. Issue several object requests concurrently and combine the results in your application.

python
1import boto3
2from concurrent.futures import ThreadPoolExecutor
3
4s3 = boto3.client("s3")
5keys = ["a.txt", "b.txt", "c.txt"]
6
7def fetch(key):
8    return s3.get_object(Bucket="my-bucket", Key=key)["Body"].read()
9
10with ThreadPoolExecutor(max_workers=3) as pool:
11    results = list(pool.map(fetch, keys))
12
13print([len(x) for x in results])

This still uses one S3 request per object, but it reduces wall-clock latency for many workloads.

Bundle Objects Upstream When Access Patterns Are Known

If your application frequently needs the same set of small objects together, a better design is often to bundle them into a larger archive or structured container.

Examples include:

  • a ZIP file
  • a tarball
  • a parquet file containing many logical records
  • a JSON document containing related items

Then one S3 object request retrieves the whole bundle.

This can reduce per-request overhead, though it changes update and caching behavior.

Range Requests Only Work Within One Object

S3 supports byte-range requests, but only for a single object.

python
1response = s3.get_object(
2    Bucket="my-bucket",
3    Key="big-archive.bin",
4    Range="bytes=0-1023"
5)

This is useful if you packed many logical items into one physical object or need only part of a large file. It is not a multi-object API.

Features People Often Confuse With Multi-Get

Several S3-related features sound similar but do not solve the same problem.

  • S3 Select reads part of one object using SQL-like filtering
  • S3 Batch Operations performs actions across many objects, but not as a single low-latency fetch response
  • list operations enumerate keys, but they do not download object bodies

So while S3 has batch-style management features, it does not expose a general single-call multi-object download in the way some databases expose batch reads.

When Parallel Fetching Is Good Enough

For many systems, parallel requests are the right tradeoff.

This works well when:

  • objects are independent
  • the object count is moderate
  • the client can tolerate combining results itself
  • the application benefits from connection reuse and concurrency

In these cases, trying to invent a fake multi-get layer can be more complex than just issuing several requests efficiently.

Common Pitfalls

The biggest mistake is assuming ListObjectsV2 or similar listing APIs can return object contents. They return metadata, not bodies.

Another mistake is using many tiny S3 objects when the application always needs them together. That can make request overhead dominate the workload.

A third issue is confusing range requests with multi-object retrieval. Ranges work only within one object.

Finally, S3 Batch Operations is not a low-latency multi-download endpoint for application reads.

Summary

  • Standard S3 does not provide a general single-request multi-object download API.
  • 'GetObject fetches one key at a time.'
  • The usual workaround is parallel client-side fetching.
  • If objects are always read together, bundling them into one larger object is often better.
  • Range requests and S3 Select operate on one object, not many unrelated keys.
  • Design object layout around access patterns if multi-object retrieval is a frequent requirement.

Course illustration
Course illustration

All Rights Reserved.