Introduction

Amazon S3 is a good place to store individual objects, but users often want a single downloadable archive. A common pattern is to let an AWS Lambda function read several S3 objects, stream them into a ZIP archive, and write the archive back to S3 without provisioning a server.

When Lambda Is a Good Fit

This approach works well when the archive is created on demand, the input files are already in S3, and the total runtime fits Lambda limits. The key design choice is to stream data instead of downloading everything into memory first.

If you load every object into a buffer, memory usage grows with the total archive size. A better pattern is:

1. Read each source object as a stream.
2. Append each stream to a ZIP writer.
3. Pipe the ZIP output directly to an S3 upload stream.

That keeps memory usage predictable and avoids filling the /tmp directory unless you explicitly need it.

Streaming ZIP Data Back to S3

In modern Node.js Lambda functions, the AWS SDK for JavaScript v3 is the usual choice. For large output, @aws-sdk/lib-storage is useful because it performs multipart upload while data is still being produced.

The example below assumes the event contains a list of source keys and one destination key. It reads from one bucket and writes the ZIP file to another location in S3.

1import { S3Client, GetObjectCommand } from "@aws-sdk/client-s3";
2import { Upload } from "@aws-sdk/lib-storage";
3import archiver from "archiver";
4import { PassThrough } from "node:stream";
5
6const s3 = new S3Client({});
7
8export const handler = async (event) => {
9  const sourceBucket = process.env.SOURCE_BUCKET;
10  const targetBucket = process.env.TARGET_BUCKET;
11  const targetKey = event.targetKey;
12  const sourceKeys = event.sourceKeys;
13
14  if (!Array.isArray(sourceKeys) || sourceKeys.length === 0) {
15    throw new Error("event.sourceKeys must contain at least one S3 key");
16  }
17
18  const zipStream = new PassThrough();
19  const archive = archiver("zip", { zlib: { level: 9 } });
20
21  const upload = new Upload({
22    client: s3,
23    params: {
24      Bucket: targetBucket,
25      Key: targetKey,
26      Body: zipStream,
27      ContentType: "application/zip",
28    },
29  });
30
31  archive.on("warning", (err) => {
32    if (err.code !== "ENOENT") throw err;
33  });
34
35  archive.on("error", (err) => {
36    throw err;
37  });
38
39  archive.pipe(zipStream);
40
41  for (const key of sourceKeys) {
42    const response = await s3.send(
43      new GetObjectCommand({
44        Bucket: sourceBucket,
45        Key: key,
46      })
47    );
48
49    const fileName = key.split("/").pop() || "file";
50    archive.append(response.Body, { name: fileName });
51  }
52
53  await archive.finalize();
54  await upload.done();
55
56  return {
57    bucket: targetBucket,
58    key: targetKey,
59    fileCount: sourceKeys.length,
60  };
61};

Trigger and Event Design

Many teams trigger this workflow from API Gateway, SQS, or Step Functions rather than directly from an S3 upload event. That makes it easier to pass an explicit list of keys and avoids accidental recursion.

If you do use an S3 trigger, keep the output ZIP in a different prefix or bucket. Otherwise the newly written ZIP file may trigger the same Lambda again.

A minimal test payload looks like this:

1{
2  "targetKey": "archives/invoice-bundle.zip",
3  "sourceKeys": [
4    "invoices/january.pdf",
5    "invoices/february.pdf",
6    "invoices/march.pdf"
7  ]
8}

Permissions and Packaging

The Lambda execution role needs permission to read source objects and write the archive. In practice that means s3:GetObject on the input prefix and s3:PutObject on the output prefix. If the function is listing keys itself, it also needs s3:ListBucket.

You also need to package archiver with the function or include it through your deployment process. For large dependency trees, keeping the deployment bundle small still matters because cold starts are affected by package size.

Common Pitfalls

The most common failure is buffering too much data. Using await response.Body.transformToByteArray() for every file works for tiny archives but breaks down quickly.

Another issue is forgetting that Lambda has time and memory limits. If users may request very large archives, step back and ask whether Lambda is still the right compute target. In some systems, a container task or batch job is the safer choice.

A third problem is duplicate file names inside the ZIP. If two keys end with the same base name, one entry can overwrite the other in the archive. Use a stable naming scheme, such as preserving part of the prefix, when collisions are possible.

Summary

• Stream S3 objects into the ZIP instead of loading them all into memory.
• Use archiver plus a streaming S3 upload for efficient archive creation.
• Keep output in a separate prefix or bucket to avoid recursive triggers.
• Grant only the S3 permissions the function actually needs.
• Reevaluate Lambda if archive size or runtime becomes unpredictable.