Node FTP Multiple asynchronous calls inside loop

Node.js

FTP

Asynchronous Programming

JavaScript

Loops

Node FTP Multiple asynchronous calls inside loop

Master System Design with Codemia

Enhance your system design skills with over 120 practice problems, detailed solutions, and hands-on exercises.

Start Practicing Learn More

Introduction

When your Node.js application needs to upload or download multiple files from an FTP server, you typically iterate over a list of file paths and perform an FTP operation for each one. This is where asynchronous programming patterns become critical. Naively firing off all FTP calls at once can overwhelm the server, exhaust connections, or produce race conditions. This article walks through three approaches for handling multiple asynchronous FTP calls inside a loop: callbacks, promises with async/await, and controlled concurrency with p-limit.

The Problem with Naive Loops

Consider a list of files you need to upload to an FTP server. A common first instinct is to loop through them and call the upload function for each one.

javascript

1const Client = require('ftp');
2const fs = require('fs');
3
4const files = ['file1.txt', 'file2.txt', 'file3.txt'];
5const client = new Client();
6
7client.on('ready', () => {
8    for (const file of files) {
9        client.put(fs.createReadStream(file), `/remote/${file}`, (err) => {
10            if (err) console.error(`Failed to upload ${file}:`, err);
11            else console.log(`Uploaded ${file}`);
12        });
13    }
14});
15
16client.connect({ host: 'ftp.example.com', user: 'user', password: 'pass' });

This code fires all three uploads simultaneously on the same connection. Many FTP servers do not support concurrent transfers on a single connection, so this can fail with protocol errors or corrupted transfers. You need to either serialize the operations or manage concurrency properly.

Approach 1: Sequential Callbacks

The callback-based ftp library requires you to chain operations manually to ensure one completes before the next starts.

javascript

1const Client = require('ftp');
2const fs = require('fs');
3
4function uploadFiles(client, files, index, callback) {
5    if (index >= files.length) {
6        return callback(null, 'All files uploaded');
7    }
8
9    const file = files[index];
10    client.put(fs.createReadStream(file), `/remote/${file}`, (err) => {
11        if (err) return callback(err);
12        console.log(`Uploaded ${file}`);
13        uploadFiles(client, files, index + 1, callback);
14    });
15}
16
17const client = new Client();
18client.on('ready', () => {
19    const files = ['file1.txt', 'file2.txt', 'file3.txt'];
20    uploadFiles(client, files, 0, (err, result) => {
21        if (err) console.error('Upload failed:', err);
22        else console.log(result);
23        client.end();
24    });
25});
26
27client.connect({ host: 'ftp.example.com', user: 'user', password: 'pass' });

This recursive approach processes one file at a time. Each upload starts only after the previous one finishes. It works, but the nested callback structure becomes harder to read as complexity grows.

Approach 2: Promises with async/await

The promise-ftp library wraps FTP operations in promises, letting you use async/await for cleaner sequential code.

javascript

1const PromiseFtp = require('promise-ftp');
2const fs = require('fs');
3
4async function uploadFiles(files) {
5    const ftp = new PromiseFtp();
6    await ftp.connect({ host: 'ftp.example.com', user: 'user', password: 'pass' });
7
8    for (const file of files) {
9        const stream = fs.createReadStream(file);
10        await ftp.put(stream, `/remote/${file}`);
11        console.log(`Uploaded ${file}`);
12    }
13
14    await ftp.end();
15    console.log('All files uploaded');
16}
17
18const files = ['file1.txt', 'file2.txt', 'file3.txt'];
19uploadFiles(files).catch(console.error);

The for...of loop with await processes files one at a time, just like the callback approach, but the code reads like synchronous logic. Error handling uses a standard try/catch or .catch() on the returned promise.

Adding Error Handling

Wrap the loop body in a try/catch to handle individual file failures without aborting the entire batch.

javascript

1async function uploadFiles(files) {
2    const ftp = new PromiseFtp();
3    await ftp.connect({ host: 'ftp.example.com', user: 'user', password: 'pass' });
4
5    const results = { success: [], failed: [] };
6
7    for (const file of files) {
8        try {
9            const stream = fs.createReadStream(file);
10            await ftp.put(stream, `/remote/${file}`);
11            results.success.push(file);
12            console.log(`Uploaded ${file}`);
13        } catch (err) {
14            results.failed.push({ file, error: err.message });
15            console.error(`Failed to upload ${file}:`, err.message);
16        }
17    }
18
19    await ftp.end();
20    return results;
21}

Approach 3: Controlled Concurrency

Sequential processing is safe but slow. If you can open multiple FTP connections, you can upload files in parallel with a concurrency limit. The p-limit package provides a simple way to cap the number of concurrent operations.

javascript

1const PromiseFtp = require('promise-ftp');
2const fs = require('fs');
3const pLimit = require('p-limit');
4
5const limit = pLimit(3); // Max 3 concurrent uploads
6
7async function uploadFile(file) {
8    const ftp = new PromiseFtp();
9    await ftp.connect({ host: 'ftp.example.com', user: 'user', password: 'pass' });
10    const stream = fs.createReadStream(file);
11    await ftp.put(stream, `/remote/${file}`);
12    await ftp.end();
13    console.log(`Uploaded ${file}`);
14}
15
16async function uploadAll(files) {
17    const promises = files.map(file =>
18        limit(() => uploadFile(file))
19    );
20    await Promise.all(promises);
21    console.log('All files uploaded');
22}
23
24const files = ['file1.txt', 'file2.txt', 'file3.txt', 'file4.txt', 'file5.txt'];
25uploadAll(files).catch(console.error);

Each call to uploadFile opens its own FTP connection, uploads the file, and closes the connection. The pLimit(3) wrapper ensures that at most 3 uploads run concurrently. This balances speed against server load.

Reusing a Connection Pool

For high-volume scenarios, opening a new connection per file is expensive. You can create a pool of FTP connections and distribute work across them.

javascript

1async function createPool(size, config) {
2    const pool = [];
3    for (let i = 0; i < size; i++) {
4        const ftp = new PromiseFtp();
5        await ftp.connect(config);
6        pool.push(ftp);
7    }
8    return pool;
9}
10
11async function uploadWithPool(files, poolSize) {
12    const config = { host: 'ftp.example.com', user: 'user', password: 'pass' };
13    const pool = await createPool(poolSize, config);
14    const limit = pLimit(poolSize);
15    let poolIndex = 0;
16
17    const promises = files.map(file =>
18        limit(async () => {
19            const ftp = pool[poolIndex++ % poolSize];
20            const stream = fs.createReadStream(file);
21            await ftp.put(stream, `/remote/${file}`);
22            console.log(`Uploaded ${file}`);
23        })
24    );
25
26    await Promise.all(promises);
27    for (const ftp of pool) await ftp.end();
28}

Common Pitfalls

Using forEach with async callbacks. Array.forEach does not await async callbacks. All iterations fire immediately, defeating the purpose of sequential processing. Use a for...of loop with await instead.
Not closing connections. Forgetting to call ftp.end() leaves connections open. Over time, this exhausts the FTP server's connection limit and can cause your application to hang.
Ignoring server concurrency limits. Most FTP servers limit the number of simultaneous connections per user (often 3 to 10). Exceeding this limit results in connection refused errors. Always match your concurrency cap to the server's limits.
Swallowing errors in Promise.all. If one promise in Promise.all rejects, the entire batch rejects. Use Promise.allSettled if you want to process all files and collect errors afterward rather than failing fast.
Not handling stream errors. fs.createReadStream can emit errors (file not found, permission denied). Attach an error handler to the stream or wrap the operation in try/catch to avoid unhandled exceptions.

Summary

When making multiple FTP calls inside a loop in Node.js, choose between sequential processing (safe for single connections), async/await with for...of (clean and readable), or controlled concurrency with p-limit (fast with parallel connections). Avoid forEach with async functions, always close connections, and respect server concurrency limits. For large batches, consider a connection pool to reduce the overhead of repeated connect/disconnect cycles.