Nodejs
Stream Parsing
Object Conversion
Programming
JavaScript

Parse stream to object in Nodejs

Master System Design with Codemia

Enhance your system design skills with over 120 practice problems, detailed solutions, and hands-on exercises.

Introduction

Parsing a stream into an object in Node.js means turning chunked input into a structured value such as JSON, CSV records, or line-based objects. The exact technique depends on the stream format. The main rule is simple: do not assume one chunk equals one logical object, because streams can split data arbitrarily.

For Small JSON Payloads, Buffer Then Parse

If the stream represents one JSON document of manageable size, the simplest solution is to collect the chunks and parse once at the end.

javascript
1import { Readable } from 'node:stream';
2
3async function streamToJson(stream) {
4  const chunks = [];
5  for await (const chunk of stream) {
6    chunks.push(Buffer.isBuffer(chunk) ? chunk : Buffer.from(chunk));
7  }
8  const text = Buffer.concat(chunks).toString('utf8');
9  return JSON.parse(text);
10}
11
12const stream = Readable.from(['{"name":"Ava","age":30}']);
13console.log(await streamToJson(stream));

This is fine for API responses or files that are small enough to hold in memory. It is not the right approach for unbounded or very large streams.

Chunk Boundaries Are Not Object Boundaries

A common mistake is trying to parse each chunk individually.

javascript
stream.on('data', chunk => {
  const obj = JSON.parse(chunk); // unsafe assumption
});

This fails because one JSON object can arrive across several chunks, or several objects can arrive in a single chunk. Streams are transport units, not semantic-message units.

That is why you need either full buffering, line framing, or a streaming parser depending on the protocol.

Use Line-Based Parsing for NDJSON and Logs

If the stream contains newline-delimited JSON or line-based records, split by lines instead of buffering everything.

javascript
1import readline from 'node:readline';
2import { Readable } from 'node:stream';
3
4async function parseNdjson(stream) {
5  const rl = readline.createInterface({ input: stream, crlfDelay: Infinity });
6  for await (const line of rl) {
7    if (line.trim()) {
8      console.log(JSON.parse(line));
9    }
10  }
11}
12
13const input = Readable.from([
14  '{"id":1}\n',
15  '{"id":2}\n'
16]);
17
18await parseNdjson(input);

This is a strong fit for logs, message exports, and streaming APIs that explicitly use one record per line.

Use Transform Streams for Structured Pipelines

When you want a reusable parser in a stream pipeline, a Transform in object mode is often the right abstraction.

javascript
1import { Transform, Readable } from 'node:stream';
2
3class JsonLineParser extends Transform {
4  constructor() {
5    super({ readableObjectMode: true });
6    this.buffer = '';
7  }
8
9  _transform(chunk, encoding, callback) {
10    this.buffer += chunk.toString('utf8');
11    const lines = this.buffer.split('\n');
12    this.buffer = lines.pop();
13
14    try {
15      for (const line of lines) {
16        if (line.trim()) {
17          this.push(JSON.parse(line));
18        }
19      }
20      callback();
21    } catch (err) {
22      callback(err);
23    }
24  }
25
26  _flush(callback) {
27    try {
28      if (this.buffer.trim()) {
29        this.push(JSON.parse(this.buffer));
30      }
31      callback();
32    } catch (err) {
33      callback(err);
34    }
35  }
36}
37
38Readable.from(['{"id":1}\n{"id":2}\n'])
39  .pipe(new JsonLineParser())
40  .on('data', obj => console.log(obj));

Now downstream consumers receive JavaScript objects instead of raw byte chunks.

Pick the Parser That Matches the Format

The correct implementation depends on the wire format:

  • one JSON document: buffer then JSON.parse
  • newline-delimited records: parse line by line
  • CSV: use a CSV parser
  • huge JSON arrays: use a streaming JSON parser

The mistake is trying to solve every format with the same chunk-based logic.

This is also why third-party parsers are often the right choice. If the data format is real CSV or large JSON, specialized parsers will handle edge cases better than a hand-written chunk loop.

Common Pitfalls

  • Assuming each stream chunk already contains one complete logical object.
  • Parsing large streams by buffering everything when a streaming parser is needed.
  • Forgetting to handle the trailing partial record when splitting chunks by delimiters.
  • Ignoring stream errors while focusing only on the parsing code.
  • Using a generic JSON approach when the actual data format is NDJSON, CSV, or another framed protocol.

Summary

  • Streams deliver chunks, not guaranteed object boundaries.
  • For small single JSON payloads, buffer all chunks and parse once.
  • For line-based formats, parse by line rather than by chunk.
  • Use object-mode transform streams when you want a reusable parsing pipeline.
  • Match the parsing strategy to the actual stream format instead of forcing one generic pattern onto all inputs.

Course illustration
Course illustration

All Rights Reserved.