Introduction

When you need to extract binary values from a stream with low memory use, the main rule is simple: do not load the full stream into memory. Instead, read fixed-size chunks, keep only a small bit buffer, and yield values as soon as enough bits are available.

Process the Stream Incrementally

Suppose the stream contains tightly packed fixed-width values, such as 5-bit or 12-bit numbers. You can decode them using:

• a byte chunk from the stream
• a small integer buffer holding unread bits
• a counter telling you how many valid bits are currently buffered

That lets memory usage stay essentially constant regardless of total stream size.

A Generator-Based Python Example

Here is a minimal decoder for fixed-width unsigned values:

1def extract_values(byte_iterable, bit_width):
2    buffer = 0
3    bits_in_buffer = 0
4    mask = (1 << bit_width) - 1
5
6    for chunk in byte_iterable:
7        for byte in chunk:
8            buffer = (buffer << 8) | byte
9            bits_in_buffer += 8
10
11            while bits_in_buffer >= bit_width:
12                shift = bits_in_buffer - bit_width
13                value = (buffer >> shift) & mask
14                yield value
15                bits_in_buffer -= bit_width
16                buffer &= (1 << bits_in_buffer) - 1

And here is how you might use it:

1with open("data.bin", "rb") as f:
2    while True:
3        chunk = f.read(4096)
4        if not chunk:
5            break
6        for value in extract_values([chunk], bit_width=12):
7            print(value)

This keeps memory low because the code never stores the whole file. It only keeps the current chunk and the small carry-over buffer.

A Cleaner Streaming Wrapper

To avoid reinitializing the decoder on each chunk, wrap the file reading loop in one generator.

1def stream_fixed_width_values(file_obj, bit_width, chunk_size=4096):
2    buffer = 0
3    bits_in_buffer = 0
4    mask = (1 << bit_width) - 1
5
6    while True:
7        chunk = file_obj.read(chunk_size)
8        if not chunk:
9            break
10
11        for byte in chunk:
12            buffer = (buffer << 8) | byte
13            bits_in_buffer += 8
14
15            while bits_in_buffer >= bit_width:
16                shift = bits_in_buffer - bit_width
17                yield (buffer >> shift) & mask
18                bits_in_buffer -= bit_width
19                buffer &= (1 << bits_in_buffer) - 1

This is the form you would usually keep in real code.

Why This Uses Little Memory

The memory footprint depends mostly on:

• the chunk size
• the small bit buffer
• whatever downstream consumer does with the yielded values

So if you choose a modest chunk size and process values immediately, memory use stays low even for very large inputs.

The important design principle is streaming, not a particular language trick.

Variable-Length Formats Are Harder

If the stream uses prefixes, delimiters, or variable-length binary encodings, you still use the same basic idea: incremental reads plus a small carry-over buffer. The parsing logic becomes more complex, but the low-memory principle does not change.

The one case where memory can grow unexpectedly is when your parser needs unbounded lookahead or when downstream logic collects all decoded values instead of processing them incrementally.

Common Pitfalls

The biggest pitfall is decoding the entire file into a giant bit string first. That may be simple to write, but it defeats the low-memory requirement immediately.

Another common mistake is picking an unnecessarily huge read buffer. A stream parser usually benefits more from steady incremental processing than from trying to maximize chunk size.

Developers also forget about leftover bits at the end of a chunk. If the parser does not preserve the partial value between reads, decoded results become misaligned and corrupt.

Summary

• Read the stream in chunks instead of loading it all at once.
• Keep a small bit buffer for carry-over across chunk boundaries.
• Yield decoded values as soon as enough bits are available.
• The same streaming pattern works for fixed-width and more complex binary formats.
• Low memory usage depends on incremental parsing all the way through the pipeline, not just on the file read call.