Introduction

Splitting a 100 GB file into 1 GB pieces is a streaming job, not a memory job. The right tool reads the input sequentially and writes fixed-size chunks to disk without trying to load the whole file into RAM. The safest workflow also includes reassembly and checksum verification, because the split is only useful if you can prove the file comes back intact.

Use split on Unix-Like Systems

On Linux and macOS, the standard tool is split.

split -b 1G -d -a 3 --additional-suffix=.part bigfile.bin chunk_

This command means:

• '-b 1G: write chunks of 1 GB each'
• '-d: use numeric suffixes instead of letters'
• '-a 3: reserve three digits such as 000, 001, and 002'
• '--additional-suffix=.part: append a predictable suffix'

The output files look like:

chunk_000.part
chunk_001.part
chunk_002.part

That is usually the cleanest answer because split already handles large files efficiently.

Reassemble and Verify the Output

Any serious file-splitting workflow should document the reverse command.

cat chunk_*.part > bigfile_restored.bin

Then verify the restored file against the original.

sha256sum bigfile.bin
sha256sum bigfile_restored.bin

If the checksums match, the round trip was lossless. This matters especially when the chunks are copied across machines, uploaded separately, or archived for later use.

PowerShell Option on Windows

If you are on Windows and want a built-in approach, PowerShell can stream the file in chunks.

1$inputPath = "C:\data\bigfile.bin"
2$chunkSize = 1GB
3$buffer = New-Object byte[] $chunkSize
4$stream = [System.IO.File]::OpenRead($inputPath)
5$index = 0
6
7try {
8    while (($bytesRead = $stream.Read($buffer, 0, $buffer.Length)) -gt 0) {
9        $outputPath = "C:\data\chunk_{0:D3}.part" -f $index
10        $outStream = [System.IO.File]::Create($outputPath)
11        try {
12            $outStream.Write($buffer, 0, $bytesRead)
13        }
14        finally {
15            $outStream.Dispose()
16        }
17        $index++
18    }
19}
20finally {
21    $stream.Dispose()
22}

This is useful when split is not available. It also handles the final partial chunk correctly if the file size is not an exact multiple of the chunk size.

Python as a Portable Fallback

If you need a scriptable solution across environments, Python is a good fallback.

1from pathlib import Path
2
3source = Path("bigfile.bin")
4chunk_size = 1024 * 1024 * 1024
5
6with source.open("rb") as infile:
7    index = 0
8    while True:
9        data = infile.read(chunk_size)
10        if not data:
11            break
12
13        output = Path(f"chunk_{index:03d}.part")
14        with output.open("wb") as outfile:
15            outfile.write(data)
16        index += 1

This is easy to understand, but be aware that reading a full 1 GB chunk at once requires a large allocation. On a constrained system, use a nested loop with a smaller buffer size.

Fixed Size Versus Record Boundaries

If the file is plain text and you need each output file to end cleanly on line boundaries, fixed byte splitting may not be the right tool. But if the requirement is exactly “hundred 1 GB files,” then byte-oriented splitting is the correct model. Human readability and exact byte size are different goals.

Also remember to check free disk space first. Splitting a 100 GB file creates roughly another 100 GB of data while the original still exists.

Common Pitfalls

• Using a method that tries to load too much of the file into memory.
• Forgetting the reassembly step and later guessing the correct file order.
• Not verifying the reconstructed file with a checksum.
• Confusing fixed-size binary splitting with line-based text splitting.
• Running the split on a target filesystem that does not have room for both the original and all chunks.

Summary

• For Unix-like systems, split -b 1G is usually the best built-in solution.
• PowerShell and Python both work when you need a Windows or portable workflow.
• The correct implementation streams the file instead of loading it all at once.
• Reassembly and checksum verification should be part of the plan from the start.
• Use byte-based splitting when exact chunk size matters more than record boundaries.