Introduction

The efficient way to split a large text file in Python is to stream it incrementally instead of reading the whole file into memory. The exact splitting strategy depends on whether you want chunks by line count, by approximate size, or by some logical separator.

Split by Line Count

If the file should be split into parts with a fixed number of lines, iterate through the file and open a new output file whenever the line limit is reached.

1from pathlib import Path
2
3def split_by_lines(source: str, lines_per_file: int) -> None:
4    source_path = Path(source)
5    part_index = 0
6    line_count = 0
7    out_file = None
8
9    with source_path.open("r", encoding="utf-8") as infile:
10        for line in infile:
11            if line_count % lines_per_file == 0:
12                if out_file is not None:
13                    out_file.close()
14                part_index += 1
15                out_file = source_path.with_name(f"{source_path.stem}_part_{part_index}.txt").open(
16                    "w", encoding="utf-8"
17                )
18
19            out_file.write(line)
20            line_count += 1
21
22    if out_file is not None:
23        out_file.close()
24
25
26split_by_lines("huge.log", 100000)

This approach keeps memory usage low because it processes one line at a time.

Split by Approximate Byte Size

Sometimes the target is not a number of lines but a rough file size. You can still stream the input and rotate output files as you cross a threshold.

1from pathlib import Path
2
3def split_by_size(source: str, bytes_per_file: int) -> None:
4    source_path = Path(source)
5    part_index = 1
6    current_size = 0
7    out_file = source_path.with_name(f"{source_path.stem}_part_{part_index}.txt").open(
8        "w", encoding="utf-8"
9    )
10
11    with source_path.open("r", encoding="utf-8") as infile:
12        for line in infile:
13            encoded = line.encode("utf-8")
14            if current_size + len(encoded) > bytes_per_file and current_size > 0:
15                out_file.close()
16                part_index += 1
17                current_size = 0
18                out_file = source_path.with_name(f"{source_path.stem}_part_{part_index}.txt").open(
19                    "w", encoding="utf-8"
20                )
21
22            out_file.write(line)
23            current_size += len(encoded)
24
25    out_file.close()

This keeps lines intact while still creating files close to the requested size.

Why Streaming Beats read()

The inefficient pattern is:

text = open("huge.log").read()

That loads the entire file into memory. It may be fine for a small file, but it becomes a serious problem when the file is gigabytes in size.

Streaming line by line avoids that memory spike and works well even for very large text files.

Use the Operating System When Python Is Not Necessary

If you only need to split a file and do not need custom Python logic, the system split command is often faster:

split -l 100000 huge.log huge_part_

That is worth knowing, but when you do need Python-side processing, the streaming pattern above is still the right foundation.

It is also a useful benchmark. If your Python implementation is dramatically slower than the OS tool, the extra time is usually the price of your custom logic rather than a surprise bug in file I/O.

Common Pitfalls

The biggest pitfall is reading the entire file into memory before splitting it. That defeats the whole point of an efficient approach.

Another issue is forgetting to close the previous output file before opening the next one. That can leak file handles or leave partially flushed data.

If you split by bytes, remember that text encoding matters. Counting characters is not the same as counting bytes in UTF-8.

Summary

• Stream the input file incrementally instead of calling read() on the whole thing.
• Split by line count when chunk boundaries should align to complete lines.
• Split by byte size when you need output parts of roughly equal size.
• Close each output file before opening the next one.
• If no custom Python logic is needed, the OS split command can be even simpler and faster.