Introduction

Finding common strings in two very large files is mostly a scaling problem, not a string-comparison problem. The right solution depends on whether one file fits in memory, whether the files are already sorted, and whether exact results matter more than speed or disk usage.

Start with the Simplest Useful Case

If one file is small enough to fit comfortably into memory, load it into a hash set and stream the other file line by line.

1def common_lines(file_a, file_b):
2    with open(file_a, 'r', encoding='utf-8') as f:
3        lookup = {line.rstrip('\n') for line in f}
4
5    with open(file_b, 'r', encoding='utf-8') as f:
6        for line in f:
7            value = line.rstrip('\n')
8            if value in lookup:
9                yield value
10
11
12for match in common_lines('a.txt', 'b.txt'):
13    print(match)

This runs in roughly linear time and gives fast membership tests, but memory usage is proportional to the number of unique strings in the in-memory file.

Use Sorting and Merge for Truly Large Inputs

If neither file fits in memory, sorting plus a merge pass is often the best exact strategy. The idea is the same as the merge step in merge sort.

1. Sort both files externally if needed.
2. Open both sorted files.
3. Compare the current string from each file.
4. Advance the pointer for the smaller string.
5. Emit the string when both lines match.

Here is a streaming merge for already sorted files:

1def common_sorted_lines(file_a, file_b):
2    with open(file_a, 'r', encoding='utf-8') as fa, open(file_b, 'r', encoding='utf-8') as fb:
3        a = fa.readline()
4        b = fb.readline()
5
6        while a and b:
7            a_value = a.rstrip('\n')
8            b_value = b.rstrip('\n')
9
10            if a_value == b_value:
11                yield a_value
12                a = fa.readline()
13                b = fb.readline()
14            elif a_value < b_value:
15                a = fa.readline()
16            else:
17                b = fb.readline()

The merge step uses little memory because it keeps only the current lines in memory.

How to Sort Large Files

When files are bigger than RAM, external sorting is the standard answer. On Unix-like systems, the sort command already implements an external sort strategy efficiently:

sort file1.txt -o file1.sorted.txt
sort file2.txt -o file2.sorted.txt
comm -12 file1.sorted.txt file2.sorted.txt

comm -12 prints lines present in both sorted files. This is often the easiest production-grade answer if shell tools are allowed.

If you need the logic inside an application, chunked sorting plus temporary files can reproduce the same pattern, but it is much more code.

Think About Duplicates

Decide what “common strings” means for your use case.

• Unique intersection: each shared string appears once.
• Multiset intersection: shared strings appear as many times as they overlap.

The hash-set approach naturally leans toward unique intersection unless you track counts. The sorted merge approach can support either interpretation, depending on how you advance pointers and emit results.

Approximate Approaches Exist, But Use Them Intentionally

For extremely large data where exact intersection is too expensive, probabilistic structures such as Bloom filters can help screen likely matches. That can reduce I/O or memory use, but it introduces false positives unless followed by exact verification.

For most engineering tasks, exact set-based or sort-based methods are easier to explain and safer to trust.

Common Pitfalls

The most common mistake is reading both huge files fully into memory without checking actual size first. That can turn a simple job into a process that swaps or crashes.

Another issue is ignoring whether the files are already sorted. If they are sorted, you should exploit that and use a merge-based scan instead of building a large hash table.

People also forget about line normalization. Trailing spaces, newline differences, or case sensitivity can make logically identical strings appear different.

Finally, define duplicate behavior up front. “Common strings” sounds simple until someone asks whether repeated values should appear once or many times.

Summary

• Use a hash set when one file fits in memory and you need a simple exact solution.
• Use external sort plus a merge pass when both files are too large for memory.
• 'sort and comm are often the most practical exact tools on Unix-like systems.'
• Normalize lines consistently before comparing them.
• Decide early whether you want unique intersection or duplicate-sensitive overlap.