Introduction

Finding duplicate phrases in text can mean several different problems, and the algorithm depends on what you call a phrase. You might want repeated fixed-length token windows, repeated variable-length substrings, or near-duplicate expressions after normalization. Most production systems start with exact fixed-length duplicates using hash maps, then add more advanced methods only when requirements demand it. For production systems, phrase-definition clarity matters as much as algorithmic complexity because output semantics drive downstream decisions.

Define Phrase Rules Before Choosing an Algorithm

Before coding, lock down these rules:

• phrase length fixed or variable
• token-based or character-based matching
• case-sensitive or case-insensitive
• punctuation and whitespace normalization

If these rules are vague, two implementations can both look correct and still produce different results.

Algorithm 1: Hash Map on Token n-grams

For fixed phrase length such as 2-word or 3-word duplicates, token n-gram counting is simple and fast.

1import re
2from collections import Counter
3
4
5def duplicate_phrases_ngram(text: str, n: int = 3):
6    tokens = re.findall(r"[a-z0-9']+", text.lower())
7    if len(tokens) < n:
8        return {}
9
10    ngrams = [tuple(tokens[i:i+n]) for i in range(len(tokens) - n + 1)]
11    counts = Counter(ngrams)
12
13    return {
14        " ".join(gram): count
15        for gram, count in counts.items()
16        if count > 1
17    }
18
19
20sample = "The quick brown fox jumps over the quick brown fox again"
21print(duplicate_phrases_ngram(sample, n=3))

Complexity is linear in token count for fixed n, which scales well for large corpora.

Algorithm 2: Rolling Hash for Fixed-Length Windows

Rolling hash is useful for high-throughput character-window duplicate detection.

1def duplicate_windows(text: str, k: int):
2    if k <= 0 or len(text) < k:
3        return {}
4
5    base = 257
6    mod = 1_000_000_007
7    high = pow(base, k - 1, mod)
8
9    seen = {}
10    duplicates = {}
11    h = 0
12
13    for i, ch in enumerate(text):
14        h = (h * base + ord(ch)) % mod
15        if i >= k:
16            h = (h - ord(text[i-k]) * high) % mod
17        if i >= k - 1:
18            start = i - k + 1
19            window = text[start:start+k]
20            if h in seen and any(text[p:p+k] == window for p in seen[h]):
21                duplicates[window] = duplicates.get(window, 1) + 1
22            seen.setdefault(h, []).append(start)
23
24    return duplicates

Hash collisions are possible, so direct substring verification is required for exact correctness.

Algorithm 3: Suffix Structures for Variable-Length Repeats

If phrase length is unknown and you need longest repeated substrings, suffix arrays or suffix automata are strong options. They are more complex but powerful for deep text analysis.

High-level suffix-array workflow:

1. normalize text
2. build suffix array
3. compute longest-common-prefix values between adjacent suffixes
4. extract repeated substrings above minimum length

This is often the right choice for advanced plagiarism checks or long-document mining.

Normalization Pipeline Has Huge Impact

Algorithm quality depends heavily on input normalization. Inconsistent normalization often creates false negatives.

Typical steps:

• lowercase conversion
• punctuation cleanup
• unicode normalization
• whitespace compression

1import re
2import unicodedata
3
4
5def normalize_text(s: str) -> str:
6    s = unicodedata.normalize("NFKC", s)
7    s = s.lower()
8    s = re.sub(r"\s+", " ", s)
9    return s.strip()

Keep normalization rules versioned so downstream reports stay reproducible.

Include Positions, Not Just Counts

Duplicate detection output is far more useful when it includes locations.

1from collections import defaultdict
2
3
4def duplicate_positions(tokens, n):
5    pos = defaultdict(list)
6    for i in range(len(tokens) - n + 1):
7        pos[tuple(tokens[i:i+n])].append(i)
8    return {k: v for k, v in pos.items() if len(v) > 1}

Position data helps UI highlighting, reviewer workflows, and debugging.

Choosing the Right Method in Practice

Use this rule of thumb:

• exact fixed token length duplicates: n-gram hash map
• exact fixed character windows at high volume: rolling hash plus verification
• variable-length repeated substrings: suffix array or suffix automaton

Most business products can ship successfully with n-gram counting first, then upgrade if needed.

Common Pitfalls

A common mistake is skipping requirement definition and arguing about algorithm choice too early.

Another mistake is treating normalization as optional. Without it, duplicate quality is unstable across input sources.

A third mistake is using rolling hash results directly without collision checks.

Teams also return only aggregate counts and forget phrase positions, which limits usefulness for review tools.

Summary

• Duplicate phrase detection starts with clear phrase and normalization rules.
• Hash-map n-gram counting is usually the best first algorithm for fixed-length phrases.
• Rolling hash helps high-throughput window scanning when collision verification is added.
• Suffix-based structures are better for variable-length repeated substring analysis.
• Position-aware output makes duplicate findings more actionable.