Introduction

Detecting duplicate text in PHP can mean several different things. Sometimes you want exact duplicates, sometimes near-duplicates after normalization, and sometimes rough similarity scoring. The right solution depends on whether your text must match character-for-character or only match closely enough to be considered effectively the same.

Exact Duplicate Detection

If the goal is exact duplicate detection, the fastest approach is usually to normalize the input a little and then compare hashes or string values directly.

For example:

1<?php
2
3$texts = [
4    "Hello world",
5    "Hello world",
6    "Goodbye world",
7];
8
9$seen = [];
10$duplicates = [];
11
12foreach ($texts as $text) {
13    if (isset($seen[$text])) {
14        $duplicates[] = $text;
15    } else {
16        $seen[$text] = true;
17    }
18}
19
20print_r($duplicates);

This works well when duplicate means literally identical text.

If the strings are large, using a hash key can be more memory-friendly in some workflows:

1<?php
2
3$text = "Example text";
4$key = md5($text);

The idea is the same: store what you have already seen and flag repeats.

Normalize Before Comparing

Many practical duplicate checks should ignore superficial differences such as extra spaces, uppercase versus lowercase, or line-ending variation.

A simple normalization function might be:

1<?php
2
3function normalize_text(string $text): string
4{
5    $text = trim($text);
6    $text = preg_replace('/\s+/', ' ', $text);
7    $text = mb_strtolower($text);
8    return $text;
9}
10
11$a = " Hello   World ";
12$b = "hello world";
13
14var_dump(normalize_text($a) === normalize_text($b));

This catches duplicates that are visually the same even when raw input differs slightly.

Normalization is often more important than the comparison function itself.

Detect Near-Duplicates With Similarity Scores

If you need approximate matching, PHP has a couple of built-in tools.

similar_text() gives a percentage-style similarity estimate:

1<?php
2
3$a = "The quick brown fox";
4$b = "The quick brown fox jumps";
5
6similar_text($a, $b, $percent);
7echo $percent, PHP_EOL;

levenshtein() measures how many single-character edits are required to turn one string into another:

1<?php
2
3$a = "kitten";
4$b = "sitting";
5
6echo levenshtein($a, $b), PHP_EOL;

These are useful for short strings such as titles, names, or user-entered labels. They are less ideal for large documents because character-level distance can become noisy and relatively expensive.

Token-Based Duplicate Detection

For paragraphs or longer text, token-based comparison is often more meaningful than raw character comparison. A simple approach is to compare the sets of normalized words.

1<?php
2
3function token_set(string $text): array
4{
5    $text = mb_strtolower($text);
6    $text = preg_replace('/[^\p{L}\p{N}\s]+/u', ' ', $text);
7    $parts = preg_split('/\s+/', trim($text));
8    $parts = array_filter($parts, fn($x) => $x !== '');
9    return array_values(array_unique($parts));
10}
11
12function jaccard_similarity(string $a, string $b): float
13{
14    $setA = token_set($a);
15    $setB = token_set($b);
16
17    $intersection = array_intersect($setA, $setB);
18    $union = array_unique(array_merge($setA, $setB));
19
20    if (count($union) === 0) {
21        return 1.0;
22    }
23
24    return count($intersection) / count($union);
25}
26
27$text1 = "PHP duplicate text detection is useful";
28$text2 = "Duplicate text detection in PHP is very useful";
29
30echo jaccard_similarity($text1, $text2), PHP_EOL;

This is still simple, but it often behaves better than character-based distance when word order changes slightly.

Choosing the Right Strategy

Use exact matching when:

• duplicates must be truly identical
• performance matters
• the input is already normalized

Use normalized exact matching when:

• whitespace and case should not matter
• user-entered text is inconsistent

Use approximate matching when:

• you want to flag near-duplicates
• the same text may appear with minor edits
• moderation or content quality is the goal

The definition of duplicate is a business rule first and a coding problem second.

Common Pitfalls

The biggest pitfall is comparing raw strings before deciding what should count as “the same.” If one user writes Hello World and another writes hello world, exact comparison will miss what may be a practical duplicate.

Another common issue is using similar_text() or levenshtein() on long documents and expecting strong semantic detection. These functions work on text form, not meaning.

People also often skip normalization for punctuation, case, and repeated whitespace. That creates noisy results even in simple duplicate-detection jobs.

Finally, hash-based exact matching is great for exact duplicates, but it does nothing for near-duplicates. Do not use a hash when the actual problem is similarity rather than identity.

Summary

• Exact duplicate detection in PHP is easiest with direct string or hash comparison.
• Normalize text first when whitespace, punctuation, or case should be ignored.
• Use similar_text() or levenshtein() for approximate matching on shorter text.
• Token-based similarity can work better for sentence or paragraph level comparisons.
• Decide what “duplicate” means for your application before choosing the algorithm.