Unicode characters that look alike but have different code points are called homoglyphs. To compare them reliably, you need Unicode normalization (NFC, NFD, NFKC, NFKD) and, when that is not enough, confusable detection via the Unicode Consortium's security mechanisms. Simple equality checks will fail because two visually identical strings can have entirely different byte representations.

Why Visual Equality Does Not Mean Byte Equality

Consider the Latin letter "A" (U+0041) and the Cyrillic letter "A" (U+0410). They render identically in most fonts, but they are different code points. A standard == comparison will return false because the underlying bytes differ.

This is not a corner case. There are thousands of homoglyph pairs across Latin, Cyrillic, Greek, Armenian, and other scripts. The problem shows up in domain name spoofing, username impersonation, password bypass, and search deduplication.

1latin_a = "A"       # U+0041
2cyrillic_a = "А"  # U+0410
3
4print(latin_a == cyrillic_a)        # False
5print(latin_a.encode("utf-8"))      # b'A'
6print(cyrillic_a.encode("utf-8"))   # b'\xd0\x90'

These two characters are indistinguishable on screen but completely different to the interpreter.

Unicode Normalization Forms

Normalization handles a different class of look-alikes: characters that have multiple valid encodings. The accented character "e" (U+00E9) can also be represented as "e" (U+0065) followed by a combining acute accent (U+0301). Normalization collapses these equivalent representations into a single canonical form.

Unicode defines four normalization forms:

1import unicodedata
2
3# Two ways to encode "e" (e with acute accent)
4precomposed = "é"           # single code point
5decomposed = "é"          # base + combining accent
6
7print(precomposed == decomposed)  # False
8
9nfc_a = unicodedata.normalize("NFC", precomposed)
10nfc_b = unicodedata.normalize("NFC", decomposed)
11print(nfc_a == nfc_b)             # True

NFC is the most common choice for storage and comparison. NFKC is more aggressive and useful when you want "fi" (the ligature) to match "fi" (two separate letters) or when comparing user input against a canonical database.

1import unicodedata
2
3ligature = "fi"  # fi ligature
4normal = "fi"
5
6print(ligature == normal)  # False
7
8nfkc_a = unicodedata.normalize("NFKC", ligature)
9nfkc_b = unicodedata.normalize("NFKC", normal)
10print(nfkc_a == nfkc_b)   # True

Detecting Homoglyphs Across Scripts

Normalization solves the problem of equivalent encodings within the same script. It does not solve cross-script homoglyphs like Latin "o" vs. Cyrillic "o". For that, you need confusable detection.

The Unicode Consortium publishes a confusables mapping (part of Unicode Technical Standard #39) that maps visually similar characters to a common skeleton. Python's confusables library and ICU's SpoofChecker implement this.

1# Using the confusables library (pip install confusables)
2from confusables import is_confusable, normalize as confusable_normalize
3
4result = is_confusable("paypal", "pаypаl")  # second string uses Cyrillic 'а'
5print(bool(result))  # True
6
7# Skeleton-based comparison
8skeleton_a = confusable_normalize("paypal")
9skeleton_b = confusable_normalize("pаypаl")
10print(skeleton_a == skeleton_b)  # True

In Java, ICU4J provides SpoofChecker:

1import com.ibm.icu.text.SpoofChecker;
2
3SpoofChecker checker = new SpoofChecker.Builder()
4    .setChecks(SpoofChecker.CONFUSABLE)
5    .build();
6
7String latin = "paypal";
8String spoofed = "pаypаl";  // Cyrillic 'а'
9
10String skeletonA = checker.getSkeleton(latin);
11String skeletonB = checker.getSkeleton(spoofed);
12System.out.println(skeletonA.equals(skeletonB));  // true

Real-World Security Example: Domain Spoofing

A domain like apple.com can be spoofed by registering аррӏе.com using Cyrillic characters. Browsers have largely mitigated this with IDN homograph attack protection (displaying the punycode form when mixed scripts are detected), but application-layer code still needs to handle it.

1legit = "apple.com"
2spoof = "аррӏе.com"
3
4print(legit == spoof)        # False
5print(legit)                 # apple.com
6print(spoof)                 # apple.com (visually identical)
7
8from confusables import normalize as confusable_normalize
9print(confusable_normalize(legit) == confusable_normalize(spoof))  # True

Handling Homoglyphs in JavaScript and Go

The same principles apply in other languages. In JavaScript, normalization is built into the String prototype:

1// JavaScript normalization
2const precomposed = "é";       // e (single code point)
3const decomposed = "é";      // e + combining accent
4
5console.log(precomposed === decomposed);                    // false
6console.log(precomposed.normalize("NFC") === decomposed.normalize("NFC"));  // true

In Go, the golang.org/x/text/unicode/norm package provides the same normalization forms:

1package main
2
3import (
4    "fmt"
5    "golang.org/x/text/unicode/norm"
6)
7
8func main() {
9    a := norm.NFC.String("café")
10    b := norm.NFC.String("café")
11    fmt.Println(a == b) // true
12}

For confusable detection across scripts, ICU libraries exist for most languages.

Comparison of Approaches

Common Pitfalls

• Assuming normalization handles all look-alike problems. NFC and NFKC only collapse equivalent encodings within the Unicode standard. They do not map Latin "A" to Cyrillic "A".
• Using raw string comparison for usernames, emails, or URLs without any normalization. This allows both encoding-variant duplicates and homoglyph spoofing.
• Normalizing once at comparison time but storing the original unnormalized form. Normalize on input, store the normalized form, and compare normalized values.
• Forgetting that confusable detection can produce false positives. The skeleton mapping is intentionally broad, so use it as a flag rather than an automatic reject.
• Applying NFKC when you need to preserve formatting distinctions. NFKC collapses ligatures, superscripts, and other compatibility characters, which may not be appropriate for display-oriented text.

Summary

• Characters that look identical on screen can have completely different Unicode code points. Simple equality checks will miss these matches.
• Use NFC or NFKC normalization to collapse equivalent encodings before comparison. NFC is the standard choice; NFKC adds compatibility decomposition.
• For cross-script homoglyphs (Latin vs. Cyrillic, Greek, etc.), use confusable/skeleton detection from the Unicode Consortium's confusables data.
• Always normalize on input and store the normalized form. Comparing unnormalized strings is a recurring source of bugs and security vulnerabilities.
• Domain spoofing, username impersonation, and phishing attacks all exploit homoglyph confusion, making this a security-critical topic, not just a text-processing curiosity.