Introduction

The Unix diff command uses the Myers algorithm (an optimized LCS-based approach) and is hard to beat for general line-by-line text comparison. However, it can be outperformed in specific scenarios: git diff uses the patience and histogram algorithms for cleaner output on source code, delta and difftastic provide syntax-aware diffs, and custom algorithms can beat diff on structured data like JSON, XML, or ASTs. The key insight is that diff treats input as lines of text — any tool with knowledge of the input structure can produce more meaningful results.

How diff Works: The Myers Algorithm

diff finds the longest common subsequence (LCS) between two files, then reports lines not in the LCS as additions or deletions:

1# Basic diff
2diff file1.txt file2.txt
3
4# Unified format (most common)
5diff -u file1.txt file2.txt
6
7# Side-by-side
8diff -y file1.txt file2.txt

The Myers algorithm runs in O(N*D) time, where N is the total number of lines and D is the size of the edit script (number of changes). For small diffs on large files, this is nearly linear.

Where diff Falls Short

1. Moved Blocks

1# file1.txt          # file2.txt
2Line A               Line C
3Line B               Line D
4Line C               Line A
5Line D               Line B

diff reports all four lines as changed. It cannot detect that lines A and B were moved, not modified.

2. Structured Data

1# diff treats JSON as plain text — reordering keys looks like a full rewrite
2echo '{"b":2,"a":1}' > a.json
3echo '{"a":1,"b":2}' > b.json
4diff a.json b.json  # Reports the entire line as changed

3. Syntax-Unaware Output

# diff doesn't know this is just a renamed variable
- int calculateTotal(int price, int quantity) {
+ int computeTotal(int cost, int count) {

diff shows the whole line changed. A syntax-aware tool would highlight only the changed identifiers.

Tools That Beat diff

git diff (Patience and Histogram Algorithms)

1# Default (Myers)
2git diff
3
4# Patience algorithm — better at matching function signatures
5git diff --patience
6
7# Histogram algorithm — faster on large files with many duplicate lines
8git diff --histogram
9
10# Minimal — smallest possible diff
11git diff --minimal

The patience algorithm matches unique lines first, producing cleaner diffs when code blocks are rearranged.

difftastic (Syntax-Aware Diff)

1# Install
2brew install difftastic
3
4# Compare files using AST parsing
5difft old.py new.py
6
7# Use as git difftool
8git config diff.external difft

difftastic parses source code into an abstract syntax tree and diffs the tree structure, so it can detect moved functions, renamed variables, and formatting-only changes.

delta (Enhanced Diff Viewer)

1# Install
2brew install git-delta
3
4# Use as git pager
5git config core.pager delta
6
7# Features: syntax highlighting, side-by-side, line numbers
8git diff  # Now renders through delta

jq for JSON Diff

1# Normalize JSON (sort keys) before diffing
2diff <(jq -S . a.json) <(jq -S . b.json)
3
4# Or use dedicated tools
5npm install -g json-diff
6json-diff a.json b.json

xmllint for XML Diff

# Normalize XML before diffing
diff <(xmllint --c14n a.xml) <(xmllint --c14n b.xml)

Writing a Custom Diff

For domain-specific data, a custom comparator can dramatically outperform diff:

1import json
2
3def json_diff(a, b, path=""):
4    """Structural diff for JSON objects."""
5    diffs = []
6    if type(a) != type(b):
7        diffs.append(f"{path}: type changed {type(a).__name__} -> {type(b).__name__}")
8    elif isinstance(a, dict):
9        for key in set(a) | set(b):
10            if key not in a:
11                diffs.append(f"{path}.{key}: added")
12            elif key not in b:
13                diffs.append(f"{path}.{key}: removed")
14            else:
15                diffs.extend(json_diff(a[key], b[key], f"{path}.{key}"))
16    elif isinstance(a, list):
17        for i in range(max(len(a), len(b))):
18            if i >= len(a):
19                diffs.append(f"{path}[{i}]: added")
20            elif i >= len(b):
21                diffs.append(f"{path}[{i}]: removed")
22            else:
23                diffs.extend(json_diff(a[i], b[i], f"{path}[{i}]"))
24    elif a != b:
25        diffs.append(f"{path}: {a!r} -> {b!r}")
26    return diffs
27
28a = {"name": "Alice", "scores": [90, 85], "active": True}
29b = {"name": "Alice", "scores": [90, 95], "active": False}
30for d in json_diff(a, b):
31    print(d)
32# .scores[1]: 85 -> 95
33# .active: True -> False

Algorithm Comparison

Common Pitfalls

• Assuming diff output is the smallest possible edit: The Myers algorithm produces a minimal edit script by default, but diff implementations may use heuristics that sacrifice minimality for speed on large files. Use diff --minimal or git diff --minimal to force the smallest output.
• Using line-based diff on minified or single-line files: diff compares line by line. A minified JavaScript file is one line, so any change shows the entire file as changed. Reformat with a tool like prettier or jq before diffing.
• Diffing binary files with text diff: diff produces meaningless output on binary files. Use specialized tools like hexdump for binary comparison, exiftool for image metadata, or pdftotext for PDF content.
• Ignoring whitespace-only changes in code reviews: Whitespace changes inflate diffs. Use diff -w (ignore all whitespace) or diff -b (ignore trailing whitespace) to focus on meaningful changes. In git, use git diff -w.
• Not normalizing structured data before diffing: Diffing JSON with reordered keys or XML with different attribute ordering shows false changes. Normalize (sort keys, canonicalize) before running diff to see only semantic differences.

Summary

• diff uses the Myers algorithm and is optimal for general line-by-line text comparison
• git diff --patience and --histogram produce cleaner diffs for source code
• difftastic parses ASTs for syntax-aware diffs that ignore formatting changes
• For structured data (JSON, XML), normalize before diffing or use format-specific tools
• Custom diff algorithms beat diff when you have knowledge of the input structure