Speed up millions of regex replacements in Python 3

Python

Regex

Optimization

Performance

String Manipulation

Speed up millions of regex replacements in Python 3

Data Structures & Algorithms practice on Codemia

Step through 300 algorithm problems with animated visualisers that show the data structure changing as the code runs.

Practice algorithms

Complexity & Performance Python

Introduction

When performing millions of regex replacements in Python, the key optimizations are: compile regex patterns once with re.compile(), combine multiple patterns into a single alternation pattern, use str.replace() or str.translate() instead of regex when possible, and process text in bulk rather than line-by-line. For pure word-to-word substitutions, building a single compiled regex from all words and using a dictionary lookup in the replacement function is orders of magnitude faster than looping through individual patterns.

The Slow Approach

python

1import re
2
3# SLOW: Looping through patterns one at a time
4replacements = {"foo": "bar", "baz": "qux", "hello": "world"}
5text = "foo baz hello " * 100_000  # Large text
6
7for old, new in replacements.items():
8    text = re.sub(old, new, text)  # Recompiles pattern each time, scans entire text each time

This makes N full passes over the text, one per replacement. With 1000 patterns and a 10MB text, you scan 10GB of data.

Fix 1: Compile Patterns Once

python

1import re
2
3# Compile once, use many times
4pattern = re.compile(r"\bfoo\b")
5
6texts = ["foo bar foo", "no match here", "foo again"] * 1_000_000
7
8# Fast: compiled pattern avoids re-parsing the regex each time
9results = [pattern.sub("bar", t) for t in texts]

re.compile() parses the regex once and returns a reusable pattern object. Without it, re.sub() parses the regex string on every call.

Fix 2: Single Combined Pattern with Dictionary Lookup

The fastest approach for many word replacements — combine all patterns into one regex:

python

1import re
2
3replacements = {
4    "foo": "bar",
5    "baz": "qux",
6    "hello": "world",
7    "old_func": "new_func",
8    # ... potentially thousands of entries
9}
10
11# Build a single pattern matching any key
12# Sort by length (longest first) to avoid partial matches
13pattern = re.compile(
14    "|".join(re.escape(k) for k in sorted(replacements, key=len, reverse=True))
15)
16
17def replace_match(match):
18    return replacements[match.group(0)]
19
20text = "foo baz hello old_func " * 100_000
21result = pattern.sub(replace_match, text)  # Single pass over the text

This scans the text exactly once, regardless of how many replacement patterns exist.

With Word Boundaries

python

1# Match whole words only
2pattern = re.compile(
3    r"\b(" +
4    "|".join(re.escape(k) for k in sorted(replacements, key=len, reverse=True)) +
5    r")\b"
6)
7
8result = pattern.sub(lambda m: replacements[m.group(0)], text)

Fix 3: Use str.replace() for Literal Strings

If patterns are literal strings (no regex metacharacters), str.replace() is faster:

python

1# str.replace is ~3-5x faster than re.sub for literal strings
2text = "foo bar baz " * 1_000_000
3
4# Fast for a small number of replacements
5text = text.replace("foo", "bar")
6text = text.replace("baz", "qux")

Fix 4: Use str.translate() for Character-Level Replacements

python

1# For single-character replacements, str.translate is the fastest
2table = str.maketrans({
3    "a": "1",
4    "b": "2",
5    "c": "3",
6})
7
8text = "abcabc" * 1_000_000
9result = text.translate(table)  # Single pass, C-level speed

Fix 5: Process in Chunks with multiprocessing

python

1import re
2from multiprocessing import Pool
3
4replacements = {"foo": "bar", "baz": "qux"}
5pattern = re.compile("|".join(re.escape(k) for k in replacements))
6
7def process_chunk(text):
8    return pattern.sub(lambda m: replacements[m.group(0)], text)
9
10# Split work across CPU cores
11lines = ["foo baz hello"] * 10_000_000
12
13with Pool() as pool:
14    chunk_size = len(lines) // pool._processes
15    chunks = ["\n".join(lines[i:i+chunk_size])
16              for i in range(0, len(lines), chunk_size)]
17    results = pool.map(process_chunk, chunks)
18
19final = "\n".join(results)

Benchmark Comparison

python

1import re
2import time
3
4replacements = {f"word{i}": f"repl{i}" for i in range(1000)}
5text = " ".join(replacements.keys()) * 100
6
7# Method 1: Loop with re.sub (slow)
8start = time.time()
9result1 = text
10for old, new in replacements.items():
11    result1 = re.sub(re.escape(old), new, result1)
12print(f"Loop re.sub: {time.time() - start:.3f}s")
13
14# Method 2: Combined pattern (fast)
15pattern = re.compile("|".join(re.escape(k) for k in replacements))
16start = time.time()
17result2 = pattern.sub(lambda m: replacements[m.group(0)], text)
18print(f"Combined pattern: {time.time() - start:.3f}s")
19
20# Typical results for 1000 patterns, 100K words:
21# Loop re.sub: ~12.5s
22# Combined pattern: ~0.03s

Common Pitfalls

Not compiling regex patterns when reusing them: Each re.sub(pattern_str, ...) call parses the regex string into an internal automaton. In a loop processing millions of strings, this parsing overhead accumulates significantly. Always use re.compile() outside the loop.
Building a combined pattern without re.escape(): If replacement keys contain regex metacharacters (., *, +, (), the combined pattern breaks or matches unintended text. Always escape keys with re.escape() when building the alternation.
Not sorting patterns by length in the alternation: Regex alternation a|ab matches a before trying ab, causing partial matches. Sort patterns longest-first so ab is tried before a.
Using regex when str.replace() suffices: For literal string replacements (no wildcards, no word boundaries), str.replace() is 3-5x faster than re.sub() because it avoids regex engine overhead entirely.
Processing text line-by-line when bulk processing is possible: Reading a file line-by-line and applying regex to each line incurs function call overhead per line. Read the entire file into a single string (if it fits in memory) and apply the regex once.

Summary

Combine all patterns into a single re.compile("|".join(...)) for a single-pass replacement
Use a dictionary lookup function as the replacement argument to re.sub()
Compile patterns with re.compile() and reuse the compiled object
Use str.replace() for literal strings and str.translate() for character-level replacements
For massive text, split into chunks and use multiprocessing.Pool for parallel processing

DSA Fundamentals

Master algorithmic patterns and data structures through hands-on LeetCode-style problems - from arrays and hashing to dynamic programming and advanced graphs.

View the course

Track what you have practised

A free account saves your progress, solutions and study plan across every problem on Codemia.

Data Structures & Algorithms practice on Codemia

Step through 300 algorithm problems with animated visualisers that show the data structure changing as the code runs.

Practice algorithms

Speed up millions of regex replacements in Python 3

Data Structures & Algorithms practice on Codemia

Introduction

The Slow Approach

Fix 1: Compile Patterns Once

Fix 2: Single Combined Pattern with Dictionary Lookup

With Word Boundaries

Fix 3: Use str.replace() for Literal Strings

Fix 4: Use str.translate() for Character-Level Replacements

Fix 5: Process in Chunks with multiprocessing

Benchmark Comparison

Common Pitfalls

Summary

Related reading

DSA Fundamentals

Track what you have practised

Data Structures & Algorithms practice on Codemia