Introduction

When you need to split text into words using several delimiters at once, simple single-character splitting is not enough. The common solution is a regular expression that defines which characters count as boundaries, followed by a cleanup step for empty results.

Start with the Delimiters

Suppose the input may contain spaces, commas, periods, semicolons, or exclamation marks:

Hello, world! Split this; text.into words

A regex character class is a good fit because it can describe multiple delimiters in one pattern.

That keeps the parsing rule in one place instead of chaining several separate split calls and trying to clean up the intermediate results afterward.

Python Example

1import re
2
3text = "Hello, world! Split this; text.into words"
4words = re.split(r"[ ,;!.]+", text)
5
6print(words)

Output:

['Hello', 'world', 'Split', 'this', 'text', 'into', 'words']

The + matters because it treats runs of delimiters as one split boundary instead of creating empty strings between repeated punctuation or spaces.

JavaScript Example

The same idea works in JavaScript:

1const text = "one,two; three...four";
2const words = text.split(/[ ,;.]+/).filter(Boolean);
3
4console.log(words);

The filter(Boolean) is a simple cleanup step if the pattern or input might still produce empty entries.

Decide What Counts as a Word

This is where the problem becomes less trivial. Should these stay intact:

• 'can't'
• 'high-quality'
• 'e-mail'

If yes, then apostrophes or hyphens should not be treated as boundaries. Your delimiter regex should reflect that decision.

For example, this Python pattern keeps apostrophes and hyphens inside words:

1import re
2
3text = "It's a high-quality parser."
4words = re.findall(r"[A-Za-z]+(?:['-][A-Za-z]+)*", text)
5print(words)

This returns words rather than splitting on a delimiter list. For many text-processing tasks, matching the words directly is cleaner than splitting.

Split Versus Extract

There are really two strategies:

• split on delimiters
• extract valid word tokens

Splitting is simpler when delimiters are obvious and word rules are loose. Extraction is better when the definition of "word" matters, such as contractions, hyphenated terms, or Unicode-aware tokenization.

That is why NLP pipelines often tokenize by matching words, not just by splitting on punctuation.

In other words, the more linguistic your definition becomes, the less you should think in terms of delimiters alone.

Empty Strings and Repeated Delimiters

If you split on a single delimiter without handling repetition, input like "a,,b" may produce empty tokens. Sometimes that is correct, as in CSV-like data. For ordinary word splitting, it usually is not.

So for word parsing, repeated delimiters are often best handled by:

• using + in the regex
• filtering empty results afterward

Common Pitfalls

The biggest mistake is assuming all punctuation should be treated as a boundary. That breaks words such as "can't" and "high-quality" if those should remain whole.

Another mistake is splitting text when you really need token extraction. The more complex the word rules become, the less attractive plain splitting gets.

A third issue is forgetting about Unicode and locale behavior. If the text contains non-ASCII letters, ASCII-only regex patterns may silently drop valid words.

Summary

• Use a regex when multiple delimiters define word boundaries.
• Add + to collapse repeated delimiters into one boundary.
• Filter empty results when needed.
• Consider matching words directly instead of splitting when apostrophes or hyphens matter.
• Be explicit about what your application considers a word before choosing the pattern.