regex
text processing
awk
GNU awk
delimiter handling

Split on regex more than a character, maybe variable width and keep the separator like GNU awk

Master System Design with Codemia

Enhance your system design skills with over 120 practice problems, detailed solutions, and hands-on exercises.

Introduction

Splitting text on a regular expression is easy in many languages. Splitting on a variable-width regex and keeping the matched separators is where things become more interesting. In GNU Awk, the useful tool is the split function with the optional seps array, which stores the separators separately instead of discarding them.

Use split with a Regex Field Separator

GNU Awk lets split take a regular expression as the separator argument.

awk
1BEGIN {
2    text = "alpha -- beta ::: gamma"
3    n = split(text, parts, /[[:space:]]*(--|:::[[:space:]]*)[[:space:]]*/)
4
5    for (i = 1; i <= n; i++)
6        printf "part[%d] = [%s]\n", i, parts[i]
7}

This gives you the pieces between matches, but by itself it does not preserve the separators. If you need the delimiters too, use the fourth argument.

Keep Separators with the seps Array

GNU Awk extends split with a separator array.

awk
1BEGIN {
2    text = "alpha -- beta ::: gamma"
3    n = split(text, parts, /[[:space:]]*(--|:::)[[:space:]]*/, seps)
4
5    for (i = 1; i <= n; i++) {
6        printf "part[%d] = [%s]\n", i, parts[i]
7        printf "sep[%d]  = [%s]\n", i, seps[i]
8    }
9}

The parts array contains the text pieces, and the seps array contains the matched separators that occurred between them. This is the GNU Awk feature that gets closest to "split and keep delimiters" behavior.

Why This Is Better Than Re-Matching Later

You could try to split first and then run another regex pass to recover delimiters, but that makes the code more fragile. Once you already know the split points, it is cleaner to keep both streams at the same time.

That matters most when:

  • delimiters have meaning of their own
  • delimiters have variable width
  • you need to reconstruct the original structure later

GNU Awk's seps support makes this much easier than juggling two separate scans.

Reconstruct or Analyze Structure Explicitly

Once you have both arrays, you can interleave them to reconstruct or analyze the text layout.

awk
1BEGIN {
2    text = "A==B---C"
3    n = split(text, parts, /(==|---)/, seps)
4
5    for (i = 1; i <= n; i++) {
6        printf "token: [%s]\n", parts[i]
7        if (i in seps)
8            printf "delimiter after token: [%s]\n", seps[i]
9    }
10}

This is often more useful than a plain split result because you preserve the structural boundary markers.

Consider patsplit for the Inverse Problem

Sometimes the real task is not "split on separators" but "extract the tokens that match a pattern." In GNU Awk, patsplit is useful for that inverse problem.

Use split when you know the separators. Use patsplit when you know the token pattern you want to keep.

That distinction helps keep the code readable instead of trying to force one function into every text-processing job.

Common Pitfalls

A common mistake is assuming standard awk behavior everywhere. The seps array is a GNU Awk feature, so portability to other awk implementations may be limited.

Another is writing an overly broad separator regex that consumes surrounding characters you actually wanted to preserve.

Developers also sometimes reach for capturing groups expecting split output to include them automatically, as in some other regex environments. In GNU Awk, the cleaner answer is the seps array.

Summary

  • GNU Awk can split on multi-character and variable-width regex separators.
  • Use split(text, parts, regex, seps) when you need to keep the matched separators.
  • The parts array stores the text segments and seps stores the delimiters between them.
  • This is cleaner than performing a second pass just to recover delimiters.
  • Remember that the separator-array behavior is a GNU Awk extension, not universal awk behavior.

Course illustration
Course illustration

All Rights Reserved.