Practice Now
Python
string manipulation
text processing
quoted substrings
Python tips

Split a string by spaces -- preserving quoted substrings -- in Python

ML System Design practice on Codemia

Design recommenders, ranking systems and training pipelines the way ML interviews actually ask for them, with worked solutions.

Practice ML system design
Natural Language ProcessingPython

Introduction

Splitting a string by spaces while keeping quoted substrings intact is a common need when parsing command-line arguments, CSV-like data, or search queries. Python's str.split() breaks on every space, destroying quoted phrases. The standard library provides shlex.split() which handles this correctly, and csv.reader works for CSV-style quoting. Understanding when to use each saves you from writing fragile regex solutions.

The Problem

python
1text = 'hello world "foo bar" baz'
2
3# Naive split breaks the quoted phrase
4text.split()
5# ['hello', 'world', '"foo', 'bar"', 'baz']
6
7# We want:
8# ['hello', 'world', 'foo bar', 'baz']

"foo bar" should stay together as one token, but split() does not understand quotes.

The shlex module parses strings using shell-like syntax, handling single quotes, double quotes, and escape characters:

python
1import shlex
2
3text = 'hello world "foo bar" baz'
4result = shlex.split(text)
5print(result)
6# ['hello', 'world', 'foo bar', 'baz']

shlex.split() removes the quotes and keeps the content together. It also handles:

python
1# Single quotes
2shlex.split("name 'John Doe' age")
3# ['name', 'John Doe', 'age']
4
5# Escaped quotes inside strings
6shlex.split('say "it\\'s fine" now')
7# ['say', "it's fine", 'now']
8
9# Mixed quotes
10shlex.split("""he said "it's a 'test'" done""")
11# ['he', 'said', "it's a 'test'", 'done']
12
13# Backslash escaping
14shlex.split(r'path "C:\\Users\\me" ok')
15# ['path', 'C:\\Users\\me', 'ok']

Method 2: csv.reader

For CSV-style quoting (where only double quotes are special):

python
1import csv
2import io
3
4text = 'hello world "foo bar" baz'
5reader = csv.reader(io.StringIO(text), delimiter=' ')
6result = next(reader)
7print(result)
8# ['hello', 'world', 'foo bar', 'baz']

csv.reader handles doubled quotes for escaping ("" inside a quoted field becomes a single "):

python
1text = 'name "John ""JD"" Doe" age'
2result = next(csv.reader(io.StringIO(text), delimiter=' '))
3print(result)
4# ['name', 'John "JD" Doe', 'age']

Note: csv.reader may produce empty strings for consecutive spaces, unlike shlex.split().

Method 3: Regular Expressions

When you need custom quoting rules:

python
1import re
2
3text = 'hello world "foo bar" baz'
4
5# Match quoted strings or non-space sequences
6tokens = re.findall(r'"([^"]*)"|\S+', text)
7print(tokens)
8# ['hello', 'world', 'foo bar', 'baz']

This regex has two alternatives:

  • "([^"]*)" — matches a double-quoted string and captures the content (without quotes)
  • \S+ — matches any non-space sequence
python
1# Support both single and double quotes
2tokens = re.findall(r"""(?:"([^"]*)")|(?:'([^']*)')|(\S+)""", text)
3# Returns tuples — flatten and filter empty strings
4result = [next(g for g in groups if g) for groups in tokens]

Method 4: Custom Parser

For full control, write a state-machine parser:

python
1def split_preserving_quotes(text):
2    tokens = []
3    current = []
4    in_quotes = False
5    quote_char = None
6
7    for char in text:
8        if in_quotes:
9            if char == quote_char:
10                in_quotes = False
11            else:
12                current.append(char)
13        elif char in ('"', "'"):
14            in_quotes = True
15            quote_char = char
16        elif char == ' ':
17            if current:
18                tokens.append(''.join(current))
19                current = []
20        else:
21            current.append(char)
22
23    if current:
24        tokens.append(''.join(current))
25
26    return tokens
27
28result = split_preserving_quotes('hello "foo bar" baz')
29print(result)  # ['hello', 'foo bar', 'baz']

Comparison

MethodHandles single quotesHandles escapesEmpty tokensSpeed
shlex.split()YesYes (backslash)NoModerate
csv.readerNoYes (doubled "")YesFast
RegexConfigurableManualNoFast
Custom parserConfigurableConfigurableConfigurableVaries

Practical Examples

Parsing Search Queries

python
1import shlex
2
3query = 'python "machine learning" -java site:github.com'
4tokens = shlex.split(query)
5print(tokens)
6# ['python', 'machine learning', '-java', 'site:github.com']

Parsing Command-Line Strings

python
1import shlex
2
3cmd = 'docker run -e "MY_VAR=hello world" --name "my container" nginx'
4parts = shlex.split(cmd)
5print(parts)
6# ['docker', 'run', '-e', 'MY_VAR=hello world', '--name', 'my container', 'nginx']

Keeping Quotes in Output

If you need to preserve the quote characters:

python
1import re
2
3text = 'hello "foo bar" baz'
4tokens = re.findall(r'"[^"]*"|\S+', text)
5print(tokens)
6# ['hello', '"foo bar"', 'baz']  — quotes preserved

Common Pitfalls

  • shlex.split() on Windows paths: Backslashes are treated as escape characters. Use shlex.split(text, posix=False) on Windows or raw strings to preserve backslashes.
  • Unclosed quotes: shlex.split('hello "foo bar') raises ValueError: No closing quotation. Wrap in try/except or validate input first.
  • Empty strings between spaces: csv.reader produces empty strings for 'a b' (two spaces) → ['a', '', 'b']. Filter with [t for t in tokens if t].
  • Unicode quotes: shlex.split() does not recognize Unicode smart quotes (\u201c, \u201d). Replace them with ASCII quotes first: text.replace('\u201c', '"').replace('\u201d', '"').
  • Performance on large input: shlex.split() parses character-by-character and is slower than str.split(). For millions of lines, use csv.reader or compiled regex.

Summary

  • Use shlex.split() for shell-like parsing with quote and escape support
  • Use csv.reader with delimiter=' ' for CSV-style quoting
  • Use re.findall(r'"([^"]*)"|\S+', text) for simple double-quote preservation
  • shlex.split() removes quotes from output; use regex if you need to keep them
  • Handle edge cases: unclosed quotes, Windows backslashes, consecutive spaces
Related reading
Free course
Beginner
7 lessons
2 hours
Tackling System Design Interview Problems

A short course that equips you with the skills to approach system design interviews methodically.

Start the free course
Track what you have practised

A free account saves your progress, solutions and study plan across every problem on Codemia.

ML System Design practice on Codemia

Design recommenders, ranking systems and training pipelines the way ML interviews actually ask for them, with worked solutions.

Practice ML system design
Codemia
© 2026 Codemia
Resources
Legal
Social
All Rights Reserved.