Introduction
In pandas, substring filtering is usually done with the str accessor, most often str.contains. The important details are not the basic syntax but the edge cases: missing values, case sensitivity, whether the pattern should be treated as a regular expression, and whether you want contains, starts-with, or ends-with behavior.
Basic Substring Filtering
For “keep rows where a column contains some text,” use str.contains.
1import pandas as pd
2
3
4df = pd.DataFrame(
5 {
6 "name": ["Alice Blue", "Bob Redford", "Charlie Grey", None],
7 "city": ["Toronto", "Boston", "Calgary", "Berlin"],
8 }
9)
10
11filtered = df[df["name"].str.contains("Red", na=False)]
12print(filtered)
na=False matters. Without it, missing values can propagate NaN into the boolean mask and make filtering less predictable.
Case Sensitivity and Literal Matching
By default, str.contains is case-sensitive and treats the pattern as a regular expression.
That means these two questions are different:
For a literal search, disable regex mode:
filtered = df[df["name"].str.contains("red", case=False, regex=False, na=False)]
print(filtered)
This is often the safest default when you are searching for ordinary user-provided text.
startswith and endswith
If the rule is positional rather than “contains anywhere,” use the dedicated helpers.
1starts = df[df["name"].str.startswith("Alice", na=False)]
2ends = df[df["name"].str.endswith("Grey", na=False)]
3
4print(starts)
5print(ends)
These methods are clearer than writing equivalent regular expressions by hand.
Filter Multiple Columns
Sometimes the substring can appear in more than one column. In that case, build a combined boolean mask.
mask = (
df["name"].str.contains("o", case=False, na=False)
| df["city"].str.contains("o", case=False, na=False) ) print(df[mask]) ``` Using masks explicitly keeps the logic readable when multiple text conditions are involved. It is also a good habit to normalize the column before matching when data quality is uneven. Converting values to string, trimming spaces, or standardizing case once up front is often cleaner than embedding those fixes into every filter expression. ## Regex Filtering When You Actually Need It Regular expressions are still useful when the matching rule is more complex than a plain substring. For example, filter names that start with `A`, `B`, or `C`: ```python regex_filtered = df[df["name"].str.contains(r"^[ABC]", na=False)] print(regex_filtered) ``` Use regex deliberately. Do not leave it on by default if the search text comes from users and should be treated literally. For larger data-cleaning pipelines, it is often worth storing the mask in a named variable instead of filtering inline. That makes debugging easier because you can inspect how many rows matched before applying the final selection. That same pattern also helps when multiple substring rules need to be combined with `&` and `|`, because each intermediate mask can be tested independently. ## Common Pitfalls The most common mistake is forgetting `na=False`. Missing values then create a mask with null entries, which often causes confusing behavior. Another mistake is forgetting that `str.contains` uses regex by default. A pattern such as `.` or `+` has special meaning unless `regex=False` is set. A third mistake is filtering non-string columns without converting them first. If the column contains mixed types, clean or cast the data before applying string methods. ## Summary - Use `str.contains` for general substring filtering in pandas. - Add `na=False` so missing values do not break the filter mask. - Use `case=False` for case-insensitive matching. - Set `regex=False` when the search term should be treated as a literal string. - Prefer `startswith` and `endswith` when the matching rule is positional rather than “contains anywhere.”
