Introduction

Searching for a string in text files can mean very different things depending on the scale of the task. For a quick command-line search across a directory, a shell tool is usually best. For application logic inside a program, reading the files in Python gives you more control over matching rules and output format.

Use Command-Line Tools for Fast Ad Hoc Searches

On Unix-like systems, grep is the classic tool:

grep -n "ERROR" app.log

This prints matching lines and includes line numbers because of -n. To search recursively through a directory:

grep -R -n "Connection refused" /var/log/myapp

If performance matters on large codebases, ripgrep is often faster and has cleaner defaults:

rg -n "TODO" .

Command-line tools are ideal when you need answers immediately and do not need to embed the logic in your application.

Search Files in Python

If you want to search programmatically, iterate through files line by line instead of loading everything into memory at once.

1from pathlib import Path
2
3
4def search_in_files(folder, needle):
5    for path in Path(folder).rglob("*.txt"):
6        with path.open("r", encoding="utf-8", errors="ignore") as f:
7            for line_number, line in enumerate(f, start=1):
8                if needle in line:
9                    print(f"{path}:{line_number}: {line.strip()}")
10
11
12search_in_files("logs", "timeout")

This approach scales well enough for many scripts and gives you full control over filters, output formatting, and error handling.

Exact Matches and Regular Expressions

Sometimes you want substring matching, and sometimes you want pattern matching. Those are different tasks.

Substring matching is simple:

if "error" in line:
    ...

Regular expressions are better when the search criteria is structured:

1import re
2from pathlib import Path
3
4
5pattern = re.compile(r"ERROR\\s+\\d{3}")
6
7for path in Path("logs").rglob("*.log"):
8    with path.open("r", encoding="utf-8", errors="ignore") as f:
9        for number, line in enumerate(f, start=1):
10            if pattern.search(line):
11                print(f"{path}:{number}: {line.strip()}")

Use regex only when you need it. Exact string matching is easier to read and often faster.

Decide How Broad the Search Should Be

A good search tool answers four questions clearly:

• which files should be searched
• whether the match is case-sensitive
• whether you want exact text or a regex pattern
• what result format you need

For example, case-insensitive matching with grep looks like this:

grep -R -n -i "warning" ./reports

In Python, you can normalize case or use regex flags:

if needle.lower() in line.lower():
    ...

Spending a minute defining the match behavior usually matters more than the code itself.

Handling Large or Messy Files

Real text files are not always tidy. Logs may contain mixed encodings, long lines, or partial binary content. That is why the Python examples above open files with errors="ignore". It lets the scan continue instead of dying on a single problematic character sequence.

For very large directories, prefer a tool built for search, such as rg, unless you need custom application behavior. Reimplementing a fast recursive searcher in Python is rarely worth it for simple investigative work.

Common Pitfalls

• Loading entire large files into memory when a line-by-line scan would be simpler and safer.
• Using regular expressions for plain substring searches, which adds complexity without benefit.
• Forgetting about case sensitivity and then missing legitimate matches.
• Searching all files blindly, including binary or generated files that add noise.
• Ignoring encoding errors in real-world text data and letting one bad file stop the whole search.

Summary

• Use grep or rg for quick command-line searches across files.
• Use Python when the search logic belongs inside a script or application.
• Prefer simple substring checks unless a regex is genuinely needed.
• Search line by line to handle large files efficiently.
• Define file scope, case sensitivity, and result format before writing the search logic.