Introduction

Recursive folder reading means walking a directory tree and visiting files inside nested subdirectories. In modern Python, you usually do not need to write recursion yourself because the standard library already provides good traversal tools such as pathlib.Path.rglob and os.walk.

Use Path.rglob for simple recursive file discovery

If your real task is "find every file matching this pattern below a root directory", pathlib is usually the cleanest choice.

1from pathlib import Path
2
3root = Path("logs")
4
5for path in root.rglob("*.log"):
6    print(path)

This yields Path objects, which makes later work such as reading text, checking suffixes, or accessing metadata straightforward.

A more practical example:

1from pathlib import Path
2
3root = Path("content")
4
5for path in root.rglob("*.md"):
6    with path.open("r", encoding="utf-8") as file:
7        first_line = file.readline().strip()
8    print(path, "->", first_line)

For many scripts, that is all you need.

Use os.walk when you need more control

os.walk is a better fit when the traversal itself matters. It gives you the current directory, the subdirectory list, and the filenames for each step of the walk.

1import os
2
3for dirpath, dirnames, filenames in os.walk("project"):
4    print("Directory:", dirpath)
5    print("Subdirectories:", dirnames)
6    print("Files:", filenames)

Its biggest advantage is that you can prune the recursion by editing dirnames in place:

1import os
2
3for dirpath, dirnames, filenames in os.walk("project"):
4    dirnames[:] = [d for d in dirnames if d not in {".git", ".venv", "__pycache__"}]
5
6    for filename in filenames:
7        if filename.endswith(".py"):
8            print(os.path.join(dirpath, filename))

That is the key feature to remember. Modifying dirnames tells os.walk which subdirectories it should not descend into.

Open files carefully during traversal

Walking the directory tree is only part of the job. You often also need to read or parse each file. The safe pattern is to keep error handling close to the file operation so one unreadable file does not stop the whole scan.

1from pathlib import Path
2
3root = Path("documents")
4
5for path in root.rglob("*.txt"):
6    try:
7        text = path.read_text(encoding="utf-8")
8    except OSError as exc:
9        print(f"Skipping {path}: {exc}")
10        continue
11
12    print(path.name, len(text))

This is a good pattern for search tools, simple indexers, and migration scripts.

It is also a good default for large trees because you process one file at a time instead of building a giant in-memory list of file contents before doing any real work.

Manual recursion is still possible

You can still write the walk recursively yourself, but that is usually only worth it when the traversal logic is custom enough that rglob and os.walk do not fit.

1from pathlib import Path
2
3def visit(directory: Path) -> None:
4    for child in directory.iterdir():
5        if child.is_dir():
6            visit(child)
7        else:
8            print(child)
9
10visit(Path("data"))

This works, but it is more code and easier to get wrong than the standard traversal helpers.

Common Pitfalls

• Reading every file eagerly into memory instead of processing one file at a time.
• Forgetting to skip irrelevant directories such as .git, node_modules, or virtual environments.
• Following symbolic links carelessly and ending up with repeated or confusing traversal paths.
• Assuming every discovered file is UTF-8 text and then failing on binary or differently encoded files.
• Writing manual recursion when rglob or os.walk already solves the problem more clearly.

Summary

• Use Path.rglob when you mainly need recursive file matching.
• Use os.walk when you need directory-level control and pruning.
• Handle file read errors near the file operation so one bad file does not stop the walk.
• Manual recursion is possible, but it is rarely the best default.
• Good traversal code is as much about exclusions and error handling as it is about recursion.