Introduction

Directory iteration is a core task in automation, data engineering, and build tooling. Python offers multiple APIs, each with tradeoffs in readability, performance, and metadata access. Choosing the right approach helps you write faster and more maintainable file traversal code.

Basic Traversal with os.walk

os.walk is a reliable recursive iterator that yields directory paths, subdirectory names, and file names.

1import os
2
3root = "./sample_data"
4
5for current_dir, subdirs, files in os.walk(root):
6    print("DIR:", current_dir)
7    for name in files:
8        full_path = os.path.join(current_dir, name)
9        print("  FILE:", full_path)

This is often the best default for recursive processing.

Modern Path Handling with pathlib

pathlib gives an object-oriented API and cleaner path operations.

1from pathlib import Path
2
3root = Path("./sample_data")
4
5for path in root.rglob("*.csv"):
6    print(path, path.stat().st_size)

Path objects improve readability and reduce path-join mistakes.

Fast Non-Recursive Scanning with os.scandir

When you only need one directory level and file metadata, os.scandir is fast and efficient.

1import os
2
3for entry in os.scandir("./sample_data"):
4    if entry.is_file():
5        print("file", entry.name, entry.stat().st_size)
6    elif entry.is_dir():
7        print("dir", entry.name)

It avoids some extra system calls compared with basic list and stat loops.

Practical Filtered Traversal

Real workflows usually need include and exclude rules. Keep those rules explicit.

1from pathlib import Path
2
3root = Path("./sample_data")
4allowed_ext = {".py", ".md"}
5ignored_dirs = {".git", "venv", "node_modules"}
6
7for path in root.rglob("*"):
8    if path.is_dir() and path.name in ignored_dirs:
9        continue
10    if path.is_file() and path.suffix in allowed_ext:
11        print(path)

A clear filter strategy prevents hidden bugs and slow scans.

Error Handling for Robust Scripts

File traversal can fail on permission errors, deleted files, or broken links. Handle exceptions where they occur and log useful context.

1from pathlib import Path
2
3for path in Path("./sample_data").rglob("*"):
4    try:
5        if path.is_file():
6            _ = path.stat().st_mtime
7    except OSError as exc:
8        print(f"Skipping {path}: {exc}")

This keeps long-running scans resilient instead of failing on one problematic path.

Build a Reusable Traversal Utility

For repeatable tooling, wrap traversal logic in a utility function that accepts filters and callback behavior.

1from pathlib import Path
2from typing import Callable
3
4
5def walk_files(root: Path, include: Callable[[Path], bool]):
6    for path in root.rglob("*"):
7        if path.is_file() and include(path):
8            yield path
9
10
11for file_path in walk_files(Path("."), lambda p: p.suffix in {".py", ".md"}):
12    print(file_path)

This keeps file discovery logic consistent across scripts.

Parallel Processing Pattern

Traversal is often only the first step. If per-file work is expensive, process files with a bounded worker pool.

1from concurrent.futures import ThreadPoolExecutor
2from pathlib import Path
3
4
5def read_size(path: Path) -> tuple[str, int]:
6    return str(path), path.stat().st_size
7
8paths = [p for p in Path("./sample_data").rglob("*.log") if p.is_file()]
9
10with ThreadPoolExecutor(max_workers=8) as pool:
11    for name, size in pool.map(read_size, paths):
12        print(name, size)

Bounded concurrency improves throughput while avoiding uncontrolled system load.

Make Traversal Deterministic

When script output is compared in CI, deterministic ordering matters. Sort discovered paths before processing.

paths = sorted(Path("./sample_data").rglob("*.csv"))
for path in paths:
    print(path)

Stable ordering makes tests and diffs easier to review.

Consistent traversal utilities also simplify testing and cross-platform maintenance for automation scripts.

Common Pitfalls

• Building paths with manual string concatenation.
• Scanning huge trees without directory filters.
• Assuming every traversed path is readable.
• Ignoring symbolic links and creating unexpected traversal behavior.

Summary

• Use os.walk for straightforward recursive directory traversal.
• Use pathlib for cleaner path handling, globbing, and readable path composition.
• Use os.scandir for fast local directory scans.
• Add explicit filters, pruning, and robust error handling for production scripts.