Introduction

When you need to collect files with multiple extensions in C#, a naive approach often causes duplicate paths, unnecessary disk scans, or brittle filtering logic. This usually happens when developers call Directory.GetFiles repeatedly (for example once for *.csv and once for *.txt) and then concatenate results without deduplication, normalization, or recursion controls. The issue becomes more visible in large directories, network shares, or pipelines where file order and uniqueness matter.

A robust implementation should answer a few questions clearly: Which extensions are allowed? Should search be recursive? Should extension matching be case-insensitive? Should symbolic links or hidden folders be skipped? And most importantly, how do we avoid duplicates when multiple patterns can match the same file path? This guide covers an efficient pattern that scales well and produces deterministic output.

Core Sections

1. Why duplicates happen and how to avoid them

Multiple calls to GetFiles can overlap, especially when extensions are provided in mixed forms such as .TXT, txt, and *.txt, or when your logic appends fallback wildcard searches. The fix is simple: normalize extensions first, enumerate once where possible, and use HashSet<string> with a case-insensitive comparer.

1using System;
2using System.Collections.Generic;
3using System.IO;
4using System.Linq;
5
6public static class FileFinder
7{
8    public static IReadOnlyList<string> GetFilesByExtensions(
9        string root,
10        IEnumerable<string> extensions,
11        bool recursive = true)
12    {
13        var normalized = new HashSet<string>(
14            extensions
15                .Where(e => !string.IsNullOrWhiteSpace(e))
16                .Select(e => e.Trim().TrimStart('*').ToLowerInvariant())
17                .Select(e => e.StartsWith(".") ? e : "." + e),
18            StringComparer.OrdinalIgnoreCase);
19
20        var option = recursive ? SearchOption.AllDirectories : SearchOption.TopDirectoryOnly;
21        var result = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
22
23        foreach (var path in Directory.EnumerateFiles(root, "*", option))
24        {
25            if (normalized.Contains(Path.GetExtension(path)))
26            {
27                result.Add(Path.GetFullPath(path));
28            }
29        }
30
31        return result.OrderBy(p => p, StringComparer.OrdinalIgnoreCase).ToList();
32    }
33}

This performs one enumeration pass and guarantees unique results, even if extensions are repeated in the input.

2. Prefer EnumerateFiles over GetFiles for large trees

GetFiles materializes all matches before returning. For large directories, this can spike memory and delay the first usable result. EnumerateFiles streams items lazily, which is better for responsiveness and memory footprint.

1foreach (var file in Directory.EnumerateFiles(rootPath, "*", SearchOption.AllDirectories))
2{
3    // Process immediately instead of waiting for full list allocation.
4    Console.WriteLine(file);
5}

If you need cancellation or time limits, you can wrap enumeration in your own loop and stop early based on application criteria.

3. Handle permission and path edge cases safely

Production directory scans often hit unauthorized folders, long paths, or transient IO issues. A resilient implementation traverses directories manually and catches exceptions per directory so one bad subtree does not fail the entire operation.

1public static IEnumerable<string> SafeEnumerate(string root)
2{
3    var dirs = new Stack<string>();
4    dirs.Push(root);
5
6    while (dirs.Count > 0)
7    {
8        var current = dirs.Pop();
9
10        IEnumerable<string> files = Array.Empty<string>();
11        IEnumerable<string> children = Array.Empty<string>();
12
13        try { files = Directory.EnumerateFiles(current); } catch { }
14        try { children = Directory.EnumerateDirectories(current); } catch { }
15
16        foreach (var f in files) yield return f;
17        foreach (var d in children) dirs.Push(d);
18    }
19}

You can combine this with extension filtering for robust behavior in mixed-permission environments.

Common Pitfalls

• Using GetFiles once per extension and concatenating lists without deduplication, which leads to repeated file paths.
• Comparing extensions with case-sensitive logic on systems where users expect .TXT and .txt to be treated the same.
• Accepting raw extension input (txt, .txt, *.txt) without normalization, causing missed or duplicated matches.
• Assuming recursive scans always succeed; permission-denied directories can throw and stop the whole operation if uncaught.
• Returning unsorted results and then depending on implicit order, which causes nondeterministic behavior across machines.

Summary

To retrieve files by multiple extensions without duplicates, normalize extension inputs, stream files with EnumerateFiles, and store matches in a case-insensitive HashSet. This approach is faster, more memory-efficient, and easier to reason about than repeated wildcard scans. For production use, add per-directory exception handling and deterministic ordering before returning results. With those safeguards in place, your file discovery pipeline becomes reliable even on large or imperfect directory trees.