Introduction

Removing duplicates while keeping the first occurrence order is a common data-cleaning task in Python. The best solution depends on your Python version and on whether the list items are hashable, but for most modern code the simplest answer is dict.fromkeys.

Use dict.fromkeys in Modern Python

Since modern Python dictionaries preserve insertion order, you can build a dictionary from the list values and turn the keys back into a list.

items = ["a", "b", "a", "c", "b", "d"]
unique_items = list(dict.fromkeys(items))
print(unique_items)

This keeps the first occurrence of each value and removes later duplicates. It is concise, idiomatic, and usually the cleanest choice in Python 3.7 and later.

Use a Set and a Loop When You Need More Control

The explicit loop is slightly longer, but it lets you combine de-duplication with other logic.

1def remove_duplicates(items):
2    seen = set()
3    result = []
4
5    for item in items:
6        if item not in seen:
7            seen.add(item)
8            result.append(item)
9
10    return result
11
12print(remove_duplicates(["a", "b", "a", "c", "b", "d"]))

This is a good pattern when you want to validate, transform, or log values while preserving order.

Handle Older Python Versions with OrderedDict

If you are maintaining older Python where regular dictionaries do not guarantee insertion order, use collections.OrderedDict.

1from collections import OrderedDict
2
3items = ["a", "b", "a", "c", "b", "d"]
4unique_items = list(OrderedDict.fromkeys(items))
5print(unique_items)

This is mostly a compatibility technique now, but it is still worth knowing for legacy codebases.

Unhashable Items Need a Different Approach

The fast approaches rely on hash-based membership checks, so they do not work directly for unhashable values such as lists or dictionaries. In that case, use a manual loop and compare values directly.

1def remove_unhashable_duplicates(items):
2    result = []
3    for item in items:
4        if item not in result:
5            result.append(item)
6    return result
7
8print(remove_unhashable_duplicates([[1], [2], [1], [3]]))

This preserves order, but it is slower because each membership check scans the result list.

Use a Key Function for Normalized Uniqueness

Sometimes uniqueness is based on a normalized version of the value, such as lowercase email addresses or trimmed strings. In that case, track the normalized key but keep the original value.

1def unique_by_key(items, key):
2    seen = set()
3    result = []
4
5    for item in items:
6        marker = key(item)
7        if marker not in seen:
8            seen.add(marker)
9            result.append(item)
10
11    return result
12
13emails = ["[email protected]", "[email protected]", "[email protected]"]
14print(unique_by_key(emails, key=str.lower))

That pattern is often more useful in real data pipelines than exact-value de-duplication.

Think About What Duplicate Means

Order-preserving de-duplication always keeps one representative value. Usually that is the first occurrence, but not always. If the business rule says the last value should win, you need a different implementation. The code should reflect that rule explicitly instead of assuming that any de-duplication behavior is acceptable.

Common Pitfalls

• Using a plain set directly and losing the original order of the list.
• Assuming dict.fromkeys works for every item type, even when elements are unhashable.
• Forgetting that de-duplication usually keeps the first occurrence, not the last one.
• Comparing raw values when the real uniqueness rule should be case-insensitive or normalized.
• Optimizing for brevity and hiding important business rules about which value should be preserved.

Summary

• Use list(dict.fromkeys(items)) as the default modern Python solution.
• Use a loop with a seen set when you need extra processing or clarity.
• Use OrderedDict.fromkeys only for older Python compatibility.
• Fall back to list-based checking for unhashable items.
• Make the uniqueness rule explicit when normalized or business-specific matching is required.