Introduction

Removing duplicates from a list sounds easy until you care about order, data type, or performance. The best technique depends on whether the elements are hashable, whether the original order matters, and whether you want to keep the first duplicate or the last one.

In Python, the most common answers are a set, dict.fromkeys, or an explicit loop with a seen set. Each one is correct for a different version of the problem.

Use set When Order Does Not Matter

If the elements are hashable and the output order is irrelevant, the shortest answer is:

numbers = [1, 2, 2, 3, 4, 4, 5]
unique = list(set(numbers))
print(unique)

This is fast and simple, but it does not preserve the original order. If the list order has meaning, this approach is too destructive.

Preserve First-Seen Order With dict.fromkeys

In modern Python, dictionaries preserve insertion order, so dict.fromkeys is a clean order-preserving solution for hashable items:

numbers = [1, 2, 2, 3, 4, 4, 5]
unique = list(dict.fromkeys(numbers))
print(unique)

This keeps the first occurrence of each value and removes later duplicates. For many everyday Python tasks, this is the best default.

Use an Explicit Loop for More Control

If you want custom logic while deduplicating, use a seen set plus a loop:

1items = ["a", "b", "a", "c", "b"]
2seen = set()
3unique = []
4
5for item in items:
6    if item not in seen:
7        seen.add(item)
8        unique.append(item)
9
10print(unique)

This is especially useful when you want to log skipped values, normalize items, or apply additional rules during the pass.

Structured Data Often Needs an Explicit Key

If the list contains dictionaries or objects, "duplicate" usually means duplicate by some key, not by full object identity:

1records = [
2    {"id": 1, "name": "Ana"},
3    {"id": 2, "name": "Ben"},
4    {"id": 1, "name": "Ana Updated"},
5]
6
7seen = set()
8unique = []
9
10for record in records:
11    key = record["id"]
12    if key not in seen:
13        seen.add(key)
14        unique.append(record)
15
16print(unique)

This keeps the first record for each id. If your business rule should keep the last occurrence instead, use a dictionary keyed by id and let later entries overwrite earlier ones.

Unhashable Items Need a Different Strategy

Lists of lists cannot go directly into a set because lists are unhashable:

1rows = [[1, 2], [1, 2], [3, 4]]
2unique = []
3
4for row in rows:
5    if row not in unique:
6        unique.append(row)
7
8print(unique)

This is acceptable for small inputs. If possible, convert to a hashable representation first:

rows = [[1, 2], [1, 2], [3, 4]]
unique = [list(t) for t in dict.fromkeys(tuple(row) for row in rows)]
print(unique)

That keeps order while allowing hash-based deduplication.

Common Pitfalls

The biggest mistake is using set and then being surprised when the original order is lost. If order matters, use an order-preserving approach.

Another common issue is assuming all elements are hashable. Lists, dictionaries, and many custom objects cannot go straight into a set.

Developers also sometimes forget to define which duplicate should win. Keeping the first and keeping the last are different business rules.

Finally, be careful with large unhashable collections. A repeated item not in unique check can become slow, so explicit key extraction is often the better path.

Summary

• Use set when order does not matter and the items are hashable.
• Use dict.fromkeys when you want to preserve first-seen order.
• Use an explicit loop with a seen set when you need custom control.
• For structured records, deduplicate by an explicit key rather than by raw object identity.
• Decide whether the first or last duplicate should win before choosing the method.