Introduction

Removing duplicate dictionaries from a Python list sounds simple until you define what "duplicate" means. Sometimes the whole dictionary must match, and sometimes only one key such as id matters. The right solution depends on that rule and on whether you want to preserve the first item, keep the last one, or just return unique records in any order.

Define Equality Before Writing Code

With plain dictionaries, Python cannot put them directly into a set because dictionaries are mutable and unhashable. That means you need a transformation step or a lookup structure.

There are three common duplicate rules:

• Full dictionary equality.
• Equality by one key such as id.
• Equality by a subset of keys such as name and email.

Pick the rule first. Otherwise, you will write code that removes the wrong records.

Keep First Occurrence by Full Dictionary Value

If two dictionaries are duplicates only when every key and value matches, one practical approach is converting each item into a hashable representation.

1def dedupe_full_dicts(items):
2    seen = set()
3    result = []
4
5    for item in items:
6        marker = tuple(sorted(item.items()))
7        if marker not in seen:
8            seen.add(marker)
9            result.append(item)
10
11    return result
12
13
14rows = [
15    {"id": 1, "name": "Ava"},
16    {"id": 2, "name": "Noa"},
17    {"id": 1, "name": "Ava"},
18]
19
20print(dedupe_full_dicts(rows))

This preserves input order and keeps the first copy of each unique dictionary.

Deduplicate by Key

In many real systems, records are unique by one field such as id. In that case, deduplication is simpler and more explicit.

1def dedupe_by_key(items, key):
2    seen = set()
3    result = []
4
5    for item in items:
6        value = item[key]
7        if value not in seen:
8            seen.add(value)
9            result.append(item)
10
11    return result
12
13
14users = [
15    {"id": 10, "name": "Ava"},
16    {"id": 20, "name": "Noa"},
17    {"id": 10, "name": "Ava Updated"},
18]
19
20print(dedupe_by_key(users, "id"))

This keeps the first item for each id. That behavior should be documented, because some applications need the last item instead.

Keep Last Occurrence by Key

If later records should override earlier ones, use a dictionary keyed by the dedupe field.

1def dedupe_keep_last(items, key):
2    by_key = {}
3    for item in items:
4        by_key[item[key]] = item
5    return list(by_key.values())
6
7
8users = [
9    {"id": 10, "name": "Ava"},
10    {"id": 20, "name": "Noa"},
11    {"id": 10, "name": "Ava Updated"},
12]
13
14print(dedupe_keep_last(users, "id"))

Because dictionaries preserve insertion order, the final values reflect the last record seen for each key.

Deduplicate by Multiple Fields

Sometimes one field is not enough. Build a tuple from the fields that define uniqueness.

1def dedupe_by_fields(items, fields):
2    seen = set()
3    result = []
4
5    for item in items:
6        marker = tuple(item[field] for field in fields)
7        if marker not in seen:
8            seen.add(marker)
9            result.append(item)
10
11    return result
12
13
14contacts = [
15    {"name": "Ava", "email": "[email protected]", "team": "sales"},
16    {"name": "Ava", "email": "[email protected]", "team": "sales"},
17    {"name": "Ava", "email": "[email protected]", "team": "ops"},
18]
19
20print(dedupe_by_fields(contacts, ["name", "email"]))

This is usually clearer than trying to compare whole dictionaries when only a subset matters.

Performance and Memory Tradeoffs

All of these approaches are typically O(n) time because each item is processed once. They also use extra memory for the seen set or lookup dictionary.

That tradeoff is usually acceptable, but for very large datasets:

• Deduplicate during ingestion instead of after building a huge list.
• Stream data if possible.
• Choose the smallest marker needed for uniqueness.

A small tuple such as (id,) is cheaper than sorting every dictionary item for every row.

When Data Is Nested

If dictionaries contain nested lists or dictionaries, tuple(sorted(item.items())) may fail because nested structures are still unhashable. In those cases, either:

• Normalize records into a simpler key.
• Serialize consistently with json.dumps(..., sort_keys=True).
• Deduplicate by domain key instead of full nested structure.

For example:

1import json
2
3
4def dedupe_nested(items):
5    seen = set()
6    result = []
7
8    for item in items:
9        marker = json.dumps(item, sort_keys=True)
10        if marker not in seen:
11            seen.add(marker)
12            result.append(item)
13
14    return result

This works, but key-based dedupe is often faster and easier to reason about.

Common Pitfalls

• Removing duplicates without first defining what counts as duplicate.
• Using full-dictionary equality when only id should matter.
• Forgetting whether the code keeps first or last occurrence.
• Building markers from unordered fields inconsistently.
• Using expensive full serialization when a simple key would solve the problem.

Summary

• Deduplication strategy depends on how your application defines equality.
• Use a hashable marker and a seen set to preserve first occurrence.
• Use a dictionary keyed by id or another field to preserve last occurrence.
• Prefer key-based dedupe over full-dictionary comparison when domain rules allow it.
• Be careful with nested structures, because plain dictionaries are unhashable.