Remove duplicate dictionaries from a list in Python

A dictionary cannot go into a set, because it can change. So set(records) fails with TypeError: unhashable type: 'dict'. Instead, keep a set of something that identifies each record, and skip any record you have already seen.

The program

records = [
    {"id": 1, "name": "Ada"},
    {"id": 2, "name": "Linus"},
    {"id": 1, "name": "Ada"},
    {"id": 3, "name": "Grace"},
    {"id": 2, "name": "Linus"},
]

seen = set()
unique = []
for record in records:
    if record["id"] not in seen:
        seen.add(record["id"])
        unique.append(record)

for record in unique:
    print(record)

{'id': 1, 'name': 'Ada'}
{'id': 2, 'name': 'Linus'}
{'id': 3, 'name': 'Grace'}

How it works

• seen holds the ids already added. Checking in seen on a set is fast, even for a long list.
• The first record with each id is kept and later ones are skipped, so the order of the list does not change.
• Use this version when one field (here id) decides whether two records are the same.

Comparing every field

If two records are duplicates only when all their fields match, use the whole record as the identity. tuple(sorted(record.items())) turns a dictionary into a tuple of pairs, which can go into a set:

records = [
    {"id": 1, "name": "Ada"},
    {"id": 1, "name": "Ada Lovelace"},
    {"name": "Ada", "id": 1},
]

seen = set()
unique = []
for record in records:
    key = tuple(sorted(record.items()))
    if key not in seen:
        seen.add(key)
        unique.append(record)

print(unique)

[{'id': 1, 'name': 'Ada'}, {'id': 1, 'name': 'Ada Lovelace'}]

The first and third records have the same keys and values in a different order, so they count as the same. sorted() puts the pairs in the same order first. This works only when every value is hashable (strings, numbers, tuples). A record with a list inside would raise a TypeError.