PythonMastery
Python programs · Lists · beginner

Remove duplicates from a list in Python

Remove repeated items from a list in Python while keeping the original order, with dict.fromkeys(), with set(), and with a loop.

The shortest way to remove duplicates and keep the original order is list(dict.fromkeys(items)).

The program

python
numbers = [3, 1, 3, 2, 1, 5, 2]

unique = list(dict.fromkeys(numbers))
print("Original:", numbers)
print("Unique:  ", unique)
output
Original: [3, 1, 3, 2, 1, 5, 2]
Unique:   [3, 1, 2, 5]

How it works

Another way: set()

set(numbers) also removes duplicates, but a set has no fixed order. Use it when the order does not matter, and sort the result if you want a predictable order:

python
numbers = [3, 1, 3, 2, 1, 5, 2]
print(sorted(set(numbers)))
output
[1, 2, 3, 5]

Ignoring upper and lower case

With a loop you can decide what counts as "the same". Here "Ada" and "ada" count as one name, and the first spelling is kept:

python
names = ["Ada", "Linus", "ada", "Grace", "LINUS"]

seen = set()
unique = []
for name in names:
    key = name.lower()
    if key not in seen:
        seen.add(key)
        unique.append(name)

print(unique)
output
['Ada', 'Linus', 'Grace']

Checking key not in seen is fast because seen is a set. Checking in on a list gets slower as the list grows.

Learn the idea: Lists & Sequences, Sets: Unordered, Unique, Lightning Fast, Dictionaries

A longer explanation: How do I remove duplicates from a list in Python?