Remove Duplicates from a List in Python (Keep Order)
Keep the first copy of every value in the original order, using a set to remember what you have seen.
- Tracking seen values with a set
- Building an ordered result
- The dict.fromkeys shortcut
Add a value only the first time
The problem
Data often contains repeats: the same email signed up twice, the same tag added again. Converting to a set removes them, but sets lose the original order, which matters for things like the order people signed up.
Think of it like this
Removing duplicates while keeping order is like a guest book at a party. As each guest arrives, you check the list of names already written. If the name is new, you add it; if it is already there, you skip it. The book keeps arrival order.
seen = set()📖 The names already writtenvalue in seen🔎 Is this guest already in the book?seen.add(value)✍️ Write a new guest's nameunique📋 Guests in arrival order: [1, 2, 3, 4]Removing duplicates while preserving order keeps the first occurrence of each value and drops later repeats.
Key points
seenis a set that remembers which values have appeared.value in seenis a fast check for a value that was already kept.seen.add(value)returns None, sovalue in seen or seen.add(value)is False for new values and adds them at the same time.uniqueis[1, 2, 3, 4], in the order the values first appeared.
When should you use it?
- Cleaning lists of emails, IDs or tags while keeping their order.
- Practicing sets for fast membership checks.
- Removing repeated events or log lines.
When not to use it
- Order does not matter:
set(values)is shorter. - Readability matters more than one line:
list(dict.fromkeys(values))or a normal loop is clearer than theseen.addtrick.
Example code
values = [1, 2, 1, 3, 2, 4]
seen = set()
# Add each value only on its first occurrence.
unique = [value for value in values if not (value in seen or seen.add(value))]
print(unique)
Result of running the code
[1, 2, 3, 4]
Code explanation
seen starts as an empty set and records each value the first time it appears. For every value, value in seen or seen.add(value) is True only when the value was seen before; for a new value, seen.add returns None, so the condition is False, the value is kept, and it is recorded at the same moment. The result keeps the first occurrence of each value in the original order.
Change the values and run the program in the CodeUtility online Python compiler without installing Python locally.
Practice exercises
Modify the runnable example with the exercises below to build understanding beyond copying the result.
- Change the input and predict the output before running.
- Handle empty or invalid input.
- Wrap the logic in a function and add more test cases.
Common mistakes
Using a set when order matters
Sets do not keep insertion order.
unique = list(set(values)) # order is not guaranteed
unique = list(dict.fromkeys(values))
Finished this lesson?
Mark it as learned to track your progress.
Practice with a challenge
Use what this lesson covers to solve a real problem. Each challenge runs in your browser with instant tests and an AI review.
All challenges