Remove Duplicates from a Python List While Keeping Order
Use a set for fast "have I seen this?" checks and a list to keep the first-seen order.
- Membership tests with a set
- Building the result list
- Why set(values) alone loses the order
Seen set plus ordered result
The problem
Duplicate entries break reports, emails and counts. You often need each value once while keeping the order in which values first appeared, and you want it to stay fast for long lists.
Think of it like this
Removing duplicates is like a doorman with a clicker and a guest list. For each person arriving, the doorman checks the list. New faces are written down and let in; repeat visitors are turned away. The room fills in arrival order.
seen = set()📋 The guest listif value not in seen:🔎 A new face?seen.add(value)✍️ Write the name downresult.append(value)🚪 Let them in, in arrival orderRemoving duplicates in order keeps the first occurrence of each value, using a set for fast checks and a list to preserve the order.
Key points
seenis a set, sovalue not in seenis a fast check even for long lists.seen.add(value)records the value the first time it appears.result.append(value)keeps values in the order they first appeared.- The result is
[3, 1, 2, 4]: the second 3 and the second 1 are skipped.
When should you use it?
- Cleaning lists of IDs, emails or tags while keeping their order.
- Removing repeated events from logs.
- Learning when to combine a set and a list.
When not to use it
- Order does not matter:
set(values)is enough. - Values that cannot go in a set, such as lists or dictionaries: convert them to tuples first or compare differently.
Example code
def unique_in_order(values):
seen = set()
result = []
for value in values:
if value not in seen:
seen.add(value)
result.append(value)
return result
print(unique_in_order([3, 1, 3, 2, 1, 4]))
Result of running the code
[3, 1, 2, 4]
Code explanation
Each value is checked once. New values enter both the set and output list, giving O(n) average time.
Change the values and run the program in the CodeUtility online Python compiler without installing Python locally.
Practice exercises
Modify the runnable example with the exercises below to build understanding beyond copying the result.
- Try empty, single-item, and duplicate-value inputs.
- Print state after every step to observe the algorithm.
- Benchmark 100, 1,000, and 10,000 items, then compare another approach.
Common mistakes
Checking membership in the result list
value in result scans the whole list each time, which is slow for long inputs.
if value not in result:
result.append(value)
if value not in seen:
seen.add(value)
result.append(value)
Finished this lesson?
Mark it as learned to track your progress.
Practice with a challenge
Use what this lesson covers to solve a real problem. Each challenge runs in your browser with instant tests and an AI review.
- Remove Duplicates Keep only the first occurrence of each value. Easy
- Move Zeroes Move every zero to the end while keeping the other order. Easy