Python Algorithms

Remove Duplicates from a Python List While Keeping Order

Use a set for fast "have I seen this?" checks and a list to keep the first-seen order.

  • Intermediate
  • 2 min read
  • 1 runnable example
What you'll learn
  • Membership tests with a set
  • Building the result list
  • Why set(values) alone loses the order
Input, then result
3
1
3
2
1
4
3
1
2
4
The highlighted repeats are skipped; order is kept.

Seen set plus ordered result

The problem

Duplicate entries break reports, emails and counts. You often need each value once while keeping the order in which values first appeared, and you want it to stay fast for long lists.

Think of it like this

Removing duplicates is like a doorman with a clicker and a guest list. For each person arriving, the doorman checks the list. New faces are written down and let in; repeat visitors are turned away. The room fills in arrival order.

From a doorman to code
seen = set()📋 The guest list
if value not in seen:🔎 A new face?
seen.add(value)✍️ Write the name down
result.append(value)🚪 Let them in, in arrival order
Each value is checked once; only first appearances are kept.
In one sentence

Removing duplicates in order keeps the first occurrence of each value, using a set for fast checks and a list to preserve the order.

Key points

  1. seen is a set, so value not in seen is a fast check even for long lists.
  2. seen.add(value) records the value the first time it appears.
  3. result.append(value) keeps values in the order they first appeared.
  4. The result is [3, 1, 2, 4]: the second 3 and the second 1 are skipped.

When should you use it?

  • Cleaning lists of IDs, emails or tags while keeping their order.
  • Removing repeated events from logs.
  • Learning when to combine a set and a list.

When not to use it

  • Order does not matter: set(values) is enough.
  • Values that cannot go in a set, such as lists or dictionaries: convert them to tuples first or compare differently.

Example code

Run code →
main.py
def unique_in_order(values):
    seen = set()
    result = []
    for value in values:
        if value not in seen:
            seen.add(value)
            result.append(value)
    return result

print(unique_in_order([3, 1, 3, 2, 1, 4]))
Get a Run button for this example

Result of running the code

[3, 1, 2, 4]

Code explanation

Each value is checked once. New values enter both the set and output list, giving O(n) average time.

Change the values and run the program in the CodeUtility online Python compiler without installing Python locally.

Practice exercises

Modify the runnable example with the exercises below to build understanding beyond copying the result.

  1. Try empty, single-item, and duplicate-value inputs.
  2. Print state after every step to observe the algorithm.
  3. Benchmark 100, 1,000, and 10,000 items, then compare another approach.
Run in the Python compiler →

Common mistakes

Checking membership in the result list

value in result scans the whole list each time, which is slow for long inputs.

Problem
if value not in result:
    result.append(value)
Fix
if value not in seen:
    seen.add(value)
    result.append(value)

Finished this lesson?

Mark it as learned to track your progress.

Practice with a challenge

Use what this lesson covers to solve a real problem. Each challenge runs in your browser with instant tests and an AI review.

All challenges