Python Remove Duplicate Elements from List
Python3 ExamplesIn this chapter, we will learn how to remove duplicate elements from a list.
Key knowledge points:
-
Python Set: A set is an unordered sequence of unique elements.
-
Python List: A list is a finite sequence of data items, that is, a collection of data items arranged in a certain linear order. The basic operations performed on this data structure include searching, inserting, and deleting elements.
Example
print(list(set(list_1)))
After executing the above code, the output is:
[1, 2, 4, 6]
In the above example, we first convert the list to a set, and then convert it back to a list. A set cannot contain duplicate elements, so set() removes the duplicate elements.
If you need to preserve the order of the elements in the original list, you can use an auxiliary set to track the elements that have already been seen, and then build a new list.
Example
def remove_duplicates(lst):
seen = set()
unique_list = []
for item in lst:
if item not in seen:
seen.add(item)
unique_list.append(item)
return unique_list
# Example
original_list = [1, 2, 2, 3, 4, 4, 5]
unique_list = remove_duplicates(original_list)
print(unique_list) # Output: [1, 2, 3, 4, 5]
Using dict.fromkeys()
The dict.fromkeys() method can also be used for deduplication while preserving order, because dictionaries maintain insertion order in Python 3.7 and above.
Example
def remove_duplicates(lst):
return list(dict.fromkeys(lst))
# Example
original_list = [1, 2, 2, 3, 4, 4, 5]
unique_list = remove_duplicates(original_list)
print(unique_list) # Output: [1, 2, 3, 4, 5]
After executing the above code, the output is:
[1, 2, 3, 4, 5]
Using List Comprehension
List comprehension combined with conditional checks can also achieve deduplication.
Example
def remove_duplicates(lst):
unique_list = []
[unique_list.append(item) for item in lst if item not in unique_list]
return unique_list
# Example
original_list = [1, 2, 2, 3, 4, 4, 5]
unique_list = remove_duplicates(original_list)
print(unique_list) # Output: [1, 2, 3, 4, 5]
Remove Duplicate Elements from Two Lists
In the following example, elements that exist in both lists are removed.
Example
list_2 = [7, 8, 2, 1]
print(list(set(list_1) ^ set(list_2)))
First, use set() to convert the two lists into two sets, in order to remove duplicate elements from the lists.
Then, use the^operator to obtain the symmetric difference of the two lists.
After executing the above code, the output is:
[4, 6, 7, 8]
First, convert the two lists into two sets to remove duplicates from each list.
Then,^gives the symmetric difference of the two lists (excluding the overlapping elements of the two sets).
Other Extensions