Pandas df.drop_duplicates() Function

Common Pandas Functions

df.drop_duplicates()It is a function in Pandas used to remove duplicate rows.

During data collection and integration, duplicate data is a common problem.drop_duplicates()It can help you identify and remove duplicate rows based on specified columns or all columns, keeping the first or last occurrence. This is very useful for data deduplication and ensuring data uniqueness.

Basic Syntax and Parameters

drop_duplicates()It is a member function of DataFrame, called via the dot operator.to invoke.

Syntax Format

DataFrame.drop_duplicates(subset=None, keep='first', inplace=False, ignore_index=False)

Parameter Description

Parameter Type Required Description Default Value
subset column label or list Optional Specify the column(s) used to determine duplicates. If it isNone, all columns are used. It can be a single column name or a list of column names. None
keep str or False Optional Specify which duplicate record to keep.'first'Keep the first one;'last'Keep the last one;FalseRemove all duplicate records. 'first'
inplace bool Optional If it isTrue, modify the original DataFrame directly and do not return a new object; if it isFalse, return a new DataFrame, and the original data remains unchanged. False
ignore_index bool Optional If it isTrue, reset the index of the result, starting from 0; if it isFalse, keep the original index. False

Return Value Description

  • Returns a new DataFrame (ifinplace=False), orNone(ifinplace=True）。
  • The returned DataFrame does not contain duplicate rows.

Examples

Let us thoroughly master, through a series of examples,drop_duplicates()the usage.

Example 1: Remove Fully Duplicate Rows

By default, all columns are used to determine whether a row is a duplicate.

Example

import pandas as pd
import numpy as np

# Create a DataFrame containing duplicate rows
data = {
    'Name': ['Zhang San', 'Li Si', 'Zhang San', 'Wang Wu', 'Li Si'],
    'Age': [25, 30, 25, 35, 30],
    'Department': ['Technology', 'Marketing', 'Technology', 'Technology', 'Marketing']
}
df = pd.DataFrame(data)

print(Original data:)
print(df)
print("=" * 50)

# Remove duplicate rows, keep the first occurrence
df_cleaned = df.drop_duplicates()

print(Data after removing duplicate rows:)
print(df_cleaned)

Expected output:

原始数据：
    姓名  年龄  部门
0  张三   25  技术
1  李四   30  市场
2  张三   25  技术
3  王五   35  技术
4  李四   30  市场
==================================================
删除重复行后的数据：
    0  姓名  年龄  部门
0  张三   25  技术
1  李四   30  市场
3  王五   35  技术

Code explanation:

  1. In the original data, row 0 and row 2 are exactly the same (Zhang San, 25 years old, Technology department).
  2. Row 1 and row 4 are also exactly the same (Li Si, 30 years old, Marketing department).
  3. Usingdrop_duplicates()with default parameters, duplicate rows are removed and the first occurrence is kept.

Example 2: Remove Duplicate Rows Based on Specified Columns

You can usesubsetparameter to determine duplicates based only on specific columns.

Example

import pandas as pd
import numpy as np

# Create a DataFrame
data = {
    'Name': ['Zhang San', 'Li Si', 'Zhang San', 'Wang Wu'],
    'Age': [25, 30, 28, 35],  # Zhang San appears twice, but with different ages
    'Department': ['Technology', 'Marketing', 'Technology', 'Marketing']
}
df = pd.DataFrame(data)

print(Original data:)
print(df)
print("=" * 50)

# Determine duplicates based only on the "Name" column
df_cleaned = df.drop_duplicates(subset=['Name'])

print(Data after removing duplicates based on the Name column:)
print(df_cleaned)
print("=" * 50)

# Determine duplicates based on the "Name" and "Department" columns
df_cleaned2 = df.drop_duplicates(subset=['Name', 'Department'])

print(Data after removing duplicates based on the Name and Department columns:)
print(df_cleaned2)

Expected output:

原始数据：
    姓名  年龄  部门
0  张三   25  技术
1  李四   30  市场
2  张三   28  技术  # 虽然姓名重复，但年龄不同
3  王五   35  市场
==================================================
根据姓名列删除重复后的数据：
    姓名  年龄  部门
0  张三   25  技术  # 保留第一条张三的记录
2  张三   28  技术
3  王五   35  市场
==================================================
根据姓名和部门列删除重复后的数据：
     参数
姓名  年龄  部门
0  张三   25  技术
1  李四   30  市场
3  王五   35  市场

Code explanation:

  • When determining duplicates based on the Name column, rows 0 and 2 both have the name "Zhang San" and are considered duplicate rows, so only row 0 is kept.
  • When determining duplicates based on both Name and Department columns, rows 0 and 2 have the same Name and Department, so row 0 is kept.

Example 3: Keep the Last Duplicate Record

Usingkeep='last'parameter keeps the last occurring duplicate record.

Example

import pandas as pd
import numpy as np

# Create a DataFrame containing duplicate rows
data = {
    'Student ID': ['S001', 'S002', 'S001', 'S003', 'S002'],
    'Name': ['Zhang San', 'Li Si', 'Zhang San', 'Wang Wu', 'Li Si'],
    'Score': [85, 90, 88, 92, 85]  # The same Student ID may have different scores
}
df = pd.DataFrame(data)

print(Original data:)
print(df)
print("=" * 50)

# Keep the first record (default)
df_first = df.drop_duplicates(subset=['Student ID'], keep='first')

print(Keeping the first duplicate record:)
print(df_first)
print("=" * 50)

# Keep the last record
df_last = df.drop_duplicates(subset=['Student ID'], keep='last')

print(Keeping the last duplicate record:)
print(df_last)

Expected output:

原始数据：
    学号  姓名  成绩
0  S001  张三     85
1  S002  李四     90
2  S001  张三     88  # 同一个人，成绩不同
3  S王五   92
4  S002  李四     85  # 同一个人，成绩不同
==================================================
保留第一条重复记录：
    学号  姓名  成绩
0  S001 张三     85  # 保留第一个85分
1  S002  李四     90  # 保留第一个90分
3  王五   92
==================================================
保留最后一条重复记录：
    学号  姓名  成绩
2  S001  张三     88  # 保留最后一个88分
3  王五   92
4  S002  李四     85  # 保留最后一个85分

Code explanation:

  • Student ID S001 has two records with scores 85 and 88; usingkeep='first'keeps the one with score 85,keep='last'keeps the one with score 88.
  • Student ID S002 also has two records, with scores 90 and 85.
  • Choose whether to keep the first or last record according to business needs.

    • Example 4: Remove All Duplicate Records

    Usingkeep=Falseremoves all duplicate records, keeping only rows that are completely unique.

    Example

    import pandas as pd
    import numpy as np

    # Create a DataFrame
    data = {
        'A': [1, 1, 2, 2, 3],
        'B': [1, 1, 2, 2, 3],
        'C': [1, 2, 3, 3, 5]
    }
    df = pd.DataFrame(data)

    print(Original data:)
    print(df)
    print("=" * 50)

    # Remove all duplicate rows (keep no duplicates)
    df_cleaned = df.drop_duplicates(keep=False)

    print(Data after removing all duplicate rows:)
    print(df_cleaned)

    Expected output:

    原始数据：
   A  B  C
0  1  1  1
1  1  1  2  # 与第0行A和B相同，是重复行
2  2  2  3
3  2  2  3  # 与第2行完全相同，是重复行
4  3  3  5
==================================================
删除所有重复行后的数据：
   A  B  C
4  3  3  5

    Code explanation:

    • Rows 0 and 1 have the same values in columns A and B, making them duplicate rows; sincekeep=Falseboth are deleted.
    • Rows 2 and 3 are also completely duplicated and are both deleted.
    • Only row 4 is completely unique and is kept.

    Example 5: Reset Index

    After removing duplicate rows, the original index may become non-contiguous; you can useignore_index=Trueto reset the index.

    Example

    import pandas as pd
    import numpy as np

    # Create a DataFrame containing duplicate rows
    data = {
        'Name': ['Zhang San', 'Li Si', 'Zhang San', 'Wang Wu'],
        'City': ['Beijing', 'Shanghai', 'Beijing', 'Guangzhou']
    }
    df = pd.DataFrame(data)

    print(Original data:)
    print(df)
    print("=" * 50)

    # Remove duplicate rows without resetting the index
    df_cleaned1 = df.drop_duplicates()

    print(Remove duplicate rows (keep original index):)
    print(df_cleaned1)
    print("=" * 50)

    # Remove duplicate rows and reset the index
    df_cleaned2 = df.drop_duplicates(ignore_index=True)

    print(Remove duplicate rows (reset index):)
    print(df_cleaned2)

    Expected output:

    原始数据：
    姓名  城市
0  张三  北京
1  李四  上海
2  张三  北京
3  王五  广州
==================================================
删除重复行（保留原始索引）：
    姓名  城市
0  张三  北京
1  李四  上海
3  王五  广州
==================================================
删除重复行（重置索引）：
    姓名  城市
0  张三  北京
1  李四  上海
2  王五  广州

    Code explanation:

    • Without usingignore_index, after deleting row 2, the index is 0, 1, 3, which is non-contiguous.
    • Usingignore_index=True, the index restarts from 0, resulting in 0, 1, 2.

    Example 6: Combine with Other Operations

    drop_duplicates()It can be used in combination with other DataFrame operations.

    Example

    import pandas as pd
    import numpy as np

    # Simulate data retrieved from a database query
    data = {
        'Order ID': ['O001', 'O002', 'O001', 'O003', 'O002', 'O004'],
        'Customer Name': ['Zhang San', 'Li Si', 'Zhang San', 'Wang Wu', 'Li Si', 'Zhao Liu'],
        'Amount': [100, 200, 100, 300, 200, 400],
        'Date': ['2024-01-01', '2024-01-02', '2024-01-01', '2024-01-03', '2024-01-02', '2024-01-04']
    }
    df = pd.DataFrame(data)

    print("Original order data:")
    print(df)
    print("=" * 50)

    # Check how many duplicate orders there are
    print(f"Total rows: {len(df)}")
    print(f"Rows after deduplication: {len(df.drop_duplicates())}")
    print(f"Duplicate rows: {len(df) - len(df.drop_duplicates())}")
    print("=" * 50)

    # Actually deduplicate, keep the first record, and only keep the needed columns
    df_unique = df.drop_duplicates(subset=['Order ID'])[['Order ID', 'Customer Name', 'Amount']]

    print("Deduplicated order data:")
    print(df_unique)

    Expected output:

    原始订单数据：
    订单号  客户名  金额   日期
0  O001    张三  100  2024-01-01
1  O002    李四  200  2024-01-02
2  O001    张三  100  2024-01-01  # 重复订单
3  O003    王五  300  2024-01-03
4  O002    李四  200  2024-01-02  # 重复订单
5  O004    赵六  400  2024-01-04
================================================++
总行数: 6
去重后行数: 4
重复行数: 2

    Code explanation:

    1. The original data has 6 rows, with 2 duplicate orders (O001 and O002 each appear twice).
    2. Usedrop_duplicates(subset=['订单号'])Deduplicate based on the order number.
    3. Combined with column selection[['Order Number', 'Customer Name', 'Amount']], only keep the needed columns.

    Precautions

    • drop_duplicates()By default, the original DataFrame is not modified; if you want to modify it in place, useinplace=Trueparameter.
    • UsesubsetWhen using the parameter, duplicates are determined only by the specified columns; values in other columns do not affect the determination.
    • Usekeep=Falsewill delete all duplicate rows, which may cause a large amount of data loss; use with caution.
    • Before deleting duplicate data, it is recommended to analyze the cause of duplicates first to ensure the deletion operation conforms to business logic.
    • If there are missing values (NaN) in the data, they will be treated as the same value for comparison by default.

    Common Pandas Functions

    Other Extensions