Introduction

Duplicate rows in a DataFrame are one of the most common data quality issues you will encounter. Whether you are cleaning survey responses, merging tables from different sources, or validating ETL output, you need a reliable way to find and inspect every duplicate. Pandas gives you several tools for this, and understanding how each one behaves will save you from silently dropping rows or missing hidden duplicates.

Using duplicated() with keep Options

The duplicated() method returns a boolean Series that marks each row as True if it is a duplicate. The keep parameter controls which occurrence is considered the "original."

1import pandas as pd
2
3df = pd.DataFrame({
4    'Name': ['Alice', 'Bob', 'Alice', 'Charlie', 'Bob', 'Alice'],
5    'Age': [30, 25, 30, 35, 25, 30],
6    'City': ['NYC', 'LA', 'NYC', 'Chicago', 'LA', 'NYC']
7})
8
9# keep='first' (default): marks later occurrences as duplicates
10print(df[df.duplicated(keep='first')])
11#      Name  Age City
12# 2   Alice   30  NYC
13# 4     Bob   25   LA
14# 5   Alice   30  NYC
15
16# keep='last': marks earlier occurrences as duplicates
17print(df[df.duplicated(keep='last')])
18#     Name  Age City
19# 0  Alice   30  NYC
20# 1    Bob   25   LA
21# 2  Alice   30  NYC
22
23# keep=False: marks ALL occurrences, including the first and last
24print(df[df.duplicated(keep=False)])
25#      Name  Age City
26# 0   Alice   30  NYC
27# 1     Bob   25   LA
28# 2   Alice   30  NYC
29# 4     Bob   25   LA
30# 5   Alice   30  NYC

Use keep=False when you want to see every row involved in a duplication, not just the "extra" copies. This is the option most people actually need during data auditing.

You can also check duplicates across specific columns only:

1# Duplicates based on Name column alone
2name_dupes = df[df.duplicated(subset=['Name'], keep=False)]
3print(name_dupes)
4#      Name  Age City
5# 0   Alice   30  NYC
6# 2   Alice   30  NYC
7# 1     Bob   25   LA
8# 4     Bob   25   LA
9# 5   Alice   30  NYC

Using value_counts() to Spot Duplicates

Sometimes you do not need the full rows. You just want to know which values appear more than once. The value_counts() method counts occurrences and sorts by frequency:

1# Count how many times each Name appears
2counts = df['Name'].value_counts()
3print(counts)
4# Alice      3
5# Bob        2
6# Charlie    1
7
8# Filter to only names that appear more than once
9duplicated_names = counts[counts > 1]
10print(duplicated_names.index.tolist())
11# ['Alice', 'Bob']

This is a quick way to get a summary without pulling entire rows. It works well on single columns and is especially useful in exploratory analysis.

Using groupby().filter()

When you need to retrieve all rows belonging to groups that have more than one member, groupby().filter() gives you a clean one-liner:

1# Return all rows where the Name appears more than once
2dupes = df.groupby('Name').filter(lambda group: len(group) > 1)
3print(dupes)
4#      Name  Age City
5# 0   Alice   30  NYC
6# 1     Bob   25   LA
7# 2   Alice   30  NYC
8# 4     Bob   25   LA
9# 5   Alice   30  NYC

This is functionally similar to duplicated(subset=['Name'], keep=False), but groupby().filter() is more flexible. You can apply any condition to the group, for example filtering groups where the Age values are not all the same:

1# Groups where the same Name has different Ages
2inconsistent = df.groupby('Name').filter(
3    lambda g: g['Age'].nunique() > 1
4)

Using drop_duplicates()

Once you have identified duplicates, you often want to remove them. The drop_duplicates() method mirrors the keep parameter from duplicated():

1# Keep the first occurrence of each duplicate set
2cleaned = df.drop_duplicates(keep='first')
3print(cleaned)
4#       Name  Age     City
5# 0    Alice   30      NYC
6# 1      Bob   25       LA
7# 3  Charlie   35  Chicago
8
9# Drop duplicates based on a subset of columns
10cleaned_by_name = df.drop_duplicates(subset=['Name'], keep='last')
11print(cleaned_by_name)
12#       Name  Age     City
13# 3  Charlie   35  Chicago
14# 4      Bob   25       LA
15# 5    Alice   30      NYC

If you pass keep=False, every row that has a duplicate is dropped, leaving only rows that were unique to begin with:

1unique_only = df.drop_duplicates(keep=False)
2print(unique_only)
3#       Name  Age     City
4# 3  Charlie   35  Chicago

Common Pitfalls

• Forgetting that keep='first' is the default: Many developers call df[df.duplicated()] expecting to see all duplicated rows, but this hides the first occurrence of each group. Use keep=False for a complete picture.
• Checking duplicates on all columns when you only care about a key: Without the subset parameter, duplicated() compares every column. Two rows with the same ID but different timestamps will not be flagged. Always specify subset when your definition of "duplicate" is based on specific columns.
• Mutating the DataFrame while iterating over duplicates: Dropping rows inside a loop that references the original DataFrame can cause index errors or skipped rows. Build the list of duplicates first, then drop them in a single operation.
• Ignoring case and whitespace differences: "Alice" and " alice " are different strings to pandas. Normalize your data with str.strip().str.lower() before checking for duplicates if case or whitespace varies.
• Using drop_duplicates() without resetting the index: After dropping rows, the index has gaps. Downstream code that relies on sequential indexing (like iloc with a range) may break. Call .reset_index(drop=True) after deduplication.

Summary

• duplicated(keep=False) is the most useful variant for auditing because it flags every row involved in a duplication, not just the extras.
• value_counts() gives a quick frequency table to identify which values appear more than once without pulling full rows.
• groupby().filter() retrieves complete rows for any group matching a custom condition, making it more flexible than duplicated() alone.
• drop_duplicates() removes duplicate rows and supports the same keep and subset parameters for precise control.
• Always specify subset when your duplicate definition is based on key columns rather than entire rows.