Introduction

Getting rows from one DataFrame that do not exist in another is an anti-join problem. In pandas, the most reliable general solution is to merge the two frames on the columns that define row identity and then keep only the rows that appear on the left side but not on the right.

The Standard Anti-Join With merge

1import pandas as pd
2
3left = pd.DataFrame({
4    "id": [1, 2, 3, 4],
5    "name": ["Ada", "Bob", "Cara", "Dan"],
6})
7
8right = pd.DataFrame({
9    "id": [2, 4],
10    "name": ["Bob", "Dan"],
11})
12
13result = (
14    left.merge(right, how="left", indicator=True)
15        .query('_merge == "left_only"')
16        .drop(columns="_merge")
17)
18
19print(result)

Output:

   id  name
0   1   Ada
2   3  Cara

The indicator=True flag adds a _merge column that tells you whether each row matched the left frame only, the right frame only, or both.

Compare on Specific Columns

Often you do not want full-row equality. You want equality based on a subset of key columns:

1result = (
2    left.merge(right[["id"]], on="id", how="left", indicator=True)
3        .query('_merge == "left_only"')
4        .drop(columns="_merge")
5)

Now rows are considered "present in the other frame" based only on id, regardless of other column differences.

This distinction matters a lot in real data-cleaning workflows.

Another Approach: MultiIndex Membership

If you want to compare full rows and are comfortable with index operations, you can convert the relevant columns into a MultiIndex:

1mask = ~left.set_index(["id", "name"]).index.isin(
2    right.set_index(["id", "name"]).index
3)
4
5result = left[mask]
6print(result)

This can be neat for row-identity problems, though the merge approach is usually easier to read and extend.

It is also handy when you want a pure membership test without bringing along extra columns from a merge result.

Duplicates Matter

Think carefully about duplicates. If left contains repeated rows and right contains one matching row, do you want:

• all matching left duplicates removed
• only some duplicates removed

The merge approach usually removes all left rows whose join keys appear in right, which is often the correct business interpretation but not always.

Missing Values and Comparison Semantics

Null handling can affect the result. If key columns contain missing values, define the intended behavior explicitly before comparing. In many workflows, cleaning or filling key columns first is safer than letting missing-value comparison rules decide the result implicitly.

This matters most when missing keys have business meaning, such as "unknown user" versus "invalid row," because those cases should not be mixed accidentally.

Why isin Alone Is Often Not Enough

People often try:

left[~left["id"].isin(right["id"])]

That works only for a single-column comparison. It is not a general row-difference solution when multiple columns define identity.

Use it when a single key column is truly the comparison rule. Use merge or index-based comparison when row identity is more complex.

Common Pitfalls

The biggest mistake is not deciding which columns define "the same row." Full-row difference and key-based difference are not the same operation.

Another mistake is forgetting about duplicates, which can change the meaning of the result.

A third issue is using single-column isin logic when the task actually requires multi-column comparison.

Summary

• In pandas, finding rows that are not in another DataFrame is usually an anti-join.
• 'merge(..., indicator=True) is the clearest general solution.'
• Decide whether equality means full-row equality or key-column equality.
• Be careful with duplicates and missing values.
• Use simple isin only when one column really is the whole comparison rule.

That small definition step around row identity usually determines whether the result is correct or subtly wrong.