Introduction

The Pandas error KeyError: ... not in index usually means your code is trying to select columns or rows that do not exist in the current DataFrame. In machine-learning workflows with scikit-learn, this often happens after feature selection, one-hot encoding, train-test splitting, or column reordering changes the available labels.

Understand what the error is really saying

Pandas indexes rows and columns by labels. If you ask for labels that are missing, Pandas raises KeyError.

A simple example:

1import pandas as pd
2
3df = pd.DataFrame({
4    "age": [30, 25],
5    "salary": [70000, 50000],
6})
7
8print(df[["age", "height"]])

This fails because height is not a column in the frame.

The first debugging step is always to inspect the current labels:

print(df.columns.tolist())

Do that at the point of failure, not only earlier in the pipeline.

Common scikit-learn workflow cause: train and test columns drift

One of the most frequent machine-learning causes is fitting transformations on one frame and trying to index another frame with columns that no longer match.

For example:

1train = pd.get_dummies(train_df, columns=["city"])
2test = pd.get_dummies(test_df, columns=["city"])
3
4test = test[train.columns]

This can fail if the test set does not contain every dummy column that appeared in training.

A safer fix is to align the frames:

train, test = train.align(test, join="left", axis=1, fill_value=0)

Now both frames share the same columns, with missing columns filled by zeros.

Check for dropped or renamed columns

Another common pattern is:

• select features
• drop a column
• later try to access the old feature list again

features = ["age", "salary", "city"]
X = df.drop(columns=["city"])
X = X[features]

That raises KeyError because city was already removed.

The fix is either to update the feature list or to derive it from the current frame:

features = [col for col in features if col in X.columns]
X = X[features]

That pattern is defensive, though in many production code paths it is even better to fail early with a clear custom message.

Reindex when you want missing columns filled

If your intent is "select these columns if they exist, and create missing ones as zeros," use reindex instead of direct indexing.

expected_columns = ["age", "salary", "city_Boston", "city_Paris"]
X = X.reindex(columns=expected_columns, fill_value=0)

This is especially useful after one-hot encoding or feature engineering pipelines.

Prefer scikit-learn pipelines for stable feature handling

A robust long-term fix is to let scikit-learn manage preprocessing consistently through a pipeline and ColumnTransformer.

1from sklearn.compose import ColumnTransformer
2from sklearn.pipeline import Pipeline
3from sklearn.preprocessing import OneHotEncoder, StandardScaler
4from sklearn.linear_model import LogisticRegression
5
6numeric_features = ["age", "salary"]
7categorical_features = ["city"]
8
9preprocessor = ColumnTransformer(
10    transformers=[
11        ("num", StandardScaler(), numeric_features),
12        ("cat", OneHotEncoder(handle_unknown="ignore"), categorical_features),
13    ]
14)
15
16model = Pipeline([
17    ("preprocessor", preprocessor),
18    ("classifier", LogisticRegression()),
19])

With handle_unknown="ignore", unseen categories in test data do not create the same column-mismatch pain that manual get_dummies workflows often do.

Index mistakes are not only about columns

The error can also come from row-label selection:

df.loc[[10, 11]]

If row labels 10 and 11 do not exist, Pandas raises KeyError.

Make sure you know whether you mean:

• label-based access with .loc
• position-based access with .iloc

That distinction matters a lot after filtering and splitting data.

Common Pitfalls

The biggest mistake is inspecting the original DataFrame instead of the transformed one that actually raised the error. In data pipelines, the label set changes frequently.

Another issue is using pd.get_dummies separately on train and test data without aligning the resulting columns. That is a classic cause of missing dummy features.

Developers also confuse .loc with .iloc. If you want positional selection, label-based indexing will fail as soon as the index labels differ from the integer positions you expect.

Finally, a feature list copied into code can drift away from reality over time. If the pipeline changes, make sure the selection logic changes with it.

Summary

• 'KeyError: ... not in index means the labels you asked for do not exist in the current Pandas index.'
• Print the current columns or index at the failure point before guessing.
• Train-test feature drift after dummy encoding is a very common source of this error.
• Use align, reindex, or scikit-learn pipelines to keep feature sets consistent.
• Be clear about whether you are selecting by label with .loc or by position with .iloc.