Introduction
pd.get_dummies() creates dummy columns only for categories present in the data, which causes problems when test data has fewer categories than training data. To specify a fixed list of possible values, convert the column to a Categorical dtype with predefined categories before calling get_dummies(). This ensures consistent columns across training and test sets. For ML pipelines, prefer sklearn.preprocessing.OneHotEncoder which stores categories from fit() and applies them consistently during transform().
The Problem
1import pandas as pd
2
3# Training data has 3 categories
4train = pd.DataFrame({'color': ['red', 'blue', 'green', 'red']})
5train_dummies = pd.get_dummies(train['color'])
6print(train_dummies.columns.tolist())
7# ['blue', 'green', 'red']
8
9# Test data has only 2 categories — different columns!
10test = pd.DataFrame({'color': ['red', 'blue', 'blue']})
11test_dummies = pd.get_dummies(test['color'])
12print(test_dummies.columns.tolist())
13# ['blue', 'red'] — missing 'green' column!
The model trained on 3 columns receives 2 columns at prediction time, causing a shape mismatch error.
Fix 1: Use pd.Categorical with Fixed Categories
Convert the column to Categorical dtype with all possible values before encoding:
1import pandas as pd
2
3all_colors = ['red', 'blue', 'green']
4
5# Training data
6train = pd.DataFrame({'color': ['red', 'blue', 'green', 'red']})
7train['color'] = pd.Categorical(train['color'], categories=all_colors)
8train_dummies = pd.get_dummies(train['color'])
9print(train_dummies)
10# blue green red
11# 0 False False True
12# 1 True False False
13# 2 False True False
14# 3 False False True
15
16# Test data — missing 'green' gets a column of False
17test = pd.DataFrame({'color': ['red', 'blue', 'blue']})
18test['color'] = pd.Categorical(test['color'], categories=all_colors)
19test_dummies = pd.get_dummies(test['color'])
20print(test_dummies)
21# blue green red
22# 0 False False True
23# 1 True False False
24# 2 True False False
Both DataFrames now have identical columns.
Fix 2: Reindex After get_dummies
Apply get_dummies normally, then reindex to match training columns:
1import pandas as pd
2
3train = pd.DataFrame({'color': ['red', 'blue', 'green', 'red']})
4train_dummies = pd.get_dummies(train['color'])
5
6test = pd.DataFrame({'color': ['red', 'blue', 'blue']})
7test_dummies = pd.get_dummies(test['color'])
8
9# Reindex to match training columns, fill missing with 0
10test_dummies = test_dummies.reindex(columns=train_dummies.columns, fill_value=0)
11print(test_dummies)
12# blue green red
13# 0 0 0 1
14# 1 1 0 0
15# 2 1 0 0
This also handles test data with unseen categories — extra columns are dropped, missing columns are added as zeros.
Fix 3: Use sklearn OneHotEncoder (Recommended for ML)
OneHotEncoder stores categories from fit() and applies them consistently:
1from sklearn.preprocessing import OneHotEncoder
2import pandas as pd
3import numpy as np
4
5train = pd.DataFrame({'color': ['red', 'blue', 'green', 'red']})
6test = pd.DataFrame({'color': ['red', 'blue', 'blue']})
7
8encoder = OneHotEncoder(sparse_output=False, handle_unknown='ignore')
9encoder.fit(train[['color']])
10
11train_encoded = encoder.transform(train[['color']])
12test_encoded = encoder.transform(test[['color']])
13
14print(encoder.categories_)
15# [array(['blue', 'green', 'red'], dtype=object)]
16
17print(train_encoded.shape, test_encoded.shape)
18# (4, 3) (3, 3) — same number of columns!
19
20# Get feature names
21print(encoder.get_feature_names_out())
22# ['color_blue', 'color_green', 'color_red']
handle_unknown='ignore' ensures unseen categories produce all-zero rows instead of errors.
Multiple Categorical Columns
1import pandas as pd
2
3categories = {
4 'color': ['red', 'blue', 'green'],
5 'size': ['S', 'M', 'L', 'XL']
6}
7
8train = pd.DataFrame({
9 'color': ['red', 'blue', 'green'],
10 'size': ['S', 'M', 'L'],
11 'price': [10, 20, 30]
12})
13
14# Apply Categorical to each column
15for col, cats in categories.items():
16 train[col] = pd.Categorical(train[col], categories=cats)
17
18# get_dummies on all categorical columns
19dummies = pd.get_dummies(train, columns=['color', 'size'])
20print(dummies.columns.tolist())
21# ['price', 'color_red', 'color_blue', 'color_green', 'size_S', 'size_M', 'size_L', 'size_XL']
Helper Function
1import pandas as pd
2
3def get_dummies_with_categories(df, column, categories):
4 """Create dummies with a fixed set of categories."""
5 df = df.copy()
6 df[column] = pd.Categorical(df[column], categories=categories)
7 return pd.get_dummies(df, columns=[column])
8
9# Usage
10all_colors = ['red', 'blue', 'green', 'yellow']
11
12train = pd.DataFrame({'color': ['red', 'blue'], 'value': [1, 2]})
13test = pd.DataFrame({'color': ['green'], 'value': [3]})
14
15train_enc = get_dummies_with_categories(train, 'color', all_colors)
16test_enc = get_dummies_with_categories(test, 'color', all_colors)
17
18print(train_enc.columns.tolist())
19# ['value', 'color_red', 'color_blue', 'color_green', 'color_yellow']
20print(test_enc.columns.tolist())
21# ['value', 'color_red', 'color_blue', 'color_green', 'color_yellow']
drop_first for Multicollinearity
1import pandas as pd
2
3df = pd.DataFrame({'color': pd.Categorical(['red', 'blue', 'green'],
4 categories=['red', 'blue', 'green'])})
5
6# Drop first category to avoid multicollinearity in linear models
7dummies = pd.get_dummies(df['color'], drop_first=True)
8print(dummies)
9# blue green
10# 0 False False # red (reference category)
11# 1 True False
12# 2 False True
Common Pitfalls
Not specifying categories on the test set: get_dummies() on test data produces fewer columns if some categories are absent, causing column mismatch errors when feeding to a trained model. Always use pd.Categorical or reindex to enforce consistent columns.
Forgetting handle_unknown='ignore' with OneHotEncoder: If test data contains a category not seen during fit(), the default behavior raises a ValueError. Set handle_unknown='ignore' to produce an all-zero row for unseen categories.
Using get_dummies in ML pipelines instead of OneHotEncoder: get_dummies does not have a fit/transform API, so it cannot remember categories from training. For reproducible ML pipelines, use sklearn.preprocessing.OneHotEncoder inside a Pipeline.
Passing the entire DataFrame to pd.Categorical: pd.Categorical works on a single Series, not a DataFrame. Apply it column by column, or use OneHotEncoder which accepts multiple columns at once.
Not dropping the first dummy variable for linear models: Linear regression and logistic regression are sensitive to multicollinearity. Use drop_first=True in get_dummies or drop='first' in OneHotEncoder to remove one redundant column per feature.
Summary
Convert columns to pd.Categorical(col, categories=[...]) before get_dummies() to ensure all categories appear
Use reindex(columns=train_columns, fill_value=0) as an alternative alignment method
For ML pipelines, use sklearn.preprocessing.OneHotEncoder with handle_unknown='ignore'
Use drop_first=True to avoid multicollinearity in linear models
Store the category list from training and apply it to all future data consistently
