Practice Now
pandas
get_dummies
data manipulation
python
data analysis

Specify list of possible values for Pandas get_dummies

ML System Design practice on Codemia

Design recommenders, ranking systems and training pipelines the way ML interviews actually ask for them, with worked solutions.

Practice ML system design
Data SciencePython

Introduction

pd.get_dummies() creates dummy columns only for categories present in the data, which causes problems when test data has fewer categories than training data. To specify a fixed list of possible values, convert the column to a Categorical dtype with predefined categories before calling get_dummies(). This ensures consistent columns across training and test sets. For ML pipelines, prefer sklearn.preprocessing.OneHotEncoder which stores categories from fit() and applies them consistently during transform().

The Problem

python
1import pandas as pd
2
3# Training data has 3 categories
4train = pd.DataFrame({'color': ['red', 'blue', 'green', 'red']})
5train_dummies = pd.get_dummies(train['color'])
6print(train_dummies.columns.tolist())
7# ['blue', 'green', 'red']
8
9# Test data has only 2 categories — different columns!
10test = pd.DataFrame({'color': ['red', 'blue', 'blue']})
11test_dummies = pd.get_dummies(test['color'])
12print(test_dummies.columns.tolist())
13# ['blue', 'red']  — missing 'green' column!

The model trained on 3 columns receives 2 columns at prediction time, causing a shape mismatch error.

Fix 1: Use pd.Categorical with Fixed Categories

Convert the column to Categorical dtype with all possible values before encoding:

python
1import pandas as pd
2
3all_colors = ['red', 'blue', 'green']
4
5# Training data
6train = pd.DataFrame({'color': ['red', 'blue', 'green', 'red']})
7train['color'] = pd.Categorical(train['color'], categories=all_colors)
8train_dummies = pd.get_dummies(train['color'])
9print(train_dummies)
10#    blue  green  red
11# 0  False  False  True
12# 1  True   False  False
13# 2  False  True   False
14# 3  False  False  True
15
16# Test data — missing 'green' gets a column of False
17test = pd.DataFrame({'color': ['red', 'blue', 'blue']})
18test['color'] = pd.Categorical(test['color'], categories=all_colors)
19test_dummies = pd.get_dummies(test['color'])
20print(test_dummies)
21#    blue  green  red
22# 0  False  False  True
23# 1  True   False  False
24# 2  True   False  False

Both DataFrames now have identical columns.

Fix 2: Reindex After get_dummies

Apply get_dummies normally, then reindex to match training columns:

python
1import pandas as pd
2
3train = pd.DataFrame({'color': ['red', 'blue', 'green', 'red']})
4train_dummies = pd.get_dummies(train['color'])
5
6test = pd.DataFrame({'color': ['red', 'blue', 'blue']})
7test_dummies = pd.get_dummies(test['color'])
8
9# Reindex to match training columns, fill missing with 0
10test_dummies = test_dummies.reindex(columns=train_dummies.columns, fill_value=0)
11print(test_dummies)
12#    blue  green  red
13# 0     0      0    1
14# 1     1      0    0
15# 2     1      0    0

This also handles test data with unseen categories — extra columns are dropped, missing columns are added as zeros.

OneHotEncoder stores categories from fit() and applies them consistently:

python
1from sklearn.preprocessing import OneHotEncoder
2import pandas as pd
3import numpy as np
4
5train = pd.DataFrame({'color': ['red', 'blue', 'green', 'red']})
6test = pd.DataFrame({'color': ['red', 'blue', 'blue']})
7
8encoder = OneHotEncoder(sparse_output=False, handle_unknown='ignore')
9encoder.fit(train[['color']])
10
11train_encoded = encoder.transform(train[['color']])
12test_encoded = encoder.transform(test[['color']])
13
14print(encoder.categories_)
15# [array(['blue', 'green', 'red'], dtype=object)]
16
17print(train_encoded.shape, test_encoded.shape)
18# (4, 3) (3, 3) — same number of columns!
19
20# Get feature names
21print(encoder.get_feature_names_out())
22# ['color_blue', 'color_green', 'color_red']

handle_unknown='ignore' ensures unseen categories produce all-zero rows instead of errors.

Multiple Categorical Columns

python
1import pandas as pd
2
3categories = {
4    'color': ['red', 'blue', 'green'],
5    'size': ['S', 'M', 'L', 'XL']
6}
7
8train = pd.DataFrame({
9    'color': ['red', 'blue', 'green'],
10    'size': ['S', 'M', 'L'],
11    'price': [10, 20, 30]
12})
13
14# Apply Categorical to each column
15for col, cats in categories.items():
16    train[col] = pd.Categorical(train[col], categories=cats)
17
18# get_dummies on all categorical columns
19dummies = pd.get_dummies(train, columns=['color', 'size'])
20print(dummies.columns.tolist())
21# ['price', 'color_red', 'color_blue', 'color_green', 'size_S', 'size_M', 'size_L', 'size_XL']

Helper Function

python
1import pandas as pd
2
3def get_dummies_with_categories(df, column, categories):
4    """Create dummies with a fixed set of categories."""
5    df = df.copy()
6    df[column] = pd.Categorical(df[column], categories=categories)
7    return pd.get_dummies(df, columns=[column])
8
9# Usage
10all_colors = ['red', 'blue', 'green', 'yellow']
11
12train = pd.DataFrame({'color': ['red', 'blue'], 'value': [1, 2]})
13test = pd.DataFrame({'color': ['green'], 'value': [3]})
14
15train_enc = get_dummies_with_categories(train, 'color', all_colors)
16test_enc = get_dummies_with_categories(test, 'color', all_colors)
17
18print(train_enc.columns.tolist())
19# ['value', 'color_red', 'color_blue', 'color_green', 'color_yellow']
20print(test_enc.columns.tolist())
21# ['value', 'color_red', 'color_blue', 'color_green', 'color_yellow']

drop_first for Multicollinearity

python
1import pandas as pd
2
3df = pd.DataFrame({'color': pd.Categorical(['red', 'blue', 'green'],
4                                            categories=['red', 'blue', 'green'])})
5
6# Drop first category to avoid multicollinearity in linear models
7dummies = pd.get_dummies(df['color'], drop_first=True)
8print(dummies)
9#    blue  green
10# 0  False  False   # red (reference category)
11# 1  True   False
12# 2  False  True

Common Pitfalls

  • Not specifying categories on the test set: get_dummies() on test data produces fewer columns if some categories are absent, causing column mismatch errors when feeding to a trained model. Always use pd.Categorical or reindex to enforce consistent columns.
  • Forgetting handle_unknown='ignore' with OneHotEncoder: If test data contains a category not seen during fit(), the default behavior raises a ValueError. Set handle_unknown='ignore' to produce an all-zero row for unseen categories.
  • Using get_dummies in ML pipelines instead of OneHotEncoder: get_dummies does not have a fit/transform API, so it cannot remember categories from training. For reproducible ML pipelines, use sklearn.preprocessing.OneHotEncoder inside a Pipeline.
  • Passing the entire DataFrame to pd.Categorical: pd.Categorical works on a single Series, not a DataFrame. Apply it column by column, or use OneHotEncoder which accepts multiple columns at once.
  • Not dropping the first dummy variable for linear models: Linear regression and logistic regression are sensitive to multicollinearity. Use drop_first=True in get_dummies or drop='first' in OneHotEncoder to remove one redundant column per feature.

Summary

  • Convert columns to pd.Categorical(col, categories=[...]) before get_dummies() to ensure all categories appear
  • Use reindex(columns=train_columns, fill_value=0) as an alternative alignment method
  • For ML pipelines, use sklearn.preprocessing.OneHotEncoder with handle_unknown='ignore'
  • Use drop_first=True to avoid multicollinearity in linear models
  • Store the category list from training and apply it to all future data consistently
Related reading
Free course
Beginner
7 lessons
2 hours
Tackling System Design Interview Problems

A short course that equips you with the skills to approach system design interviews methodically.

Start the free course
Track what you have practised

A free account saves your progress, solutions and study plan across every problem on Codemia.

ML System Design practice on Codemia

Design recommenders, ranking systems and training pipelines the way ML interviews actually ask for them, with worked solutions.

Practice ML system design
Codemia
© 2026 Codemia
Resources
Legal
Social
All Rights Reserved.