Introduction

Categorical encoding converts text labels into numbers that machine learning models can process. Python offers four main tools for this: pd.factorize (simple integer encoding), pd.get_dummies (one-hot encoding in pandas), LabelEncoder (integer encoding in scikit-learn), and OneHotEncoder (one-hot encoding in scikit-learn). Each has different strengths regarding handling unknown categories, integration with ML pipelines, and output format.

pd.factorize

Assigns a unique integer to each unique value. Returns codes and unique values:

1import pandas as pd
2
3colors = pd.Series(["red", "blue", "green", "blue", "red"])
4codes, uniques = pd.factorize(colors)
5
6print(codes)    # [0, 1, 2, 1, 0]
7print(uniques)  # Index(['red', 'blue', 'green'], dtype='object')

Key characteristics:

• Assigns integers in order of first appearance (not alphabetical)
• Returns (codes_array, unique_values_array)
• Does not remember the mapping — cannot transform new data
• Handles NaN as -1 by default

1# NaN handling
2data = pd.Series(["a", None, "b", "a"])
3codes, uniques = pd.factorize(data)
4print(codes)  # [0, -1, 1, 0]  (-1 for NaN)

LabelEncoder (sklearn)

Similar to factorize but remembers the mapping and can transform new data:

1from sklearn.preprocessing import LabelEncoder
2
3le = LabelEncoder()
4colors = ["red", "blue", "green", "blue", "red"]
5
6# Fit and transform
7encoded = le.fit_transform(colors)
8print(encoded)       # [2, 0, 1, 0, 2]  (alphabetical order)
9print(le.classes_)   # ['blue', 'green', 'red']
10
11# Transform new data
12new_data = ["green", "red"]
13print(le.transform(new_data))  # [1, 2]
14
15# Inverse transform
16print(le.inverse_transform([0, 1, 2]))  # ['blue', 'green', 'red']

Key characteristics:

• Assigns integers in alphabetical order (sorted)
• Remembers the mapping via fit() — can transform() new data
• Raises an error for unseen categories during transform()
• Designed for encoding target variables (y), not features (X)

pd.get_dummies

Creates binary columns (one-hot encoding) directly in pandas:

1import pandas as pd
2
3df = pd.DataFrame({"color": ["red", "blue", "green", "blue"]})
4dummies = pd.get_dummies(df, columns=["color"])
5
6print(dummies)
7#    color_blue  color_green  color_red
8# 0       False        False       True
9# 1        True        False      False
10# 2       False         True      False
11# 3        True        False      False
12
13# Get integer 0/1 instead of boolean
14dummies = pd.get_dummies(df, columns=["color"], dtype=int)
15
16# Drop first to avoid multicollinearity (for linear models)
17dummies = pd.get_dummies(df, columns=["color"], drop_first=True)

Key characteristics:

• Returns a DataFrame with new binary columns
• Does not remember the mapping — cannot transform new data consistently
• drop_first=True avoids the dummy variable trap for linear models
• Handles NaN by skipping it (no column created) unless dummy_na=True

OneHotEncoder (sklearn)

Scikit-learn's pipeline-compatible one-hot encoder:

1from sklearn.preprocessing import OneHotEncoder
2import numpy as np
3
4ohe = OneHotEncoder(sparse_output=False, handle_unknown="ignore")
5colors = np.array([["red"], ["blue"], ["green"], ["blue"]])
6
7# Fit and transform
8encoded = ohe.fit_transform(colors)
9print(encoded)
10# [[0. 0. 1.]
11#  [1. 0. 0.]
12#  [0. 1. 0.]
13#  [1. 0. 0.]]
14
15print(ohe.categories_)  # [array(['blue', 'green', 'red'], dtype=object)]
16
17# Transform new data (including unknown categories)
18new_data = np.array([["red"], ["yellow"]])
19print(ohe.transform(new_data))
20# [[0. 0. 1.]
21#  [0. 0. 0.]]  ← unknown "yellow" → all zeros

Key characteristics:

• Remembers the mapping via fit() — works in sklearn pipelines
• handle_unknown="ignore" produces all-zero rows for unseen categories
• Returns a sparse matrix by default (sparse_output=True)
• Expects 2D input (column vector), not 1D
• drop="first" avoids multicollinearity

Comparison Table

When to Use Each

1# pd.factorize — quick integer encoding during exploration
2codes, uniques = pd.factorize(df["category"])
3
4# LabelEncoder — encoding target variable (y)
5le = LabelEncoder()
6y_encoded = le.fit_transform(y)
7
8# pd.get_dummies — quick one-hot for EDA or simple models
9X = pd.get_dummies(df, columns=["color", "size"])
10
11# OneHotEncoder — production ML pipelines
12from sklearn.compose import ColumnTransformer
13from sklearn.pipeline import Pipeline
14
15preprocessor = ColumnTransformer([
16    ("cat", OneHotEncoder(handle_unknown="ignore"), ["color", "size"]),
17])
18pipeline = Pipeline([
19    ("preprocess", preprocessor),
20    ("model", LogisticRegression()),
21])
22pipeline.fit(X_train, y_train)
23pipeline.predict(X_test)  # Handles unknown categories gracefully

Common Pitfalls

• Using LabelEncoder on features: LabelEncoder assigns ordinal integers (0, 1, 2) which imply an ordering. Tree-based models handle this fine, but linear models and neural networks interpret "red=2" as twice "blue=1". Use OneHotEncoder for nominal features in these models.
• pd.get_dummies creating inconsistent columns: If the training set has ["red", "blue"] and the test set has ["red", "green"], get_dummies creates different columns for each. Use OneHotEncoder.fit() on training data to ensure consistent columns.
• LabelEncoder crashing on unknown categories: le.transform(["yellow"]) raises ValueError if "yellow" was not in the training data. OneHotEncoder with handle_unknown="ignore" handles this gracefully by outputting all zeros.
• Forgetting drop_first for linear models: Including all one-hot columns creates perfect multicollinearity (the columns sum to 1). This causes issues for linear regression and logistic regression. Use drop_first=True or drop="first".
• OneHotEncoder expecting 2D input: ohe.fit_transform(["red", "blue"]) fails because OneHotEncoder expects a 2D array. Reshape with np.array(data).reshape(-1, 1) or pass a DataFrame column as df[["color"]].

Summary

• pd.factorize: Quick integer encoding, no memory of mapping, order-of-appearance labels
• LabelEncoder: Integer encoding with fit/transform, alphabetical order, best for target variables
• pd.get_dummies: Quick one-hot encoding in pandas, no fit/transform, good for exploration
• OneHotEncoder: Pipeline-compatible one-hot encoding, handles unknown categories, production-ready
• Use OneHotEncoder in sklearn pipelines for consistent encoding between train and test sets
• Use drop_first=True or drop="first" for linear models to avoid multicollinearity