Introduction

One-hot encoding must be learned from the training data and then reused unchanged for validation, test, and production inputs. If you fit separate encoders on each split, the column meanings can drift and your model will see inconsistent feature vectors.

The Core Rule: Fit Once, Reuse Everywhere

For categorical features, the encoder needs a fixed mapping from category name to output columns. The correct workflow is:

1. fit the encoder on the training set only
2. transform validation and test data with that fitted encoder
3. deploy the same fitted encoder with the model in production

This avoids leakage from validation or test data into training and keeps the feature space stable.

Why Separate Fitting Is Wrong

Suppose the training set contains categories red, blue, and green, but the validation set happens to contain only blue and green.

If you fit one-hot encoding separately on each split, the encoded columns can differ in both count and order. The model then receives incompatible inputs even though the raw column name looks the same.

That is the real reason the encoder must be treated as part of the trained pipeline.

A Correct scikit-learn Example

1import pandas as pd
2from sklearn.preprocessing import OneHotEncoder
3
4train = pd.DataFrame({"color": ["red", "blue", "green", "blue"]})
5valid = pd.DataFrame({"color": ["green", "blue"]})
6test = pd.DataFrame({"color": ["red", "green"]})
7
8encoder = OneHotEncoder(handle_unknown="ignore", sparse_output=False)
9encoder.fit(train[["color"]])
10
11x_train = encoder.transform(train[["color"]])
12x_valid = encoder.transform(valid[["color"]])
13x_test = encoder.transform(test[["color"]])
14
15print(encoder.get_feature_names_out(["color"]))
16print(x_train)
17print(x_valid)
18print(x_test)

The same fitted encoder is reused on every split, so the column layout stays consistent.

What About Production Data?

Production data should not create a new encoding scheme. It should be transformed by the same encoder artifact that was fitted during training.

That usually means saving the full preprocessing pipeline and loading it alongside the model.

1import joblib
2from sklearn.pipeline import Pipeline
3from sklearn.preprocessing import OneHotEncoder
4from sklearn.linear_model import LogisticRegression
5
6pipeline = Pipeline([
7    ("encoder", OneHotEncoder(handle_unknown="ignore")),
8    ("model", LogisticRegression(max_iter=200)),
9])
10
11joblib.dump(pipeline, "model_pipeline.joblib")
12loaded = joblib.load("model_pipeline.joblib")

Now production requests go through the same encoder configuration automatically.

Handling Unseen Categories

Production data may contain categories that never appeared during training. That is why handle_unknown="ignore" is often a good choice for one-hot encoding.

With that setting, unseen categories map to all zeros across that feature's learned columns instead of crashing the pipeline.

That behavior is usually safer operationally, but it also means the model cannot distinguish different unseen values from one another. If unseen categories are common, the data collection or feature design may need improvement.

Validation And Test Sets Still Matter

Even though you do not fit on validation or test data, those sets are where you discover whether the training-time category coverage was adequate.

If many rows in validation or test hit unseen categories, that is a signal that:

• the training sample may be too narrow
• the feature may be too high-cardinality
• a different encoding strategy might be better

So the rule is not "ignore validation categories." The rule is "do not let them redefine the encoding."

Pipelines Prevent Mistakes

In practice, the cleanest solution is to put preprocessing and modeling in one pipeline. That prevents accidental split-specific preprocessing.

1import pandas as pd
2from sklearn.compose import ColumnTransformer
3from sklearn.pipeline import Pipeline
4from sklearn.preprocessing import OneHotEncoder
5from sklearn.linear_model import LogisticRegression
6
7X = pd.DataFrame({
8    "color": ["red", "blue", "green", "blue"],
9    "weight": [1.2, 0.9, 1.5, 1.1],
10})
11y = [1, 0, 1, 0]
12
13preprocess = ColumnTransformer([
14    ("cat", OneHotEncoder(handle_unknown="ignore"), ["color"]),
15    ("num", "passthrough", ["weight"]),
16])
17
18model = Pipeline([
19    ("preprocess", preprocess),
20    ("clf", LogisticRegression(max_iter=200)),
21])
22
23model.fit(X, y)

This is the production-friendly approach because the exact same transformations are bundled with the estimator.

Common Pitfalls

The most common mistake is fitting separate one-hot encoders on train, validation, and test data. That creates incompatible feature spaces.

Another mistake is fitting the encoder before the train-validation-test split, which leaks category knowledge from held-out data.

Developers also forget about unseen categories in production and deploy an encoder that throws errors on new values.

Finally, do not treat preprocessing as disposable notebook code. The encoder is part of the trained model contract and needs to be versioned and deployed with it.

Summary

• Fit the one-hot encoder on the training set only.
• Reuse that fitted encoder for validation, test, and production data.
• Do not fit separate encoders on different splits.
• Use handle_unknown="ignore" when production data may contain unseen categories.
• Save and deploy the preprocessing pipeline together with the model.