Introduction

Spark drops the last one-hot category by default to avoid a redundant feature column. In linear models, keeping every category column plus an intercept creates perfect multicollinearity, so Spark uses a reference category representation unless you tell it otherwise.

What Dropping The Last Category Actually Means

Suppose a categorical feature has three indexed values:

• '0 for red'
• '1 for green'
• '2 for blue'

If you keep all three one-hot columns, the vectors look like this:

• red becomes [1, 0, 0]
• green becomes [0, 1, 0]
• blue becomes [0, 0, 1]

With dropLast=True, Spark uses only two columns:

• red becomes [1, 0]
• green becomes [0, 1]
• blue becomes [0, 0]

The missing category is still represented. It is simply encoded as the all-zero baseline vector.

Why Spark Does This By Default

For linear regression, logistic regression, and other linear models, full one-hot encoding plus an intercept creates a dependency between columns. One category can always be derived from the others, so the design matrix is not full rank.

That redundancy is often called the dummy-variable trap. Dropping one category solves it cleanly because the omitted category becomes the reference level against which the other categories are measured.

This is why Spark's default is sensible for many modeling pipelines. It reduces dimensionality slightly and avoids a common source of unstable coefficients in linear models.

Example In Spark

In PySpark, the usual pipeline is StringIndexer followed by OneHotEncoder:

1from pyspark.sql import SparkSession
2from pyspark.ml.feature import StringIndexer, OneHotEncoder
3
4spark = SparkSession.builder.getOrCreate()
5
6df = spark.createDataFrame(
7    [("red",), ("green",), ("blue",)],
8    ["color"]
9)
10
11indexer = StringIndexer(inputCol="color", outputCol="color_index")
12indexed = indexer.fit(df).transform(df)
13
14encoder = OneHotEncoder(
15    inputCols=["color_index"],
16    outputCols=["color_vec"],
17    dropLast=True
18)
19
20encoded = encoder.fit(indexed).transform(indexed)
21encoded.select("color", "color_index", "color_vec").show(truncate=False)

If you set dropLast=False, Spark keeps the full set of indicator positions:

1encoder = OneHotEncoder(
2    inputCols=["color_index"],
3    outputCols=["color_vec"],
4    dropLast=False
5)

That can be useful when you explicitly want a full indicator vector, for example when exporting features to another system or working with models that do not care about collinearity in the same way.

When You Might Want dropLast=False

Tree-based models are generally less sensitive to the same linear-algebra issue that affects linear models. In those cases, keeping all categories is often acceptable.

You may also prefer dropLast=False when:

• interpretability requires every category to have its own explicit bit
• downstream consumers expect a full one-hot vector
• you are not fitting a linear model with an intercept term

The important point is that Spark's default is a modeling convenience, not a universal mathematical law.

Do Not Confuse Indexing Order With Category Importance

OneHotEncoder works on indexed values, usually produced by StringIndexer. The category that gets dropped is the last indexed category, which depends on how indexing was assigned.

That means the reference category is not chosen semantically unless you control the indexing behavior yourself. If the baseline category matters for interpretation, inspect the indexer model and verify which label maps to which index.

Common Pitfalls

• Thinking the dropped category disappears entirely instead of becoming the all-zero reference vector.
• Forgetting that the dropped category depends on the index order produced earlier in the pipeline.
• Using full one-hot encoding with linear models and then being surprised by redundant columns.
• Assuming tree models and linear models react to encoded features in the same way.
• Changing dropLast without considering how downstream feature consumers interpret the vector size.

Summary

• Spark drops the last category by default to avoid redundant one-hot columns.
• The omitted category is still represented as the all-zero baseline vector.
• This default is especially helpful for linear models, where full one-hot encoding can cause multicollinearity.
• If you need every category explicitly, set dropLast=False and make that choice deliberately.