Introduction

In the realm of supervised machine learning, classification models are pivotal for tasks that require categorizing data points into discrete classes. While tuning these models, you may have encountered the term "sample weights." Sample weights allow you to assign different levels of importance to different data points during model training. This can be particularly useful for handling class imbalance, prioritizing recent data, or emphasizing more reliable observations. This article takes a deep dive into how sample weights function in classification models, their technical implementation, and practical applications.

What are Sample Weights?

Sample weights are a vector of non-negative scalar multipliers that are applied to each observation in your dataset. They modify the role each data point plays in the training process by adjusting the error function or loss function's contribution.

Mathematical Representation

Let's consider a simple binary classification problem where you have a dataset $D = {(x_1, y_1), (x_2, y_2), \ldots, (x_n, y_n)}$, with $x_i$ representing feature vectors and $y_i \in {0, 1}$ representing the corresponding labels. Assume a weight vector $w = [w_1, w_2, \ldots, w_n]$ where each $w_i \geq 0$ corresponds to an observation $(x_i, y_i)$.

If you're using a loss function $L(y_i, f(x_i))$ in your model, where $f$ is the function predicted by the model, the weighted loss becomes:

$$L\_{\text{weighted}} = \sum\_{i=1}^{n} w\_i \cdot L(y\_i, f(x\_i)).$$

In this way, observations with higher weights contribute more to the total loss function than those with lower weights.

Why Use Sample Weights?

Addressing Class Imbalance

In real-world datasets, you might often encounter class imbalance where one class significantly outnumbers the others. Sample weights can correct this imbalance by assigning higher weights to rarer class instances.

For example, in a dataset with a 1:10 ratio of positive to negative examples, simply classifying every instance as negative would yield 90% accuracy. Weighting the positive class higher adjusts the model focus to maximize precision across both classes.

Incorporating Observation Reliability

Not all data points are created equal. Some may result from noisy measurements or unreliable sources. Assigning lower weights to such points makes them less significant in the model's understanding.

Temporal Importance

In time series data or streaming data, newer observations might be more relevant to current predictions. Weights allow you to emphasize recent data over older data.

Implementation in Popular Libraries

Most modern machine learning libraries such as Scikit-learn, XGBoost, and TensorFlow/Keras support sample weights during model training:

Scikit-Learn

In Scikit-learn, you can provide sample weights using the sample_weight parameter available in many estimators. Here's an example of using sample weights with a logistic regression model: