Implement logistic regression from scratch

Question

Write a clean implementation of logistic regression without using ML libraries.

Codemia · Accepted Answer

Logistic regression is a supervised learning algorithm used for binary classification problems. Unlike linear regression, which predicts continuous outputs, logistic regression predicts the probability that a given input belongs to a particular category. The core of logistic regression is the logistic function (or sigmoid function) defined as:

$$
\sigma(z) = \frac{1}{1 + e^{-z}}
$$

where $ z = w^T x + b $ (with $ w $ being the weights, $ x $ the input features, and $ b $ the bias). The output of the sigmoid function is a value between 0 and 1, representing the probability of the positive class. The model is trained using the maximum likelihood estimation to find the optimal weights that maximize the likelihood of the observed data.

The training of logistic regression involves optimizing the cost function, typically the binary cross-entropy loss:

$$
L(w, b) = -\frac{1}{N} \sum_{i=1}^{N} [y_i \log(h(x_i)) + (1 - y_i) \log(1 - h(x_i))]
$$

where $ h(x_i) = \sigma(w^T x_i + b) $ is the predicted probability for the $ i^{th} $ instance and $ N $ is the total number of samples. We use gradient descent to minimize this loss by iteratively updating the weights and bias:

$$
w := w - \alpha \frac{\partial L}{\partial w}, \quad b := b - \alpha \frac{\partial L}{\partial b}
$$

where $ \alpha $ is the learning rate. Regularization techniques such as L1 (Lasso) and L2 (Ridge) can be applied to avoid overfitting by adding penalties to the cost function.

In a company like Meta, logistic regression can be used for various tasks, such as detecting spam content or classifying user interactions. For example, suppose we want to predict whether a user will click on an advertisement based on their profile and behavior data. Deploying logistic regression in this context requires careful consideration of class imbalance, which is common in click-through data where negative samples vastly outnumber positive samples. Techniques like oversampling, undersampling, or using appropriate evaluation metrics (like AUC-ROC) are crucial in these scenarios.

Logistic regression is relatively simple and interpretable, making it a good starting point for binary classification. However, it may underfit complex datasets with non-linear relationships between features and the target variable. Regularization (L1 or L2) can help mitigate overfitting, but the choice between them depends on the feature set; L1 can promote sparsity, useful in high-dimensional datasets, while L2 tends to better handle multicollinearity. In cases where the relationship is non-linear, it may be wise to consider more complex models like decision trees or neural networks.

A strong candidate would discuss hyperparameter tuning strategies, such as grid search or random search for the learning rate and the regularization strength. They might also address the importance of cross-validation for robust model evaluation, especially in the presence of class imbalance. Furthermore, discussing how to handle feature scaling (e.g., standardization) and dimensionality reduction (e.g., PCA) can enhance the model's performance. Finally, integrating logistic regression with ensemble methods like bagging or boosting could be a valuable consideration for improving predictive power in real-world applications.

Implement logistic regression from scratch

Meta

What the Interviewer Expects

Key Topics to Cover

How to Approach This

Possible Follow-up Questions

Sharpen Your Skills on Codemia

Sample Answer

Core Concept: Logistic Regression

How It Works: Mathematical Mechanism

Submit Your Answer

Meta Data Scientist Interview Guide

Related Questions

Compare attention mechanism vs cross-validation

Design an ML pipeline for fraud detection

Implement naive bayes from scratch

Explain dropout and its applications

Compare feature importance vs RLHF