Implementing dropout from scratch

deep learning

dropout

neural networks

machine learning

programming tutorial

Implementing dropout from scratch

ML System Design practice on Codemia

Design recommenders, ranking systems and training pipelines the way ML interviews actually ask for them, with worked solutions.

Practice ML system design

Deep Learning Machine Learning

Introduction

Dropout is a regularization technique that randomly sets a fraction of neuron outputs to zero during training, forcing the network to learn redundant representations and reducing overfitting. During inference, all neurons are active but their outputs are scaled down. Implementing dropout from scratch requires generating a random binary mask during the forward pass, applying it to the activations, and scaling the output so that expected values remain consistent between training and inference.

How Dropout Works

During training, each neuron's output is independently set to zero with probability p (the dropout rate). During inference, no neurons are dropped, but outputs are multiplied by (1 - p) to compensate for the missing neurons during training:

Training:  output = activation * mask / (1 - p)    [inverted dropout]
Inference: output = activation                       [no change needed]

The most common implementation is "inverted dropout," which scales during training (dividing by 1 - p) so that no scaling is needed at inference time.

NumPy Implementation

python

1import numpy as np
2
3class Dropout:
4    def __init__(self, p=0.5):
5        """
6        p: probability of dropping a neuron (dropout rate)
7        """
8        self.p = p
9        self.mask = None
10
11    def forward(self, x, training=True):
12        if not training:
13            return x
14
15        # Generate binary mask: 1 with probability (1 - p), 0 with probability p
16        self.mask = (np.random.rand(*x.shape) > self.p).astype(np.float64)
17
18        # Apply mask and scale by 1/(1-p) (inverted dropout)
19        return x * self.mask / (1 - self.p)
20
21    def backward(self, grad_output):
22        # Gradient flows through kept neurons, blocked for dropped neurons
23        return grad_output * self.mask / (1 - self.p)

Usage in a simple network:

python

1# Forward pass
2x = np.array([[1.0, 2.0, 3.0, 4.0, 5.0]])
3
4dropout = Dropout(p=0.5)
5
6# Training mode — some neurons are zeroed out
7out_train = dropout.forward(x, training=True)
8print(out_train)  # e.g., [0.0, 4.0, 6.0, 0.0, 10.0]
9
10# Inference mode — all neurons active, no scaling
11out_test = dropout.forward(x, training=False)
12print(out_test)   # [1.0, 2.0, 3.0, 4.0, 5.0]

PyTorch Implementation

python

1import torch
2import torch.nn as nn
3
4class MyDropout(nn.Module):
5    def __init__(self, p=0.5):
6        super().__init__()
7        self.p = p
8
9    def forward(self, x):
10        if not self.training:
11            return x
12
13        # Bernoulli mask: 1 with probability (1 - p)
14        mask = torch.bernoulli(torch.full_like(x, 1 - self.p))
15
16        # Apply inverted dropout
17        return x * mask / (1 - self.p)
18
19# Compare with PyTorch's built-in
20model_custom = nn.Sequential(
21    nn.Linear(10, 20),
22    MyDropout(p=0.3),
23    nn.Linear(20, 5)
24)
25
26model_builtin = nn.Sequential(
27    nn.Linear(10, 20),
28    nn.Dropout(p=0.3),
29    nn.Linear(20, 5)
30)
31
32# Both behave identically
33x = torch.randn(32, 10)
34model_custom.train()
35out_train = model_custom(x)
36
37model_custom.eval()
38out_test = model_custom(x)  # No dropout applied

Full Neural Network with Dropout

python

1import numpy as np
2
3class SimpleNetwork:
4    def __init__(self, input_dim, hidden_dim, output_dim, dropout_rate=0.5):
5        # Xavier initialization
6        self.W1 = np.random.randn(input_dim, hidden_dim) * np.sqrt(2.0 / input_dim)
7        self.b1 = np.zeros((1, hidden_dim))
8        self.W2 = np.random.randn(hidden_dim, output_dim) * np.sqrt(2.0 / hidden_dim)
9        self.b2 = np.zeros((1, output_dim))
10        self.dropout = Dropout(p=dropout_rate)
11
12    def forward(self, X, training=True):
13        # Layer 1
14        self.z1 = X @ self.W1 + self.b1
15        self.a1 = np.maximum(0, self.z1)  # ReLU
16
17        # Dropout after activation
18        self.a1_dropped = self.dropout.forward(self.a1, training=training)
19
20        # Layer 2
21        self.z2 = self.a1_dropped @ self.W2 + self.b2
22        return self.z2
23
24    def predict(self, X):
25        return self.forward(X, training=False)
26
27# Training
28net = SimpleNetwork(784, 256, 10, dropout_rate=0.5)
29output_train = net.forward(X_train, training=True)   # Dropout active
30output_test = net.predict(X_test)                      # Dropout inactive

Standard vs Inverted Dropout

python

1# Standard dropout (scale at test time)
2class StandardDropout:
3    def __init__(self, p=0.5):
4        self.p = p
5
6    def forward(self, x, training=True):
7        if training:
8            mask = (np.random.rand(*x.shape) > self.p).astype(np.float64)
9            return x * mask  # No scaling during training
10        else:
11            return x * (1 - self.p)  # Scale down at test time
12
13# Inverted dropout (scale at train time) — PREFERRED
14class InvertedDropout:
15    def __init__(self, p=0.5):
16        self.p = p
17
18    def forward(self, x, training=True):
19        if training:
20            mask = (np.random.rand(*x.shape) > self.p).astype(np.float64)
21            return x * mask / (1 - self.p)  # Scale up during training
22        else:
23            return x  # No change at test time

Inverted dropout is preferred because inference requires no modification — the model outputs the correct values without any scaling step.

Dropout Variants

python

1# Spatial Dropout (for CNNs) — drops entire feature maps
2class SpatialDropout2D:
3    def __init__(self, p=0.5):
4        self.p = p
5
6    def forward(self, x, training=True):
7        # x shape: (batch, channels, height, width)
8        if not training:
9            return x
10        # One mask value per channel, broadcast across spatial dims
11        mask_shape = (x.shape[0], x.shape[1], 1, 1)
12        mask = (np.random.rand(*mask_shape) > self.p).astype(np.float64)
13        return x * mask / (1 - self.p)
14
15# DropConnect — drops weights instead of activations
16class DropConnect:
17    def __init__(self, p=0.5):
18        self.p = p
19
20    def forward(self, x, W, training=True):
21        if not training:
22            return x @ W
23        mask = (np.random.rand(*W.shape) > self.p).astype(np.float64)
24        W_dropped = W * mask / (1 - self.p)
25        return x @ W_dropped

Common Pitfalls

Forgetting to disable dropout during inference: If dropout stays active during evaluation, outputs are noisy and accuracy drops. Always call model.eval() in PyTorch or pass training=False in custom implementations before running inference or validation.
Applying dropout before the activation function: Dropout should be applied after the activation (e.g., after ReLU), not before. Applying it before means the activation function sees scaled inputs, which changes the nonlinearity's behavior.
Using too high a dropout rate: Dropout rate above 0.5 drops more than half the neurons, causing severe underfitting. Common rates are 0.2-0.3 for input layers and 0.5 for hidden layers. Start with 0.5 and reduce if training loss is too high.
Not using inverted dropout: Standard dropout requires multiplying outputs by (1 - p) at test time. Forgetting this scaling step means test predictions are systematically too large. Inverted dropout avoids this by scaling during training.
Applying dropout to the output layer: Dropping neurons in the final layer randomly zeroes out class predictions, degrading performance. Dropout belongs in hidden layers only — never on the input or output layer.

Summary

Dropout randomly zeroes neuron outputs during training to prevent overfitting
Inverted dropout scales by 1/(1-p) during training so inference needs no modification
The backward pass blocks gradients through dropped neurons (gradient * mask)
Use dropout rates of 0.2-0.5 for hidden layers, never on the output layer
Always disable dropout during inference (model.eval() or training=False)
Spatial dropout drops entire feature maps in CNNs; DropConnect drops weights instead of activations

Tackling System Design Interview Problems

A short course that equips you with the skills to approach system design interviews methodically.

Start the free course

Track what you have practised

A free account saves your progress, solutions and study plan across every problem on Codemia.

ML System Design practice on Codemia

Design recommenders, ranking systems and training pipelines the way ML interviews actually ask for them, with worked solutions.

Practice ML system design

Implementing dropout from scratch

ML System Design practice on Codemia

Introduction

How Dropout Works

NumPy Implementation

PyTorch Implementation

Full Neural Network with Dropout

Standard vs Inverted Dropout

Dropout Variants

Common Pitfalls

Summary

Related reading

Tackling System Design Interview Problems

Track what you have practised

ML System Design practice on Codemia