Introduction

L1 regularization adds a penalty proportional to the absolute value of model weights, which encourages sparsity and can make feature selection easier. In TensorFlow, the exact implementation depends on whether you want the penalty attached automatically through a layer regularizer or added manually to the training loss.

The L1-Regularized Loss

Conceptually, the total loss is:

total_loss = data_loss + lambda * sum(abs(weights))

The lambda term controls how strongly the model is pushed toward zero-valued weights. A small value nudges the weights. A large value can dominate training and underfit the data.

The main practical question is which variables should be regularized. In most neural networks, you regularize kernel weights and leave biases alone.

The Easiest Approach In Keras

For TensorFlow 2 projects built with Keras layers, use kernel_regularizer:

1import tensorflow as tf
2
3model = tf.keras.Sequential([
4    tf.keras.layers.Dense(
5        64,
6        activation="relu",
7        kernel_regularizer=tf.keras.regularizers.L1(1e-4),
8        input_shape=(20,),
9    ),
10    tf.keras.layers.Dense(
11        1,
12        activation="sigmoid",
13        kernel_regularizer=tf.keras.regularizers.L1(1e-4),
14    ),
15])
16
17model.compile(
18    optimizer="adam",
19    loss="binary_crossentropy",
20    metrics=["accuracy"],
21)

When you use layer regularizers, TensorFlow automatically adds the penalty terms to model.losses. During fit, those extra terms are included in the optimized loss.

This is the cleanest option when your model is built from standard layers and you want the regularization logic attached directly to the layer definitions.

Adding The L1 Term Manually

If you write a custom training loop, calculate the penalty yourself so you can see exactly what is being optimized:

1import tensorflow as tf
2
3lambda_l1 = 1e-4
4loss_fn = tf.keras.losses.BinaryCrossentropy()
5optimizer = tf.keras.optimizers.Adam()
6
7
8for x_batch, y_batch in train_ds:
9    with tf.GradientTape() as tape:
10        predictions = model(x_batch, training=True)
11        data_loss = loss_fn(y_batch, predictions)
12
13        l1_penalty = tf.add_n([
14            tf.reduce_sum(tf.abs(var))
15            for var in model.trainable_variables
16            if "bias" not in var.name
17        ])
18
19        total_loss = data_loss + lambda_l1 * l1_penalty
20
21    gradients = tape.gradient(total_loss, model.trainable_variables)
22    optimizer.apply_gradients(zip(gradients, model.trainable_variables))

This approach is explicit and easy to adapt. For example, you can regularize only certain layers or log the penalty separately from the data loss.

Choosing Which Variables To Penalize

You do not have to regularize every trainable variable. In many models, regularizing kernels is enough:

1l1_penalty = tf.add_n([
2    tf.reduce_sum(tf.abs(var))
3    for var in model.trainable_variables
4    if "kernel" in var.name
5])

That pattern keeps the penalty focused on the main weights while skipping biases, normalization parameters, and other variables that usually should not be shrunk aggressively.

If you use subclassed models, another robust option is to gather weights directly from layers you choose to regularize instead of filtering only by variable name.

Inspecting The Loss Components

It helps to print both parts of the loss during tuning:

1with tf.GradientTape() as tape:
2    predictions = model(x_batch, training=True)
3    data_loss = loss_fn(y_batch, predictions)
4    l1_penalty = tf.add_n([
5        tf.reduce_sum(tf.abs(var))
6        for var in model.trainable_variables
7        if "kernel" in var.name
8    ])
9    total_loss = data_loss + lambda_l1 * l1_penalty
10
11print(
12    "data_loss=", float(data_loss.numpy()),
13    "l1_penalty=", float(l1_penalty.numpy()),
14    "total_loss=", float(total_loss.numpy()),
15)

If lambda_l1 * l1_penalty is tiny compared with data_loss, the regularizer is probably too weak to matter. If it is much larger, the model may collapse toward overly sparse weights.

Common Pitfalls

The most common mistake is double-counting the penalty by using both kernel_regularizer and a manual L1 term on the same variables. Pick one approach unless you intentionally want both and have verified the math.

Another issue is regularizing the wrong parameters. Biases and normalization scales usually do not benefit from the same L1 treatment as dense or convolution kernels.

Developers also often compare lambda values across different codebases without checking loss scaling. The effective strength depends on batch size, reduction mode, and whether the base loss is averaged or summed. Tune it in the context of your training setup.

Finally, L1 regularization encourages sparse weights, not guaranteed feature selection in every deep network. It is a useful bias, but it does not replace good data preparation or model validation.

Summary

• L1 regularization adds lambda * sum(abs(weights)) to the training loss.
• In Keras, kernel_regularizer=tf.keras.regularizers.L1(...) is the simplest implementation.
• In custom loops, compute the penalty explicitly and add it to the base loss.
• Regularize kernel weights intentionally instead of blindly penalizing every variable.
• Watch out for double-counting and tune the regularization strength against your actual loss scale.