FastAI is a powerful deep learning library that sits on top of PyTorch, providing high-level APIs to facilitate rapid prototyping and development of deep learning models. One of the key features of FastAI is its ability to simplify complex training processes and deliver state-of-the-art results with minimal code. In this article, we dive into one such sophisticated feature: fit_one_cycle() and the role of the slice(lr) parameter.

Overview of fit_one_cycle()

fit_one_cycle() is a training method in FastAI that leverages the "1-cycle policy," a training schedule introduced by Leslie N. Smith in 2018. This method involves a cyclical variation of the learning rate over the epochs, which often results in faster convergence and improved model performance compared to using a fixed learning rate.

The 1-cycle policy can be broken down into two phases:

1. A "warm-up" phase, where the learning rate is gradually increased.
2. A "cool-down" phase, where the learning rate is decreased.

Additionally, the cyclical schedule can also adjust other hyperparameters, such as momentum, to optimize the learning process.

Understanding the slice(lr)

Parameter

In FastAI, when you call fit_one_cycle() , you typically pass a learning rate (lr ). This is where the slice(lr) parameter plays a crucial role. The slice() function in FastAI provides flexibility in setting different learning rates for different layers of the model, effectively implementing a discriminative learning rate strategy.

What slice()

Does

• slice(lr) Single Learning Rate: When you pass a single value with slice(lr) , FastAI applies this learning rate uniformly to all layers.
• slice(lr1, lr2) Linear Discriminative LR: When two values are passed, such as slice(lr1, lr2) , the learning rates are applied across the model's layers in a linearly spaced manner from lr1 to lr2 . This allows for more fine-grained control, typically applying smaller learning rates to earlier layers of the model and larger rates to the later layers.
• **Choosing lr1 and lr2 **: In practice, it's effective to set lr1 to a fraction (e.g., 1/3) of lr2 , especially useful in transfer learning scenarios where earlier layers have more general features and might not need as aggressive updates.

Technical Explanation

Let's consider a convolutional neural network (CNN) with three layers. Here's how the learning rates might be distributed when using slice(1e-3, 1e-2) :

In this case, the learning rates smoothly transition from 1e-3 to 1e-2 , accommodating a balanced learning approach for each layer.

Example Usage