Introduction

The AdaDelta optimization algorithm, an extension of the classic Gradient Descent method, is often praised for its adaptive learning rate and efficient memory usage. However, its performance can vary depending on the neural network architecture and other components like loss functions and activation layers. One noteworthy scenario is when AdaDelta fails to converge when applied to a Denoising AutoEncoder (DAE) with Mean Squared Error (MSE) loss and ReLU activation. This article delves into the reasons behind this, supported by technical explanations.

Background on AdaDelta

AdaDelta was proposed as an improvement over the AdaGrad algorithm. While AdaGrad scales the learning rate according to the square root of the sum of gradients squared, AdaDelta introduces two main concepts to address AdaGrad's diminishing learning rates over time:

1. RMS Propagation:
   • Unlike AdaGrad, which uses the accumulated sum of previous squared gradients, AdaDelta uses the Root Mean Squared (RMS) of previous gradients. This method adjusts the parameters without requiring a manual learning rate.
2. Delta Accumulation:
   • AdaDelta also keeps track of the updates, using an exponentially decaying average of squared parameter updates, allowing it to adaptively adjust the learning rates.

These characteristics typically make AdaDelta robust across different training scenarios, except in circumstances like the one discussed here.

The Denoising AutoEncoder with MSE `Loss` & ReLU Activation

A Denoising AutoEncoder is a type of autoencoder that reconstructs the original input from a corrupted version. It typically uses a simple architecture containing an encoder and a decoder. The DAE with MSE loss and ReLU activation face challenges when trained with the AdaDelta method.

Key Elements

1. Mean Squared Error (MSE):
   • The MSE loss measures the average squared difference between predicted and actual values. While suitable for regression tasks, its behavior with AdaDelta might result in inefficient learning in certain cases.
2. ReLU Activation:
   • The Rectified Linear Unit (ReLU) is a popular activation function that allows networks to converge more quickly. However, it has a well-known issue called "dying ReLU" where neurons can permanently deactivate.

Technical Explanations

The Challenges with AdaDelta

1. ReLU Induced Sparsity:
   • Sparsity caused by ReLU activations can drastically reduce the gradient signal in forward and backward passes. AdaDelta's reliance on RMS propagation means it might inadequately adjust learning due to the presence of zeros.
2. Insensitivity to Scale in MSE:
   • When using MSE, errors might be too small to produce significant updates. In combination with AdaDelta's adaptive updates, this can cause exceedingly small or zero updates, hindering convergence.
3. Zero Learning Effect:
   • In layers dominated by dead neurons (due to ReLU) and small gradient flow (due to MSE), AdaDelta's no-tuning learning rate can cease any effective learning.

Illustrative Example

Consider a simple DAE with a single hidden layer. Assume the network encounters quiet large input noise. In the early stages of training, ReLU activations may cause many neurons to output zero, while MSE provides nearly zero gradient updates for inactive neurons. The combination causes AdaDelta to quickly decrease learning updates, ultimately ceasing meaningful learning.

Conclusion

While AdaDelta is robust in many scenarios, its over-dependence on gradient magnitudes and adaptive scaling can lead to convergence issues in models with specific properties like Denoising AutoEncoders with MSE loss & ReLU activation. Designing models with activation functions that do not lead to dead neurons, or scaling and normalizing inputs, may ameliorate these problems.

Key Points Summary

Additional Considerations

1. Alternative Activation Functions:
   • Employing alternative activation functions, such as Leaky ReLU or Parametric ReLU, can mitigate the dead neuron problem.
2. Regularization Techniques:
   • Incorporating regularization and dropout can prevent overfitting and promote efficient use of neurons, facilitating convergence.

By understanding the interplay between loss functions, activations, and optimization algorithms like AdaDelta, practitioners can make informed choices that optimize network design and training efficacy.