Theano's HiddenLayer is a core component in building neural networks using the Theano library. Theano, although no longer under active development, laid the foundation for many modern deep learning frameworks such as TensorFlow and PyTorch. It's crucial to understand the workings of its components, such as the HiddenLayer, for historical context and for understanding some foundational aspects of deep learning.

Understanding HiddenLayer in Theano

A HiddenLayer is essentially a fully connected layer that consists of neurons, each receiving input, applying a linear transformation followed by a non-linear activation function. The linear transformation is typically parameterized by weights and biases that are learned during the training of the network. The essential purpose of this layer is to introduce non-linearity into the model, which enhances its ability to make complex predictions.

Components of HiddenLayer

1. Weights ($W$) and Biases ($b$):
   Each neuron in the hidden layer has associated weights and a bias that help in determining the neuron's output.
   • Weights ($W$): These are parameters associated with every input feature that gets adjusted during training. They're initialized using different strategies such as zero initialization, random initialization, Xavier initialization, etc.
   • Biases ($b$): Bias is an additional parameter used to shift the activation function. It helps the model fit the data better by adjusting the output independent of the input.
2. Activation Function ($g$):
   The non-linear activation function applied to the weighted sum of inputs and bias. Activations like sigmoid, tanh, ReLU are commonly used. The choice of activation function impacts the learning and predictive power of the neural network significantly.
3. Linear Transformation:
   The output of the linear transformation for a neuron is typically given by: $$z = Wx + b$$ where $z$ is the scalar input to the activation function, $x$ is the input vector, and $W$ and $b$ are the weights and bias respectively.
4. Output ($a$) from the Activation Function:
   The activated output is computed as: $$a = g(z)$$ where $g$ is the chosen activation function.

Common Activation Functions

Sigmoid Function

• Formula:
  $$g(z) = \frac{1}{1 + e^{-z}}$$
• Properties:
  The sigmoid function maps inputs to a range between 0 and 1, making it useful for binary classification.
• Drawbacks:
  Susceptible to issues like vanishing gradients for large positive or negative inputs which leads to slow convergence during training.

Hyperbolic Tangent (Tanh)

• Formula:
  $$g(z) = \tanh(z) = \frac{e^{z} - e^{-z}}{e^{z} + e^{-z}}$$
• Properties:
  Outputs values between -1 and 1, centered around zero, which generally leads to faster convergence compared to sigmoid.

Rectified Linear Unit (ReLU)

• Formula:
  $$g(z) = \max(0, z)$$
• Properties:
  ReLU introduces non-linearity while maintaining a simple nature, promoting sparsity of activations and computational efficiency.
• Drawbacks:
  It can suffer from the "dying ReLU" problem, where neurons can become inactive during training if they output zero for any input.

Example: Implementing a HiddenLayer in Theano