Back-Propagation Notes

Back-Propagation in Neural Networks

COMP3211 - The Department of Computer Science & Engineering

Introduction to Back-Propagation

Back-propagation is a fundamental concept in neural networks, integral to training multi-layer networks using nonlinear activation functions. As the complexity of networks increases, so does the need for more sophisticated learning algorithms. Back-propagation specifically employs the idea of gradient descent, an optimization technique that adjusts the weights of the network based on the calculated error.

Concept of Gradient Descent

The core principle of back-propagation is gradient descent, which involves the following steps:

1. Initialization: Start with an initial value for the weights, denoted as w.

2. Iterative Process: Repeat the process until convergence is achieved.

3. Compute Gradient: Calculate the gradient vector of the error (loss) function for the current weights w.

4. Update Weights: Adjust the weights by moving in the opposite direction of the gradient.

Computing the Gradient

The process of calculating the gradient involves the following units:

• Input Units

• Hidden Units

• Output Units
  For the network, let’s denote:

• Output from the output unit as O_i

• Weight connecting the input unit to the output unit as W_{j,i}

• Activation of the hidden unit as a_j

• Weight connecting hidden unit to output unit as W_{k,j}

• Input to the network as I_k

To define an error function, it is essential that the function is differentiable with respect to the network outputs. Since neural networks use differentiable activation functions, the outputs become differentiable as functions of the inputs and weights. Consequently, the error can be expressed as a differentiable function of the weights, allowing the application of the chain rule:
$\frac{dy}{dx} = \frac{dy}{du} \cdot \frac{du}{dx}$

Weight Update Rule

The process of back-propagation effectively involves propagating the error backward through the network. This requires the computation of a gradient vector based on the error, allowing adjustments to the weights from the output layer back to the input layer. The flow of activity is as follows:

• Out put/Response: The response generated is a result of the error being propagated backward.

• Middle or Hidden Layer: Adjustments and modifications to weights occur here based on the back-propagated error.

• Input Layer: Although indirect, the input also plays a role through the weights adjusted by this back-propagation process.

Optimization Considerations

Optimization in neural networks can be complex due to various factors. The loss function, denoted as TrainLoss(V, w), is subject to optimization as outlined:

• Prediction Models: Linear predictors exhibit a convex optimization landscape, while neural networks typically result in a non-convex landscape. Optimization in neural networks is in principle hard due to the numerous local minima.

Practical Details in Back-Propagation

When implementing back-propagation, several practical considerations must be taken into account:

• Weight Initialization: It is recommended to initialize weight values to small random values to enhance convergence.

• Speeding Up Training:

  1. Momentum Term: Introduce inertia or momentum in the weight updates as follows:
     $\Delta w<em>{ji}(t + 1) = -\eta \frac{\partial E}{\partial w</em>{ji}} + \alpha \Delta w_{ji}(t)$
     Here, 0 < α < 1 is the momentum parameter (for example, α = 0.9).

  2. Dynamic Adaptation of Learning Rate: Adapt the learning rate η dynamically during training.

  3. Higher-Order Information: Utilize higher-order derivatives for analyzing the error surface.

  4. Sophisticated Optimization Algorithms: Implement advanced optimization algorithms to prevent complications during training.

Addressing Local Minima

One significant challenge involved in neural network training is that the error surface may contain multiple local minima, leading to solutions that are not necessarily globally optimal. The gradient descent method guarantees convergence to a local minimum; however, not to the global minimum. To escape locally optimal solutions, the following strategy can be employed:

• Training Multiple Networks: Initiate multiple networks with varying random weights using the same dataset, thus promoting diversification in exploring the error surface.

Summary

In summary, computation graphs are instrumental in visualizing and comprehending gradients throughout the back-propagation process. Back-propagation serves as a general-purpose algorithm for calculating gradients in neural networks, thus playing a pivotal role in efficiently training neural models and enhancing their performance.