L4_Training deep neural networks

Neural Network Parameters

The parameters of a neural network are crucial for its functionality and learning process. These parameters are adjusted during training to minimize the difference between the network's predictions and the actual values.

• Weights: Each connection between neurons has an associated weight. These weights determine the strength of the connection, influencing how much a particular input affects the neuron's output.

• Biases: Each neuron has a bias term. The bias allows the neuron to activate even when the weighted sum of its inputs is zero. It shifts the activation function, providing additional flexibility.

Parameter Optimisation

Optimizing parameters is a critical task, especially for large networks with numerous weights and biases. Efficient optimization techniques are necessary to train these networks effectively.

For large networks, a clever way to optimize parameters simultaneously is needed. Backpropagation is the solution, which relies on the network output being differentiable with respect to its parameters. Differentiability ensures that gradients can be computed to update the parameters.

Backpropagation Steps

Backpropagation is an algorithm used to iteratively adjust the parameters of a neural network to minimize the loss function. It involves several steps:

1. Initialization: Randomly initialize parameters. Proper initialization helps in faster convergence and avoids issues like vanishing or exploding gradients.

2. Forward Pass: Run a forward pass on a mini-batch and compute predictions, keeping track of the output for each node. The forward pass calculates the output of the network for a given input.

3. Loss Computation: Compute the loss, which measures the error between the predicted output and the actual target. Common loss functions include mean squared error (MSE) and cross-entropy loss.

4. Gradient Computation: Compute how much each parameter contributed to the loss by computing the gradient for each parameter. Start at the last layer and move backwards (backwards pass). This involves calculating the partial derivatives of the loss function with respect to each parameter.

5. Gradient Descent: Run one step of gradient descent to find better values for all parameters. Update the parameters in the opposite direction of the gradient to minimize the loss.

Gradient Descent

Gradient descent is an optimization algorithm used to find the minimum of a function. In the context of neural networks, it is used to update the parameters to minimize the loss function.

The gradient points towards the direction of the maximum increase of the loss function. To find the minimum, the negative gradient is needed.

\nabla L(\theta) = \begin{bmatrix} \frac{\partial L}{\partial \theta0} \ \vdots \ \frac{\partial L}{\partial \thetan} \end{bmatrix}

Where:

• \nabla L(\theta) is the gradient vector in parameter space, indicating the direction of the steepest ascent of the loss function.

Learning Rate

The learning rate ($\eta$) is a hyperparameter that controls the step size during gradient descent. Selecting an appropriate learning rate is crucial for effective training.

The learning rate ($\eta$) is a hyperparameter. It determines the step size towards the optimal solution:

\theta{n+1} = \thetan - \eta \nabla L

• Too small: Slow convergence. The training process takes a long time to reach the optimal solution.

• Just right: Optimal convergence. The training process efficiently reaches the optimal solution.

• Too high: No convergence. The training process oscillates and fails to converge to the optimal solution.

Practical Problems

During optimization, several practical problems can arise, affecting the training process and the quality of the model.

• Local Minimum: Leads to bad predictions. The optimization process gets stuck in a local minimum, which is not the global minimum of the loss function.

• Plateau: Leads to slow convergence. The optimization process slows down significantly in flat regions of the loss function.

Vanishing and Exploding Gradients

Vanishing and exploding gradients are common problems that occur during the training of deep neural networks. These issues can hinder the learning process and prevent the network from converging.

During backpropagation, gradients can experience amplification effects:

1. Vanishing Gradients: Gradients go towards zero, resulting in no learning. This occurs when the gradients become extremely small, preventing the weights from being updated effectively.

2. Exploding Gradients: Gradients go towards infinity, resulting in no learning. This occurs when the gradients become extremely large, causing instability in the training process.

Improved training stability can be achieved if the variance of the output of a layer is similar to the variance of the input. Maintaining consistent variance helps in propagating gradients effectively.

Tricks to Avoid Vanishing/Exploding Gradients

Several techniques can be employed to mitigate the vanishing and exploding gradients problems, enhancing the stability and efficiency of neural network training.

1. Choose a non-saturating activation function (like ReLU). ReLU and its variants (e.g., Leaky ReLU) help in maintaining healthy gradients by avoiding saturation.

2. Initialize each layer’s parameters according to the number of input and output connections. Proper initialization strategies ensure that the initial gradients are well-behaved.

   • Glorot initialization: Used with None, tanh, sigmoid, softmax activation functions. This initialization is designed to keep the variance of the activations consistent across layers.

   • He (Kaiming) initialization: Used with ReLU, GELU, Swish activation functions. This initialization is tailored for ReLU-based networks to maintain stable gradients.

3. Add normalization layers to the model. Normalization layers, such as Batch Normalization, help in stabilizing the learning process by normalizing the inputs to each layer.

Normalization Layers

Normalization layers are used to standardize the inputs to a layer, which can help in faster convergence and better generalization.

• Batch Normalization: A common normalization layer that normalizes the activations of each layer per mini-batch. It reduces internal covariate shift and stabilizes training.

Regularisation

Regularization techniques are used to prevent overfitting by adding a penalty term to the loss function, discouraging complex models.

L1 and L2 regularization can be applied to neural network nodes. These methods add a penalty term to the loss function based on the magnitude of the weights.

Regularization Options:

layer = keras.layers.Dense(
    units=64,
    kernel_regularizer=keras.regularizers.L1L2(l1=1e-5, l2=1e-4),
    bias_regularizer=keras.regularizers.L2(1e-4),
    activity_regularizer=keras.regularizers.L1(1e-5)
)

Dropout Regularisation

Dropout regularization is a technique where, during training, randomly selected neurons are ignored. This prevents the network from relying too much on any single neuron and promotes more robust learning.

At each training step, randomly remove a fraction of the neurons to prevent neuron co-adaptation. This forces the network to learn redundant representations, improving generalization.

Spatial dropout is recommended after convolutional layers, which drops entire feature maps. This is particularly effective in reducing overfitting in convolutional neural networks.

Dropout Layers:

keras.layers.Dropout(rate, noise_shape=None, seed=None, **kwargs)
keras.layers.SpatialDropout2D(
    rate, data_format=None, seed=None, name=None, dtype=None
)

Putting Together an Improved Network

Constructing an effective neural network involves careful selection and configuration of layers, activation functions, and regularization techniques.

from keras.layers import (
    Input, Rescaling, Conv2D, BatchNormalization,
    MaxPooling2D, Activation, Dropout, Dense
)

model = keras.Sequential([
    keras.Input(shape=(128, 128, 3)),
    Rescaling(1.0 / 255),
    Conv2D(128, kernel_size=3, kernel_initializer="he_uniform", padding="same"),
    BatchNormalization(),
    Activation("relu"),
    MaxPooling2D(3, padding="same"),
    #
    # (more layers)
    #
    Conv2D(128, kernel_size=3, kernel_initializer="he_uniform", padding="same"),
    BatchNormalization(),
    layers.Activation("relu"),
    MaxPooling2D(3, padding="same"),
    Flatten(),
    Dropout(0.3),
    Dense(num_classes, activation="softmax"),
])

Best Practices

Finding the optimal neural network configuration is challenging due to the vast search space of possible architectures and hyperparameters.

The search space for finding the optimal solution is huge, considering:

• Architecture (layers and nodes): The arrangement and size of layers in the network.

• Parameter initializers: Methods for setting the initial weights and biases.

• Activation functions: Functions that introduce non-linearity into the network.

• Regularization: Techniques to prevent overfitting.

• Optimization algorithm: Algorithms used to update the network's parameters.

• Learning rate and scheduling: The step size for updating parameters and how it changes over time.

Guidelines are meant to save time but are not absolute. These are general recommendations and may need to be adjusted based on the specific problem and dataset.