Week 5 - Regularisation & Neural networks

What is regularisation?

It consists of applying constraints to the amplitude of estimated model parameters.

It is a way to keep the complexity of the model under control.

Objective is to reduce the variance of the model and help with predictor selections.

What does regularisation do?

Penalises large values of parameters in optimisation.

Reduces the sensitivity of the predicted response to variations in testing predictor data.

Automatically carries out best predictor selections.

How does regularisation do this?

Changes the cost function by adding a Jreg = theta^T*theta term to the SSE.

This provides a trade off between fit to data and smoothness.

What is the formula for the best choice of coefficient using regularisation?

How to choose optimal rho?

What is the basic structure of a single layer network?

Composed of nodes connected by directed links.

Each link has a numeric weight associated with it.

Each node first computes a weighted sum of its inputs.

The node applies an activation function to this sum to derive the output.

What are the different types of activation functions?

Threshold function (step).

Piecewise linear.

Sigmoid.

Tanh.

What is error correction learning?

The x’s in the neural network are the features of the dataset and the idea is to tune the weights to achieve an objective function.

This leads to a closed loop feedback learning system.

We tune the weights with the update rule in the image.

What is the structure of a multiple layer network?

How do we train a multiple layer network?

We use backpropagation in combination with the supervised error-correction learning rule.

Backpropagation passes error signals backwards through the network during training to update the weights of the network.

What is the backpropagation formula?

How does back propagation look like in a diagram?

What are the advantages of neural networks?

Flexibility - can be used for a wide range of data and complexity.

Versatility - can be used for a wide range of problems e.g. classification, regression, clustering, pattern recognition, etc.

What is the disadvantage of using neural networks?

Flexibility tends to lead to overfitting.