Week 5 - Regularisation and Neural Networks

What is Regularisation?

Applying constraints to the amplitude of estimated model parameters

A way to keep the complexity of the model under control

The objective is to reduce the variance of the model, and at the same time, help with predictor selections.

Applying constraints to the estimated model parameters can:

1. Reduce the sensitivity of the predicted response to variations in testing predicto data

2. Automatically carry out best predictor selections

   1. Those that do not depend on how strong the coefficient-parameter association is

How does Regularisation work?

• Penalise large values of parameters in optimisation

• Change the cost function to include J_reg = 𝜽^T 𝜽

  • When this is used it is known as L2 or “ridge” or “weight decay”

• J_tot = J_e𝑚𝑝 + 𝜌J_reg with 𝜌 ≥ 0 and J_e𝑚𝑝 = (𝐘 − 𝚿𝜽)^T ( 𝐘 − 𝚿𝜽)

  • Trade off between fit to data and smoothness

  • small 𝜌 = fit to data more important

  • large 𝜌 = smoothness more important

Uses of Artificial Neural Networks

• Classification

  • (Multi-layer perceptron)

• Pattern recognition

  • (Multi-layer perceptron, time-delay neural networks and recurrent nets etc)

• Regression/Function Approximation

  • (Feedforward Architecture)

• Clustering

  • (Self Organising Map Network)

• Pattern association

• Control

• Filtering time series data

Because they are very flexible they tend to overfit

Different Network Methodologies

• Self-organising map networks

• Radial Basis Functions

• Single/Multi layer perceptron

• Hopfield Networks

• Feedforward/backpropagation Architecture

• Deep learning structures

• Support Vector Machines

• Time-delay neural networks

• Recurrent nets

Depending on the type of methodology chosen, it can be used in supervised or unsupervised learning