L2: Rise of neural networks and deep learning

Training

1. Input data into neural networks

2. Choose algorithms

3. Feed data and adjust parameters

4. Evaluate generalisation to real data

Prediction

1. Receive data

2. Generate predictions based on patterns

Structure of a deep network

• Hidden layer: Columns of middle neurons

• Deep network: Machine learning model with multiple hidden layers of interconnected nodes between input and output layers

Hyperparameters

• Define the structure of the neural network

• E.g. number of hidden layers, neuron per layer

Forward propogation

• Synapse takes value from input and multiplies it by its weight

• Neuron adds output of all synapses and applies activation function

• Synapse weight → Connection strength

Trained network

• Gives correct score for all training data

• Weights = Trainable parameters

• Neural network is like a machine with knobs (e.g. synthesizer)

  • Weights = knobs

  • Training = turning knobs til output sounds right

• Number of trainable parameters (500B+)

How do you train large neural networks?

1. Initialise with random weights

2. Measure the error

   1. J = how wrong the model is

3. Minimise error → Adjust parameter so J gets smaller

Naive (Brute-force) approach

• Try every knob position

• Find where J is lowest

• IMPOSSIBLE

Minimisation: Backpropagation

• Use backpropagation and update weights with gradient descent

• Works for deep networks

How to do backpropagattion?

1. Randomly pick value for w

2. Pick two more around the point

3. Compute gradient (slope of error)

4. Update weights in downhill direction

5. Repeat until error is near 0

Prediction

Learning to predict the future from the past