Artificial Neural Networks CONT.

Learning Rate Schedules

• A fixed learning rate is often used in the first few epochs followed by a decreasing schedule

  • The learning rate may be adapted over epochs t to give a learning rate schedule

Back Propagation - Pros

• Big errors lead to big changes in the weights

• Small errors lead to weights relatively unchanged

• Over many iterations, a given weight keeps changing until the error associated with that weight is negligible, at which point weights change little

Choosing the Training Set

• Number of features

  • Too many features require larger networks, hence risk of overfitting

• Coverage of values for all features

  • The training set need to cover the full range of values for all features and output

Size of the Training Set

• With I input nodes, H hidden nodes, and 1 output node, there are I*(H)+H = (I+1)*H weights in the network (~I*H)

• Consider about 100 records each weight(at least 30 weights)

• If a model is too complex it will be overfit then it will lose generality

Rule of thumb for number of nodes in the hidden layer

• Do not use # of nodes in the hidden layers larger than the number of nodes in input layer

  • Start with smaller number of nodes in the hidden layer. If network is overfitted, reduce the SIZE OF THE LAYER.

Overfitting

• Neural networks can overfit with increasing number of iterations as indicated by error via ross-validation or on the unseen data

  • To avoid overfitting:

    • Limit iteration

    • Limit complexity of the networks: reduce number of hidden layers, reduce number of nodes in a hidden layer

  • Common criteria to stop changing weights

    • When weights change very little from on iteration to the next

    • When the misclassification rated reaches a required threshold

    • When a threshold on runs is reached

Data Mining using ANN

1. Identify the input and output features

2. Scale the inputs and outputs so they are in the range

3. Set up a network with an appropriate technology

4. Train the network on a representative set of training records

5. Evaluate the network using the test set to see how well it performs

6. Apply the model to predict outcomes for unseen inputs

The input needs to be scaled

For Continuous Values

Normalized within (0 and 1):

Y = (X-Min)Range → X = Min + Y * Range

Normalized within (-1,1)

Y = (2 (X-Min)/Range)) - 1 → X = Min + (Y+1) Range/2

Standardized value