P5:

Memorization vs Generalization in Machine Learning

• Definitions:

  • Memorization: Refers to the process by which a neural network learns to recognize patterns from input data by storing specific weights (w values) that represent connections between neurons.
  • Generalization: The ability of a neural network to apply learned knowledge to unseen data, even when that data includes noise or variations.

• Neural Network Structure:

  • Comprises layers of neurons: Input layer, Hidden layer, and Output layer.
  • Connections between neurons have weights, denoted as w, that adjust during training.

• Example of Learning Process:

  • Start with a network of 3 neurons in the input layer leading to 2 output neurons.
  • Inputs like (1, 0) are applied, causing adjustments in weights that strengthen certain connections based on the input values.
  • The process includes applying varying inputs (like 0.99, 0.01, 0.1, etc.) to train the network and update weights accordingly.

• Weight Adjustments:

  • Strong weights (high positive values) facilitate connection strength, while weak weights (close to zero) signal less connection relevance.
  • For effective memory, the network must update its weights in accordance with the specific training dataset used.

• Understanding Classifications:

  • For given input values, the output can be directed through strong connections to achieve expected classifications (e.g., expected output (1, 0) or (0, 1)).

• Memorization vs. Generalization:

  • Memorization: Requires clearly defined w values from training to classify inputs correctly. The network's ability to recall these values forms its memory parts.
  • Generalization: The network’s capability to accurately predict outputs even when small noises or variances are added to the input.

The Role of Weights in Memory Retention

• Weights as Memory:

  • Weights (w values) store learned knowledge, acting as memory for the network. Higher weighted connections imply a stronger memory of that input.
  • Classifying new inputs relies on the network's ability to retrieve and use these weights effectively.

• Example of Output Generation:

  • Different input scenarios are presented (like (0, 1)) to illustrate how weights influence output while some inputs may be nullified or not significantly affect the classification outcome.

The Importance of Basics in Learning

• To effectively utilize neural networks, it is crucial to memorize fundamental mathematical principles similar to basic arithmetic axioms that support complex learning processes.

Epochs and Iterations in Neural Network Training

• Definitions:

  • Epoch: One complete pass through the entire training dataset.
  • Iteration: Each update of weights in the neural network based on a subset of data points.

• Different Approaches:

  • Single Iteration for Entire Dataset: Each data point updates weights in a straightforward manner.
  • For N data points, N iterations equate to the number of data points since each point individually adjusts weights.
  • Batch Processing: Similar to a cumulative method where weights are updated only after processing the entire batch of data points at once (low iteration counts).
  • Batch Update: Once all data points are used, weights get updated based on the cumulative effect, thereby averaging the updates from individual weights.

• The trade-off lies between the granularity of weight updates and computational efficiency based on the adopted method for feeding data (e.g., individual vs. batch methods).