Neural Networks: Connectionism, Models, and Learning Processes

Module Overview

• Focus on neural networks (connectionism) and their workings.

• Emphasis on a gentle mathematical description to aid understanding.

• Plan to cover:

  • Non-cognitive example: sonar detection.

  • Cognitive example: past tense acquisition in children.

  • Development of tools for modeling and simulating brain processes, specifically neural networks.

The Need for Neural Networks

• Current challenges in studying the brain:

  • Available techniques:

  • Coarse grained: mapping between brain regions and cognitive functions.

  • Fine grained: measuring single neuron activity.

  • Gaps: Lack of intermediate techniques to study neural connectivity (interconnected patterns of neuron activation) relevant for perception, memory, motor control, and cognition.

Neural Network Models (Connection Models)

• Definition: Models that approximate populations of neurons to study connectivity.

• Nature of Models:

  • Abstractions of real neuronal processes.

  • Relaxation of some biological constraints while simulating basic principles of neuron information processing.

• Neuron Function:

  • Neurons receive inputs, send outputs via neurotransmitters, changing electrical voltage.

  • A typical neuron can receive input from about 10,000 other neurons.

  • Input can be positive (excitatory) or negative (inhibitory).

  • Neuron fires and sends output if input strength exceeds a threshold.

Simplified Representation of Neural Networks

• Diagram Description: Circles depict neurons interconnected across input-output levels.

  • Computational neural network as a set of mathematical operations.

  • Activation levels: Range from 0 to 1.

• Activation Transfer: Signals sent to output units based on weighted input.

  • Weights adjust the strength of each connection (positive or negative).

  • Input summed at each unit; if it meets a certain threshold, activation passes forward.

• Hidden Layer:

  • Detailed examination of a single unit in the hidden layer (middle layer).

  • Input nodes: $i1$ (input 1), $i2$ (input 2), $i_3$ (input 3).

  • Each input multiplied by corresponding weights: ( W1, W2 ) (weight 1, weight 2).

  • Examples of transformations:

  • Input of 1 × Weight of -1 = -1

  • Input of 0.5 × Weight of 0.5 = 0.25

  • Summation as total input X; leading to output activation.

  • If X < 0: output activation < 0.5.

  • If X > 0: output activation > 0.5.

Information Processing in Neural Networks

• Question raised: How do these models process information effectively?

• Transitioning to practical examples.

Non-Cognitive Example: Sonar Detection

• Similarity drawn in learning mechanisms:

  • A boat uses sonar waves to detect surrounding objects.

  • Echo profile generated from reflections captures unique frequency changes.

• Challenge: Distinguishing between different echo profiles (e.g., mine vs. rock);

  • Echo profiles often overlap due to noise, complicating detection.

  • Neural networks excel in identifying complex patterns:

  • Start with a neural network initialized with random weights.

  • Given an echo profile from a mine, it makes a prediction (e.g., erroneously predicts rock).

Learning Process: Backpropagation

• Key concept: Backpropagation algorithm involved in model learning.

• Process of learning from errors:

  • Corrects weights that led to incorrect predictions.

  • Weights contributing most to the wrong prediction are adjusted (e.g., reduced).

  • Requires feedback from a teacher or dataset indicating correctness.

• Training Process:

  • Multiple known examples (echo profiles for rocks and mines) provided for learning.

  • Model adjusts weights iteratively based on feedback.

• Outcome: After extensive training, the model can classify unknown echo profiles effectively, distinguishing mine from rock based on learned weights.