CH2: Neural Network Foundations

What defines a neural network as a "deep" network?

Having 2 or more hidden layers between the input and output layers.

What are three alternative names for an Artificial Neuron?

Semi-Linear Unit / Binary Neuron / McCulloch-Pitts (MCP) Neuron.

What three actions does a single node in a hidden layer perform?

Receive data, process data, and send data.

What is the primary limitation of a single-layer neural network?

It is limited to solving linear classification problems (linearly separable datasets).

What roles do weights and biases play in a perceptron?

Weights adjust the influence of input values, while the bias shifts the output to provide flexibility.

What is the purpose of an Activation Function?

It transforms the weighted sum to determine whether a neuron should "fire" (be activated) and produce a final output.

What are some common activation functions?

Step Function, Sigmoid, Tanh, ReLU, Leaky ReLU, Softmax, Swish, ELU.

What is the purpose of a Loss Function?

It measures how wrong the model’s prediction is by evaluating how well predictions match actual target values.

What are common loss functions?

Mean Squared Error (MSE), Mean Absolute Error (MAE), Binary Cross-Entropy, Categorical Cross-Entropy, Huber Loss.

What do Learning Mechanisms like Backpropagation do?

They define how the network learns from errors and adjusts weights and biases during training to minimize mistakes.

What is the role of an Optimizer Function?

It controls the training process by deciding exactly how weights and parameters are updated using gradients.

What are some common optimizer functions?

SGD, Momentum, Adagrad, RMSprop, Adam, Nadam.