P4:

Logistic Function

• The logistic function varies with the input $x$ and produces outputs between 0 and 1.
  • When $x = 0$, the function evaluates to $y = \frac{1}{1 + 0} = 1$.
  • As $x$ approaches $+ ext{infinity}$, the logistic function approaches 1.
  • As $x$ approaches $- ext{infinity}$, the function approaches 0.
  • The output ($y$) never actually reaches 1 or 0, making it valuable as an activation function.

Activation Function and Decision Rule

• Used as an activation function in neural networks to classify data points.
• Decision rule:
  • For the sigmoid function, if $y < 0.5$, classify as Class A; if $y > \beta$ (where $\beta = ext{lambda} imes 0.5$), classify as Class A; otherwise, classify as Class B.
• Issues with this method arise in logistic regression.

Softmax Activation Function

• In contrast to the sigmoid, the softmax function deals with multiple neurons and outputs values that sum to 1.
• Calculation involves:
  • $y_1 = \frac{e^{z_1}}{e^{z_1} + e^{z_2}}$
  • $y_2 = \frac{e^{z_2}}{e^{z_1} + e^{z_2}}$
• Decision rule for softmax: If $y_1 > y_2$ after normalization, the output is 1 (Class A), else 0 (Class B).

Key Considerations for Neural Networks

• Ensure that label formats for $y$ (true value) and $\hat{y}$ (predicted value) are consistent. $\hat{y}$ should be a subset of $y$.
• Input layer neurons should exceed the number of features in the dataset; output layer neurons typically match the number of classes.
• Output predictions should be accounted using one-hot encoding for multi-class scenarios.
  • Example: If Class A (0) and Class B (1), output should be a vector like [1, 0] for Class A.
• Class output for regression is always a single neuron using a linear activation function.

Fundamental Requirements for Errors and Predictions

• Errors from predictions (e.g., $\hat{y}$) should utilize loss functions like:
  • Mean Squared Error (MSE): $E = \frac{1}{n} \sum<em>{i=1}^{n} (y</em>i - \hat{y}_i)^2$
  • Mean Absolute Error (MAE): $E = \frac{1}{n} \sum<em>{i=1}^{n} |y</em>i - \hat{y}_i|$
• These errors help guide the optimization of weights during training.

Learning Algorithm Notes

• The learning algorithm updates weights based on calculated errors and uses backpropagation.
• Notation for weights is $w_{ij}$ (weight linking neuron $j$ in layer $i$ to neuron in subsequent layer).

Concluding Thoughts

• Understanding the nuances of activation functions and error handling is crucial for modeling neural networks effectively. Consistency in label formats and careful handling of neuron counts relative to data dimensions are foundational principles.