Machine Learning Fundamentals

KNN (K-Nearest Neighbors)

A machine learning algorithm highly sensitive to noise; a small $k$ value tends toward overfitting, while a very large $k$ results in a very smooth but underfitted decision boundary.

Overfitting

A situation where a model learns training data (including noise) too well, essentially "memorizing" it instead of learning general rules, leading to failure on new data.

Underfitting

A condition occurring when a model is not complex enough to capture the patterns in the data, often associated with high bias.

Self-information $I(p)$

A measure where information decreases as the probability of an event increases, defined by the function $I(p_i) = -\text{log}(p_i)$, satisfying the property that information of independent events is additive.

Shannon's Entropy

The average self-information or expected value over all possible values of $x$, calculated as $$H(X) = - extstyle extstyle extstyle extstyle extstyle extstyle extstyle extstyle extstyle extstyle extstyle extstyle extstyle extstyle extstyle extstyle extstyle extstyle extstyle extstyle extstyle extstyle extstyle extstyle extstyle extstyle extstyle extstyle extstyle extstyle extstyle extstyle extstyle extstyle extstyle extstyle extstyle extstyle extstyle extstyle extstyle extstyle ext