Machine Learning: Classification, Linear Regression, and Decision Trees

Comprehensive vocabulary flashcards covering supervised machine learning, linear regression metrics, classification via logistic regression, statistical feature selection, bias-variance trade-off, and decision tree splitting criteria.

Binary Classification

A supervised learning problem that classifies data into two distinct classes, such as spam/not spam or fraudulent/not fraudulent.

Multinomial Classification

A supervised classification problem that categorizes data into three or more classes, such as document classification, product categorization, or malware classification.

Hard Assignments vs. Soft Assignments

Hard assignments place data directly into specific categories or classes, whereas soft assignments output the predicted probability that each class applies to the given data.

Logistic Regression

A classification algorithm that estimates the probability of an event occurring based on a dataset of independent variables, bounding the dependent variable between $0$ and $1$.

Odds

The ratio of the probability of success to the probability of failure, defined mathematically as $\frac{P}{1 - P}$.

ANOVA F-test

A collection of statistical models used in feature selection to analyze differences among group means for numerical variables; larger $F$-scores and $p$-values less than $0.05$ indicate statistically significant differences.

Pearson's Chi-Squared Test

A statistical test used in categorical feature selection to determine whether there is a statistically significant difference between expected frequencies and observed frequencies in a contingency table.

Dummy Variable

A numerical indicator variable (such as $0$ and $1$) used in regression analysis to represent categorical subgroups or treatment groups within a single equation.

SMOTE (Synthetic Minority Over-sampling Technique)

An oversampling algorithm from $imblearn.over\_sampling$ used to balance imbalanced training datasets by synthesizing new samples for the minority class.

Machine Learning

A field coined around 1960 focused on exploring and constructing algorithms that learn from historical data to make predictions on new input data.

Supervised Learning

A category of machine learning where models are trained using historical datasets containing both independent variables (features) and known target variables (labels).

Features

The independent or explanatory variables in a dataset that serve as inputs for a predictive machine learning model.

Label

The dependent or target response variable that a supervised machine learning model aims to predict.

Linear Regression

A linear modeling approach that fits coefficients $w = (w_1, \dots, w_p)$ to model the relationship between a scalar dependent variable $y$ and one or more explanatory variables by minimizing the residual sum of squares.

Mean Absolute Error (MAE)

A regression metric measuring the average absolute difference between actual target values and predicted values, expressed as $\text{MAE} = \frac{1}{n} \sum_{i=1}^{n} |\hat{y}_i - y_i|$.

Mean Squared Error (MSE)

A regression performance metric calculated as the average squared difference between predicted and actual values; unlike MAE, it disproportionately penalizes large errors.

Total Sum of Squares (TSS)

The measure of total variation of observed actual values relative to their sample mean, formulated as $\text{TSS} = \sum_{i=1}^{n} (y_i - \bar{y})^2$.

Residual Sum of Squares (RSS)

The sum of squared differences between observed values and the values predicted by a regression function, calculated as $\text{RSS} = \sum_{i=1}^{n} (y_i - f(x_i))^2$.

R-Squared ($R^2$)

The proportion of variance in the dependent variable explained by the model, calculated as $R^2 = 1 - \frac{\text{RSS}}{\text{TSS}}$, bounded between $0$ and $1$.

Correlation Coefficient

A numerical measure ranging from $-1.0$ to $1.0$ that indicates the strength and direction of a linear relationship between two variables, where $0$ indicates no relationship.

Polynomial Regression

A regression approach that models non-linear relationships between independent and dependent variables by using an $n^{\text{th}}$-degree polynomial function.

Bias (in Machine Learning)

The difference between the average prediction of a model and the correct value; high bias results from oversimplifying models and causes high training and test error.

Variance (in Machine Learning)

The variability of model predictions for a given data point; models with high variance pay excessive attention to training data and fail to generalize to unseen test data.

Overfitting

A modeling flaw that occurs when an algorithm fits training points and noise too closely, exhibiting high variance and leading to poor generalization on test data.

Underfitting

A modeling flaw that occurs when a model is too simple to capture the underlying pattern of data, exhibiting high bias and low variance.

Total Error

The overall prediction error of a machine learning model, calculated as the sum of bias error and variance error.

Decision Tree

A flowchart-like structure used for classification and prediction where internal nodes denote attribute tests, branches denote test outcomes, and leaf nodes hold class labels.

Root Node

The top node in a decision tree selected via attribute selection techniques to divide the initial dataset into two or more subsets.

Leaf Node (Terminal Node)

The terminal node of a decision tree branch that carries a final class label and cannot be divided into further sub-nodes.

Pruning

The process of removing sub-nodes or branches from a decision tree to reduce model complexity and avoid overfitting.

Gini Impurity (Gini Index)

A split metric measuring the probability of randomly mislabeling an element, calculated as $\text{GiniIndex} = 1 - \sum_j p_j^2$, ranging from $0$ (pure node) to $0.5$.

Pure Node

A decision tree node containing elements belonging entirely to a single unique class, resulting in a Gini Index of $0$ and requiring no further splitting.

Entropy

A measure of information unpredictability calculated as $E(S) = \sum_{i=1}^{c} -p_i \log_2(p_i)$, taking values inside the interval $[0, 1]$.

Information Gain

The reduction in entropy achieved after partitioning dataset instances using a decision tree attribute split, calculated as $\text{Gain} = E_1 - E_2$.