Foundations of Machine Learning

Flashcards covering key terms and definitions from the foundations of machine learning concepts and techniques.

Machine Learning

Using algorithms to learn patterns from data and make predictions.

Supervised Learning

Learning using labeled data (input + known output).

Unsupervised Learning

Learning patterns from unlabeled data.

Classification

Predicting categories, such as spam vs not spam.

Regression

Predicting continuous values like height or price.

Feature Matrix (X)

Input variables used to make predictions.

Target Vector (y)

Output variable being predicted.

Linear Regression

Models relationship between variables using a straight line.

Least Squares Method

Minimizes squared error between predicted and actual values.

Error (Residual)

Difference between actual and predicted value.

Mean Squared Error (MSE)

Average of squared errors.

Root Mean Squared Error (RMSE)

Square root of MSE; average prediction error.

R² (R-Squared)

Percentage of variability explained by the model.

Training Set

Data used to build the model.

Validation Set

Data used to tune and evaluate during training.

Test Set

Final dataset to evaluate model performance.

Overfitting

Model performs well on training data but poorly on new data.

Cross-Validation (CV)

Repeatedly splitting data into folds to evaluate model performance.

K-Fold Cross Validation

Splitting data into k groups and rotating validation sets.

Missing Data

Data points with no value (NaN) that must be handled.

Imputation

Filling missing values using mean, median, or most frequent value.

Mean Imputation

Replace missing values with average.

Median Imputation

Replace missing values with median (better for skewed data).

Standardization (Z-score)

Scale data to mean = 0 and SD = 1.

Normalization (Min-Max Scaling)

Scale values between 0 and 1.

Feature Scaling

Ensuring all features are on a similar scale to improve models.

K-Nearest Neighbors (KNN)

Classifies data based on nearest neighbors.

K Value

Number of neighbors used for classification.

Distance (Norm)

Measure of similarity between data points.

Curse of Dimensionality

Performance decreases as features increase.

Naive Bayes

Probabilistic classifier using Bayes’ theorem.

Bayes Theorem

Calculates probability of a class given features.

Prior Probability

Initial probability of a class.

Likelihood

Probability of features given class.

Posterior

Final probability after considering evidence.

Naive Assumption

Features are independent given class.

Joint Probability

Probability of multiple events occurring together.

Conditional Probability

Probability of one event given another.

Support Vector Machines (SVM)

Finds optimal boundary (hyperplane) separating classes.

Hyperplane

Decision boundary.

Margin

Distance between boundary and closest data points.

Support Vectors

Points closest to boundary.

Hard Margin

No misclassification allowed.

Soft Margin

Allows some errors.

Kernel

Function defining shape of decision boundary.

RBF Kernel

Nonlinear boundary (curved separation).

Regularization Parameter (C)

Controls tradeoff between accuracy and generalization.

Decision Tree

Model that splits data based on features.

Entropy

Measure of randomness/impurity.

Information Gain

Reduction in entropy after a split.

Gini Index

Measure of impurity used in trees.

Bagging (Bootstrap Aggregation)

Combine multiple models trained on random samples.

Random Forest

Ensemble of decision trees.

Boosting

Sequentially improving weak models.

AdaBoost

Adjusts weights to focus on errors.

Gradient Boosting

Builds models based on previous errors.

Learning Rate

Controls how much each new model learns.

Hyperparameter

Parameter set before training that controls model behavior.

Grid Search

Testing multiple hyperparameter values to find the best one.

Regularization

Prevent overfitting by penalizing large coefficients.

Ridge Regression (L2)

Shrinks coefficients evenly.

Lasso Regression (L1)

Can reduce coefficients to zero.

Time Series Data

Data ordered over time.

Trend

Long-term direction.

Seasonality

Repeating pattern.

Noise

Random variation.

Lag Feature

Previous value used for prediction.

Moving Average

Average of past values to smooth data.

Autocorrelation

Correlation of a variable with its past values.

Stationarity

Statistical properties remain constant over time.

Differencing

Subtracting previous values to remove trend.

Autoregressive Model (AR)

Uses past values to predict future.

Moving Average Model (MA)

Uses past errors to predict.

ARIMA Model

Combines AR + I (integration) + MA.

Ontologies

Structured representation of knowledge.

Ontology Evaluation

Assessing quality of ontology.

Accuracy (Ontology)

Correctness of representation.

Consistency

No contradictions in ontology.

Completeness

Covers domain fully.

Clarity

Easy to understand.

Adaptability

Can be extended.

Semantic Web

Web of linked structured data.

Web Ontology Language (OWL)

A formal language for representing ontologies.

SKOS

Vocabulary system for concepts.