Machine Learning

Machine Learning

A field of Artificial Intelligence where systems learn from data to make predictions or decisions without being explicitly programmed. It involves the study of algorithms that improve performance at a task through experience.

Supervised Learning

A type of training that uses a series of labeled examples with direct feedback, where the training data includes the desired outputs.

Unsupervised Learning

A type of learning with no feedback, where the training data does not include desired outputs. It involves learning "what normally happens" or grouping similar instances.

Reinforcement Learning

A type of learning involving indirect feedback after many examples, where rewards are received from a sequence of actions. It focuses on learning a policy (a sequence of outputs).

Regression

A task where the goal is to predict a continuous numeric value based on input features (e.g., predicting house prices or temperature).

Classification

A task where the goal is to predict categories or nominal outputs.

Linear Regression

A regression algorithm that fits data with a hyperplane (or line in 2D). It is the simplest model for function approximation.

Logistic Regression

A common algorithm used when the dependent variable is binary (e.g., disease vs. no disease). It fits data with a sigmoidal or logistic curve rather than a line to output a probability approximation.

Delta Rule (Least Mean Squares Rule)

An update rule used in supervised learning (specifically for neural networks) to minimize error by adjusting weights based on the difference between actual and predicted outputs.

Sum of Squared Error (SSE)

An objective function used in simple linear regression that sums the squared differences between predicted and actual values. It creates a parabolic error surface ideal for gradient descent.

Mean Absolute Error (MAE)

The average absolute difference between predicted and actual values.

Mean Squared Error (MSE)

The average squared difference between predicted and actual values; it penalizes larger errors more than MAE.

R² Score (Coefficient of Determination)

A metric that measures how well the model explains the variance in the data.

Binary Classification

A type of classification involving exactly two classes (e.g., Pass/Fail, Yes/No).

Multiclass Classification

Classification involving more than two classes (e.g., Cat, Dog, Bird).

Multilabel Classification

A scenario where each instance can belong to multiple classes simultaneously.

Threshold

The decision boundary (e.g., 0.5 or 0.9) that converts a model’s probability output into a specific class label.

Decision Tree

A hierarchical structure that makes decisions based on feature values, used for classification and regression.

Random Forest

An ensemble method consisting of multiple decision trees.

K-Nearest Neighbors (KNN)

An algorithm that classifies instances based on their nearest data points.

Support Vector Machine (SVM)

An algorithm that finds the best boundary between classes.

Naïve Bayes

A classifier based on probability and Bayes’ theorem.

Confusion Matrix

A table comparing the actual (true) labels with the labels predicted by the classification model.

Precision

The number of correctly classified positive examples divided by the total number of examples classified as positive.

Recall (Sensitivity)

The number of correctly classified positive examples divided by the total number of actual positive examples in the test set.

Specificity

Also known as the True Negative Rate (TNR).

Cross-validation

A method where data is partitioned into n subsets to train and test the model n times to estimate accuracy.

Leave-one-out Cross-validation

A special case of cross-validation for small datasets where each fold has only a single test example.

Scoring

Assigning a probability estimate (PE) to an instance rather than a definite class label.

ROC Curve (Receiver Operating Characteristics)

A plot of the True Positive Rate (TPR) against the False Positive Rate (FPR).

AUC (Area Under the Curve)

A performance measure where a value of 1 represents a perfect classifier and 0.5 represents random guessing.

Lift Analysis

An analysis performed by ranking examples by their score and dividing them into bins to observe the distribution of positive examples.

Overfitting

A situation where a tree fits the training data well but performs poorly on test data, often characterized by the tree being too deep or having too many branches.

Pre-pruning

A method to avoid overfitting by halting the construction of the tree early.

Post-pruning

A method to avoid overfitting by removing branches or sub-trees from a "fully grown" tree.

Association Rule Mining

A task focused on finding relationships between items, such as identifying that customers who buy coffee likely buy bread.

Dimensionality Reduction

The process of simplifying large datasets into fewer variables while retaining most of the important information.

Clustering

The process of grouping data based on similarity or distance (e.g., Euclidean distance) without knowing labels ahead of time.

K-Means

An algorithm that partitions data into k clusters by minimizing the distance between data points and cluster centers (centroids).

Hierarchical Clustering

An algorithm that builds a tree-like structure of clusters, often visualized as a dendrogram.

DBSCAN

A density-based algorithm that groups points lying close together and marks outliers as noise.

Elbow Method

A technique used to select the optimal number of clusters (k) by plotting the Sum of Squared Errors (SSE) and finding the "elbow point" where the error curve starts to flatten.

PCA

A dimensionality reduction technique that transforms data into a new coordinate system using principal components to simplify the dataset while keeping the maximum variance.

Principal Components

New variables created by PCA that are linear combinations of original features, designed to capture the most variance in the data.

Eigenvalues and Eigenvectors

Mathematical properties used in PCA to determine the directions (components) with the most variance.

Silhouette Score

A metric (ranging from –1 to +1) that measures how well data points match their own cluster compared to others; values closer to +1 indicate good clustering.

Davies–Bouldin Index (DBI)

A metric measuring cluster separation and compactness, where values closer to 0 indicate better clustering.

Calinski–Harabasz Index (CH Index)

A metric where higher values indicate better clustering, signifying that between-cluster variance is much greater than within-cluster variance.

fit()

A method used to learn or train parameters from the data.

transform()

A method used to apply learned parameters to data, typically used for test or new data.

fit_transform()

A method that learns parameters and applies the transformation in a single step, commonly used with preprocessing transformers.

Scalers

Tools used to normalize features, which is essential for distance-based algorithms.

StandardScaler

Standardizes features to have a mean of 0 and a standard deviation of 1.

MinMaxScaler

Scales data to a specific range, usually [0, 1].

RobustScaler

Scales data using the Interquartile Range (IQR), making it resistant to outliers.

MaxAbsScaler

Scales data to the range [-1, 1], often used for sparse data.

QuantileTransformer

Transforms data to follow a uniform or Gaussian distribution.

Imputers

Tools for handling missing values.

SimpleImputer

Fills missing values with basic statistics like mean, median, or mode.

KNNImputer

Fills values based on the similarity of nearest neighbors.

IterativeImputer

A regression-based method for modeling complex missing values.

MissingIndicator

Adds a binary indicator to denote where values were missing.

Encoders

Techniques to convert categorical data into numbers.

One-Hot Encoding

Converts categories into binary columns.

Label Encoding

Assigns each category a unique integer label.

Ordinal Encoding

Converts categories into integers based on a specific order.

Binary Encoding

Converts categories into binary digits.

Target Encoding

Replaces categories with the mean of the target variable.

Hashing Encoding

Maps categories to fixed-length hash values.

Text Vectorizers

Tools that convert text into numeric vectors, such as CountVectorizer and TfidfVectorizer.

Regularization

Techniques to prevent overfitting in linear models by penalizing complexity.

Ridge

Shrinks coefficients but keeps all features.

Lasso

Shrinks coefficients and sets some to zero, performing feature selection.

Elastic Net

Combines Ridge and Lasso penalties.

Gini Impurity

Measures the impurity of a class distribution (used to select splits).

Entropy

A measure of uncertainty or disorder used in splitting.

Gaussian NB

Assumes continuous features are normally distributed.

Multinomial NB

Used for discrete counts (e.g., text classification).

Bernoulli NB

Used for binary features.

Kernel Trick

A method to transform data into higher dimensions without calculating the transformation explicitly.

Types

Linear, Polynomial, RBF (Radial Basis Function), and Sigmoid.

Activation Functions

Operations determining neuron output.

ReLU (Rectified Linear Unit)

Common for hidden layers, outputs input if positive, else 0.

Softmax

Used for multi-class classification.

Optimizers

Algorithms to minimize loss.

MLP Regressor

A Multi-Layer Perceptron specifically for predicting continuous values.

Manhattan Distance

Grid-like distance.

Minkowski Distance

Generalization of Euclidean and Manhattan.

Hamming Distance

Used for binary/string data.

Cosine Similarity

Measures the angle between vectors (text similarity).

ARIMA

AutoRegressive Integrated Moving Average for forecasting.

SARIMA

Seasonal ARIMA, which includes seasonal components for data with cyclic patterns.