ML

Machine learning

Learning patterns from data to make predictions on new examples.

Traditional programming vs ML

Traditional programming follows human-written rules. ML learns patterns from examples.

ML for complex problems

Useful when patterns are too complicated to describe with fixed rules.

Sample, feature, target

Sample = one data row; feature = input information; target = correct output.

Supervised learning

Learning from labeled examples with known answers.

Unsupervised learning

Finding patterns or groups in data without labels.

Semi-supervised learning

Learning from a few labeled examples and many unlabeled examples.

Classification

Predicting a category, such as approved or rejected.

Regression

Predicting a numerical value, such as a house price.

Tom Mitchell's T, E, P

T = task; E = experience or training data; P = performance measure.

T/E/P example: Spam detection

Task = classify spam; Experience = labeled emails; Performance = accuracy or F1 score.

Data preparation

Cleaning and transforming data so models can learn meaningful patterns.

Data quality problems

Missing values, duplicates, invalid entries, outliers and inconsistent labels.

Data inconsistency

Different labels for the same thing, such as US, USA and United States.

Numerical imputation

Replacing missing numbers with the mean or median.

Mean, median and mode

Mean = average; median = middle value; mode = most frequent value. Median resists outliers.

Outlier

A value unusually far from most observations. Investigate before removing it.

Min-Max normalization

Rescaling values, usually to 0–1. Formula: (x − min)/(max − min).

Standardization

Rescaling a feature to have mean 0 and standard deviation 1.

One-hot encoding

Turning categories into separate binary 0/1 columns.

Ordinal encoding

Assigning numbers to ordered categories, such as small, medium and large.

Feature selection vs extraction

Selection keeps useful original features; extraction creates new features.

Exploratory Data Analysis (EDA)

Using summaries and plots to understand data before training.

EDA charts

Histogram = distribution; boxplot = spread/outliers; scatter = relationship; bar = categories.

Data leakage

Information from validation or test data influences training. Fit preprocessing on training data only.

Training set

Data used to learn model parameters.

Validation set

Data used to compare models and choose settings.

Test set

Untouched data used for final evaluation on unseen examples.

K-fold cross-validation

Training and validating repeatedly using different parts of the dataset.

Overfitting

Learning training details or noise and performing poorly on new data.

Underfitting

Being too simple to learn patterns; performance is poor on training and validation data.

Overfitting prevention

Simpler models, regularization, limited tree depth or more suitable training data.

Generalization

A model's ability to perform well on unseen data.

K-Nearest Neighbors (KNN)

Predicting from the K closest training examples.

KNN classification

Predicting the majority class among nearby examples.

KNN regression

Predicting the average target value of nearby examples.

KNN: Small vs large K

Small K may overfit; large K may oversmooth and underfit.

Feature scaling in KNN

Important because features with larger scales can dominate distance calculations.

True Positive (TP)

A correctly predicted positive.

True Negative (TN)

A correctly predicted negative.

False Positive (FP)

A negative incorrectly predicted as positive.

False Negative (FN)

A positive incorrectly predicted as negative.

Accuracy

The proportion of all predictions that are correct.

Precision

Of predicted positives, the proportion that are actually positive.

Recall

Of actual positives, the proportion correctly detected.

F1-score

A measure that balances precision and recall.

Class imbalance

When one class is much more common; accuracy alone may be misleading.

MAE, MSE and RMSE

Regression error measures. MAE uses absolute errors; MSE squares errors; RMSE is the square root of MSE.

R² score

Measures how much variation in the target a regression model explains.

Linear regression

Predicting a number using a weighted combination of input features.

Residual

Actual value minus predicted value.

Logistic regression

A classification model that predicts class probabilities.

Sigmoid function

Converts a score into a value between 0 and 1.

Loss function

Measures how wrong a model's predictions are.

Gradient descent

Updating model parameters in a direction that reduces loss.

Learning rate

Controls the size of parameter updates; too large may overshoot, too small may be slow.

Decision tree

A model that asks questions at nodes, follows branches and predicts at leaves.

Gini impurity

Measures how mixed the classes are; 0 means a completely pure node.

Entropy

Measures uncertainty or class mixing in a node.

Information gain

Reduction in entropy after splitting; higher gain is better.

Pre-pruning

Stopping a tree from growing too complex.

Post-pruning

Removing unnecessary branches after a tree has grown.

Maximum depth and maximum leaves

Limits on tree levels and final leaves to prevent overfitting.

Minimum samples per leaf

The minimum number of examples allowed in each leaf.

Minimum samples for split

The minimum number of examples needed before a node can split.

Classification vs regression trees

Classification reduces class impurity; regression groups similar numerical values.

Random Forest

Multiple decision trees whose predictions are combined.

Random Forest advantage

Usually more stable and less prone to overfitting than a single tree.

Dimensionality reduction

Reducing the number of features while keeping useful information.

Principal Component Analysis (PCA)

Creating new features called principal components that capture the greatest variation first.

Neuron

Combines weighted inputs and a bias, then applies an activation function.

Input layer

Receives input features.

Hidden layer

Transforms information to learn patterns.

Output layer

Produces the final prediction.

Connections and weights

Links between neurons with learned values controlling signal strength.

Activation function

Introduces nonlinearity so neural networks can learn complex patterns.

Network without nonlinear activations

Multiple linear layers behave like one linear layer, limiting the patterns learned.

ReLU

An activation function that replaces negative values with zero.

Sigmoid vs softmax

Sigmoid often gives a binary-class probability; softmax gives multiclass probabilities.

Five steps of neural network learning

Initialize weights → predict → calculate error → backpropagate → update weights.

Forward propagation

Passing data through the network to produce a prediction.

Backpropagation

Calculating gradients that show how weights affect the error.

Weight updates

Adjusting weights using gradients to reduce future errors.

Epoch vs batch

Epoch = one full pass through training data; batch = a smaller group processed together.

Support Vector Machine (SVM)

A supervised model that finds a separating boundary with the largest possible margin.

SVM hyperplane

The decision boundary separating classes; a line in two dimensions.

SVM margin

The perpendicular gap from the decision boundary to the closest training points.

Support vectors

The closest training points that determine the separating boundary.

SVM decision score

w·x + b; positive predicts +1, negative predicts −1, zero is on the boundary.

SVM functional margin

m = y(w·x + b), where y is the actual class, +1 or −1.

SVM margin m = 1

Correctly classified and exactly on its class margin boundary.

SVM margin m > 1

Correctly classified and outside the margin.

SVM margin 0 < m < 1

Correctly classified but inside the margin.

SVM margin m = 0 or m < 0

Zero means on the decision boundary; negative means misclassified.

Hard-margin SVM

Requires perfect linear separation of training examples.

Soft-margin SVM

Allows some margin violations or misclassification to handle noisy data.