ML

What is the bias-variance tradeoff?

Expected error = bias^2 + variance + irreducible noise. Simple models have high bias (underfit). Flexible models have high variance (overfit). Aim for the complexity that minimizes total error.

Signs of overfitting vs underfitting

Overfitting: low train error, much higher val error. Underfitting: both train and val error high.

How do you fix overfitting?

Get more data or augment it, add regularization (L1/L2, dropout), simplify the model, use early stopping, do feature selection, use ensembling/bagging, or add noise.

How do you fix underfitting?

Use a bigger or more expressive model, add features, reduce regularization, train longer, or tune the learning rate.

Supervised vs unsupervised vs self-supervised learning

Supervised: learn from labeled (x, y). Unsupervised: find structure in unlabeled x (clustering, density, dim. reduction). Self-supervised: create labels from the data itself (next-token prediction, masked modeling, contrastive learning).

Generative vs discriminative models

Discriminative models learn p(y|x) or a decision boundary (logistic regression, SVM). Generative models learn p(x|y)p(y) or p(x) and can sample data (Naive Bayes, GMM, VAE, diffusion, GPT).

Parametric vs non-parametric models

Parametric: a fixed number of parameters regardless of data size (linear regression, NN). Non-parametric: complexity grows with the data (kNN, decision trees, kernel methods, GPs).

Train / validation / test split purpose

Train: fit the parameters. Validation: tune hyperparameters and select the model. Test: a final unbiased estimate, touched once.

What is k-fold cross-validation?

Split the data into k folds. Train on k-1 folds and validate on the remaining one, rotating through all k. Average the scores. Useful for small data and gives a variance estimate. Use stratified folds for imbalanced classes and time-based splits for time series.

What is data leakage? Examples?

Information from the target or test set sneaking into training. Examples: fitting the scaler or imputer on the full dataset, features computed from the future, duplicate rows across splits, a target-derived feature. Fix: split first and fit preprocessing only on train.

Curse of dimensionality

In high dimensions, data becomes sparse, distances concentrate (all points are roughly equally far apart), and you need exponentially more data. This hurts kNN and clustering. Fix with dimensionality reduction or feature selection.

No Free Lunch theorem

No single algorithm is best across all problems. Performance depends on how well the model's assumptions match the data.

What is a hyperparameter vs a parameter?

Parameters are learned from data (weights). Hyperparameters are set before training (learning rate, depth, k, regularization strength) and tuned on validation data.

Hyperparameter tuning methods

Grid search, random search (often better in high dimensions), Bayesian optimization (e.g., Optuna), Hyperband or successive halving, and early stopping of bad trials.

L1 vs L2 regularization

L1 (Lasso) adds lambdasum|w|. It produces sparse weights and does feature selection. L2 (Ridge) adds lambdasum w^2. It shrinks weights smoothly and handles collinearity. Elastic Net combines both.

Why does L1 produce sparsity?

The L1 ball has corners on the axes, so the optimum often lands where some weights are exactly 0. Equivalently, the L1 gradient is a constant magnitude that pushes small weights all the way to zero.

Bayesian interpretation of L1/L2

L2 is a MAP estimate with a Gaussian prior on the weights. L1 is a MAP estimate with a Laplace prior.

What is dropout?

During training, randomly zero each activation with probability p, which prevents co-adaptation and acts like an ensemble. At inference all units are used, with activations scaled (inverted dropout scales by 1/(1-p) during training instead).

What is early stopping?

Stop training when the validation loss stops improving (with a patience window), and keep the best checkpoint. This is implicit regularization.

What is weight decay? Same as L2?

Weight decay shrinks the weights each step. It is equivalent to L2 for plain SGD but NOT for Adam. AdamW decouples weight decay from the adaptive gradient update.

What is data augmentation?

Creating label-preserving transformations of the training data (flips, crops, color jitter, noise, mixup, back-translation) to improve generalization.

Linear regression assumptions

Linearity, independent errors, homoscedasticity (constant variance), normally distributed errors (needed for inference), and no strong multicollinearity.

Closed-form solution for linear regression

w = (X^T X)^(-1) X^T y (the normal equation). The cost is O(d^3) for the inverse. Use gradient descent when d is large or X^T X is ill-conditioned.

Why use MSE for regression?

MSE equals maximum likelihood under Gaussian noise. It is differentiable and convex for linear models. Its weakness is sensitivity to outliers (use MAE or Huber loss instead).

Logistic regression: how does it work?

Logistic regression applies a sigmoid to a linear score: p = sigma(w^T x + b). It is trained by minimizing binary cross-entropy (log loss), which is convex. It produces a linear decision boundary and the output is a probability.

Why not use MSE for logistic regression?

Sigmoid plus MSE gives a non-convex loss with vanishing gradients when predictions are confidently wrong. Cross-entropy is the MLE for a Bernoulli outcome and gives the clean gradient (p - y)x.

Sigmoid vs softmax

Sigmoid maps a scalar to (0, 1), for binary or multi-label problems. Softmax maps a vector to a probability distribution that sums to 1, for multiclass with exactly one label.

How does kNN work? Pros/cons

kNN predicts by majority vote or average of the k nearest training points. It has no training step and is flexible. However, prediction is slow (O(nd) per query), it needs feature scaling, and it suffers from the curse of dimensionality. Small k means high variance; large k means high bias.

Naive Bayes

Naive Bayes applies Bayes' rule assuming features are conditionally independent given the class. It is fast, works well for text (spam filtering), and needs little data. The independence assumption is usually false, but rankings are often still good.

SVM: core idea

An SVM finds the hyperplane that maximizes the margin between classes. Only the support vectors matter. A soft margin (parameter C) allows violations via hinge loss. A large C means less regularization.

What is the kernel trick?

Compute inner products in a high-dimensional feature space via a kernel function K(x, x') without explicitly mapping the data. Common kernels are RBF and polynomial. This lets linear methods learn nonlinear boundaries.

How does a decision tree split?

Greedily choose the feature and threshold that maximizes impurity reduction. For classification that means Gini or entropy (information gain); for regression, variance or MSE. Recurse until a stopping criterion is met (max depth, min samples).

Gini impurity vs entropy

Gini = 1 - sum p_i^2. Entropy = -sum p_i log p_i. Both measure node impurity and give similar splits. Gini is slightly cheaper to compute.

Bagging vs boosting

Bagging trains models in parallel on bootstrap samples and averages them, which reduces variance (Random Forest). Boosting trains models sequentially, each fixing the errors of the previous ones, which reduces bias (AdaBoost, Gradient Boosting).

Random forest: what makes it work?

It bags many deep trees and also uses a random subset of features at each split, which decorrelates the trees. Averaging decorrelated high-variance trees reduces variance. It also provides out-of-bag error and feature importance.

Gradient boosting: how does it work?

Start with a constant prediction. Each new shallow tree fits the negative gradient of the loss (residuals for MSE). Add it with a learning rate (shrinkage). XGBoost, LightGBM, and CatBoost add regularization, second-order gradients, and speedups.

Why do tree models not need feature scaling?

Splits depend only on the ordering of feature values, not their magnitude.

Why are GBMs often best on tabular data?

They handle mixed feature types, nonlinearity, interactions, and missing values. They are robust to scaling and need relatively little tuning compared to NNs on structured data.

Stacking

Train a meta-model on the out-of-fold predictions of several base models to learn how best to combine them.

k-means algorithm

Initialize k centroids (k-means++). Then repeat: assign each point to its nearest centroid, and move each centroid to the mean of its points. Stop at convergence. It minimizes within-cluster squared distance and finds a local optimum.

k-means limitations

You must pick k (elbow method or silhouette score). It assumes spherical, similarly sized clusters, is sensitive to initialization and outliers, and needs feature scaling.

k-means vs GMM

GMM is soft clustering with Gaussian components (each with its own covariance), fit by EM. k-means is hard assignment and a special case of GMM with equal spherical variance.

DBSCAN

Density-based clustering with parameters eps and minPts. It finds arbitrarily shaped clusters, labels outliers as noise, and does not need k. It struggles when clusters have varying densities.

PCA: what and how

PCA projects data onto the orthogonal directions of maximum variance. These are the eigenvectors of the covariance matrix, computed via SVD of the centered data. Center (and usually scale) the data first. Choose the number of components by explained variance.

PCA vs t-SNE / UMAP

PCA is linear, fast, deterministic, preserves global variance, and can transform new data. t-SNE and UMAP are nonlinear and preserve local neighborhoods, so they are good for visualization, but distances between clusters are not meaningful.

What is an autoencoder?

An encoder compresses the input to a bottleneck latent and a decoder reconstructs it, trained on reconstruction loss. Uses: dimensionality reduction, denoising, anomaly detection (high reconstruction error flags anomalies).

What is the EM algorithm?

EM handles latent-variable MLE by iterating two steps. E-step: compute the posterior over the latent variables given the current parameters. M-step: maximize the expected log-likelihood. It is guaranteed not to decrease the likelihood (used for GMMs, HMMs).

Precision vs recall

Precision = TP/(TP+FP): of the predicted positives, how many are right. Recall = TP/(TP+FN): of the actual positives, how many were found.

When to prioritize precision vs recall?

Prioritize precision when false positives are costly (spam filter, recommendations). Prioritize recall when false negatives are costly (cancer screening, fraud, defect detection).

F1 score

The harmonic mean of precision and recall: 2PR/(P+R). It is useful for imbalanced data. The F-beta score weights recall beta times as much as precision.

ROC curve and AUC

The ROC curve plots TPR vs FPR across thresholds. AUC is the probability that a random positive is ranked above a random negative: 0.5 = random, 1 = perfect. It is threshold-independent.

ROC-AUC vs PR-AUC

For heavy class imbalance prefer PR-AUC. ROC-AUC can look optimistic because the large number of true negatives keeps the FPR low.

Why is accuracy misleading?

With class imbalance (e.g., 99% negatives), predicting all-negative gets 99% accuracy while being useless.

Confusion matrix

A table of actual vs predicted classes: TP, FP, FN, TN. All classification metrics derive from it.

Macro vs micro vs weighted averaging

Macro: average the per-class metrics equally (treats rare classes as important). Micro: pool all TP/FP/FN (dominated by frequent classes). Weighted: macro average weighted by class support.

Regression metrics: MSE, RMSE, MAE, R^2

MSE penalizes large errors heavily. RMSE is MSE in the target's units. MAE is robust to outliers. R^2 = 1 - SS_res/SS_tot is the fraction of variance explained (it can be negative).

Log loss

Log loss = -mean[y log p + (1-y) log(1-p)]. It heavily penalizes confident wrong predictions and measures probability quality, not just ranking.

What is calibration?

A model is calibrated when its predicted probabilities match observed frequencies (of the 0.8 predictions, 80% are positive). Check with reliability diagrams. Fix with Platt scaling or isotonic regression. Boosted trees and deep nets are often miscalibrated.

How to choose a classification threshold?

Pick it based on business cost, a precision/recall target, or by maximizing F1 or Youden's J on validation data. The default of 0.5 is rarely optimal for imbalanced problems.

Ranking metrics

Precision@k, Recall@k, MAP, MRR (mean reciprocal rank of the first relevant item), NDCG (graded relevance with a position discount).

How to handle class imbalance?

Use better metrics (PR-AUC, F1), class weights or a weighted loss, oversampling (SMOTE) or undersampling, focal loss, threshold tuning, or more minority data. Resample only the training split.

How to handle missing data?

Drop rows or columns, impute (mean, median, mode, kNN, model-based), add a 'was missing' indicator, or use models that handle NaN natively (XGBoost). Check whether data is missing at random.

Normalization vs standardization

Min-max normalization scales to [0, 1] and is sensitive to outliers. Standardization (z-score) gives mean 0 and std 1. It is needed for distance-based methods and gradient descent, but not for trees.

Encoding categorical variables

One-hot (low cardinality), ordinal (ordered categories), target/mean encoding (high cardinality, watch for leakage), hashing, or learned embeddings.

Feature selection methods

Filter (correlation, mutual information, chi-squared), wrapper (recursive feature elimination), embedded (L1, tree importance), and SHAP or permutation importance.

Permutation importance vs impurity importance

Impurity (Gini) importance is biased toward high-cardinality features and computed on training data. Permutation importance measures the score drop when a feature is shuffled on validation data, which is more reliable.

What are SHAP values?

SHAP gives each feature's contribution to an individual prediction based on Shapley values from game theory. The contributions add up to the prediction minus the baseline. It provides both local and global explanations.

Multicollinearity: problem and fix

Correlated features make linear coefficients unstable and hard to interpret, though prediction is often fine. Detect with VIF or a correlation matrix. Fix with ridge regression, dropping features, or PCA.

Data drift vs concept drift

Data (covariate) drift: the input distribution p(x) changes. Concept drift: the relationship p(y|x) changes. Monitor feature distributions and model performance, and retrain.

Gradient descent: batch vs SGD vs mini-batch

Batch uses the full dataset: an exact gradient but slow. SGD uses one sample: noisy but cheap, and the noise can escape bad minima. Mini-batch (32-512 samples) is the standard middle ground and GPU-friendly.

Effect of learning rate

Too high: divergence or oscillation. Too low: slow training or getting stuck. Use LR schedules (step decay, cosine, warmup) and an LR range test.

What is momentum?

Momentum accumulates an exponentially decaying average of past gradients. This speeds movement along consistent directions and dampens oscillation in ravines.

Adam optimizer

Adam combines momentum (first moment) with RMSProp-style per-parameter scaling (second moment), plus bias correction. It is robust with little tuning. AdamW is the standard for transformers.

Why use learning rate warmup?

Early on, Adam's second-moment estimates are noisy and weights are random, so large steps can destabilize training. Ramping the LR up gradually avoids this (standard for transformers).

Convex vs non-convex optimization

Convex problems: any local minimum is global (linear and logistic regression, SVM). Neural nets are non-convex, but in high dimensions saddle points are the bigger issue than bad local minima.

What is gradient clipping?

Gradient clipping rescales gradients whose norm exceeds a threshold, which prevents exploding gradients (common in RNNs and transformer training).

Effect of batch size

Larger batches give smoother gradients and better hardware use, but may generalize slightly worse (sharp minima) and need a higher LR (linear scaling rule). Smaller batches are noisier and act as a regularizer.

What is backpropagation?

Backpropagation applies the chain rule to compute the gradient of the loss with respect to every weight. It does a forward pass to compute and cache activations, then a backward pass that propagates gradients from the output layer to the input. Cost is about 2x the forward pass.

Why do we need nonlinear activations?

Without them, stacked linear layers collapse into a single linear map. Nonlinearity lets networks approximate complex functions (universal approximation).

ReLU vs sigmoid vs tanh

Sigmoid and tanh saturate, which causes vanishing gradients, and sigmoid is not zero-centered. ReLU = max(0, x) is cheap and does not saturate for x > 0, but can cause dead neurons. Variants: Leaky ReLU, GELU (used in transformers), SiLU/Swish.

Vanishing / exploding gradients

Repeated multiplication through many layers shrinks or blows up gradients. Fixes: ReLU-family activations, good initialization (He/Xavier), normalization, residual connections, gradient clipping, and LSTM/GRU for RNNs.

Xavier vs He initialization

Both scale initial weights to keep activation variance stable across layers. Xavier (Glorot) uses variance about 1/fan_avg, for tanh/sigmoid. He uses variance 2/fan_in, for ReLU. All-zero init fails because of symmetry.

Batch normalization

BatchNorm normalizes each feature across the batch to mean 0 and variance 1, then applies a learned scale and shift. It stabilizes and speeds training and allows higher LR. At inference it uses running statistics. It is batch-size dependent.

BatchNorm vs LayerNorm

BatchNorm normalizes across the batch dimension (common in CNNs). LayerNorm normalizes across the features of each sample, so it is independent of batch size. It is used in transformers and RNNs.

What are residual (skip) connections?

A residual block computes y = x + F(x). Gradients flow directly through the identity path, which enables very deep networks (ResNet, transformers). Layers learn a residual correction.

Softmax with cross-entropy gradient

The gradient with respect to the logits is simply (p - y), where p is the softmax output and y is the one-hot label. Clean and stable when fused (log-softmax).

Universal approximation theorem

A network with one hidden layer and enough units can approximate any continuous function on a compact set. It says nothing about learnability or efficiency. Depth is more parameter-efficient in practice.

Debugging a NN that won't learn

Overfit a single tiny batch first. Check data and labels, the loss/activation pairing, the learning rate, gradient norms (NaN, zero), input normalization, train vs eval mode, and the shapes.

Train vs eval mode in PyTorch

model.eval() switches dropout off and makes BatchNorm use running statistics. Also wrap inference in torch.no_grad() to skip gradient tracking.

Why CNNs for images?

Local connectivity plus weight sharing exploit spatial structure. This gives far fewer parameters than fully connected layers and translation equivariance. Stacked layers build hierarchical features (edges, then textures, then parts, then objects).

Conv output size formula

out = floor((W - K + 2P) / S) + 1, where W is the input size, K the kernel size, P the padding, and S the stride.

What does pooling do?

Pooling downsamples (max or average), which grows the receptive field, reduces computation, and adds small translation invariance. Global average pooling replaces fully connected heads.

What is a 1x1 convolution?

A 1x1 conv mixes channels at each pixel. It is used to reduce or expand channels cheaply (bottlenecks in Inception and ResNet).

What is the receptive field?

The region of the input that affects a given activation. It grows with depth, kernel size, stride, pooling, and dilation.

What is transfer learning / fine-tuning?

Start from a model pretrained on large data and adapt it to a new task: freeze the backbone and train the head, or fine-tune all layers with a low LR. It works well with small target datasets.

Object detection vs segmentation vs classification

Classification: one label per image. Detection: bounding boxes plus classes (YOLO, Faster R-CNN). Semantic segmentation: a class per pixel (U-Net). Instance segmentation: separates individual objects (Mask R-CNN).

What is IoU?

Intersection over Union = area of overlap / area of union between predicted and true boxes or masks. Used for detection matching and mAP.

Non-max suppression (NMS)

NMS keeps the highest-scoring box and removes boxes overlapping it above an IoU threshold, then repeats. It dedupes detections.

RNN core idea and problem

An RNN carries a hidden state updated at each time step: h_t = f(W h_(t-1) + U x_t). It struggles with long-range dependencies (vanishing gradients) and is sequential, so it is slow to train.

LSTM vs GRU

Both use gates to control memory flow and mitigate vanishing gradients. An LSTM has input, forget, and output gates plus a cell state. A GRU has update and reset gates, is simpler, has fewer parameters, and often performs similarly.