pin acc

Logistic regression: what does it model?

P(y=1|x) = sigmoid(w·x + b). A linear model whose output is squashed into a probability. Decision boundary is linear (w·x + b = 0).

Sigmoid function formula and range

σ(z) = 1 / (1 + e^(-z)). Range (0, 1). σ(0) = 0.5. Derivative: σ(z)(1 − σ(z)).

Logistic regression loss function

Binary cross-entropy (log loss): −[y log p + (1−y) log(1−p)]. Equal to the negative log-likelihood under a Bernoulli model.

Why not use MSE for logistic regression?

MSE + sigmoid makes the loss non-convex in w (local minima) and gives weak gradients when predictions are confidently wrong. Log loss is convex and penalizes confident mistakes heavily.

Gradient of log loss w.r.t. weights in logistic regression

∇w = (p − y)·x (averaged over samples). Same form as linear regression's gradient, with p = σ(w·x).

Interpret a logistic regression coefficient

A one-unit increase in feature x_j multiplies the odds of y=1 by e^(w_j), holding other features fixed. w_j is the change in log-odds.

Log-odds (logit)

log(p / (1−p)). Logistic regression assumes log-odds are linear in the features.

Does logistic regression have a closed-form solution?

No. Optimized iteratively (gradient descent, Newton's method / IRLS, L-BFGS). The loss is convex, so it converges to the global optimum.

What happens to logistic regression on linearly separable data with no regularization?

Weights grow without bound (to infinity) because the loss keeps decreasing as predictions approach 0/1. Regularization fixes this.

Multiclass extension of logistic regression

Softmax regression: p_k = e^(z_k) / Σ e^(z_j), trained with categorical cross-entropy. Or one-vs-rest binary classifiers.

Bias (in bias-variance tradeoff)

Error from wrong assumptions / a model too simple to capture the true pattern. High bias → underfitting (high train AND test error).

Variance (in bias-variance tradeoff)

Error from sensitivity to the specific training set. High variance → overfitting (low train error, high test error).

Bias-variance decomposition of expected MSE

Expected error = Bias² + Variance + Irreducible noise (σ²).

How to fix high bias (underfitting)

More complex model, more/better features, less regularization, train longer, boosting.

How to fix high variance (overfitting)

More data, simpler model, more regularization, dropout, early stopping, feature selection, bagging/ensembling, data augmentation.

How do you diagnose bias vs variance?

Compare train vs validation error. Both high → bias. Train low, val much higher → variance. Learning curves (error vs training-set size) make this visible.

Learning curve: what does a large, persistent gap between train and val error mean?

High variance. More data will likely help.

Learning curve: train and val errors converge at a high value

High bias. More data won't help; need a more expressive model or better features.

Regularization: purpose

Add a penalty on model complexity to the loss to reduce overfitting. Loss = data loss + λ·penalty(w).

L2 regularization (Ridge)

Penalty λΣw². Shrinks weights smoothly toward zero but rarely exactly zero. Gaussian prior on weights (MAP view). Handles correlated features by spreading weight.

L1 regularization (Lasso)

Penalty λΣ|w|. Drives many weights to exactly zero → sparse model / built-in feature selection. Laplace prior on weights.

Why does L1 produce sparsity and L2 doesn't?

L1's constraint region is a diamond with corners on the axes, so the loss contours tend to touch it at a corner (some w=0). L1's gradient magnitude is constant near 0, while L2's gradient vanishes as w→0.

Effect of increasing λ (regularization strength)

Higher bias, lower variance. Too large → underfit; too small → overfit. Tune λ with cross-validation.

Other forms of regularization besides L1/L2

Dropout, early stopping, data augmentation, weight decay, batch norm (mild), max-norm constraints, tree pruning / depth limits, label smoothing.

Why scale features before regularized regression?

The penalty treats all weights equally, so features on larger scales would be penalized unfairly. Standardize first.

Decision tree: how does it split?

Greedily chooses the feature + threshold that most reduces impurity (Gini or entropy for classification, variance/MSE for regression) at each node.

Gini impurity formula

Gini = 1 − Σ p_k². 0 = pure node. Max for 2 classes = 0.5.

Entropy (decision trees)

H = −Σ p_k log₂ p_k. 0 = pure. Max for 2 classes = 1 bit.

Information gain

Parent entropy − weighted average of child entropies. The split with highest gain is chosen.

Decision tree pros

Interpretable, handles nonlinearity and feature interactions, no feature scaling needed, handles mixed feature types, fast inference.

Decision tree cons

High variance (overfits easily, unstable to small data changes), greedy splits aren't globally optimal, axis-aligned boundaries, poor extrapolation.

How to prevent decision tree overfitting

Max depth, min samples per leaf/split, min impurity decrease, max leaf nodes, cost-complexity (post-)pruning. Or ensemble them.

Do decision trees need feature scaling?

No. Splits depend only on the ordering of values, not their scale.

Bagging

Bootstrap aggregating: train many models on bootstrap samples (sample with replacement) and average/vote. Mainly reduces variance.

Random forest

Bagged decision trees + random subset of features considered at each split. Feature randomness decorrelates trees, so averaging reduces variance more.

Boosting

Train weak learners sequentially, each focusing on the previous ensemble's errors. Mainly reduces bias (can overfit if run too long).

Gradient boosting

Each new tree fits the negative gradient of the loss (the pseudo-residuals) of the current ensemble. Added with a learning rate (shrinkage). Examples: XGBoost, LightGBM, CatBoost.

Random forest vs gradient boosted trees

RF: parallel, deep trees, reduces variance, robust with little tuning. GBDT: sequential, shallow trees, reduces bias, usually higher accuracy but more hyperparameter-sensitive and easier to overfit.

Feature importance in tree models

Impurity-based (total gain from splits on that feature; biased toward high-cardinality features) or permutation importance (drop in metric when the feature is shuffled; more reliable). SHAP for per-prediction attributions.

Convex function definition

f(λx + (1−λ)y) ≤ λf(x) + (1−λ)f(y) for all x, y and λ∈[0,1]. The line segment between any two points on the graph lies on or above the graph.

Why convexity matters in ML

Any local minimum of a convex function is a global minimum, so gradient descent (with a suitable step size) reaches the optimum.

Second-order test for convexity

Twice-differentiable f is convex iff its second derivative ≥ 0 (1D), or its Hessian is positive semidefinite (multi-D).

Strictly convex vs convex

Strictly convex: strict inequality, so there's at most one (a unique) global minimum.

Examples of convex losses / models

MSE for linear regression, log loss for logistic regression, hinge loss (SVM), L1 and L2 penalties. Sum of convex functions is convex.

Are neural network losses convex?

No. Non-convex due to hidden layers / nonlinearities and weight symmetries. In practice SGD still finds good minima; saddle points are a bigger issue than bad local minima in high dimensions.

Is a sum of convex functions convex? A max of convex functions?

Yes and yes. Composition is convex under conditions (e.g., convex non-decreasing f of convex g).

Gradient descent update rule

w ← w − η ∇L(w). η = learning rate.

Batch vs stochastic vs mini-batch gradient descent

Batch: whole dataset per step (stable, slow). SGD: one example (noisy, fast, can escape saddle points). Mini-batch: compromise, standard in practice, uses GPU parallelism.

Learning rate too high vs too low

Too high: diverges or oscillates. Too low: slow convergence, can get stuck. Use schedules (decay, warmup, cosine) or adaptive optimizers.

Momentum

Accumulates an exponentially decaying average of past gradients to smooth updates and accelerate along consistent directions; dampens oscillation.

Adam optimizer

Combines momentum (1st moment estimate) with per-parameter adaptive learning rates (RMSProp-style 2nd moment), with bias correction. Good default.

Model evaluation: why split train / validation / test?

Train fits parameters, validation tunes hyperparameters and does model selection, test gives an unbiased final estimate. Using test for tuning leaks information and inflates results.

k-fold cross-validation

Split data into k folds; train on k−1, validate on the remaining one, rotate, and average. Better use of limited data; gives variance of the estimate.

Stratified k-fold

Keeps class proportions the same in each fold. Important for imbalanced classification.

When NOT to use random k-fold

Time series (use forward-chaining / time-based splits) and grouped data (users, sessions; use group k-fold) — otherwise you leak future or same-entity info.

Data leakage

Information unavailable at prediction time sneaks into training (e.g., future data, target-derived features, scaling fit on full data before split). Gives overly optimistic offline metrics.

Confusion matrix terms

TP, FP (Type I error), FN (Type II error), TN.

Precision

TP / (TP + FP). Of predicted positives, how many are correct? Prioritize when false positives are costly (e.g., spam filter flagging real email).

Recall

TP / (TP + FN). Of actual positives, how many did we catch? Prioritize when false negatives are costly (e.g., disease screening, fraud).

F1 score

Harmonic mean of precision and recall: 2PR / (P + R). Useful for imbalanced classes when you care about both.

Specificity

TN / (TN + FP). True negative rate; important for medical testing, confirmatory tests, fraud or security alerts

Why is accuracy misleading with class imbalance?

With 99% negatives, predicting all-negative gives 99% accuracy but zero recall. Use precision/recall, F1, PR-AUC instead.

ROC curve

TPR (recall) vs FPR (FP / (FP + TN)) across all classification thresholds.

ROC-AUC interpretation

Probability a random positive is ranked above a random negative. 0.5 = random, 1.0 = perfect. Threshold-independent, measures ranking quality.

PR-AUC vs ROC-AUC

PR-AUC is more informative under heavy class imbalance (rare positives), since ROC's FPR stays tiny when negatives dominate, making ROC-AUC look optimistic.

How do you choose a classification threshold?

Based on business costs of FP vs FN: maximize F1, hit a target precision or recall, or minimize expected cost. Tune on validation data, not test.

Calibration

Post-training: whether predicted probabilities match observed frequencies (of things predicted 0.7, ~70% are positive). Check with reliability diagrams; fix with Platt scaling or isotonic regression. Important for ads/CTR models.

Regression metrics

MSE (penalizes large errors), RMSE (same units as target), MAE (robust to outliers), R² (fraction of variance explained), MAPE (relative error).

MSE vs MAE

MSE squares errors → sensitive to outliers, optimal prediction is the mean. MAE → robust, optimal prediction is the median.

Log loss as an evaluation metric

Measures quality of predicted probabilities, not just ranking. Heavily penalizes confident wrong predictions.

L = −[y·log(p) + (1−y)·log(1−p)]

Offline vs online evaluation

Offline: metrics on held-out historical data (AUC, NDCG). Online: A/B test on real users measuring business metrics (engagement, saves, CTR). Offline gains don't always translate online.

Handling class imbalance

Resample (oversample minority, e.g. SMOTE; undersample majority), class weights in the loss, focal loss, better metrics (PR-AUC, F1), threshold tuning.

Feature scaling: standardization vs normalization

Standardization: (x − μ)/σ → mean 0, std 1. Min-max normalization: rescale to [0,1]. Needed for gradient-based, distance-based (kNN, k-means, SVM) and regularized models; not for trees.

Encoding categorical features

One-hot (low cardinality), target/mean encoding (watch leakage), hashing trick, or learned embeddings (high cardinality, like user/pin IDs).

Handling missing values

Drop rows/columns, impute (mean/median/mode, model-based), add a "was missing" indicator, or use models that handle missingness natively (XGBoost/LightGBM).

Curse of dimensionality

In high dimensions data becomes sparse and distances become less meaningful (nearest and farthest points similar), so distance-based methods degrade and more data is needed.

PCA

Projects data onto orthogonal directions of maximum variance (eigenvectors of the covariance matrix / via SVD). Unsupervised, linear dimensionality reduction. Standardize first. Simply: find the directions along which the data spreads out the most, then describe each point using only those directions

Linear regression assumptions

Linearity, independent errors, homoscedasticity (constant error variance), normally distributed errors (for inference), low multicollinearity.

Linear regression closed-form solution

Normal equation: w = (XᵀX)⁻¹ Xᵀy. O(d³) to invert; use gradient descent for large d or huge data. Ridge version: (XᵀX + λI)⁻¹ Xᵀy, which is always invertible.

MLE vs MAP

MLE: choose parameters maximizing P(data|θ), aka which parameters make the data I saw most likely. MAP: maximize P(data|θ)·P(θ), i.e. adds a prior, aka which parameters are most likely, given the data AND what I believed beforehand. L2 regularization = MAP with Gaussian prior; L1 = Laplace prior.

Generative vs discriminative models

Discriminative model P(y|x) directly (logistic regression, SVM, most NNs). Generative model P(x, y) or P(x|y)P(y) (Naive Bayes, GMMs) and can generate data.

Naive Bayes

Applies Bayes' rule assuming features are conditionally independent given the class. Fast, works well for text despite the unrealistic assumption. (spam filtering, sentiment analysis)

k-Nearest Neighbors (kNN)

Predict by majority vote / average of the k closest training points. No training phase; slow at inference O(n·d) per query; needs feature scaling. Small k → high variance; large k → high bias.

SVM

Finds the maximum-margin hyperplane between classes. Hinge loss + L2. Soft margin parameter C trades margin width vs misclassification. Support vectors are the points defining the margin.

Kernel trick

Compute inner products in a high-dimensional feature space via a kernel function (e.g. RBF, polynomial) without explicitly mapping, letting linear methods like SVM learn nonlinear boundaries.

k-means clustering

Alternate assigning points to nearest centroid and recomputing centroids. Minimizes within-cluster sum of squares. Must pick k (elbow method, silhouette score); sensitive to init (use k-means++) and scale; assumes roughly spherical clusters.

Neural network: why do we need nonlinear activations?

Without them, stacked linear layers collapse into a single linear transformation, no matter how deep.

ReLU vs sigmoid/tanh

ReLU max(0,x): cheap, no saturation for x>0, mitigates vanishing gradients; can cause "dead" neurons (Leaky ReLU / GELU help). Sigmoid/tanh saturate, so gradients vanish in deep nets.

Backpropagation

Applying the chain rule backward through the computation graph to compute gradients of the loss w.r.t. every parameter efficiently, reusing intermediate results.

Vanishing / exploding gradients

Gradients shrink or blow up as they're multiplied through many layers. Fixes: ReLU, careful init (He/Xavier), batch/layer norm, residual connections, gradient clipping (for exploding).

Dropout

A regularization trick for neural networks that reduces overfitting. Randomly switch off (zero) a fraction p of activations during training (scale the rest); disabled at inference. Prevents co-adaptation (neurons can’t rely on specific partners); acts like an implicit ensemble.

Early stopping

Stop training when validation loss stops improving (with some patience). Simple, effective regularization.

Hyperparameter vs parameter

Parameters are learned from data (weights). Hyperparameters are set before training (learning rate, λ, tree depth, k) and tuned via validation: grid search, random search, Bayesian optimization.

Ensembling: why does averaging help?

Averaging models whose errors are not perfectly correlated reduces variance; the more decorrelated the models, the larger the gain.

Parametric vs non-parametric models

Parametric: fixed number of parameters regardless of data size (linear/logistic regression). Non-parametric: complexity grows with data (kNN, decision trees, kernel SVM).