ML Theory

Bias-variance decomposition

Expected test error = bias² + variance + irreducible noise. High bias = underfit (too simple); high variance = overfit (too sensitive to training set). Always tie to a concrete fix (more capacity ↓bias; regularization/more data ↓variance).

Regularization menu — L1 vs L2 vs dropout vs early-stop

L1 (Lasso): sparse weights → feature selection. L2 (Ridge): smooth shrinkage, small weights. Dropout: randomly zero units → ensemble effect. Early stopping: halt before it overfits. All fight variance.

AdamW vs Adam+L2 (the gotcha)

Adam+L2 couples weight decay INTO the gradient, so the adaptive learning rate scales it unevenly (weak decay on high-gradient params). AdamW DECOUPLES weight decay — applies it directly to weights — which is why AdamW generalizes better. Common follow-up.

Precision vs recall vs F1 — which matters more?

Precision = of predicted-positive, how many right (cost of false positives). Recall = of actual-positive, how many caught (cost of false negatives). F1 = harmonic mean. Answer is scenario-driven: spam→precision, cancer screen→recall.

ROC-AUC vs PR-AUC under class imbalance

The #1 cited gotcha: on rare-positive data ROC-AUC is optimistic because the huge true-negative count inflates it. PR-AUC (precision vs recall) ignores true negatives → far more informative when positives are rare.

Cross-validation types

k-fold (average over k splits) · stratified (preserve class ratios — use for imbalance) · time-series (NO shuffle; train on past, test on future — shuffling leaks the future). Match the split to the data's structure.

Data leakage — the canonical form

Fitting preprocessing (scaler, imputer, encoder, feature selection) on the FULL dataset before the train/test split → test info bleeds into training → inflated scores that collapse in production. Fit only on train, transform test. Drill jointly with cross-validation.

Bagging vs boosting

Bagging (Random Forest): train trees in PARALLEL on bootstrap samples, average → ↓variance. Boosting (GBM/XGBoost): train SEQUENTIALLY, each learner fixes the last's errors → ↓bias. Parallel-independent vs sequential-corrective.

Class imbalance — the menu

Resample (over/under, SMOTE) · class weights in the loss · adjust the decision threshold · pick the right metric (PR-AUC/F1, not accuracy). Choice "depends on cost asymmetry" between FP and FN.

Feature scaling — which models need it?

NEEDED for distance/gradient models (k-NN, k-means, SVM, linear/logistic, neural nets). NOT needed for tree-based models (splits are scale-invariant). Know BOTH directions — the "trees don't need it" half is the tell.

Curse of dimensionality

As dimensions grow, points become nearly equidistant (distance metrics lose meaning), data gets sparse, and you need exponentially more samples. Hits k-NN/clustering hardest → motivates dimensionality reduction.

SGD vs Adam

SGD (+momentum): simple, often better final generalization, needs LR tuning. Adam: per-parameter adaptive LR (fast convergence, robust to LR), sometimes worse generalization. Adam to move fast; SGD to squeeze final quality.

Data drift vs concept drift

Data drift = the input distribution P(X) shifts (new inputs). Concept drift = the relationship P(y|X) shifts (model wrong even on familiar inputs). Monitor input stats + prediction distribution + delayed-label metrics; respond with retrain / rollback. The production tail — my eval-eng turf.