Linear Regression — Interview Question Bank

Comprehensive vocabulary flashcards covering fundamental concepts, mathematical formulations, diagnostic metrics, regularization techniques, and leakage scenarios from the Linear Regression Interview Question Bank.

Linear Regression

A statistical model that represents a continuous target as a weighted sum of input features plus an error term ($y = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \dots + \beta_p x_p + \varepsilon$), where "linear" strictly refers to being linear in the parameters ($\beta$), not necessarily in the features.

Homoscedasticity

A key assumption of linear regression stating that the model error terms have a constant variance across all fitted values.

Exogeneity

The condition in linear regression where the error term has a conditional mean of zero given the predictors ($E(\varepsilon | X) = 0$), implying that features are measured without error and are uncorrelated with the error term.

Residual

The observed difference between the actual target value and the fitted model prediction ($e = y - \hat{y}$), serving as the sample estimate of the unobservable population error term ($\varepsilon$).

Ordinary Least Squares (OLS)

An estimation technique that determines regression parameters by minimizing the sum of squared residuals: $\min_{\beta} \sum_{i=1}^n (y_i - \hat{y}_i)^2$.

Normal Equation

The analytical closed-form solution to Ordinary Least Squares given by $\hat{\beta} = (X^T X)^{-1} X^T y$, which fails when $X^T X$ is singular due to perfect multicollinearity or when features outnumber samples ($p > n$).

Gauss–Markov Theorem

A foundational theorem stating that under the conditions of linearity, exogeneity, homoscedasticity, and no autocorrelation, the OLS estimator is BLUE: the Best (minimum variance) Linear Unbiased Estimator.

Mean Absolute Error (MAE)

A regression evaluation metric reported in the same units as $y$ with low sensitivity to outliers, ideal when every unit of prediction error carries an equal cost.

Mean Squared Error (MSE)

A regression loss and evaluation metric expressed in $y^2$ units that is smooth and differentiable, disproportionately penalizing large errors.

Root Mean Squared Error (RMSE)

The square root of the Mean Squared Error, measured in the same units as $y$, which penalizes large misses while providing an interpretable scale for stakeholders.

Coefficient of Determination ($R^2$)

The proportion of variance in the target variable explained by the model, calculated as $1 - \frac{SS_{\text{res}}}{SS_{\text{tot}}}$, which can become negative on test data if the model performs worse than predicting the mean.

Adjusted $R^2$

A modified version of $R^2$ defined as $R^2_{\text{adj}} = 1 - (1 - R^2) \cdot \frac{n - 1}{n - p - 1}$ that penalizes the inclusion of unnecessary predictor variables, increasing only if a new feature improves the model more than expected by chance.

Mean Absolute Percentage Error (MAPE)

A metric that expresses error as a percentage of actual values, which fails when actual target values are zero or near zero and penalizes over-forecasts and under-forecasts asymmetrically.

Multicollinearity

High correlation among predictor variables that inflates the variance of estimated coefficients, making them unstable, prone to sign-flipping, and difficult to interpret.

Variance Inflation Factor (VIF)

A diagnostic metric used to detect multicollinearity, calculated as $\text{VIF} = \frac{1}{1 - R_j^2}$, where $R_j^2$ is obtained by regressing feature $j$ on all remaining predictors; values above $5$ or $10$ indicate concerning or severe collinearity.

Dummy Variable Trap

A state of perfect multicollinearity caused by including all $k$ dummy columns alongside an intercept term when one-hot encoding a categorical feature, resolved by dropping one category ($k - 1$ dummies) as a reference.

Ridge Regression ($L_2$ Regularization)

A regularized regression method that adds an $L_2$ penalty ($\lambda \sum \beta_j^2$) to the loss function, shrinking all coefficients smoothly toward zero without setting any exactly to zero, with closed-form solution $\hat{\beta}_{\text{ridge}} = (X^T X + \lambda I)^{-1} X^T y$.

Lasso Regression ($L_1$ Regularization)

A regularized regression technique that applies an $L_1$ penalty ($\lambda \sum |\beta_j|$), driving some coefficients to exactly zero to produce sparse models and perform feature selection.

Elastic Net

A regularized regression approach that combines $L_1$ and $L_2$ penalties, producing a sparse model while preserving groups of strongly correlated features together.

Heteroscedasticity

A condition where the variance of regression errors varies across fitted values or predictors, visible as a funnel shape on a residuals-versus-fitted plot and formally detected using the Breusch–Pagan or White test.

Durbin–Watson Test

A statistical test for detecting autocorrelation among regression residuals, where a test statistic value near $2$ indicates that residuals are independent.

Cook's Distance

A diagnostic measure that quantifies how much all fitted values or coefficients change when a specific observation is excluded, used to detect influential points.

Omitted Variable Bias

The bias introduced into coefficient estimates when a variable that affects the target $y$ and correlates with an included feature is left out of the regression specification.

Data Leakage

The unintended inclusion of information during training that would not be available at inference time, leading to overly optimistic evaluation metrics but failure on unseen production data.

Target Leakage

A specific type of data leakage in which a predictor feature is directly derived from or chronologically recorded after the occurrence of the target variable.

Train–Test Contamination

Data leakage caused when information from test data influences preprocessing steps, such as fitting scalers, imputers, or dimensionality reduction models on the full dataset before splitting.

Temporal (Look-Ahead) Leakage

A form of data leakage in time-series data where future information is used to predict past outcomes, often resulting from random cross-validation splitting or rolling features spanning forward dates.

Group / Entity Leakage

Data leakage that occurs when multiple records or observations from the same entity (such as patient IDs or near-duplicates) appear simultaneously in both the training and test sets.

Overfitting

A modeling issue characterized by high variance where the model learns sample-specific noise, resulting in low training error but substantially higher validation or test error.

Underfitting

A modeling flaw marked by high bias where the model is too simple to capture the underlying data patterns, producing high error on both training and validation sets.

Principal Component Analysis (PCA)

An unsupervised feature extraction technique that projects standardized data onto an orthogonal set of linear combinations (principal components) ordered by the variance explained.

Principal Component Regression (PCR)

A modeling strategy that fits an Ordinary Least Squares regression on the top principal components produced by PCA, removing multicollinearity and mitigating issues where $p > n$.

Prediction Interval

An estimated range covering the expected target value for a single new individual observation, which is inherently wider than a confidence interval because it accounts for irreducible error variance.

Robust Regression

An alternative regression framework (e.g., Huber, RANSAC, or Theil–Sen) designed to limit the influence of severe outliers by applying loss functions that grow slower than squared error.