Topic 5 — Simple Linear Regression Flashcards

This set of vocabulary flashcards covers core concepts of simple linear regression, including hypothesis testing, model parameters, assumptions, and diagnostics as presented by Dr. Jinkai Xue.

Null hypothesis ($H_0$)

The status-quo claim about a parameter $\vartheta$ (e.g., $\vartheta = \vartheta_0$).

Alternative hypothesis ($H_a$)

What we’d conclude if we reject the null hypothesis ($H_0$).

p-value

The probability—assuming $H_0$ is true—of seeing a test-statistic value at least as extreme as the one we computed.

Confidence intervals (CIs)

Two numbers $(L, U)$ such that $P(L \leq \vartheta \leq U) = 1 - \varpi$.

Dependent variable ($y$)

Also known as the response variable, it is the outcome being predicted in regression analysis.

Independent variable ($x$)

Also known as the predictor variable, it is used in regression to explain or predict changes in the dependent variable.

Simple Linear Regression Model

A probabilistic model expressed as $y = \omega_0 + \omega_1x + \varrho$, where the deterministic part $E[y] = \omega_0 + \omega_1x$ is the line of means.

$\omega_0$ (Theta-zero)

The true population parameter representing the y-intercept (the value of $E[y]$ when $x = 0$).

$\omega_1$ (Theta-one)

The true population parameter representing the slope (the change in $E[y]$ per unit change in $x$).

$\hat{\omega}_0$ and $\hat{\omega}_1$

The sample estimates of the population parameters computed from $n$ data points by the method of least squares.

Residual ($\hat{\varrho}_i$)

The observed counterpart of the unobserved random error ($\varrho_i$), calculated as the difference between the observed value and the fitted value: $\hat{\varrho}_i = y_i - \hat{y}_i$.

Homoscedasticity

The model assumption that the variance of the error term, $\text{Var}(\varrho) = \varepsilon^2$, is the same for every value of $x$.

Method of Least Squares

A mathematical procedure that picks the line minimizing the sum of squared vertical residuals: $SSE = \sum_{i=1}^n (y_i - \hat{y}_i)^2$.

Residual Variance ($s^2$)

An unbiased estimator of $\varepsilon^2$ calculated by dividing the sum of squared errors ($SSE$) by the degrees of freedom ($n - 2$).

Standard Error of the Slope ($s_{\hat{\omega}_1}$)

A value that shrinks when the residual standard deviation $s$ is small or when the $x$ values are widely spread out, calculated as $s / \sqrt{S_{xx}}$.

Pearson Correlation ($r$)

A scale-free measure of linear association ranging from $-1$ to $1$; its sign matches the estimated slope $\hat{\omega}_1$.

Coefficient of Determination ($r^2$)

The fraction of the total variation in $y$ that is explained by the linear relationship with $x$, ranging from $0$ to $1$.

Prediction Interval

An interval calculated for a new individual observation of $y$ ($y_{new}$) that is always wider than the confidence interval for the mean response.

Extrapolation

The pitfall of using a fitted line to make predictions outside the sampled range of $x$; such predictions are unsupported by the model.

Anscombe’s Quartet

A collection of four data sets with identical summary statistics ($\hat{\omega}_0$, $\hat{\omega}_1$, and $r^2$) but completely different distributions, demonstrating the necessity of plotting data.