SLR: Assessing the Model

What is a fitted value $\hat{y}_i$?

$\hat{y}_i = \hat\beta_0 + \hat\beta_1 x_i$, the estimated mean response at $x_i$ (a point on the fitted line).

What is a residual $e_i$?

$e_i = y_i - \hat{y}_i$ (observed minus fitted), also called the crude residual. It estimates the error $\varepsilon_i$.

What is $SS_E$, what does it measure, and how many d.f. does it have?

$SS_E = \sum (y_i - \hat{y}_i)^2 = \sum e_i^2$: variation about the fitted line; the minimum of $S(\beta_0, \beta_1)$. d.f. $n - 2$. Shortcut: $SS_E = S_{yy} - SS_R$.

What is $SS_E$ for the constant model?

$SS_E = SS_T$, since $\hat{y}_i = \bar{y}$.

What is $SS_T$, what does it measure, and how many d.f. does it have?

$SS_T = \sum (y_i - \bar{y})^2$: total variation in $y$ about its mean. d.f. $n - 1$.

What is $SS_R$, what does it measure, and how many d.f. does it have?

$SS_R = \sum (\hat{y}_i - \bar{y})^2$: variation explained by the fitted model. d.f. $1$. Shortcut: $SS_R = \hat\beta_1^2 S_{xx} = \hat\beta_1 S_{xy}$.

What is the ANOVA identity?

$SS_T = SS_R + SS_E$ (the cross term vanishes because $\sum e_i = 0$ and $\sum x_i e_i = 0$).

What is the ANOVA table for simple linear regression?

Regression: d.f. $1$, $SS_R$, $MS_R$, $F = \frac{MS_R}{MS_E}$. Residual: d.f. $n - 2$, $SS_E$, $MS_E$. Total: d.f. $n - 1$, $SS_T$.

What are degrees of freedom, and why do $SS_T$, $SS_E$ and $SS_R$ have $n-1$, $n-2$ and $1$?

The number of independent pieces of information that go into an estimate. $SS_T$: $n - 1$ (one used by $\bar{y}$). $SS_E$: $n - 2$ (two used by $\hat\beta_0, \hat\beta_1$). $SS_R$: $(n-1) - (n-2) = 1$.

What are the mean squares $MS_R$ and $MS_E$?

Sum of squares divided by its d.f.: $MS_R = \frac{SS_R}{1}$, $MS_E = \frac{SS_E}{n-2}$.

What is the variance ratio $F$, and what does it measure?

$F = \frac{MS_R}{MS_E}$: variation explained by the model relative to variation due to residuals.

How is the F distribution defined?

If $X \sim \chi^2_{\nu_1}$ and $Y \sim \chi^2_{\nu_2}$ are independent, $\frac{X/\nu_1}{Y/\nu_2} \sim F_{\nu_1, \nu_2}$. Skewed; $\nu_1$ numerator d.f., $\nu_2$ denominator d.f.

How do you carry out the F test for the slope?

$H_0: \beta_1 = 0$ vs $H_1: \beta_1 \neq 0$. Under $H_0$, $F = \frac{MS_R}{MS_E} \sim F^1_{n-2}$. Reject $H_0$ at level $\alpha$ if $F_{cal} > F^1_{n-2}(\alpha)$.

What does rejecting $H_0: \beta_1 = 0$ mean?

The slope is non-zero, so the full model $y_i = \beta_0 + \beta_1 x_i + \varepsilon_i$ is better than the constant model $y_i = \beta_0 + \varepsilon_i$.

What is $E(SS_E)$?

$E(SS_E) = (n - 2)\sigma^2$.

What is the unbiased estimator of $\sigma^2$?

$S^2 = MS_E = \frac{SS_E}{n-2} = \frac{1}{n-2}\sum (y_i - \hat{y}_i)^2$. In R it appears as "Residual standard error" $= \sqrt{MS_E}$.

Is $MS_E$ the sample variance?

Not in the full model. Only in the constant model ($\hat{y}_i = \bar{y}$, d.f. $n - 1$) is $S^2 = \frac{1}{n-1}\sum (y_i - \bar{y})^2$.

What is the coefficient of determination $R^2$?

$R^2 = \frac{SS_R}{SS_T} = 1 - \frac{SS_E}{SS_T}$ (× 100%): the percentage of total variation in $y$ explained by the fitted model.

What is the range of $R^2$, and what do $R^2 = 0$ and $R^2 = 100\%$ mean?

$R^2 \in [0, 100]\%$. $R^2 = 0$: the model explains none of the variability. $R^2 = 100\%$: $SS_E = 0$, all points on the line.

What is the main caveat when interpreting $R^2$?

It measures linear association only; a small $R^2$ does not always mean a poor relationship (e.g. it may be quadratic).

What is $R^2$ when all the $y$ values are equal?

$S_{yy} = 0$, so $R^2 = \frac{0}{0}$ is undefined; it is typically taken as $0$.

How is $R^2$ defined for a no-intercept model, and what is the caution?

Use $SS_T = \sum y_i^2$, so $R^2 = 1 - \frac{SS_E}{\sum y_i^2}$. It can be artificially high and is not comparable with the usual $R^2$.

What is the formula for adjusted $R^2$?

$R^2_{adj} = 1 - (1 - R^2)\frac{n-1}{n-k-1}$, $k$ = number of predictors. For SLR: $R^2_{adj} = 1 - \frac{n-1}{n-2}(1 - R^2)$.

Why use adjusted $R^2$ instead of $R^2$?

It penalises extra parameters, so it can compare models with different numbers of predictors. $R^2$ always increases when a regressor is added; $R^2_{adj}$ increases only if that variable's $F$ statistic is greater than $1$.

What are the properties of adjusted $R^2$?

$R^2_{adj} \le R^2$; it can be negative (model worse than the mean); it equals $1$ for a perfect fit; it is close to $R^2$ when $n$ is large.

What are the key properties of the residuals?

$\sum e_i = 0$ (so $\bar{e} = 0$), $\sum x_i e_i = 0$, $\sum \hat{y}_i e_i = 0$, and $E[e_i] = 0$.

What is the variance of the residual $e_i$?

$\text{var}(e_i) = \sigma^2(1 - v_i)$ with $v_i = \frac{1}{n} + \frac{(x_i - \bar{x})^2}{S_{xx}}$. Not constant: it depends on $i$.

What is the covariance of two residuals $e_i$ and $e_j$?

$\text{cov}(e_i, e_j) = -\sigma^2\left(\frac{1}{n} + \frac{(x_i - \bar{x})(x_j - \bar{x})}{S_{xx}}\right)$, not $0$. So residuals do not quite mimic the errors.

What is the difference between an error $\varepsilon_i$ and a residual $e_i$?

The error $\varepsilon_i$ comes from the model (unobservable); the residual $e_i$ comes from fitting the model to the data.

What is the standardised residual $d_i$, and why use it?

$d_i = \frac{e_i}{\sqrt{s^2(1 - v_i)}}$, $v_i = \frac{1}{n} + \frac{(x_i - \bar{x})^2}{S_{xx}}$. More nearly constant variance and smaller covariance than $e_i$.

Which residual plot checks linearity?

Plot $d_i$ against $x_i$.

Which residual plot checks constant variance?

Plot $d_i$ against the fitted values $\hat{y}_i$.

What does an acceptable residual plot look like?

Random scatter around zero with roughly constant spread and no pattern.

What patterns make a residual plot unacceptable?

Fan shape (non-constant variance), curve (non-linearity), outliers, systematic pattern (missing predictor), pattern in observation order (autocorrelation), influential high-leverage point, different spread by group.

How do you check the normality assumption?

Normal QQ plot of the residuals: points should lie close to a straight line. Heavy tails or skewness show as departures. A formal test is better (later in the course).

What assumption do the F and t tests need?

Normality of the errors. If it fails, the tests may not be valid.