Simple Linear Regression: Proof of the Variance of a Residual

Vocabulary practice flashcards covering the definitions, formulas, lemmas, and theorems used in the proof of the variance of a residual in simple linear regression (MA343).

Simple Linear Regression Model Setup

The model given by $y_i = \beta_0 + \beta_1 x_i + \bar{\varepsilon}_i$, where $\varepsilon_i \overset{iid}{\sim} (0, \sigma^2)$ for $i = 1, \dots, n$, with fixed and non-random $x_i$, implying the $y_i$ are independent with $Var(y_i) = \sigma^2$.

Residual ($e_i$)

The difference between the observed response and the fitted value, defined as $e_i = y_i - \hat{y}_i$.

Fitted Value ($\hat{y}_i$)

The predicted value for the $i$-th observation obtained from least squares estimates, given by $\hat{y}_i = \hat{\beta}_0 + \hat{\beta}_1 x_i$.

Sum of Squared Deviations of $x$ ($S_{xx}$)

The sum of squared deviations of the predictor variable from its sample mean, defined as $S_{xx} = \sum_{j=1}^n (x_j - \bar{x})^2$, with the assumption that $S_{xx} > 0$.

Sum of Cross-Products ($S_{xy}$)

The sum of products of deviations of $x$ and $y$ from their respective sample means, defined as $S_{xy} = \sum_{j=1}^n (x_j - \bar{x})(y_j - \bar{y})$.

Leverage ($h_{ii}$)

The $i$-th diagonal element of the hat matrix, defined as $h_{ii} = \frac{1}{n} + \frac{(x_i - \bar{x})^2}{S_{xx}}$, representing the weight placed on $y_i$ in determining its own fitted value $\hat{y}_i$.

Intercept Estimator ($\hat{\beta}_0$)

The least squares estimator of the intercept parameter, expressed in terms of sample means and the slope estimator as $\hat{\beta}_0 = \bar{y} - \hat{\beta}_1 \bar{x}$.

Slope Estimator ($\hat{\beta}_1$)

The least squares estimator of the slope parameter, given by $\hat{\beta}_1 = \frac{S_{xy}}{S_{xx}} = \frac{\sum_{j=1}^n (x_j - \bar{x})(y_j - \bar{y})}{\sum_{j=1}^n (x_j - \bar{x})^2}$.

Fact 0 (Deviations from the Mean Sum to Zero)

The property that deviations from the sample mean sum to zero for any variable: $\sum_{j=1}^n (x_j - \bar{x}) = 0$ and $\sum_{j=1}^n (y_j - \bar{y}) = 0$.

Lemma 1 (Slope Estimator as a Weighted Sum)

The formula establishing that the slope estimator can be written as a weighted linear combination of the observed values $y_j$: $\hat{\beta}_1 = \sum_{j=1}^n \frac{x_j - \bar{x}}{S_{xx}} y_j$.

Weight ($h_{ij}$)

The coefficient of $y_j$ in the linear representation of the fitted value $\hat{y}_i$, defined as $h_{ij} = \frac{1}{n} + \frac{(x_i - \bar{x})(x_j - \bar{x})}{S_{xx}}$.

Lemma 2 (Fitted Values as a Weighted Sum)

The result showing that each fitted value is a weighted linear combination of the responses: $\hat{y}_i = \sum_{j=1}^n h_{ij} y_j$, where $h_{ij} = \frac{1}{n} + \frac{(x_i - \bar{x})(x_j - \bar{x})}{S_{xx}}$.

Lemma 3 (Sum of Squared Weights)

The algebraic property that the sum of the squared weights equals the diagonal weight: $\sum_{j=1}^n h_{ij}^2 = h_{ii}$ for each $i = 1, \dots, n$.

Covariance of $y_i$ and Fitted Value $\hat{y}_i$

The covariance between the observed response and its fitted value, which evaluates to $Cov(y_i, \hat{y}_i) = h_{ii} \sigma^2$.

Variance of the Fitted Value ($Var(\hat{y}_i)$)

The variance of the predicted response $\hat{y}_i$, derived via Lemma 3 to be $Var(\hat{y}_i) = \sigma^2 \sum_{j=1}^n h_{ij}^2 = \sigma^2 h_{ii}$.

Variance of a Residual Theorem ($Var(e_i)$)

The theorem establishing that in simple linear regression, the variance of the $i$-th residual is given by $Var(e_i) = \sigma^2(1 - h_{ii})$ for each $i = 1, \dots, n$.