SLR: Model & Least Squares

What is $S_{xx}$, and what is its computational form?

$S_{xx} = \sum (x_i - \bar{x})^2 = \sum x_i^2 - n\bar{x}^2 = \sum x_i^2 - \frac{(\sum x_i)^2}{n}$. Use the last form for calculation (avoids rounding error).

What is $S_{yy}$?

$S_{yy} = \sum (y_i - \bar{y})^2 = \sum y_i^2 - \frac{(\sum y_i)^2}{n}$. Same as $SS_T$.

What is $S_{xy}$, and what is its computational form?

$S_{xy} = \sum (x_i - \bar{x})(y_i - \bar{y}) = \sum x_i y_i - n\bar{x}\bar{y} = \sum x_i y_i - \frac{(\sum x_i)(\sum y_i)}{n}$.

What is $\sum (x_i - \bar{x})$ equal to?

$0$. Hence $S_{xy} = \sum (x_i - \bar{x}) y_i$.

What is the principle of parsimony?

Use the simplest model that describes the relationship well.

What is the simple linear regression model?

$Y_i = \beta_0 + \beta_1 x_i + \varepsilon_i$, $i = 1, \dots, n$. $\beta_0$ = intercept, $\beta_1$ = slope, $\varepsilon_i$ = random error. $X$ is known (not random); $Y$ is random.

What are the three standard assumptions on the errors $\varepsilon_i$?

(1) $E(\varepsilon_i) = 0$; (2) $\text{var}(\varepsilon_i) = \sigma^2$ (constant); (3) $\text{cov}(\varepsilon_i, \varepsilon_j) = 0$ for $i \neq j$.

What do the three error assumptions mean in words?

$Y$ depends linearly on $X$, the variance of $Y$ is the same at every $x$, and the $Y_i$ are uncorrelated.

What is the normal simple linear regression model?

$y_i \sim N(\beta_0 + \beta_1 x_i, \sigma^2)$ independently, i.e. $\varepsilon_i \overset{iid}{\sim} N(0, \sigma^2)$. Under normality, zero covariance means the $Y_i$ are independent.

What is the centred form of the model, and how are its parameters interpreted?

$y_i = \alpha + \beta(x_i - \bar{x}) + \varepsilon_i$ with $\alpha = \beta_0 + \beta_1\bar{x}$, $\beta = \beta_1$. Same slope; $\alpha$ is the mean response at $x = \bar{x}$.

What does least squares minimise?

$S(\beta_0, \beta_1) = \sum [y_i - (\beta_0 + \beta_1 x_i)]^2$.

What are the normal equations?

$n\hat\beta_0 + \hat\beta_1 \sum x_i = \sum y_i$ and $\hat\beta_0 \sum x_i + \hat\beta_1 \sum x_i^2 = \sum x_i y_i$ (from setting $\frac{\partial S}{\partial \beta_0} = \frac{\partial S}{\partial \beta_1} = 0$).

What is the least squares estimator of the intercept $\beta_0$?

$\hat\beta_0 = \bar{y} - \hat\beta_1 \bar{x}$.

What is the least squares estimator of the slope $\beta_1$?

$\hat\beta_1 = \frac{S_{xy}}{S_{xx}}$.

What is the difference between an estimator and an estimate?

An estimator is a function of the random $Y_i$ (so a random variable). An estimate is its value calculated from the observed data.

Why should you not predict outside the observed range of $x$?

The relationship may not stay linear outside the range used to fit the model (extrapolation).

How do you interpret the intercept estimate $\hat\beta_0$?

The predicted value of $y$ when $x = 0$.

How do you interpret the slope estimate $\hat\beta_1$?

The average change in $y$ for a one-unit increase in $x$.

What is the no-intercept model, and what is its least squares estimator?

$y_i = \beta_1 x_i + \varepsilon_i$, so $E(Y \mid X = 0) = 0$. LSE: $\hat\beta_1 = \frac{\sum x_i y_i}{\sum x_i^2}$.

What is the constant model?

$y_i = \beta_0 + \varepsilon_i$: $Y$ does not depend on $X$. Fitted value $\hat{y}_i = \bar{y}$.

When is an estimator $\hat\theta$ unbiased for $\theta$?

When $E[\hat\theta] = \theta$.

How can $\hat\beta_1$ be written as a linear combination of the $Y_i$, and why is it normal?

$\hat\beta_1 = \sum c_i Y_i$ with $c_i = \frac{x_i - \bar{x}}{S_{xx}}$, where $\sum c_i = 0$ and $\sum c_i x_i = 1$. It is normal because it is a linear combination of normals.

What is the distribution of a linear combination of independent normals (Theorem 2.1)?

If $Y_i \sim N(\mu_i, \sigma_i^2)$ independently, then $\sum a_i Y_i \sim N\left(\sum a_i \mu_i, \sum a_i^2 \sigma_i^2\right)$.

What is the sampling distribution of $\hat\beta_1$?

$\hat\beta_1 \sim N\left(\beta_1, \frac{\sigma^2}{S_{xx}}\right)$. Unbiased.

What is the sampling distribution of $\hat\beta_0$?

$\hat\beta_0 \sim N\left(\beta_0, \sigma^2\left(\frac{1}{n} + \frac{\bar{x}^2}{S_{xx}}\right)\right)$. Unbiased.

What does it mean that the LS estimators are BLUE, and under what assumptions?

Best Linear Unbiased Estimators: smallest variance among all linear unbiased estimators. Holds under linearity, exogeneity, homoskedasticity and no autocorrelation (Gauss-Markov).

When are $\hat\beta_0$ and $\hat\beta_1$ consistent?

If $\frac{1}{n} S_{xx} \to \sigma_x^2 > 0$, then $\hat\beta_1 \xrightarrow{p} \beta_1$ and $\hat\beta_0 \xrightarrow{p} \beta_0$ (law of large numbers + continuous mapping theorem).