Lecture 23: Inference with the Linear Regression Model

Purpose of Inference in Linear Regression

• Regression models are used to estimate the true regression line within a population based on sample data.

• For example, if we consider the height of fathers and sons, the measured data is used to estimate the relationship for a larger population, such as all male University of Otago students and their fathers.

• A fitted model is only an estimate; therefore, researchers must assess the precision of the estimated parameters.

• Precision is quantified using standard errors of the regression parameters.

• These standard errors are the foundation for constructing confidence intervals and conducting hypothesis tests.

The Role of Error Variance and Uncertainty

• The the linear regression model is expressed as: $y = \beta_0 + \beta_1x + \epsilon$

• The error term $\epsilon$ is assumed to follow a normal distribution with a mean of $0$ and a variance of $\sigma^2_{\epsilon}$.

• Impact of Error Variance:

  • Large error variance (holding all other factors constant) results in a larger spread of data points around the true regression line.

  • Increased spread leads to greater uncertainty regarding the fitted regression line.

  • Greater uncertainty means the estimates $\hat{\beta}_0$ (intercept) and $\hat{\beta}_1$ (slope) are less precise.

• To quantify uncertainty, the error variance $\sigma^2_{\epsilon}$ must be estimated from the data.

Estimating Error Variance ($s^2_{\epsilon}$)

• The actual errors ($\epsilon$) are unknown, but they are estimated using residuals ($\hat{\epsilon}$).

• A good estimate for $\sigma^2_{\epsilon}$ is the sample variance of the residuals $\hat{\epsilon}_1, \hat{\epsilon}_2, . . . , \hat{\epsilon}_n$.

• Standard sample variance uses the formula $\frac{1}{n-1} \sum_{i=1}^{n} (\hat{\epsilon}_i - \bar{\hat{\epsilon}})^2$, but specific modifications are required for regression:

  • The sample mean of the residuals is always zero ($\bar{\hat{\epsilon}} = 0$).

  • The correct divisor (degrees of freedom) for simple linear regression is $n - 2$, rather than $n - 1$.

• The estimate of the error variance is denoted as $s^2_{\epsilon}$, calculated as: $s^2_{\epsilon} = \frac{1}{n - 2} \sum_{i=1}^{n} \hat{\epsilon}^2_i = \frac{RSS}{n - 2}$

• $RSS$ stands for the Residual Sum of Squares, defined as: $RSS = \sum_{i=1}^{n} \hat{\epsilon}^2_i$

Standard Error of the Slope Estimate ($\hat{\beta}_1$)

• The parameter $\beta_1$ is often the primary focus in studies as it represents the change in the expected value of $y$ for a unit change of $x$ in the population.

• The precision of the estimate $\hat{\beta}_1$ is measured by its standard error, $\sigma_{\hat{\beta}_1}$.

• The standard error represents the standard deviation of the sampling distribution of $\hat{\beta}_1$ (the variation of $\hat{\beta}_1$ if multiple data sets of the same size were taken from the population).

• The theoretical formula for the standard error is: $\sigma_{\hat{\beta}_1} = \frac{\sigma_{\epsilon}}{\sqrt{\sum_{i=1}^{n} (x_i - \bar{x})^2}}$

• This formula indicates that the standard error is proportional to the error standard deviation $\sigma_{\epsilon}$. As $\sigma^2_{\epsilon}$ increases, the standard error of $\hat{\beta}_1$ also increases.

• Because we do not know the true $\sigma_{\epsilon}$, we use the estimated standard error in practice: $s_{\hat{\beta}_1} = \frac{s_{\epsilon}}{\sqrt{\sum_{i=1}^{n} (x_i - \bar{x})^2}}$

Confidence Intervals for the Regression Slope

• Confidence intervals for parameters in linear regression follow the general form: $estimate \pm multiplier \times std. error$

• For the slope $\beta_1$, the components are:

  • Estimate: $\hat{\beta}_1$

  • Standard Error: $s_{\hat{\beta}_1}$

  • Multiplier: Derived from a $t$-distribution with degrees of freedom $\nu = n - 2$. This divisor matches the denominator used in the calculation of $s^2_{\epsilon}$.

• For a $100(1 - \alpha)\%$ confidence interval, the multiplier is $t_{1-\alpha/2, n-2}$.

• Calculation form: $\hat{\beta}_1 \pm t_{1-\alpha/2, n-2} \times s_{\hat{\beta}_1}$

Case Study: Father and Son Heights

• Data Summary and R Output Analysis:

  • The fitted model is m_height = lm(son ~ father, data = height).

  • Residual standard error: $6.13$ on $277$ degrees of freedom.

  • Estimate for Intercept: $114.0533$ with a Std. Error of $8.4979$.

  • Estimate for Father (Slope $\hat{\beta}_1$): $0.3699$ with a Std. Error of $0.0478$.

  • Total observations $n = 279$.

• Calculating Confidence Interval 'By Hand':

  • Level of confidence: $95\%$

  • Multiplier from R: qt(0.975, 277) = $1.9686$

  • Calculation: $0.3699 \pm 1.9686 \times 0.0478$

  • $0.3699 \pm 0.0941$

  • CI: $(0.2758, 0.464)$

• Interpretation: We are $95\%$ confident that for every $1\,cm$ increase in the height of the father, the expected height of the son increases by between $0.2758$ and $0.464\,cm$.

• Using the confint() function in R:

  • Result for father (95%): $(0.2758, 0.464)$

  • Result for father (99%): confint(m_height, level = 0.99) yields $(0.2459, 0.4939)$.

The Concept of Regression to the Mean

• Origin: The term "regression" was introduced by Francis Galton in his 1886 paper, "Regression Towards Mediocrity in Hereditary Stature."

• Observation: One might expect a $1\,cm$ increase in father's height to result in a $1\,cm$ increase in son's height on average. However, the data shows this is not the case ($\hat{\beta}_1 < 1$).

• The definition of Regression to the Mean:

  • The son of a short father tends to be short, but on average, he is taller than his father.

  • The son of a tall father tends to be tall, but on average, he is shorter than his father.

  • This implies that extreme traits in parents tend to regress toward the population mean in their offspring.

Hypothesis Testing for the Slope Parameter

• Hypothesis testing assesses whether two variables are linearly related in the population.

• Hypotheses:

  • Null Hypothesis ($H_0: \beta_1 = 0$): The outcome variable is not linearly related to the predictor variable; the mean of $y$ ($\mu_y$) does not depend on $x$.

  • Alternative Hypothesis ($H_A: \beta_1 \neq 0$): A linear relationship exists between $x$ and $y$.

• Test Statistic: The test statistic is calculated as: $t = \frac{estimate - null}{std. error} = \frac{\hat{\beta}_1}{s_{\hat{\beta}_1}}$

• This follows a $t$-distribution with $n-2$ degrees of freedom.

Application: PCB Accumulation in Trout

• Background Details:

  • Researchers studied polychlorinated biphenyls (PCBs), pollutants that accumulate in the food chain.

  • Subjects: $28$ trout collected from Cayuga Lake, NY in 1970.

  • Methodology: Age was known because fish were marked and stocked annually. Fish were ground and mixed, and chromatography was used to find PCB residue in ppm (parts per million).

• Scientific Question: Is there evidence that log PCB residue increases with age?

  • Null Hypothesis ($H_0: \beta_1 = 0$

  • Alternative Hypothesis ($H_A: \beta_1 \neq 0$

• R Model Output Analysis:

  • Log PCB fitted against age: m_pcb = lm(logpcb ~ age, data = pcb)

  • Residual standard error: $0.567$ on $26$ degrees of freedom.

  • Multiple R-squared: $0.731$.

  • Age parameter estimate: $0.2591$.

  • Standard error for age: $0.0308$.

  • tnd-value: $8.41$.

  • p-value ($Pr(>|t|)$): $6.8 \times 10^{-09}$.

• Interpretation Approaches:

  • Confirmatory Study: Compare p-value to $\alpha$ ($0.05$). Since $6.8 \times 10^{-09} < 0.05$, we reject $H_0$. There is evidence of a relationship between log PCB and age.

  • Exploratory Study: Interpret the p-value as the degree of incompatibility between the data and the null hypothesis. Use $\alpha$ as a guide and avoid strict binary decisions.

• Conclusion: The data are highly unusual compared to what would be expected if the null hypothesis were correct. There is strong evidence against $H_0$.