Week 11: Goodness of Fit and Regression Inference

Measures of Variance and Goodness of Fit

• Unconditional Variance vs. Error Term Variance

  • Unconditional Variance ($S_Y^2$): Measures the total variation in the dependent variable (e.g., $wage_i$). It is calculated based on the unconditional mean line ($\bar{Y}$).

  • Formula: $S_Y^2 = \frac{1}{n-1} \sum_{i=1}^{n} (Y_i - \bar{Y})^2$

  • Value in dataset: $0.2601$

  • Error Term Variance ($S_{\epsilon}^2$): Estimates the average variation in the dependent variable that remains unexplained by the regression (based on the regression line $\hat{Y}_i$).

  • Formula: $S_{\epsilon}^2 = \frac{1}{n-2} \sum_{i=1}^{n} (Y_i - \hat{Y}_i)^2$

  • Value in dataset: $0.2399$

• Understanding Variation Components

  • The regression model decomposes the total variation of the dependent variable into two distinct parts:

  • Explained Variation (Deterministic Part): Represented by the fitted values $\hat{Y}_i = \hat{\beta}_0 + \hat{\beta}_1 edu_i$. This captures how the explanatory variable (education) explains wages.

  • Unexplained Variation (Residual Part): Represented by the residuals $\hat{\epsilon}_i$. This captures the part of wages not explained by education.

  • The full equation is: $wage_i = \hat{\beta}_0 + \hat{\beta}_1 edu_i + \hat{\epsilon}_i$

• The Sum of Squares Identity

  • SST (Total Sum of Squares): The total variation in the dependent variable around its mean.

  • $SST = \sum_{i=1}^{n} (Y_i - \bar{Y})^2 = (n-1) S_Y^2$

  • SSR (Regression Sum of Squares): The variation explained by the regression line.

  • $SSR = \sum_{i=1}^{n} (\hat{Y}_i - \bar{Y})^2$

  • SSE (Sum of Squares of Error): The unexplained variation (sum of squared residuals).

  • $SSE = \sum_{i=1}^{n} \hat{\epsilon}_i^2 = \sum_{i=1}^{n} (Y_i - \hat{Y}_i)^2$

  • The relationship between these terms is: $SST = SSR + SSE$

• Coefficient of Determination ($R^2$)

  • Definition: The $R^2$ measures the goodness of fit of the regression. It represents the proportion of variation in the dependent variable explained by the explanatory variable.

  • Formulas for calculation:

  • $R^2 = \frac{\text{Variation Explained by Regression}}{\text{Total Variation of Dependent Variable}} = \frac{SSR}{SST}$

  • Equivalent form: $R^2 = 1 - \frac{SSE}{SST} = 1 - \frac{\sum (Y_i - \hat{Y}_i)^2}{\sum (Y_i - \bar{Y})^2}$

  • Properties of $R^2$:

  • Range: $0 \leq R^2 \leq 1$

  • $0$: The regression explains none of the variation.

  • $1$: The regression explains all of the variation (perfect fit).

  • Application (Returns to Education):

  • $R^2 = \frac{49.32}{528.78} = 0.093$

  • Interpretation: $9.3\%$ of the total variation in average wages is explained by education. This suggests very little of the variability in wages is explained by education alone, leading to questions about other factors (Omitted Variables).

  • Correlation in this dataset: $0.305394$

Inference About Parameters

• OLS Estimation Formulas

  • Slope Estimator: $\hat{\beta}_1 = \frac{\sum_{i=1}^{n} (X_i - \bar{X})(Y_i - \bar{Y})}{\sum_{i=1}^{n} (X_i - \bar{X})^2}$

  • Intercept Estimator: $\hat{\beta}_0 = \bar{Y} - \hat{\beta}_1 \bar{X}$

• Standard Assumptions for OLS Properties

  • To establish statistical properties, four core assumptions about the error term are required:

  1. Zero Mean: The errors average out to zero.

  2. Constant Variance (Homoskedasticity): The spread of the error term is the same across all levels of X.

  3. Independence: Errors are not correlated with each other.

  4. Exogeneity: The error term is not correlated with the explanatory variable.

• BLUE Properties

  • If the above four assumptions hold, the OLS estimator is BLUE (Best Linear Unbiased Estimator):

  • Best: Among all linear unbiased estimators, OLS has the smallest variance (most precise). This ensures that the standard errors $se(\hat{\beta}_0)$ and $se(\hat{\beta}_1)$ are correctly computed.

  • Linear: The regression is linear with respect to the parameters.

  • Unbiased: The expected value of the estimators equals the true population parameters: $E[\hat{\beta}_0] = \beta_0$ and $E[\hat{\beta}_1] = \beta_1$.

• Gaussianity and Central Limit Theorem

  • Inclusion of the normality assumption for the error term ($\epsilon_i | edu_i \sim N(0, \sigma^2)$) leads to the result that OLS estimators are normally distributed:

  • $\hat{\beta}_0 \sim N(\beta_0, Var(\hat{\beta}_0))$

  • $\hat{\beta}_1 \sim N(\beta_1, Var(\hat{\beta}_1))$

  • This normality allows for formal hypothesis testing.

Hypothesis Testing Procedures

• The Null and Alternative Hypotheses

  • For a population parameter set to a hypothesized value $b_1$:

  • $H_0: \beta_1 = b_1$

  • $H_1: \beta_1 \neq b_1$ (Two-sided test)

  • A "statistical significance" test specifically checks if the parameter is different from zero ($b_1 = 0$).

  • If $H_0$ is rejected: The variable (education) has a statistically significant effect.

  • If $H_0$ is not rejected: There is insufficient evidence of a linear relationship.

• Test Statistic and Decision Rules

  • t-ratio Test Statistic: Used to test a single restriction on a parameter.

  • Formula: $t = \frac{\hat{\beta}_1 - b_1}{se(\hat{\beta}_1)}$

  • Distribution: The test statistic follows the Student-t distribution with $n-2$ degrees of freedom ($t \sim t_{n-2}$). The two degrees of freedom are lost by estimating $\beta_0$ and $\beta_1$.

  • Decision Rule: Reject $H_0$ if |t| > t_{\alpha/2, n-2}.

• Example 1: Significance of Education

  • Hypothesis: $H_0: \beta_1 = 0$ vs $H_1: \beta_1 \neq 0$

  • Calculations: $t = \frac{0.079 - 0}{0.006} = 14.34$

  • Critical Value ($\alpha = 0.05$, $n=2000$): $t_{0.025, 1998} = 1.96$

  • Decision: 14.34 > 1.96. Reject $H_0$; education has a statistically significant effect on wages.

• Example 2: Testing Scientific Consensus

  • Context: A study by Andrew Leigh and Chris Ryan (published in Economics of Education Review, 2008) suggests a robust $10\%$ return ($b_1 = 0.1$) to education in Australia.

  • Hypothesis: $H_0: \beta_1 = 0.1$ vs $H_1: \beta_1 \neq 0.1$

  • Calculations: $t = \frac{0.079 - 0.1}{0.006} = -3.79$

  • Decision: |-3.79| > 1.96. Reject $H_0$; our estimate is significantly different from the scientific consensus of $10\%$.

Confidence Intervals

• Constructing Intervals

  • Confidence intervals (CI) provide a range of plausible values for the true population parameters based on the estimator properties.

  • Formula for Slope: $\hat{\beta}_1 \pm t_{\alpha/2, n-2} \times se(\hat{\beta}_1)$

  • Formula for Intercept: $\hat{\beta}_0 \pm t_{\alpha/2, n-2} \times se(\hat{\beta}_0)$

  • Interpretation: The interval will contain the true parameter value in $100(1-\alpha)\%$ of repeated samples.

• Wage Data Results (95% CI)

  • Slope (Education): $0.079 \pm 1.96 \times 0.006 \rightarrow [0.068, 0.09]$

  • Intercept: $1.293 \pm 1.96 \times 0.071 \rightarrow [1.15, 1.43]$

  • Because neither interval includes zero, both parameters are statistically significant at the $5\%$ level.

Prediction in Regression Analysis

• Objectives of Economic Forecasting

  1. Conditional Mean Prediction: Predicting the average outcome for a given X-value (e.g., average wage for everyone with 15.5 years of education). This is a "point prediction."

  2. Density Prediction (Non-Examinable): Providing the complete probability distribution of future individual outcomes.

• Conditional Mean Prediction Calculations

  • Estimating the population regression function at a specific value $x$:

  • $E[\hat{Y}_i | X_i = x] = \hat{\beta}_0 + \hat{\beta}_1 x$

  • Example (15.5 years of education):

  • $E[\hat{wage}_i | edu_i = 15.5] = 1.29 + 0.079 \times 15.5 = 2.51$

  • Result: The expected log-hourly wage is $2.51$

  • Example (Expected gain from college): Comparing 15 years vs. 12 years of education.

  • $E[\hat{wage}_i | edu_i = 15] - E[\hat{wage}_i | edu_i = 12] = \hat{\beta}_1 (15 - 12) = 0.079 \times 3 = 0.237$

  • Interpretation: A college graduate increases expected wage by $23.7\%$ relative to high school graduates (percentage interpretation because wages are in logs).

• Confidence Intervals for Conditional Mean Forecast

  • The CI for the mean reflects uncertainty in estimating the regression line because different samples produce different lines.

  • Standard Error ($se(\hat{E})$):

  • $se(E[\hat{wage}_i | edu_i]) = S_{\epsilon} \sqrt{\frac{1}{n} + \frac{(x - \bar{X})^2}{(n-1) S_X^2}}$

  • For $edu_i = 15.5$: $se = 0.49 \sqrt{\frac{1}{2000} + \frac{(15.5 - 12.7)^2}{1999 \times (1.992)}} = 0.019$

  • 95% CI Range: $2.51 \pm 1.96 \times 0.019 = [2.482, 2.556]$

  • This interval is very narrow, indicating precise conditional mean forecasting.

• Prediction Interval (PI) for an Individual Outcome

  • A predictive interval provides the range for an individual's future wage. It is wider than a confidence interval because it reflects two sources of error:

  1. Uncertainty in estimating the regression line (sampling uncertainty).

  2. Natural variation of individuals around that line (error term variance).

  • Standard Error ($se_{indiv}$):

  • $S_{\epsilon} \sqrt{1 + \frac{1}{n} + \frac{(x - \bar{X})^2}{(n-1) S_X^2}}$

  • For $edu_i = 15.5$: The SE is approximately $0.4902$

  • 95% PI Range: $2.51 \pm 1.96 \times 0.49 = [1.558, 3.479]$

  • This interval tells where an individual's wage is likely to fall for a given education level with $95\%$ probability.

• Summary Comparison (at 15.5 years education)

  • Confidence Interval (Mean): $[2.482, 2.556]$ (Contains the true expected value).

  • Prediction Interval (Individual): $[1.558, 3.479]$ (Contains the true individual predicted log-wage).