Chapter 5 - 2

Chapter 5: Regression

Overview

  • This chapter covers key concepts related to regression including:

    • Regression lines

    • The least-squares regression line

    • Prediction via the regression line

    • Facts about least-squares regression

    • Understanding residuals

    • Identifying outliers and influential observations

    • Cautions regarding correlation and regression

    • Importance of recognizing that association does not imply causation

The Least-Squares Method

  • The least-squares method is used to determine the regression line that minimizes the residual sum of squares (RSS).

  • Key Formulas:

    • Predicted value of the response variable: ( \hat{y} )

    • Intercept: ( a = \bar{y} - b\bar{x} )

    • Slope: ( b = r \frac{s_y}{s_x} )

    • Residual Sum of Squares: ( RSS = \sum (y - \hat{y})^2 )

    • Regression Line Equation: ( \hat{y} = a + b x )

Understanding Residuals

  • A residual represents the difference between the observed value (y) and the predicted value (( \hat{y} )):

    • ( e_i = y_i - \hat{y}_i )

    • Example:

      • In DC, the poverty rate is 5.44% higher than predicted.

      • In Rhode Island (RI), the poverty rate is 4.16% lower than predicted.

Calculation of Residuals

  • For Rhode Island:

    • HS Graduation Rate: 81%

    • Expected Poverty Rate: ( 64.68 - 0.62 * 81 = 14.46 )%

    • Residual: ( 10.3 - 14.46 = -4.16 )

  • For DC:

    • HS Graduation Rate: 86%

    • Expected Poverty Rate: ( 64.68 - 0.62 * 86 = 11.36 )%

    • Residual: ( 16.8 - 11.36 = 5.44 )

Residual Plots

  • Residual plots display the relationship between residuals and the explanatory variable to evaluate the regression fit.

  • Ideal residuals: Randomly scattered around zero.

  • A curved pattern in residuals suggests a non-linear relationship, indicating that a linear regression may not be appropriate.

Checking Residual Conditions

  • Normality of Residuals: Should be approximately normally distributed. Histograms can be used for verification.

  • Homogeneity of Variance (Homoscedasticity): Residuals should show roughly constant variability; non-constant spread indicates heteroscedasticity.

  • Linearity: If significant patterns exist within residuals, a linear model may not be suitable.

Outliers and Influential Points

  • Outliers can have a significant impact on regression lines.

    • An outlier's removal can change the slope and relationship mapped by the least-squares line.

  • Notable behaviors:

    • Outliers in the y-direction produce large residuals.

    • Outliers in the x-direction are often more influential.

Correlation vs. Causation

  • Correlation alone does not guarantee a causal relationship.

  • Cautions include:

    • Correlation describes only linear relationships.

    • Beware of ecological correlation (averages rather than individuals).

    • Extrapolation beyond the data range can lead to inaccuracies.

    • Lurking variables may affect observed correlations.

Example of Extrapolation Risks

  • Example scenario with predicted heights based on age:

    • Height at 42 months predicted accurately, but prediction for 30 years (360 months) yields unrealistic results (6'10.5"), illustrating dangers of extrapolation.

Summary of Key Takeaways

  • The least-squares regression line minimizes the vertical distance from data points.

  • The slope indicates the rate of change in y for a unit change in x.

  • The intercept is the predicted response when x=0, though its relevance depends on realistic value ranges.

  • Caution is urged before conducting regression analyses, emphasizing the role of residual plots in analyzing model fit and recognizing outliers and variables that may influence results.