Describing Bivariate Numerical Data: Correlation and Linear Regression

Correlation ($r$)

• Bivariate Data: Observations on two numerical variables $(x, y)$, where $x$ is the predictor variable and $y$ is the response variable.
• Pearson's Sample Correlation Coefficient ($r$): Measures the strength and direction of a linear relationship between two numerical variables.
• Standardized Formula:
  $z_x = \frac{x - \bar{x}}{s_x}$

  $z_y = \frac{y - \bar{y}}{s_y}$

  $r = \frac{\sum z_x z_y}{n - 1}$

• Properties of $r$:
  • Range is bounded: $-1 \le r \le 1$.
  • Sign indicates direction: positive $r$ indicates an upward slope; negative $r$ indicates a downward slope.
  • Extreme values $r = 1$ or $r = -1$ occur only when all points lie exactly on a straight line.
  • Unit-free: Independent of units of measurement for $x$ and $y$
  • Measures linear relationships only; $r \approx 0$ means no linear relationship, but a strong nonlinear relationship may still exist.
  • Correlation does not imply causation.

Least Squares Regression Line

• Regression Line Equation: Represented as $\hat{y} = a + bx$, where $\hat{y}$ is the predicted value of $y$, $a$ is the y-intercept, and $b$ is the slope.
• Slope ($b$): Amount by which $y$ changes when $x$ increases by $1\, \text{unit}$:   $b = \frac{\sum (x - \bar{x})(y - \bar{y})}{\sum (x - \bar{x})^2}$
• y-Intercept ($a$): Predicted value of $y$ when $x = 0$:   $a = \bar{y} - b\bar{x}$
• Principle of Least Squares: Finds the unique line minimizing the sum of squared vertical deviations (residuals):   $\text{SSResid} = \sum (y - \hat{y})^2$
• Residual: Difference between observed value $y$ and predicted value $\hat{y}$:   $\text{residual} = y - \hat{y}$
• Extrapolation Danger: Making predictions for $x$ values outside the range of observed data is unreliable because the linear pattern may not extend beyond observed limits.

Assessing Model Fit

• Residual Plot: A scatterplot of $(x, \text{residual})$ pairs.
  • Curvature or distinct patterns indicate a linear model is inappropriate.
  • Outliers show large vertical distances from zero.
  • Potentially influential observations have extreme $x$ values separated from main data that significantly alter slope or intercept.
• Standard Deviation About the Regression Line ($s_e$): Measures typical prediction error magnitude:   $s_e = \sqrt{\frac{\text{SSResid}}{n - 2}} = \sqrt{\frac{\sum (y - \hat{y})^2}{n - 2}}$
• Coefficient of Determination ($r^2$): Proportion of total variation in $y$ explained by the linear relationship with $x$:   $r^2 = 1 - \frac{\text{SSResid}}{\text{SSTotal}}$$\text{SSTotal} = \sum (y - \bar{y})^2$
• Model Quality Criterion: A useful linear model combines a small standard deviation about the line ($s_e$) with a high coefficient of determination ($r^2$).

Steps in Linear Regression Analysis

• Step 1: Construct a scatterplot of bivariate data.
• Step 2: Visually verify whether the relationship appears approximately linear.
• Step 3: Calculate slope $b$ and intercept $a$ for the least squares regression line $\hat{y} = a + bx$
• Step 4: Plot residuals against $x$ to confirm line appropriateness and check for outliers or influential points.
• Step 5: Calculate and interpret $s_e$ and $r^2$
• Step 6: Determine overall model suitability for predictions.
• Step 7: Calculate predictions for $x$ values within the range of observed data.

Common Pitfalls

• Concluding causation from statistical correlation.
• Assuming $r \approx 0$ implies no relationship without inspecting scatterplots for non-linear patterns.
• Swapping response ($y$) and predictor ($x$) variables; regression of $y$ on $x$ differs from $x$ on $y$
• Extrapolating beyond observed $x$ data ranges.
• Interpreting intercept $a$ when $x = 0$ lies outside the observed data range.
• Evaluating a model using only $r^2$ or $s_e$ without examining residual plots.