Correlation and Linear Regression

Overview of Association Testing

• Testing Associations (Continuous Variables):

  • Scatterplot: Visual inspection ("eyeballing") of coordinate pairs ($X_i$, $Y_i$).

  • Correlation: Quantifies the strength and direction of a linear association.

  • Regression: Quantifies how changes in predictor variables influence an outcome variable to allow prediction.

Pearson Correlation Coefficient

• Definition & Formula:

  • Ratio of signal to noise:   $r = \frac{\text{Signal}}{\text{Noise}} = \frac{\text{cov}_{xy}}{s_x s_y}$   $\text{cov}(x, y) = \frac{\sum_{i=1}^{N} (x_i - \bar{x})(y_i - \bar{y})}{N - 1}$

• Value Range:

  • $-1 \le r \le +1$

  • $r = +1$: Perfect positive linear relationship.

  • $r = -1$: Perfect negative linear relationship.

  • $r = 0$: No linear relationship (nonlinear relationships may still exist).

• Anscombe's Quartet & Linearity:

  • Evaluates linear associations only.

  • Datasets can share identical summary statistics ($\bar{X} = 9.0$, $\bar{Y} = 7.5$, $r = .82$) while having completely different patterns or outliers; visual inspection via scatterplots is required.

Causality, Confounders, and Hypothesis Testing

• Spurious Correlations:

  • Significant associations driven by a third confounding variable (e.g., age confounding arm length and reasoning ability) or sheer coincidence.

• Three Criteria for Causality:

  1. Concomitant Variation: Cause ($X$) and effect ($Y$) must empirically vary together.

  2. Temporal Precedence: Cause ($X$) must occur before effect ($Y$).

  3. Non-Spuriousness: Association cannot be explained by alternative confounding factors.

• Hypothesis Testing for Correlation:

  • Null hypothesis:   $H_0: r = 0$

  • Alternative hypothesis:   $H_a: r \ne 0$

  • Degrees of freedom:   $df = n - 2$

  • Reporting standards:

  • Significant:     $r(198) = .578$, $p < .001$

  • Non-significant:     $r(198) = .039$, $p = .587$

Simple Linear Regression Foundations

• Correlation vs. Regression:

  • Correlation: Symmetric; variables are interchangeable (similar to a $t$-test).

  • Regression: Asymmetric; directional dependency using predictors to explain an outcome (tied to ANOVA and the General Linear Model).

• Model Equation:   $Y_i = b_0 + b_1 X_i + \varepsilon_i$

  • $Y_i$: Outcome (dependent variable).

  • $b_0$: Intercept (value of $Y$ when $X = 0$).

  • $b_1$: Regression slope (change in $Y$ per 1-unit increase in $X$).

  • $X_i$: Predictor (independent variable).

  • $\varepsilon_i$: Random error component.

• Ordinary Least Squares (OLS):

  • Finds the line of best fit by minimizing the sum of squared vertical distances from data points to the line.

Evaluating Regression Models

• Partitioning Sum of Squares:   $SS_T = SS_M + SS_R$

  • $SS_T$: Total sum of squares (variation relative to grand mean $\bar{y}$).

  • $SS_M$: Model sum of squares (variance explained by fitted regression model).

  • $SS_R$: Residual sum of squares (unexplained variability / error).

• Goodness of Fit ($R^2$):

  • Proportion of total variance explained by the model:   $R^2 = \frac{SS_M}{SS_T}$

• Model Significance ($F$-Test):

  • Tests if regression model explains significantly more variance than the grand mean:   $F = \frac{MS_M}{MS_R} = \frac{\frac{SS_M}{df_M}}{\frac{SS_R}{df_R}}$

Regression Coefficients and Reporting

• Unstandardized Coefficient ($b_i$):

  • Raw change in $Y$ per 1-unit change in $X$.

  • Used for prediction:   $\hat{Y} = b_0 + b_1 X$

  • Example: If $b_0 = 34.67$ and $b_1 = 0.601$, at $X = 80$:   $\hat{Y} = 34.67 + 80 \times 0.601 = 82.75$

• Standardized Coefficient ($\beta_i$):

  • Shift in standard deviations of $Y$ per 1-SD shift in $X$.

  • Example: A 1-SD increase in $X$ with $\beta = 0.685$ and $s_y = 26.10$:   $\Delta Y = 0.685 \times 26.10 = 17.8785$

• Reporting Format:

  • Predictor:   $b = 0.601$, $t(98) = 9.314$, $p < .001$

  • Overall model:   $R^2 = .47$, $F(1, 98) = 86.74$, $p < .001$