Correlation, Linear Regression, and Causal Inference Study Guide

Review of Key Statistical Concepts

• Data distributions with skewness influence the relative positions of the mean and median:
  • When a dataset has a heavy concentration of data on the right side, the median sits in the physical center of the distribution.
  • The mean is pulled away from the center towards the direction where the bulk or tail of the data resides.
• A $z$-score of $0$ ($z = 0$) corresponds precisely to the mean of the distribution.
• In a two-way contingency table:
  • The total counts listed at the bottom row and far-right column are referred to as the marginal totals or the margins.
  • Joint Distribution: Measures how two categorical variables distribute together as a proportion or share of the overall sample table, dividing cell counts by the total sample size $N$.
  • Marginal Distribution: Focuses exclusively on the distribution of a single variable by reading row or column totals along the margins.
  • Conditional Distribution: Measures the distribution of one variable within a specific, single category of another variable.
  • A table with three categories per variable remains a two-way table, provided it evaluates two distinct variables.

Pearson Correlation Coefficient ($r$)

• Definition: The Pearson correlation coefficient, denoted as $r$, provides a single, unitless summary measure that captures both the direction and strength of a linear relationship between two quantitative variables.
• Scatterplots allow visual inspection of relationships, but visual evaluation is inherently subjective. Calculating $r$ standardizes evaluation across different datasets.
• Mathematical Basis:
  • The correlation coefficient $r$ is calculated by standardizing the raw values of the explanatory variable ($x$) and response variable ($y$) into standard deviation units ($z$-scores).
  • The product of these standardized deviations is averaged across all observations $N$:

$r = \frac{1}{N-1} \sum \left(\frac{x_i - \bar{x}}{s_x}\right) \left(\frac{y_i - \bar{y}}{s_y}\right)$

• Sign Determination:
  • When an observed $x_i$ is greater than the mean $\bar{x}$ and the corresponding $y_i$ is greater than $\bar{y}$, both standardized scores are positive, yielding a positive product.
  • When an observed $x_i$ is below $\bar{x}$ and $y_i$ is below $\bar{y}$, both standardized scores are negative, yielding a positive product.
  • When one variable is above its mean and the other is below its mean, the product is negative, driving $r$ into negative values.
• Application in Regression Analysis: Correlation matrices are evaluated during regression modeling to ensure covariates (explanatory variables) are not heavily correlated with one another, preventing multicollinearity issues where individual effects cannot be separated.

Properties of Correlation

• Bounded Range: Correlation $r$ is strictly bounded between $-1$ and $1$ ($-1 \le r \le 1$).
  • $r = 1$: Perfect positive linear correlation.
  • $r = -1$: Perfect negative linear correlation.
• Direction:
  • Positive ($r > 0$): Both variables move in the same direction; as $x$ increases, $y$ increases.
  • Negative ($r < 0$): Variables move in opposite directions; as $x$ increases, $y$ decreases.
• Magnitude: The absolute value $|r|$ indicates the strength of the linear association. Values closer to $1$ or $-1$ represent stronger linear relationships.
• Unitless Measure: $r$ is pure number and has no physical units (e.g., dollars, inches, or years). This allows direct comparison of correlation strength across disparate fields and measurements.
• Symmetry: Correlation is symmetric with respect to $x$ and $y$. Swapping the outcome variable ($y$) and explanatory variable ($x$) produces the exact same value of $r$.
• Outlier Sensitivity: $r$ is highly sensitive to extreme outlying observations. Individual points situated far from the primary cluster can drastically alter the calculated line and value of $r$.
• Linearity Limitation: $r$ measures only linear relationships. A dataset with a distinct curved or non-linear pattern may produce an $r$ value close to $0$ (e.g., $r = -0.2$ or $r = 0.08$), masking a clear non-linear dependency.

Social Research Applications and Plot Interpretation

• Common Empirical Questions in Social Science:
  • Education vs. Earnings: Does completing additional years of schooling correlate with higher income levels?
  • Social Support vs. Well-being: Do individuals reporting stronger social support networks report higher subjective well-being?
  • Screen Time vs. Stress: Does increased daily screen exposure correlate with elevated reported stress levels?
• Interpretation Rules for Scatterplots:
  • $r = 0.95$: Indicates a very strong positive linear relationship.
  • $r = -0.70$: Indicates a moderate-to-strong negative linear relationship.
  • $r = 0.08$: Represents an absence of a linear relationship.
• Correlation merely demonstrates empirical co-movement within a specific dataset; it does not explain underlying causal drivers.

Linear Regression Modeling

• Purpose: Linear regression fits a straight line through quantitative data to establish a predictive mathematical equation. This allows estimation of unobserved outcome values based on explanatory inputs.
• Least-Squares Regression Line Formula:

$\hat{y} = a + b \cdot x$

• $\hat{y}$ ($y$-hat): Predicted value of the response/outcome variable $y$
• $a$: Intercept parameter; the predicted value of $y$ when $x = 0$
• $b$: Slope parameter; the expected unit change in $y$ for every $1\,\text{unit}$ increase in $x$
• $x$: Explanatory variable
  • Asymmetry: Unlike correlation, linear regression is asymmetric. Reversing the assigned roles of $x$ and $y$ changes the slope, intercept, and fit of the model.
  • Multiple Linear Regression: Regression models can incorporate multiple explanatory variables simultaneously (e.g., predicting cardiovascular disease risk using income, education, and dog ownership together) to evaluate their joint predictive power.
  • Software packages like Stata derive least-squares parameters $a$ and $b$ automatically from uploaded observational data.

Empirical Examples and Extrapolation Limits

• Example 1: Study Hours vs. Exam Scores
  • Model equation with intercept $a = 1.67$: A student studying $0\,\text{hours}$ ($x = 0$) has a predicted exam score of $1.67\,\text{points}$.
  • Constraint Rules: Regression equations can generate non-sensical mathematical predictions if inputs fall outside realistic boundaries (e.g., predicting scores for $-5\,\text{hours}$ of study or predicting scores exceeding the maximum exam limit of $10\,\text{points}$ for $40\,\text{hours}$ of study).
• Example 2: Car Age vs. Resale Value (Toyota Corolla)
  • Correlation: $r = -0.99$, indicating an extremely strong negative linear relationship.
  • Slope parameter $b = -1.92$: For each additional year of car age, the predicted resale value decreases by $1.92\,\text{thousand dollars}$ ($\$1,920$).
  • Fractional Calculation: A half-year age increase ($0.5\,\text{years}$) predicts a resale value drop of $0.96\,\text{thousand dollars}$ ($\$960$).
  • Boundary Violations: Evaluating the model at $x = 30\,\text{years}$ yields a negative predicted resale price, which is impossible under realistic market conditions.
  • Initial Depreciation Nonlinearity: Real-world asset values often drop steeply in the first year after initial purchase before leveling off into a steady linear trend.

Coefficient of Determination ($R^2$) and Model Evaluation

• Prediction Method: To predict an outcome $\hat{y}$, substitute a specific value of $x$ into the derived regression formula.
• Extrapolation Risk: Predictions are reliable only within the observed range of the sample dataset. Applying the model far beyond observed bounds leads to severe prediction errors.
• Coefficient of Determination ($R^2$):
  • Derived by squaring the correlation coefficient ($r^2$).
  • Expressed as a percentage by multiplying by $100\%$.
  • Represents the proportion of total variation in the response variable $y$ that is explained by the linear relationship with $x$
  • Interpretation Example: If $r = 0.98$, then $R^2 = 0.96$, meaning $96\%$ of the variance in $y$ is explained by $x$. The remaining $4\%$ of variance is unexplained and attributed to factors omitted from the model.
  • In complex social science research, models frequently produce low $R^2$ values (e.g., $R^2 = 0.20$, explaining $20\%$ of variance), reflecting high unexplained social variability.

Class Exercise: Grocery Spending Model

• Model Specification:

$\hat{y} = 45 + 38 \cdot x$

• $x$: Household size (number of people)
• $\hat{y}$: Predicted weekly grocery spending in Canadian dollars ($\text{CAD}$)
  • Slope Interpretation ($b = 38$): Each additional household member increases predicted weekly grocery spending by $\$38\,\text{CAD}$.
  • Intercept Interpretation ($a = 45$): A household with zero occupants ($x = 0$) has a baseline predicted weekly overhead cost of $\$45\,\text{CAD}$ (accounting for standing fixed household running costs).
  • Prediction for Household of 4 People ($x = 4$):

$\hat{y} = 45 + 38 \times 4 = 45 + 152 = \$197$

• Prediction for Household of 15 People ($x = 15$):

$\hat{y} = 45 + 38 \times 15 = 45 + 570 = \$615$

• Reliability Evaluation: This prediction is unreliable due to extreme extrapolation. Datasets rarely contain sufficient observations for 15-person households to validate the linearity of the model at that scale.

Pitfalls in Inferring Causation

• Lurking Variables (Confounders):
  • A third, unmeasured variable influences both $x$ and $y$ simultaneously, producing a false correlation between them.
  • Example: Monthly ice cream sales positively correlate with drowning deaths. The lurking variable is the summer season / temperature. Warmer weather independently drives up both ice cream consumption and outdoor swimming participation.
• Reverse Causation:
  • An observed correlation between $x$ and $y$ is misinterpreted because the outcome variable $y$ actually causes or influences the explanatory variable $x$
  • Example: The number of firefighters sent to an incident positively correlates with the total dollar value of property damage. Sending firefighters does not cause property damage; severe fires independently require larger firefighter responses and produce greater structural damage.
• Spurious Correlation (Pure Coincidence):
  • Two completely unrelated variables exhibit high statistical correlation purely due to random chance when mining large datasets.
  • Example: Annual per capita cheese consumption strongly correlates with the annual number of deaths caused by becoming tangled in bedsheets (documented in Tyler Vigen's project Spurious Correlations).
  • High statistical correlation mandates further investigation, but never proves causal connections.

Evaluating Causal Pathways and Directed Acyclic Graphs (DAGs)

• Sweden Dog Ownership Study ($N = 3,400,000$):
  • Followed $3,400,000\,\text{adults}$ in Sweden over a $12\,\text{year}$ longitudinal period.
  • Findings: Dog owners exhibited lower rates of cardiovascular disease. The protective health association was significantly stronger among individuals living alone compared to multi-person households.
  • Mechanism Hypothesis: Owning a dog provides an external social support system and physical activity routine for single occupants, emulating cardiovascular protective benefits.
  • Methodological Precaution: Health status must be measured at baseline Time 1 ($T_1$) before dog ownership to ensure healthy individuals are not simply more likely to acquire dogs (selection bias/reverse causation) prior to measuring cardiovascular health at Time 2 ($T_2$).
• Directed Acyclic Graphs (DAGs):
  • Graphical tools used to clarify structural causal relationships prior to regression modeling.
  • Nodes: Structural points representing variables in the causal framework.
  • Arrows: Unidirectional lines pointing from explanatory variables directly toward outcome variables.
  • Three Basic Building Blocks:
  1. Chain (Mediation): $x \rightarrow M \rightarrow y$ (e.g., Dog Ownership $\rightarrow$ Social Interaction/Exercise $\rightarrow$ Improved Cardiovascular Health).
  2. Fork (Common Cause Confounding): $x \leftarrow Z \rightarrow y$ (e.g., Baseline Socioeconomic Status $Z$ influencing both Dog Ownership $x$ and Baseline Health $y$).
  3. Time Ordering: Incorporating explicit time steps ($T_1 \rightarrow T_2$) to establish chronological order and prevent reverse causation errors.

Questions & Discussion

• Question: What happens to the mean and median in a right-skewed distribution?
  • Answer: The median stays at the physical center of the data distribution, while the mean is pulled rightward toward the tail where the higher values reside.
• Question: Is a statement asserting that a $z$-score of zero equals the mean true or false?
  • Answer: True. A $z$-score of $0$ represents an observation that lies exactly at the population mean.
• Question: What are the row and column totals called in a two-way contingency table?
  • Answer: Marginal totals or margins.