Bivariate Data Analysis: Correlation, Causation, and Linear Regression Diagnostics

Matching Scatterplots and Correlation Coefficients

  • Systematic matching strategy:
    • Begin with correlation coefficients (rr) closest to −1-1 or +1+1, as these correspond to data distributions that most closely approximate a straight line.
    • Progressively evaluate values approaching 00, which exhibit greater dispersion or lack of linear structure.
  • Application to sample correlation coefficients and scatterplot pairings:
    • Correlation r=−0.85r = -0.85 (Identifier: dd):
    • Visual form: Tightly clustered data points displaying a clear downward linear trajectory.
    • Matching scatterplot: First graph.
    • Correlation r≈0.49r \approx 0.49 (Identifier: aa):
    • Visual form: Moderately dispersed, vague cluster with a discernible upward trend.
    • Matching scatterplot: Second graph.
    • Correlation r≈−0.48r \approx -0.48 (Identifier: bb):
    • Visual form: Moderately dispersed, vague cluster with a discernible downward trend.
    • Matching scatterplot: Fourth (last) graph.
    • Correlation r=−0.03r = -0.03 (Identifier: cc):
    • Visual form: Points closest to 00 typically present either as a completely random cloud of dots or as a distinct nonlinear pattern.
    • Specific case manifestation: A nonlinear curve that trends positively for an interval and then negatively for an interval. The opposing positive and negative associations cancel each other out mathematically, yielding a net linear correlation near zero.
    • Matching scatterplot: Third graph.

Correlation Versus Causation

  • Fundamental principle:
    • "Correlation does not imply causation" is a foundational tenet in science and mathematics emphasizing that a statistical relationship between two variables does not prove that one directly causes the other.
    • Rigorous causal inference can only be demonstrated through a randomized controlled experiment utilizing explicit treatment and control groups.
  • Explanatory mechanisms for observed correlations:
    • Coincidence:
    • Definition: A sequence of events that, though purely accidental, appears planned, arranged, or correlated over a given timeframe.
    • Statistical origin: In a world with millions of variables, random chance ensures that certain unrelated variables will exhibit high mathematical correlation purely by accident.
    • Example: Per capita cheese consumption correlates strongly with the number of deaths resulting from becoming tangled in bedsheets.
    • Example: The age of Miss America correlates with the number of murders or accidental deaths caused by steam, hot vapors, and hot objects.
    • Domain context: Specialized resources, such as the Spurious Correlations website, document these nonsensical coincidences.
    • Underlying cause (lurking variable or confounding factor):
    • Definition: An unmeasured or overlooked variable that simultaneously influences both observed variables, generating an apparent correlation between them.
    • Example: The correlation between ice cream sales and swimmer shark attacks is confounded by season. Warm summer weather increases both beach attendance (raising shark exposure) and ice cream purchases; colder winter weather decreases both.
    • Example: Sleeping with shoes on correlates strongly with waking up with a headache. The underlying confounding factor is alcohol consumption.
    • Example: Children with larger feet read at higher proficiencies than children with smaller feet. Foot size does not cause reading skill; the underlying cause is chronological age, as older children have larger feet and have completed more schooling.
    • Example: Cities with higher church counts also report higher homicide rates. The lurking variable is overall population size; larger metropolitan areas have both more churches and higher absolute numbers of homicides, while smaller towns have fewer of both.
    • Direct causation:
    • Definition: A relationship where the first event directly produces the second event as an explicit consequence.
    • Example: Depressing a light switch (cause) produces electrical circuit completion and illuminates a light bulb (effect).
    • Example: Greater hours worked directly increase total earned paychecks.
    • Example: Higher completed educational attainment leads to greater expected lifetime earnings.
    • Example: Regular physical exercise directly lowers resting blood pressure and reduces the incidence of cardiovascular heart disease.
    • Dual-variable interaction caveats:
    • Even when a causal relationship exists between variables, one variable is frequently only one of multiple contributing causes rather than an isolated sole driver.
    • Example: The historical decrease in deaths from falling out of fishing boats occurring alongside a drop in the Kentucky marriage rate is a pure coincidence.

Coefficient of Determination (r2r^2)

  • Definition and conceptual foundation:
    • The coefficient of determination, denoted as r2r^2, is the square of the Pearson correlation coefficient (rr).
    • It serves as a standard metric to quantify the strength and goodness of fit of a linear regression model.
    • Interpretation: r2r^2 measures the proportion (or percentage) of the total variation in the response variable (yy) that is explained by the least squares linear regression line and the explanatory variable (xx).
    • The unexplained remainder of variation (1−r21 - r^2) is attributable to unmodeled external factors, individual differences, or random error.
  • Mathematical operations between rr and r2r^2:
    • Calculating r2r^2 from correlation rr:
    • Formula: Square the correlation coefficient directly: r2=(r)2r^2 = (r)^2.
    • Numeric property: Because any real number squared is non-negative, r2r^2 is always positive and satisfies 0≤r2≤10 \le r^2 \le 1.
    • Example: If r=0.6r = 0.6, then r2=(0.6)2=0.36r^2 = (0.6)^2 = 0.36. Interpretation: 36%36\% of the variation in the response variable (yy) is explained by the linear relationship with the explanatory variable (xx).
    • Example: For a linear model where r=−0.87r = -0.87:
      • Calculation: r2=(−0.87)2=0.7569r^2 = (-0.87)^2 = 0.7569.
      • Interpretation: Approximately 75.69%75.69\% (or roughly 76%76\%) of the differences in the response variable are explained by the explanatory variable.
    • Calculating correlation rr from coefficient of determination r2r^2:
    • Formula: r=±r2r = \pm \sqrt{r^2}.
    • Methodological caveat: Taking a square root introduces both a positive and a negative root. The correct mathematical sign must be deduced from context, such as examining the slope direction of the scatterplot or textual descriptors regarding the association.
    • Example: Given r2=0.48r^2 = 0.48 for a known strong negative linear relationship:
      • Identification of sign: Negative association necessitates a negative root.
      • Calculation: r=−0.48≈−0.693r = -\sqrt{0.48} \approx -0.693.

Linear Regression Equations and Statistical Notation

  • Geometric review of linear lines:
    • Classical algebra form: y=mx+by = mx + b.
    • Slope (mm): Represents line steepness, computed as the ratio of vertical change to horizontal change (rise/run\text{rise}/\text{run}). Larger numerical magnitudes represent steeper, more vertical orientations.
    • Intercept (bb): The coordinate where the line intersects the vertical axis, representing the value of yy when x=0x = 0.
  • Standard statistical regression notation:
    • Statistical convention rearranges algebraic slope-intercept order to position the constant intercept first:     y^=b0+b1x\hat{y} = b_0 + b_1 x
    • Parameter definitions:
    • b0b_0: The point estimate for the vertical axis intercept (yy-intercept).
    • b1b_1: The point estimate for the slope coefficient.
    • Rationalization: Prioritizing the intercept before the slope allows natural algebraic extension into multiple linear regression models with multiple independent predictors (b0+b1x1+b2x2+…b_0 + b_1 x_1 + b_2 x_2 + \dots).
    • The "hat" symbol (circumflex):
    • Used exclusively over predicted response estimates (y^\hat{y}) to distinguish modeled values from actual observed sample values (yy).
    • Contextual variable naming:
    • Replacing abstract symbols xx and yy with natural language descriptions of the variables improves interpretability in applied statistical reporting.
    • Example: Presenting the equation as Temperature^=37+chirps\widehat{\text{Temperature}} = 37 + \text{chirps} instead of y^=37+x\hat{y} = 37 + x.

Predictions, Residuals, and Residual Plots

  • Definition of residuals:
    • A residual (or model error) is the vertical leftover variation remaining in an observed data point after accounting for the regression line.
    • Mathematical definition:     Residual=Actual y−Predicted y=y−y^\text{Residual} = \text{Actual } y - \text{Predicted } y = y - \hat{y}
    • Geometric interpretation:
    • If an observed point lies above the fitted line (y>y^y > \hat{y}), the residual is positive.
    • If an observed point lies below the fitted line (y<y^y < \hat{y}), the residual is negative.
    • If an observed point falls directly on the fitted line (y=y^y = \hat{y}), the residual is exactly zero.
  • Prediction and residual calculation example:
    • Empirical linear model: Temperature^=45.8+0.738×chirps\widehat{\text{Temperature}} = 45.8 + 0.738 \times \text{chirps}.
    • Given input: 4646 cricket chirps observed within a 15 second15\,\text{second} interval.
    • Predicted value computation:     y^=45.8+0.738×46=45.8+33.948=79.748∘F≈79.7∘F\hat{y} = 45.8 + 0.738 \times 46 = 45.8 + 33.948 = 79.748^\circ\text{F} \approx 79.7^\circ\text{F}
    • Observed actual value: y=78.5∘Fy = 78.5^\circ\text{F}.
    • Residual calculation:     Residual=78.5−79.7=−1.2∘F\text{Residual} = 78.5 - 79.7 = -1.2^\circ\text{F}
  • Residual plot mechanics and utility:
    • Construction: A secondary scatterplot that retains the original explanatory values (xx) along the horizontal axis, but replaces the original response values (yy) on the vertical axis with their corresponding residual values (y−y^y - \hat{y}).
    • Conceptual effect: Creating a residual plot is equivalent to rotating the original scatterplot until the diagonal regression line becomes a horizontal axis at zero (residual=0\text{residual} = 0), isolating and amplifying leftover errors.
    • Criteria for assessing model adequacy:
    • Appropriate linear fit: Residuals must appear as an unstructured, uniform horizontal band of random dots dispersed evenly above and below the zero line.
    • Inappropriate linear fit: Any discernible systematic pattern (such as curvature, bends, upward drifts, or downward drifts) signifies that the linear assumption is violated and that an alternative or nonlinear model is required.

Model Evaluation and Comparison Case Study

  • Dolbear cricket chirp temperature estimation:
    • Phenomenon: The acoustic chirp frequency of crickets fluctuates directly with ambient atmospheric temperature.
    • Historic rule of thumb: Count the number of chirps within a 15 second15\,\text{second} window and add 3737 to approximate the temperature in degrees Fahrenheit (Temperature=37+chirps\text{Temperature} = 37 + \text{chirps}).
    • Data collection protocol:
    • Ambient temperature values were established by averaging readings from two separate thermometers.
    • Chirps were tracked across five 30 second30\,\text{second} intervals, averaged, and then divided by 22 to standardize measurements to a 15 second15\,\text{second} basis.
    • Valid temperature domain in test: 50∘F50^\circ\text{F} to 80∘F80^\circ\text{F}.
    • Correlation strength: Correlation coefficient r=0.9803r = 0.9803, demonstrating an exceptionally strong positive linear association.
  • Eyeball line estimation vs. objective regression:
    • An informal visual fit choosing two arbitrary points yields a slope of approximately 11 (rise over run) and an apparent vertical intercept of 3737.
    • Limitation: Eyeballing regression lines is inherently subjective, non-reproducible, and unscientific across different analysts.
    • Requirement: Statistical linear regression models must rely on reproducible mathematical procedures (such as ordinary least squares) to ensure consistent parameter estimates.
  • Diagnostic comparison of candidate linear models:
    • Model 1: Temperature^=56+0.375×chirps\widehat{\text{Temperature}} = 56 + 0.375 \times \text{chirps}
    • Scatterplot alignment: Fails to track the slope of the data trajectory.
    • Residual plot: Residuals display an obvious linear diagonal slope rather than an unbiased horizontal dispersion.
    • Assessment: Bad model fit.
    • Model 2: Temperature^=45.8+0.738×chirps\widehat{\text{Temperature}} = 45.8 + 0.738 \times \text{chirps}
    • Scatterplot alignment: Positioned higher than the observed points at lower chirp ranges, matching only locally.
    • Residual plot: Displays a noticeable upward linear drift.
    • Assessment: Bad model fit.
    • Model 3: Temperature^=37+1×chirps\widehat{\text{Temperature}} = 37 + 1 \times \text{chirps}
    • Scatterplot alignment: Tracks overall points relatively well and functions adequately for quick mental arithmetic.
    • Residual plot: Residuals show non-random structural gaps and mild upward bias.
    • Assessment: Suboptimal fit.
    • Model 4: Temperature^=40.3+0.892×chirps\widehat{\text{Temperature}} = 40.3 + 0.892 \times \text{chirps}
    • Scatterplot alignment: Centered tightly through the data cloud.
    • Residual plot: Displays uniformly scattered, random points distributed horizontally above and below the zero axis without curvature or slope.
    • Assessment: Best fit model, generated through objective least squares mathematical computation.