Correlation & Simple Regression

Pearson r

• strength (0 to 1) + direction (+/−) • LINEAR relationships only • average cross-product of z-scores • scales don't matter • no prediction equation

r formula

• r = Σ(z_X × z_Y) / N • computational: [NΣXY − ΣXΣY] / √{[NΣX² − (ΣX)²][NΣY² − (ΣY)²]}

Shyness vs speeches example

• N = 6 • (642 − 960) / √(480 × 332) • r = −.797

Nonlinear relationship (effect on r)

• r understates the real relationship • e.g., Yerkes-Dodson inverted U • fix: scatterplot; transform (ln X, √X, X², 1/X, ln Y)

On-line outlier

• falls near the regression line • INFLATES r

Off-line outlier

• falls far from the regression line • DEFLATES r

Range restriction / truncated range

• limited range on one or both variables • DEFLATES r • e.g., correlating only among people hired

Range restriction correction

• I-O corrects predictor only (criterion's unrestricted SD rarely known) • Thorndike Case II: r_c = r(S/s) / √[1 − r² + r²(S²/s²)] • S = unrestricted SD, s = restricted SD

Sample size and r

• size of r doesn't depend on n • significance does • small n → chance correlations • huge n → trivial rs significant

Critical r at p = .05

• n = 50 → .28 • n = 100 → .20 • n = 200 → .14

Power for correlations (.80, α .05)

• ρ = .10 → N 782 • ρ = .20 → N 193 • ρ = .30 → N 84

Correlation vs causation

• X → Y? • Y → X? • third variable Z causes both?

Fisher z transformation

• z_r = ½ ln[(1 + r) / (1 − r)] • SE_z = 1 / √(n − 3) • near-normal; variance doesn't depend on ρ • not the same as z from a z-test

Why use Fisher z

• r's sampling distribution skewed for |ρ| ≥ ~.50 • r's variance depends on ρ

r to z values

• .30 → .31 • .50 → .55 • .70 → .87 • .90 → 1.47

Test 1: is ρ = 0?

• t = r√(N − 2) / √(1 − r²) • df = N − 2 • e.g., r = .25, N = 100 → t ≈ 2.56, significant

Test 2: is ρ a specific value?

• z = (z_r − z_ρ) × √(N − 3) • e.g., r = .54 vs ρ = .30, N = 200 → z = 4.13, significant

Test 3: two independent rs

• z = (z1 − z2) / √[1/(n1 − 3) + 1/(n2 − 3)] • e.g., .63 vs .70 → slide shows −1.18, ns • can run even if one r is ns

Test 4: 3+ independent rs

• Q = Σ(n − 3)(z − z̄)² • z̄ weighted by n − 3 • compare to χ², df = k − 1 • e.g., Q = 17.09 > 5.99 → not all ρ equal

Test 5: dependent rs (shared variable)

• Hotelling-Williams t • same sample • df = N − 3 • e.g., N = 101, r12 .4, r13 .6, r23 .3 → t ≈ −2.10 (my calculation; check slide)

Cosine similarity

• relationship among 3+ measures • cos = (d1 · d2) / (‖d1‖ × ‖d2‖) • 0 = orthogonal (90°) • equals r for centered, normalized vectors

Simple linear regression

• predicts Y from one X • turns a correlation into an equation • least squares: minimize Σ(Y − Y′)²

Regression equation

• Y′ = b0 + b1X (predicted) • Y = b0 + b1X + ε (actual)

Y′ (Y-hat)

• predicted score • falls on the regression line

Residual (ε)

• Y − Y′ • unexplained part of each score • unique for each person

Intercept (b0)

• predicted Y when X = 0 • b0 = mean Y − b1 × mean X

Slope (b1)

• change in Y per 1-unit change in X • in Y's units • b1 = r × (SD_Y / SD_X) = Cov(X, Y) / Var(X) • trap: needs r; never just the SD ratio

Slope rules

• r = 1 → slope = SD_Y / SD_X • r near 0 → slope near 0 • slope 0 → best prediction = mean of Y • line passes through (mean X, mean Y)

Slope worked example

• r .50, SD_Y 10, SD_X 2, mean X 5, mean Y 40 • b1 = 2.5, b0 = 27.5 • X = 8 → Y′ = 47.5

SST

• Σ(Y − mean Y)² • total variation

SSR

• Σ(Y′ − mean Y)² • explained (regression) variation

SSE

• Σ(Y − Y′)² • unexplained (residual) variation

Variance partition

• SST = SSR + SSE • same logic as ANOVA

r² (coefficient of determination)

• r² = SSR / SST = 1 − SSE / SST • proportion of Y's variance explained by X • shared variance (circle overlap) • trap: SSE / SST = unexplained share

Why residuals exist

• measurement error • missing predictors • sampling error • straight line is only an approximation

Standardized β in simple regression

• β = r

r vs r² trap

• r = .30 ≠ 30% explained • r² = .09 = 9% explain