Correlation vs Causation | data mining!

What does the correlation coefficient r measure?

The strength and direction of a linear relationship between two numeric variables, from −1 to 1.

What does the sign of r tell you?

The direction: positive means both rise together; negative means one rises as the other falls.

Which is stronger: r = −0.9 or r = 0.5?

r = −0.9. Strength is the size of r; the sign only shows direction.

Does r = 0 mean two variables are unrelated?

No. It only means there's no linear relationship; a strong curved relationship can still have r = 0.

Application: x = −2, −1, 0, 1, 2 and y = x². What is r?

0, even though y is completely determined by x. The two halves of the U shape cancel out.

Why is "r measures how correlated two variables are" a weak definition?

It's circular. Say instead: r measures the strength and direction of a linear relationship.

What is the formula for r?

r = Σ(dx · dy) / √(Σdx² · Σdy²), where dx and dy are each value minus its variable's mean.

What's the first step when computing r by hand?

Center both variables by subtracting each variable's mean. Never multiply the raw values.

After computing Σ(dx · dy), what's the last step for r?

Divide by √(Σdx² · Σdy²). Don't take a cosine of anything.

Application: x = 1, 2, 3, 4, 5 and y = 2, 1, 4, 3, 5. Find r.

dx = −2, −1, 0, 1, 2 and dy = −1, −2, 1, 0, 2. Σ(dx · dy) = 8, Σdx² = 10, Σdy² = 10, so r = 8 / 10 = 0.8.

What is a confounder?

A third variable that causes both A and B, creating a correlation between them even though A doesn't cause B.

Application: ice cream sales and drownings rise together. What explains it?

A confounder: summer weather drives both.

Are ice cream sales and drownings independent?

No. They are correlated; they just don't cause each other. Independent means no statistical relationship at all.

What is reverse causation?

When B actually causes A, rather than A causing B.

Application: towns with more firefighters at a fire see more damage. Why?

Reverse causation: bigger fires bring more firefighters.

Why is coincidence a real risk in data mining?

Testing thousands of variable pairs guarantees that some will correlate by pure chance.

How do you establish causation?

With a randomized experiment. Random assignment breaks the link to any confounders.

Application: more hours_per_week goes with earning >50K. What's a confounder?

Occupation or seniority: some jobs require more hours and also pay more.

Application: more hours_per_week goes with earning >50K. What's a reverse causation explanation?

Earning more brings more responsibility, which means working longer hours.

Why is relationship = "Husband" fine for predicting income but useless for advice?

Prediction only needs a reliable correlation; advice needs causation. Marriage doesn't cause higher income.

Application: married people tend to earn more. What's a reverse causation explanation?

A higher income makes someone a stronger dating candidate, so income leads to marriage, not the other way around.

Why look at the scatterplot before trusting r?

One point can create a strong r by itself (Anscombe Dataset IV), and r misses curves entirely.

How do you get a correlation matrix in pandas?

df[["col1", "col2", "col3"]].corr()

After centering a list (subtracting its mean), what is the new mean?

Always 0. The original mean stays what it was (e.g. 3 for 1, 2, 3, 4, 5); only the centered list averages to 0.