Correlation and Regression Notes

Correlation and Regression

Correlation Review

• Focuses on bivariate cases:
  • One predictor variable (IV).
  • One criterion variable (DV).
• Interested in:
  • Strength.
  • Direction (positive or negative).
  • Shape (linear or curvilinear).
  • Significance.
• Currently focuses on linear relationships.
• Pearson’s Product Moment Correlation Coefficient (r) quantifies the strength and direction of bivariate linear relationships.

Pearson’s Correlation Coefficient

• A statistic ranging from -1 to +1.
• Indicates the direction and strength of a linear relationship between X and Y variables.
• Negative Correlation: Increases in one variable are associated with decreases in the other.
• Positive Correlation: Increases in one variable are associated with increases in another.
• Tests for linear relationships by drawing a straight line through the data points.

TV Violence Example

• Researchers examine the relationship between:
  • Weekly viewing of violent TV programs (hours).
  • Number of violent acts toward playmates among children.
• Hypothesis: Watching TV violence is related to violent acts towards playmates.

Calculating the Correlation Coefficient $r$

• Calculate Z scores for each score ( $Z<em>x$, $Z</em>y$ ).
• Ensure the correct mean and SD are used.
• Multiply the Zx by the Zy to get the cross product ($Z<em>xZ</em>y$).
• Ensure the sign (+ or -) is correct.
• Add all the cross-products ($\sum Z<em>xZ</em>y$) and divide by the number of pairs of scores ($N$).
• Formula: $r = \frac{\sum Z<em>xZ</em>y}{N}$

Z Scores and Pearson’s $r$: The Missing Link

• Z scores indicate whether a score is above (+Z) or below (-Z) the mean (M).
• Multiplying Z values on X and Y gives cross-products.
• Positive correlation = mostly positive cross-products.
• Negative correlation = mostly negative cross-products.
• No correlation = equal number of + and - cross-products.

Positive Correlations

• Above mean on X (+$Z<em>X$), above mean on Y (+$Z</em>Y$).
• Below mean on X (-$Z<em>X$), below mean on Y (-$Z</em>Y$).
• $Z<em>XZ</em>Y$ values are mostly positive.

Negative Correlations

• Above mean on X (+$Z<em>X$), below mean on Y (-$Z</em>Y$).
• Below mean on X (-$Z<em>X$), above mean on Y (+$Z</em>Y$).
• $Z<em>XZ</em>Y$ values are mostly negative.

No Correlation

• Position on X (+$Z<em>X$ or -$Z</em>X$) not linked to position on Y (+$Z<em>Y$ or -$Z</em>Y$).
• $Z<em>XZ</em>Y$ values are equally positive and negative.

Interpreting Correlation Result

• Example: A correlation of +0.925 between hours watching violent TV and violent acts.
• Very strong positive relationship.
• As hours watching violent TV increase, violent acts increase.

Regression - Prediction

• Correlation indicates:
  • Direction of the relationship.
  • Strength of the relationship.
• Regression:
  • Plots the line of best fit.
  • Uses the line to predict scores on the DV.

The Regression Line of Best Fit

• Correlation analysis involves drawing a line of best fit.
• The correlation coefficient measures how close the data falls to the line.

Drawing the Line of Best Fit

• Requires two pieces of information:
  • The slope.
  • The Y-axis intercept.

What is the Slope?

• Indication of the gradient or steepness of the line.
• Also known as the regression coefficient.
• Mathematical terms: units of Y variable increase for every unit increase in the X variable.

What is the Y-Axis Intercept?

• The point where the line of best fit crosses the Y-axis.
• Predicted value of Y when X is zero.
• Can be positive or negative.

Developing the Regression Equation

• $\hat{Y} = a + bX$
  • $\hat{Y}$ = predicted score on the Y variable.
  • $a$ = the Y-intercept.
  • $b$ = the slope.
  • $X$ = the person’s score on the X variable.

Using the Equation to Predict Scores

• Example: Predicting violent acts based on hours watching violent TV.
• Calculations shown for predicting individual scores using the regression equation.

Plotting the Regression Line

• Plot predicted Ys with Xs on the original scatterplot.
• Join the dots to create the regression line.

Predicting Scores

• Using the regression equation to predict scores for new individuals based on their X score (IV).
• Example calculations provided.

Examining the Accuracy of Predictions

• A residual score is the difference between the actual score and the predicted score.
• Residual Score = Actual Score - Predicted Score

Residuals Example

• Example calculation of residual scores for each data point.
• Explanation of what the residual scores indicate.

Squared Residuals

• The sum of the squared residuals is at a minimum when based on the regression line.
• The regression line is the least squares regression line.

Standard Error of the Estimate (SEE)

• The average distance from data points to the regression line.
• The average amount of error when using the regression line to predict a person’s score.
• SEE formula is similar to the SD formula.

Calculating SEE

• Interpretation: when using the regression equation to predict violent acts, our estimate will be inaccurate by an average of 1.18 violent acts.

Corrected measure of accuracy: the Estimated Population Standard Error of the Estimate

• The SEE is an underestimate of the true population SEE and so a correction is made (much like one made when estimating population SDs)

Calculating the Estimated Population SEE

• Interpretation: when using the regression equation to predict violent acts, our estimate will likely be inaccurate by an average of 1.67 violent acts in the population.

Back into SS (Sums of Squares) land:

• Partitioning variability in Y scores into two components:
  • Variability due to error ($SS_{error}$).
  • Variability due to regression ($SS_{reg}$).
• The total variability ($SS<em>{total}$) will be $SS</em>Y$: the sum of the squared deviations between each Y-score and the mean of Y.
• The error variability ($SS_{error}$) = the sum of squared deviations between each Y-score and the predicted Y-score.
• The regression variability ($SS_{reg}$) = the sum of squared deviations between each predicted Y-value and the mean of Y.
• From these SS, we can calculate MS using 1 and N-2 df
• We can then calculate an F-statistic as $MS<em>{reg}/MS</em>{error}$

Noting the similarity between $r^2$ and SEE:

• $r^2$ within an ANOVA context is the equivalent of $SS<em>{regression}/SS</em>{total}$
• Example: $r^2$ = 32.40/38 = 0.85. Also, $r^2$ from r = $.925^2$ = 0.85.
• This is similar to eta-squared in ANOVA (between groups variance/total variance)

Constructing a summary table as for an ANOVA:

• ANOVA summary table includes source, df, SS, MS, and F.
• Example of ANOVA table with regression and error components.
• Determining if Fobs > Fcrit to assess the significance of the regression.

Why isn’t it significant?

• Power (the ability to obtain a significant result when an effect is real) operates as a function of both effect size and sample size
• It is possible to have a large effect size with a small sample and NOT obtain a significant result
• It is possible to have a small effect size with a large sample and obtain a significant result
• Always temper your interpretation of significance with an examination of the obtained effect size (i.e., is it a meaningful size) and the sample size (i.e., is the sample size so small that it is preventing a decent effect size from being significant or so large that it is making a negligible effect significant)

Reporting

• Example write-up of correlation and regression results including:
  • Correlation coefficient (r).
  • Coefficient of determination ($r^2$).
  • Regression coefficient (b).
  • Standard error of the estimate (SEE).
  • F-statistic and p-value.
• Mention of the small sample size (low power) contributing to the non-significant F value.

SPSS Example

• Instructions on how to perform linear regression in SPSS.
• Creating a data file.
• Selecting LINEAR from the REGRESSION submenu.
• Selecting the IV (X) and DV (Y).

Assumptions

• Normality: Both X and Y variables should have relatively normal distributions.
• Linearity: The relationship being tested should be linear.

Potential problems:

• Impact of outliers/extreme scores
• Regression equations and correlations are highly influenced by outliers
• An outlier is a data point that lies away from the rest of the pack of data
• If an outlier is in a position consistent with the pattern of the rest of the data, it is okay but if it is inconsistent it will unduly influence the positioning of the regression line
• This notion is akin to how the mean is affected by outliers
• Other issues such as restriction in range are also problems

Correlation vs. Causation

• Correlation does not imply causation.
• Movement on one variable is merely associated with movement on the other.
• Cannot infer direction of relationship, causality, or rule out the influence of other variables.