Notes on Simple Regression and Correlation Analysis

Introduction to Simple Regression and Correlation Analysis

  • Focus on understanding relationships between two variables
  • Key Objectives:
    • Joint distribution of two variables
    • Linear relationships between continuous variables

Chapter Overview

  1. Introduction
  2. The scatterplot
  3. Pearson’s correlation coefficient
  4. Regression: linear relationships
  5. Coefficient of determination
  6. Graphical representation of residuals

6.1 Introduction

  • Multivariate Dataset: Observations on two or more variables
  • Bivariate Dataset: Paired observations (x1, y1), (x2, y2), … , (xn, yn)
    • Example Variables: Length and mass of men, IQ and academic performance, Diameter and height of trees
  • Dependent and Independent Variables:
    • Changes in Y explained by changes in X
    • Example: Study hours (x) vs. Test marks (y)

6.2 Scatterplot

  • Definition: Graphical representation of independent (x) vs dependent variable (y)
  • First step to assess relationship potential between two continuous variables
  • Example: Advertisement expenditure (x) vs. Passengers (y)
    • Increasing x correlates with increasing y, indicating a potential linear relationship

6.3 Pearson’s Correlation Coefficient

  • Correlation defines linear relationship between two variables
  • Coefficient (r): Measures linear relationship degree
    • Defined as the covariance between x and y
  • Properties of Correlation Coefficient:
    • Range: -1 ≤ r ≤ 1
    • Positive r: Positive relationship
    • Negative r: Negative relationship
    • r = 1: Perfect positive relation
    • r = -1: Perfect negative relation
    • r = 0: No linear relation
  • Example with strong positive correlations: r ≈ 0.993

6.5 Regression

  • Purpose: Determine relationships and forecast values
  • Estimation model:
    • Theoretical: E(y)=a+βx+ϵE(y) = a + \beta x + \epsilon
    • Estimated: y^=a+bx\hat{y} = a + bx
  • Interpretation of coefficients:
    • a: Mean of y when x=0 (y-intercept)
    • b: Change in y with a unit increase in x (gradient)
  • Method of Least Squares:
    • Minimizes vertical distances between actual observations and regression line

6.6 Coefficient of Determination

  • Definition (R²): Indication of how well the regression curve fits the data
  • Characteristics:
    • Range: 0 ≤ R² ≤ 1
    • R² = 1: Perfect fit
    • R² = 0: No fit at all
  • Example with R² = 0.993 indicating a very good fit

6.7 Graphical Representation of Residuals

  • Residual: Difference between observed and predicted values
  • Importance: Indicates fit quality of regression line
  • Residual Plot: Scatter plot of residuals against independent variable
  • Patterns to identify:
    • No pattern: Good linear fit
    • Non-linear pattern: Indicates non-linearity
    • Increasing scatter: Indicates erratic behavior
    • Outliers: Points greatly differing from regression line

Practical Considerations

  • Ensure appropriate variable relationships are analyzed; avoid irrelevant correlations
  • Handle predictions carefully, especially extrapolation, as it can lead to errors
  • Non-linear data may necessitate transformation or non-linear regression methods

Calculator Use for Correlation and Regression

  1. Input data via specific modes for statistical calculations
  2. Calculate correlation coefficients and regression parameters using calculator functions

Self-Evaluation Exercises

  • Application of learned concepts through construction of scatter plots, correlation, regression line fitting, and interpretation
  • Example: Analyze Johan's heavy object tossing performance, reflecting on results and predicted values
  • Calculation and explanation of coefficients and residuals for understanding model effectiveness.