Comprehensive Notes on Simple Linear Regression and Correlation

Introduction to Simple Linear Regression and Correlation

• This study explores the relationship between two variables using two primary methods:

  • 1. Regression analysis.

  • 2. Computing the correlation coefficient.

• Bivariate Data Sets: These are data sets involving two variables that are measured simultaneously to explore their relationships.

The Regression Model

• Definition: A regression model is a mathematical equation that describes the relationship between two or more variables. It is used to evaluate the magnitude of change in one variable resulting from a change in another variable and to predict (estimate) the value of one variable for a given value of another.

• General Equation:   $\hat{y} = b_0 + b_1x_1 + b_2x_2 + \dots + b_nx_n + \epsilon$

  • $b_i (i = 1, \dots, n)$: Parameter estimates or coefficients.

  • $b_0$: The y-intercept (constant).

  • $\epsilon$: The error term.

• Linear vs. Nonlinear Models:

  • A simple regression model providing a straight-line relationship is a linear regression model.

  • If the relationship is not a straight line, it is a nonlinear regression model.

• Simple Linear Regression Equation:   $Y = a + bX + \epsilon$

  • $X$: Independent (explanatory) variable.

  • $Y$: Dependent (response) variable.

  • $a$: The y-intercept.

  • $b$: The slope of the regression line.

Simple Linear Regression Objectives

• The exploration of linear relationships focuses on two main issues:

  • 1. Finding the Equation: Determining the linear regression equation (the straight line that best describes the relationship). This line is then used for linear estimation to predict $Y$ for a given $X$.

  • 2. Measuring Strength: Determining the strength of the linear relationship using the linear correlation coefficient.

Scatter Diagrams

• A scatter diagram (also known as a scatter plot or scatter graph) is the first step in exploring bivariate data. It involves plotting variables on a graph to detect potential relationships.

• Example Case Study: Energy Consumption vs. Household Income.

  • Household Income ($X$) in thousands of rand.

  • Energy Consumption ($Y$) in thousands of Btu.

  • Data points for 8 households:

  • (20, 2), (72, 7), (44, 5), (96, 10), (60, 7), (32, 3), (90, 9), (52, 5).

The Least Squares Principle

• Regression analysis aims to find the line that best fits the points in a scatter diagram, providing the best possible description of the relationship.

• Least Squares Regression Line: This is the line that best fits the plotted $(X, Y)$ points based on the least squares principle.

• Least Squares Principle Rule: The "best" fit line is the one that minimizes the sum of the squares of errors (SSE).

Calculation of the Least Squares Regression Line

• Equation to be fitted:   $\hat{y} = a + bx + e$

  • $\hat{y}$: The fitted $y$ value (the value on the line, distinct from the observed $y$ value).

  • $e$: The error (residual).

• Computation Formulas:

  • Slope ($b$):     $b = \frac{S_{XY}}{S_{XX}}$     $S_{XY} = \sum XY - \frac{\sum X \sum Y}{n}$     $S_{XX} = \sum X^2 - \frac{(\sum X)^2}{n}$

  • Y-Intercept ($a$):     $a = \frac{\sum Y - b \sum X}{n}$     $OR$     $a = \bar{Y} - b\bar{X}$

Practical Example: Income and Energy Consumption

• Data Summary ($n = 8$):

  • $\sum X = 466$

  • $\sum Y = 48$

  • $\sum X^2 = 32164$

  • $\sum Y^2 = 342$

  • $\sum XY = 3310$

• Intermediate Calculations:

  • $S_{XY} = 3310 - \frac{466 \times 48}{8} = 514$

  • $S_{XX} = 32164 - \frac{466^2}{8} = 5019.5$

• Model Parameters:

  • $b = \frac{514}{5019.5} = 0.1024$

  • $a = \frac{48 - (0.1024 \times 466)}{8} = 0.0352$

• Estimated Regression Line:   $\hat{Y} = 0.0352 + 0.1024X$

• Prediction Example:

  • For a household with an income of R35,000 ($X = 35$):

  • $\hat{Y} = 0.0352 + 0.1024(35) = 3.6192$

  • Estimated consumption is approximately $3.6 \text{ thousand Btu}$.

Limitations and Practical Cautions

• Non-linear relationships: Lack of a linear relationship does not imply no relationship exists; variables might be related non-linearly.

• Practical Validity: Variables should only be explored if a practical relationship is plausible. Spurious correlations (e.g., number of vehicles in New York vs. divorces in South Africa) have no practical value despite what the data might suggest.

Standard Error of the Estimate

• Definition ($S_e$): A measure of the standard deviation of the errors in the regression model. It represents the average deviation of the observed $Y$ values from the predicted $\hat{Y}$ values.

• Random Error ($e$): Defined as the difference between actual $Y$ and estimated $\hat{Y}$.

  • $e = Y - \hat{Y}$

• The sum of errors ($\sum e$) is always zero.

• Formula for $S_e$:   $S_e = \sqrt{\frac{\sum e^2}{n - 2}} = \sqrt{\frac{\sum (Y - \hat{Y})^2}{n - 2}}$

• Accuracy: A smaller standard error indicates higher prediction accuracy.

• Calculation for Energy Example:

  • $\sum e^2 = 1.3661$

  • $S_e = \sqrt{\frac{1.3661}{8 - 2}} = 0.4772$

Residuals

• A residual is the vertical distance between an actual data point and the regression line.

• If the point is above the line, the residual is positive.

• If the point is below the line, the residual is negative.

Coefficient of Determination ($r^2$)

• Definition: A statistical measure that determines the proportion of variance in the dependent variable explained by the independent variable in the model.

• Range: $0 \le r^2 \le 1$.

• Formulas:   $r^2 = \frac{(S_{XY})^2}{S_{XX} S_{YY}}$   $OR$   $r^2 = \frac{b S_{XY}}{S_{YY}}$

• Coefficient of Alienation: Calculated as $1 - r^2$, representing the unexplained variance.

• Calculation for Energy Example:

  • $S_{YY} = 342 - \frac{48^2}{8} = 54$

  • $r^2 = \frac{514^2}{5019.5 \times 54} = 0.9747$

  • Result: $97.47\%$ of the variation in energy consumption is explained by income; the remaining $3\%$ is due to other factors.

Linear Correlation Coefficient ($r$)

• Definition: Measures the strength and direction of the linear association between two variables and how closely points cluster around the regression line.

• Range: $-1 \le r \le 1$.

  • Higher clustering near the line leads to $r$ approaching $1$ or $-1$.

  • Lower clustering leads to $r$ approaching $0$.

• Rules of Thumb for Interpretation:

  • $0.90 \text{ to } 1.00$: Very high correlation.

  • $0.70 \text{ to } 0.90$: High correlation.

  • $0.50 \text{ to } 0.70$: Moderate correlation.

  • $0.30 \text{ to } 0.50$: Low correlation.

  • $0.00 \text{ to } 0.30$: Little, if any, correlation.

• Pearson’s Linear Correlation Coefficient Formula:   $r = \frac{S_{XY}}{\sqrt{S_{XX} S_{YY}}}$

Spearman’s Rank Order Correlation ($r_s$)

• Definition: Measures the relationship between the ranks of the data rather than the raw values.

• Range: $-1 \le r_s \le 1$.

• Ranking Method:

  • 1. Rank $X$ values in increasing/decreasing order to get $x$.

  • 2. Rank $Y$ values to get $y$.

  • 3. Tied observations: Assign the mean of the ranks they would have occupied.

• Formula using Ranks ($x$ and $y$):   $r_s = \frac{S_{xy}}{\sqrt{S_{xx} S_{yy}}}$

• Simplified Formula for Rank Difference ($d$):   $r_s \approx 1 - \frac{6 \sum d^2}{n(n^2 - 1)}$

  • where $d = x - y$.

• Comparison in Energy Example:

  • both methods result in $r_s = 0.988$. Note that if there are many tied observations, results may differ between formulas.