Introduction to Regression Analysis: Predictions, Equations, and Error

Introduction for Regression Analysis

• Context within the Course: Regression is the follow-up topic to Correlation (Chapter 13). It is considered one of the more challenging topics to adapt to an online format because the presentation often varies based on student questions and feedback.
• Prerequisites for Success:
  • Slope-Intercept Form: Students must have a solid grasp of the algebraic concept of a line ($y = mx + b$). Reviewing this for 10-20 minutes is essential to avoid wasting time during the 90-minute to two-hour regression lecture.
  • Concept of Deviations: A refresher of deviations from Chapter 4 is highly recommended, as regression concepts (like residuals) are analogous to deviations from the mean.
• The Example Case: This lecture continues the example used in Chapter 13: the relationship between playing violent video games (Variable $X$) and aggressive behavior (Variable $Y$, measured in fights per week).

The Purpose and Logic of Regression

• Prediction: The primary purpose of regression is to predict specific values of $Y$ based on given values of $X$. While correlation tells us the strength and direction of a relationship, regression allows us to make concrete estimates.
• The Best Fit Line (Regression Line):
  • In a scatter plot, we can visualize a line running through the points.
  • If points go up and to the right, there is a positive correlation; if they go down and to the right, it is a negative correlation. If the points are a "blob," there is no correlation.
  • Regression uses a precise mathematical procedure to find the single best line that fits the data better than any other possible line. This is also known as the "best fit line."
• Predicting Heights ($Y$): On a graph, the regression line allows us to find an $X$ value on the horizontal axis, look up to the height of the line at that point, and find the corresponding prediced $Y$ value on the vertical axis.

Regression Notation and Symbols

• Y-Hat ($\hat{y}$): Predicted values of $Y$ are given the symbol $\hat{y}$. It is read aloud as "y-hat."
  • Metaphor: Think of the symbol as a "wizard's hat." This indicates the value is a prediction (magic/foresight) rather than an actual observed data point.
• Algebra vs. Statistics Notation:
  • Algebra: $y = mx + b$ (where $m$ = slope and $b$ = intercept).
  • Statistics (Regression): $\hat{y} = bX + a$.
  • In stats, $b$ is the symbol for slope (changing from $m$).
  • In stats, $a$ is the symbol for the intercept (changing from $b$).
  • Note of Confusion: The letter $b$ represents the intercept in algebra but represents the slope in regression statistics.
• Variables vs. Constants:
  • $X$ and $\hat{y}$ are variables. You can plug in any $X$ to get a corresponding $\hat{y}$.
  • $b$ (slope) and $a$ (intercept) are constants. For a specific dataset, these numbers are calculated once and do not change for that problem.

Solving for Slope and Intercept

To find the equation of the regression line, you use data already calculated for correlation ($M_x$, $M_y$, $SS_x$, $SS_y$, and $SP$).

• Formula for Slope ($b$):$\text{Slope } (b) = \frac{SP}{SS_x}$
• Formula for Intercept ($a$):$\text{Intercept } (a) = M_y - (b \times M_x)$
  • Restriction: You must always solve for slope ($b$) first because it is a required component of the intercept ($a$) formula.

Example Calculation (Video Games and Aggression)

Using values from the Chapter 13 example:

• $M_x = 6$ (Average hours of games per week)
• $M_y = 4$ (Average fights per week)
• $SS_x = 64$
• $SS_y = 16$
• $SP = +28$
• $r = 0.875$

1. Calculate $b$:$b = \frac{28}{64} = 0.44$
2. Calculate $a$:$a = 4 - (0.44 \times 6)$$a = 4 - 2.64 = 1.36$
3. The Regression Equation:$\hat{y} = 0.44X + 1.36$

Interpreting Slope and Intercept

• General Interpretation of Slope ($b$): It tells you how many units of increase occur in the predicted $Y$ ($\hat{y}$) for each one-unit increase in $X$.
  • Example Case: For every one additional hour of violent video games played per week ($X$), we predict the child will get into $0.44$ more fights per week ($\hat{y}$).
• General Interpretation of Intercept ($a$): It tells you the predicted value of $Y$ when $X = 0$.
  • Example Case: If a child plays zero hours of violent video games per week, we predict they will get into $1.36$ fights per week. (This accounts for other factors that might cause aggression besides games).

Using the Equation to Generate Predictions

We plug actual observed $X$ values into the equation to see how well the line predicts the actual $Y$ values (fights).

• Child 1: $X = 0 \rightarrow \hat{y} = 0.44(0) + 1.36 = 1.36$ (Actual $y = 2$)
• Child 2: $X = 10 \rightarrow \hat{y} = 0.44(10) + 1.36 = 5.76$ (Actual $y = 6$)
• Child 3: $X = 4 \rightarrow \hat{y} = 0.44(4) + 1.36 = 3.12$ (Actual $y = 2$)
• Child 4: $X = 8 \rightarrow \hat{y} = 0.44(8) + 1.36 = 4.88$ (Actual $y = 4$)
• Child 5: $X = 8 \rightarrow \hat{y} = 0.44(8) + 1.36 = 4.88$ (Actual $y = 6$)

Note: Prediction models are rarely perfect. The value of regression is being able to predict outcomes for a new child ($X = 7$) even when we don't have their fight data ($Y$).

Residuals (Error of Prediction)

• Definition: A residual is the difference between an actual observed value ($y$) and a predicted value ($\hat{y}$).
• Formula:$\text{Residual} = y - \hat{y}$
• Interpretation of the Sign:
  • Positive Residual: The person scored higher than the prediction. The point is above the line.
  • Negative Residual: The person scored lower than the prediction. The point is below the line.
• Mathematical Property: The sum of all residuals in a dataset is always zero ($\sum(y - \hat{y}) = 0$), much like the sum of deviations from the mean.
• Visual Representation: Residuals are the vertical distances between the points on a scatter plot and the regression line.

Defining the "Best Fit"

• Least Squares Criterion: The regression line is called the "best fit" because it minimizes the total vertical distance between the line and all data points.
• Sum of Squared Residuals ($SS_{residual}$): Because residuals add up to zero, we square them to make them all positive before summing them.
  • The regression line is the line that results in the smallest possible $SS_{residual}$.
  • If you tweak the slope or intercept even slightly, the $SS_{residual}$ will increase.

Standard Error of the Estimate

• Definition: The standard error of the estimate is the "average residual." It measures the average vertical distance between the actual data points and the regression line.
• Analogy: It is to the regression line what standard deviation is to the mean.
• Notation: In this course, we use the abbreviation "Standard Error of the Estimate."
• Formula:$\text{Standard Error of the Estimate} = \sqrt{\frac{SS_{residual}}{n - 2}}$
  • Note on Degrees of Freedom: We use $n - 2$ because we are dealing with two variables ($X$ and $Y$).
• Computational Shortcut for $SS_{residual}$:$SS_{residual} = (1 - r^2) \times SS_y$
  • Using the example: $SS_{residual} = (1 - 0.875^2) \times 16 = (1 - 0.766) \times 16 = 0.234 \times 16 = 3.744$
• Final Calculation for Example:$\text{Standard Error} = \sqrt{\frac{3.744}{5 - 2}} = \sqrt{\frac{3.744}{3}} = \sqrt{1.25} = 1.12$
  • Meaning: On average, our predictions of fights per week are off by $1.12$ fights.

Relationship Between Correlation and Standard Error

• Inverse Relationship: As the correlation ($r$) gets stronger (closer to $1.0$ or $-1.0$), the standard error of the estimate gets smaller.
• High Correlation = Small Error: If education and income are highly correlated, knowing education allows for very accurate (low error) predictions of income.
• Low Correlation = High Error: If IQ and shoe size have a near-zero correlation, using IQ to predict shoe size will result in massive errors, performing no better than random guessing.