Notes on Residuals, SSE, and the Least Squares Regression Line

Residuals, SSE, and the Least Squares Line

• What is a residual?

  • For each data point, residual = actual y − predicted ŷ: ri = yi -
    75
    ehat{}_i
  • Intuition: vertical distance from the data point to the fitted line.

• Sum of squared residuals (SSE)

  • To measure total error of a line, square and sum all residuals: $SSE = \sum<em>{i=1}^n (y</em>i - \hat{y}<em>i)^2 = \sum</em>{i=1}^n r_i^2$
  • Example from the transcript: if residuals are -1, -1, -4 (squared they become 1, 1, 16), then for that dataset the SSE is $1+1+16=18$.
  • Lower SSE means the line fits the data better (compared across lines).
  • In the transcript, two candidate lines gave SSEs: first line SSE = 14, second line SSE = 18. Since 14 < 18, the first line fits better by SSE.

• Least squares regression line (best fit)

  • The line that minimizes SSE is called the least squares regression line.
  • General form: $\hat{y} = a + b x$ where
  • b is the slope (how much y changes per unit x)
  • a is the intercept (the predicted y when x = 0)
  • In the transcript, the goal is to find the line with the least SSE (best approximation in the least-squares sense).

• How to compute the slope and intercept from data (using means, standard deviations, and correlation)

  • Given:
  • mean of x: $\bar{x} = 8.76\, , \quad s_x = 2.09$
  • mean of y: $\bar{y} = 65.97\, , \quad s_y = 4.51$
  • correlation: $r = 0.748$
  • Slope (the rate of change):$b = r \frac{s<em>y}{s</em>x} \approx 0.748 \cdot \frac{4.51}{2.09} \approx 1.614$
  • Intercept (line passes through the point (\bar{x}, \bar{y})):
    $a = \bar{y} - b\,\bar{x} \approx 65.97 - 1.614 \cdot 8.76 \approx 51.83$
  • Therefore, the least-squares regression line is approximately: $\hat{y} = 51.83 + 1.614\,x$

• What does the regression line predict at specific x-values?

  • Predict at the average x: plug in $x = \bar{x} = 8.76$
  • Then $\hat{y} = a + b\bar{x} \approx 65.97$ (as expected, it matches the mean of y when x is at its mean).
  • Predict one standard deviation above the average x: $x = \bar{x} + s_x = 8.76 + 2.09 = 10.85$
  • Then $\hat{y} = 51.83 + 1.614(10.85) \approx 69.3$
  • Change from the mean when x increases by one standard deviation can be seen as approximately $\Delta \hat{y} \approx s_y \cdot r \approx 4.51 \cdot 0.748 \approx 3.37$
  • This is consistent with the predicted increase from 65.97 to ~69.3 (a rise of about 3.3–3.4).

• Why the regression line passes through (\bar{x}, \bar{y})

  • A common construction is to center the line so that the predicted point at the mean x equals the mean y, i.e., the line goes through (\bar{x}, \bar{y}).
  • This yields the intercept a = \bar{y} - b\bar{x} as shown above.

• Interpreting the slope and the standard-deviation relation

  • The slope b provides the average change in y for a one-unit increase in x.
  • A handy heuristic: a change of one standard deviation in x is associated with a change in y of approximately $\Delta \hat{y} \approx s<em>y \cdot r$, since b = r (sy / s_x).
  • In the example, one SD increase in x (in x units) yields about 3.37 units increase in y.

• Significance of the correlation (whether the linear relationship is reliable)

  • Degrees of freedom for testing correlation: $df = n - 2$
  • A critical value approach: compare the observed |r| to a critical value from a table with df = n - 2 and a chosen significance level.
  • Example in the transcript: with a certain df, a critical value is about $0.195$ for a 5% level with large df.
  • If |r| > 0.195, then the correlation is considered statistically significant. In the transcript, r = 0.234 led to |r| > 0.195, so the correlation was deemed significant. A follow-up example used r = -0.234 (negative correlation) with similar interpretation.
  • Alternative standard test statistic (not emphasized in the transcript but commonly used):
  • $t = r \sqrt{\frac{n-2}{1 - r^2}}$ with df = n - 2, which is equivalent to testing the Pearson correlation.

• Worked additional example themes from the transcript

  • A second scenario involved a different dataset (e.g., a study on cell phone use and activity performance):
  • Given: $s_y = 0.601\, , \quad r = -0.234$
  • The negative r indicates an inverse linear relationship.
  • The idea is to use the same least-squares framework to estimate a and b from their data and interpret the slope and intercept in context.
  • A causality discussion (ethical/philosophical/practical implications):
  • Correlation does not imply causation; studies often explore whether one variable causes another or whether both are related through a third factor.
  • The transcript discusses examples (e.g., anorexia and suicidal tendencies) to illustrate how complex relationships can be bidirectional or mediated by other factors, and why study design matters for causal claims.
  • Suggested study ideas include observing whether a condition precedes another in longitudinal designs or manipulating a hypothetical causal variable (where ethical) to infer causality. The key takeaway is to be cautious about drawing causal conclusions from correlation alone.

• Applying the regression to predictions: an applied workflow

  • Once a least-squares line is established, you can predict outcomes for any x within the data range by computing $\hat{y} = a + b x$.
  • Example interpretation from the transcript: predicting GPA for a student given an amount of a predictor (e.g., meds/day, phone usage, etc.). The general message is that the slope tells you the direction and approximate magnitude of the expected change in the outcome per unit change in the predictor; the intercept anchors the line on the y-axis.
  • The transcript also notes that real-world predictions may be imperfect and not all relationships are perfectly realistic, but the regression provides the best linear summary under the least-squares criterion given the data.

• Quick recap of key formulas

  • Residual for point i: $r<em>i = y</em>i - \hat{y}_i$
  • SSE: $SSE = \sum<em>{i=1}^n r</em>i^2 = \sum<em>{i=1}^n (y</em>i - \hat{y}_i)^2$
  • Slope: $b = r \frac{s<em>y}{s</em>x}$
  • Intercept: $a = \bar{y} - b \bar{x}$
  • Regression line: $\hat{y} = a + b x$
  • Change in y per 1 SD change in x: $\Delta \hat{y} \approx s_y r$
  • Degrees of freedom for correlation: $df = n - 2$
  • Significance test (alternative form): $t = r \sqrt{\frac{n-2}{1 - r^2}}$

• Important caution

  • Always keep track of what x and y stand for in your dataset (which variable is the predictor and which is the response) and ensure you interpret the intercept in the context of whether x = 0 is meaningful for your scenario.