3.2: Least Squares Regression and Determination

Regression Lines

• Regression line: a straight line that describes how a response variable (y) changes as an explanatory variable (x) changes
• The distinction between explanatory and response variables is essential with regression
  • Response (y): what you’re trying to predict
  • Explanatory (x): what you’re using to make a prediction
  • You will get a different slope and y-intercept if you change the x- and y-values
• ŷ = a + bx
  • ŷ: the predicted value of x from the regression equation
  • y would be the actual observed y value, hence the importance of the distinction
  • a: the y-intercept (predicted value of y when x=0)
  • b: slope (as x increases by one unit, the predicted y value changes by this amount)
• Least-squares regression line: the line that makes the sum of the squares of the vertical distances of the data points from the line as small as possible
  • There is a close connection between correlation and slope; they have the same sign, but not the same value
  • Always pass through (x̄, ȳ)
  • Extrapolation: the use of a regression line for prediction far outside the range of the data
  • Predictions made using extrapolation are often inaccurate
• Interpreting slope and y-intercept in context—how to write
  • Slope: “The predicted y increases/decreases on average by [value] for each increase in x by [value]”
  • Y-intercept: “A y-intercept of [value] means that for a x of 0, the predicted y is [value]”

Residuals

• Residual = observed - predicted = y - ŷ
  • We want a line to predict y using x, so we will use a line that is as close as possible to our points, which minimizes the residuals (vertical distance from the line to our points)
• Positive residual: actual value is above the line; under-predicted
• Negative residual: actual value is below the line; over-predicted
• Coefficient of determination — r^2
  • The linear regression explains r^2% of the variation in y-variable
  • In other words, this means that r^2% of the variation in y-variable is explained by the linear regression model
• s is the coefficient which measures the standard deviation of the residuals when we create a least squares regression line
  • The approximate average error (residual) when using the line for prediction is s
• Individual points with large residuals are outliers in the y direction because they lie far from the line that describes the overall pattern
• Individual points that are extreme in the x direction may not have large residuals but may still be significant
  • These points become influential points if they significantly influence the correlation
  • Influential point: a point that, if removed, would markedly change the results of the calculation
• 
• This graph highlights an outlier (yellow) and an influential point (green)
• High leverage point: a point in a regression that has a substantially larger or smaller x-value than the other observations have
  • Influential points with high leverage can be difficult to locate in residual plots as they pull the line toward them
• Outliers and high leverage points are often influential
• Extrapolation: predicting a response value while using a value for the explanatory variable that is beyond the interval of x-values used to determine the regression line
  • The further a value is extrapolated, the less reliable the predicted value is as an estimate

Correlation and the Coefficient of Determination

1. Correlation = r

• The coefficient of determination is written as r^2
• The linear regression equation explains r^2% of the variation in y-variable

\