R

3.2: Least Squares Regression and Determination

Regression Lines

  • Regression line: a straight line that describes how a response variable (y) changes as an explanatory variable (x) changes
  • The distinction between explanatory and response variables is essential with regression
    • Response (y): what you’re trying to predict
    • Explanatory (x): what you’re using to make a prediction
    • You will get a different slope and y-intercept if you change the x- and y-values
  • ŷ = a + bx
    • ŷ: the predicted value of x from the regression equation
    • y would be the actual observed y value, hence the importance of the distinction
    • a: the y-intercept (predicted value of y when x=0)
    • b: slope (as x increases by one unit, the predicted y value changes by this amount)
  • Least-squares regression line: the line that makes the sum of the squares of the vertical distances of the data points from the line as small as possible
    • There is a close connection between correlation and slope; they have the same sign, but not the same value
    • Always pass through (x̄, ȳ)
    • Extrapolation: the use of a regression line for prediction far outside the range of the data
    • Predictions made using extrapolation are often inaccurate
  • Interpreting slope and y-intercept in context—how to write
    • Slope: “The predicted y increases/decreases on average by [value] for each increase in x by [value]”
    • Y-intercept: “A y-intercept of [value] means that for a x of 0, the predicted y is [value]”

Residuals

  • Residual = observed - predicted = y - ŷ
    • We want a line to predict y using x, so we will use a line that is as close as possible to our points, which minimizes the residuals (vertical distance from the line to our points)
  • Positive residual: actual value is above the line; under-predicted
  • Negative residual: actual value is below the line; over-predicted
  • Coefficient of determination — r^2
    • The linear regression explains r^2% of the variation in y-variable
    • In other words, this means that r^2% of the variation in y-variable is explained by the linear regression model
  • s is the coefficient which measures the standard deviation of the residuals when we create a least squares regression line
    • The approximate average error (residual) when using the line for prediction is s
  • Individual points with large residuals are outliers in the y direction because they lie far from the line that describes the overall pattern
  • Individual points that are extreme in the x direction may not have large residuals but may still be significant
    • These points become influential points if they significantly influence the correlation
    • Influential point: a point that, if removed, would markedly change the results of the calculation
  • This graph highlights an outlier (yellow) and an influential point (green)
  • High leverage point: a point in a regression that has a substantially larger or smaller x-value than the other observations have
    • Influential points with high leverage can be difficult to locate in residual plots as they pull the line toward them
  • Outliers and high leverage points are often influential
  • Extrapolation: predicting a response value while using a value for the explanatory variable that is beyond the interval of x-values used to determine the regression line
    • The further a value is extrapolated, the less reliable the predicted value is as an estimate

Correlation and the Coefficient of Determination

  1. Correlation = r
  • The coefficient of determination is written as r^2
  • The linear regression equation explains r^2% of the variation in y-variable