2.4

2.4 Least-Square Regression

  • The least-squares regression summarizes the overall linear pattern between two variables by drawing a regression line on the scatterplot.

Regression Line

  • A regression line is a straight line that describes how a response variable $y$ changes as an explanatory variable $x$ changes.
  • The regression line is often used to predict the value of $y$ for a given value of $x$.
  • Unlike correlation, regression requires the specification of a response variable and an explanatory variable among the two variables in analysis.

Straight Lines

  • A straight line that relates $y$ to $x$ has an equation of the form:
    y=a+bxy = a + bx
  • The slope $b$ represents the amount by which $y$ changes when $x$ increases by one unit.
  • The intercept $a$ is the value of $y$ when $x = 0.

Least-Squares Regression Line

  • The least-squares regression line of $y$ on $x$ minimizes the sum of squares of the vertical distances of the data points from the line.
  • This can be mathematically represented, where the values of the slope $b$ and intercept $a$ are found such that the following sum is minimized:
    exterror=1n∑<em>i=1n(y</em>i−(a+bxi))2ext{error} = \frac{1}{n}\sum<em>{i=1}^{n}(y</em>i - (a + bx_i))^2

Statistical Package Output

  • Call: lm(formula = Volume ~ Girth, data = trees)
  • Residuals:
    • Min: -8.0654
    • 1Q: -3.1067
    • Median: 0.1520
    • 3Q: 3.4948
    • Max: 9.5868
  • Coefficients:
    • (Intercept):
    • Estimate: -36.9435
    • Std. Error: 3.3651
    • t value: -10.98
    • p-value: 7.62e-12 ***
    • Girth:
    • Estimate: 5.0659
    • Std. Error: 0.2474
    • t value: 20.48
    • p-value: < 2e-16 ***
  • Significance codes:
    • 0 '***'
    • 0.001 '**'
    • 0.01 '*'
    • 0.05 '. '
    • 0.1 ' '
    • 1 ' '
  • Residual standard error: 4.252 on 29 degrees of freedom.
  • Multiple R-Squared: 0.9353
  • Adjusted R-squared: 0.9331
  • F-statistic: 419.4 on 1 and 29 DF, p-value: < 2.2e-16.

Prediction

  • The following equation, derived from the output of the statistical software, can be used for prediction:
    Y=−36.94+5.07xY = -36.94 + 5.07x
  • The Multiple R-squared is reported as:
    r2=0.9353r^2 = 0.9353
  • Thus, the correlation coefficient $r$ is calculated as:
    r=0.9671r = 0.9671

r^2 in Prediction

  • The square of the correlation $r^2$ (also known as the Multiple R-squared) is the fraction of the variation in the values of $y$ explained by the least-squares regression of $y$ on $x$.
  • The square of the correlation acts as a measure of how well the regression explains the response variable $y$.

Extrapolation

  • Extrapolation refers to using a regression line for prediction far outside the range of values of the explanatory variable $x$ used to obtain the line.
  • Predictions made through extrapolation are generally not accurate and may lead to misleading conclusions.

Equation of the Least-Squares Regression Line

  • The equation of the least-squares regression line of $y$ on $x$ is:
    ildey=a+bxilde{y} = a + bx
  • Where:
    • $ ilde{y}$ is the predicted value of the response variable.
    • $b$ is the slope, which can be calculated as:
      b=r<em>s(y)s</em>xb = \frac{r<em>{s(y)}}{s</em>x}
    • $a$ is the intercept, computed when:
      a=yˉ−bxˉa = \bar{y} - b \bar{x}