3.2

Chapter 3: Describing Relationships

Section 3.2: Least-Squares Regression

Learning Targets

  • By the end of this section, you should be able to:

    • Make predictions using regression lines, keeping in mind the dangers of extrapolation.

    • Calculate and interpret a residual.

    • Interpret the slope and y-intercept of a regression line.

    • Determine the equation of a least-squares regression line using technology or computer output.

    • Construct and interpret residual plots to assess whether a regression model is appropriate.

    • Interpret the standard deviation of the residuals and $r^2$ and use these values to assess how well a least-squares regression line models the relationship between two variables.

    • Describe how the least-squares regression line, standard deviation of the residuals, and $r^2$ are influenced by outliers.

    • Find the slope and y-intercept of the least-squares regression line from the means and standard deviations of $x$ and $yandtheircorrelation.</p></li></ul></li></ul><h5id="b95f28f0−0734−454a−8e6f−91f805d2304f"data−toc−id="b95f28f0−0734−454a−8e6f−91f805d2304f"collapsed="false"seolevelmigrated="true">RegressionLines</h5><ul><li><p>Linear(straight−line)relationshipsbetweentwoquantitativevariablesarecommon.</p></li><li><p>Aregressionlinesummarizestherelationshipbetweentwovariablesonlyinaspecificsetting:whenonevariablehelpsexplaintheother.</p></li><li><p><strong>Regressionlineequation</strong>:(y^=b<em>0+b</em>1x),where:</p><ul><li><p>(y^)isthepredictedvalueofand their correlation.</p></li></ul></li></ul><h5 id="b95f28f0-0734-454a-8e6f-91f805d2304f" data-toc-id="b95f28f0-0734-454a-8e6f-91f805d2304f" collapsed="false" seolevelmigrated="true">Regression Lines</h5><ul><li><p>Linear (straight-line) relationships between two quantitative variables are common.</p></li><li><p>A regression line summarizes the relationship between two variables only in a specific setting: when one variable helps explain the other.</p></li><li><p><strong>Regression line equation</strong>: (\hat{y} = b<em>0 + b</em>1 x), where:</p><ul><li><p>(\hat{y}) is the predicted value ofy$ for a given value of $x.</p></li></ul></li></ul><h5id="ab6f3f7e−8e12−4dd3−9869−847dcee24f46"data−toc−id="ab6f3f7e−8e12−4dd3−9869−847dcee24f46"collapsed="false"seolevelmigrated="true">PredictionExample</h5><ul><li><p>Arandomsampleof16usedFordF−150SuperCrew4×4sselectedfromautotrader.comhastheregressionequation:</p><ul><li><p>(price^=38257−0.1629×miles driven)</p></li></ul></li><li><p>Example:PredictthepriceofaFordF−150thathasbeendriven100,000miles:</p><ul><li><p>(price^=38257−0.1629×100000=21,967)</p></li></ul></li></ul><h5id="2ab4c0ba−9f37−410f−9fad−2d710a33fc56"data−toc−id="2ab4c0ba−9f37−410f−9fad−2d710a33fc56"collapsed="false"seolevelmigrated="true">Extrapolation</h5><ul><li><p>Extrapolationispredictingvaluesoutsidetherangeofdatausedtocreatetheregressionmodel.</p></li><li><p>Caution:predictionsfaroutsidethisintervalcanbeinaccurate.</p></li><li><p>Example:PredictingpriceforaFordF−150with300,000miles:</p><ul><li><p>(price^=38257−0.1629×300000=−10613)(nonsensicalresult,indicatingextrapolationerror).</p></li></ul></li></ul><h5id="585bc3d6−6698−4304−b4a9−3bf9df3deb33"data−toc−id="585bc3d6−6698−4304−b4a9−3bf9df3deb33"collapsed="false"seolevelmigrated="true">Residuals</h5><ul><li><p>Aresidualisthedifferencebetweentheactualvalueof.</p></li></ul></li></ul><h5 id="ab6f3f7e-8e12-4dd3-9869-847dcee24f46" data-toc-id="ab6f3f7e-8e12-4dd3-9869-847dcee24f46" collapsed="false" seolevelmigrated="true">Prediction Example</h5><ul><li><p>A random sample of 16 used Ford F-150 SuperCrew 4×4s selected from autotrader.com has the regression equation:</p><ul><li><p>(\hat{\text{price}} = 38257 - 0.1629 \times \text{miles driven})</p></li></ul></li><li><p>Example: Predict the price of a Ford F-150 that has been driven 100,000 miles:</p><ul><li><p>(\hat{\text{price}} = 38257 - 0.1629 \times 100000 = 21,967)</p></li></ul></li></ul><h5 id="2ab4c0ba-9f37-410f-9fad-2d710a33fc56" data-toc-id="2ab4c0ba-9f37-410f-9fad-2d710a33fc56" collapsed="false" seolevelmigrated="true">Extrapolation</h5><ul><li><p>Extrapolation is predicting values outside the range of data used to create the regression model.</p></li><li><p>Caution: predictions far outside this interval can be inaccurate.</p></li><li><p>Example: Predicting price for a Ford F-150 with 300,000 miles:</p><ul><li><p>(\hat{\text{price}} = 38257 - 0.1629 \times 300000 = -10613) (nonsensical result, indicating extrapolation error).</p></li></ul></li></ul><h5 id="585bc3d6-6698-4304-b4a9-3bf9df3deb33" data-toc-id="585bc3d6-6698-4304-b4a9-3bf9df3deb33" collapsed="false" seolevelmigrated="true">Residuals</h5><ul><li><p>A residual is the difference between the actual value ofy$ and the predicted value of $y$:

      • Residual = Actual $y$ - Predicted $y$ = y−y^y - \hat{y}.

    • In practice, no line will pass through all points; residuals measure prediction errors in $y.</p></li><li><p>ResidualexampleusingFordF−150driven70,583miles:</p><ul><li><p>Findpredictedprice:</p></li><li><p>(price^=38257−0.1629×70583=26759)</p></li><li><p>Iftheactualpriceis.</p></li><li><p>Residual example using Ford F-150 driven 70,583 miles:</p><ul><li><p>Find predicted price:</p></li><li><p>(\hat{\text{price}} = 38257 - 0.1629 \times 70583 = 26759)</p></li><li><p>If the actual price is21,994, then:

    • Residual = 21,994 - 26,759 = -4765.

Interpreting a Regression Line

  • A regression line is a model analogous to density curves.

  • Components of regression equation (\hat{y} = b0 + b1 x):

    • $b_0$: y-intercept (predicted $y$ when $x$ = 0).

    • $b_1$: slope (amount $y$ changes with a 1 unit increase in $x$).

  • Example Interpretation: For Ford F-150, $b_1 = -0.1629$ (the predicted price decreases by $0.1629 for each additional mile driven).

  • The value of $b_0 = 38257$ gives a meaningful prediction for mileage near 0.

The Least-Squares Regression Line

  • The least-squares regression line minimizes the sum of squared residuals, achieving the best fit.

Determining Appropriateness of Linear Models
Residual Plots

  • To assess a regression model's appropriateness, look for patterns in residual plots (vertical axis displays residuals, horizontal displays the explanatory variable).

  • Characteristics of a good fit:

    • No clear patterns in the residual plot.

    • Residuals should be relatively small in size.

Evaluating Fit with $s$ and $r^2</h5><ul><li><p>Thestandarddeviationofresiduals(s)measuresthesizeofatypicalresidual,indicatingaveragepredictionerror:</p><ul><li><p>(s=∑(yi−y^)2n−2)</p></li></ul></li><li><p>Thecoefficientofdetermination(r2)measuresthepercentageofvariabilityintheresponsevariableexplainedbytheregressionline:</p><ul><li><p>(r2=1−∑(residuals)2∑(yi−yˉ)2)</p></li></ul></li></ul><h5id="9a2ae8db−a267−4ad2−8b0f−8bbbebc78703"data−toc−id="9a2ae8db−a267−4ad2−8b0f−8bbbebc78703"collapsed="false"seolevelmigrated="true">InterpretingTechnologyOutput</h5><ul><li><p>Interpretcomputerregressionoutput,identifying:</p><ul><li><p>Slope(b1).</p></li><li><p>Y−intercept(b0).</p></li><li><p>Valueofstandarddeviationofresiduals(s).</p></li><li><p>Valueofcoefficientofdetermination(r2).</p></li></ul></li></ul><h5id="67510b26−7841−4477−9f57−38b62cdf31df"data−toc−id="67510b26−7841−4477−9f57−38b62cdf31df"collapsed="false"seolevelmigrated="true">CalculatingtheRegressionEquationfromSummaryStatistics</h5><ul><li><p>Approach:Calculatemeans(xˉ),(yˉ),standarddeviations</h5><ul><li><p>The standard deviation of residuals (s) measures the size of a typical residual, indicating average prediction error:</p><ul><li><p>(s = \sqrt{\frac{\sum (y_i - \hat{y})^2}{n-2}})</p></li></ul></li><li><p>The coefficient of determination (r^2) measures the percentage of variability in the response variable explained by the regression line:</p><ul><li><p>(r^2 = 1 - \frac{\sum (\text{residuals})^2}{\sum (y_i - \bar{y})^2})</p></li></ul></li></ul><h5 id="9a2ae8db-a267-4ad2-8b0f-8bbbebc78703" data-toc-id="9a2ae8db-a267-4ad2-8b0f-8bbbebc78703" collapsed="false" seolevelmigrated="true">Interpreting Technology Output</h5><ul><li><p>Interpret computer regression output, identifying:</p><ul><li><p>Slope (b_1).</p></li><li><p>Y-intercept (b_0).</p></li><li><p>Value of standard deviation of residuals (s).</p></li><li><p>Value of coefficient of determination (r^2).</p></li></ul></li></ul><h5 id="67510b26-7841-4477-9f57-38b62cdf31df" data-toc-id="67510b26-7841-4477-9f57-38b62cdf31df" collapsed="false" seolevelmigrated="true">Calculating the Regression Equation from Summary Statistics</h5><ul><li><p>Approach: Calculate means (\bar{x}), (\bar{y}), standard deviationssx$, $sy$, and correlation $rtofind:</p><ul><li><p>Slope:(b<em>1=r⋅s</em>ysx).</p></li><li><p>Intercept:(b<em>0=yˉ−b</em>1⋅xˉ).</p></li></ul></li></ul><h5id="540866cc−ce09−4225−aa00−dffc9309e700"data−toc−id="540866cc−ce09−4225−aa00−dffc9309e700"collapsed="false"seolevelmigrated="true">RegressiontotheMean</h5><ul><li><p>Iftheexplanatoryvariableto find:</p><ul><li><p>Slope: (b<em>1 = r \cdot \frac{s</em>y}{s_x}).</p></li><li><p>Intercept: (b<em>0 = \bar{y} - b</em>1 \cdot \bar{x}).</p></li></ul></li></ul><h5 id="540866cc-ce09-4225-aa00-dffc9309e700" data-toc-id="540866cc-ce09-4225-aa00-dffc9309e700" collapsed="false" seolevelmigrated="true">Regression to the Mean</h5><ul><li><p>If the explanatory variablex$ increases by 1 standard deviation, the response variable $y$ increases by $r$ standard deviations: this is called regression to the mean.


Correlation and Regression Wisdom

  • Correlation and regression are powerful but limited tools. Key considerations:

    • They describe only linear relationships.

    • Correlation does not imply causation; ensure understanding of variable relationships and causation implications.