Simple Linear Regression - Notes

Simple Linear Regression

Statistics in Practice: Alliance Data Systems

• Alliance Data Systems (ADS) provides transaction processing, credit services, and marketing services.

• ADS uses regression analysis to build models for direct market campaigns.

• Regression models predict the probability of purchase and the amount spent by consumers.

• Example: A retail store chain wanted to attract new customers.

• ADS analysts selected a sample, sent promotional materials, and collected transaction data.

• The consumer-specific variable that contributed most to predicting the amount purchased was the total amount of credit purchases at related stores over the past 39 months.

• Estimated regression equation: $\hat{y} = 26.7 + 0.00205x$

  • $\hat{y}$ = amount of purchase

  • $x$ = amount spent at related stores

• Someone spending $10,000 over the past 39 months at related stores would spend $47.20 when responding to the direct mail promotion.

• The final model included other variables such as the absence/presence of a bank credit card, estimated income, and the average amount spent per trip at a selected store.

Simple Linear Regression

• Regression analysis is used to develop an equation showing how variables are related.

• The variable being predicted is the dependent variable (y).

• The variable used to predict the dependent variable is the independent variable (x).

• Simple linear regression involves one independent variable and one dependent variable with a straight-line relationship.

• Multiple regression analysis involves two or more independent variables.

Simple Linear Regression Model

• Armand’s Pizza Parlors: Quarterly sales (y) are related to the size of the student population (x).

• Regression model: $y = \beta<em>0 + \beta</em>1x + \epsilon$

  • $\beta<em>0$ and $\beta</em>1$ are parameters of the model.

  • $\epsilon$ is the error term (random variable).

• Error term accounts for variability in y that cannot be explained by the linear relationship between x and y.

Simple Linear Regression Equation

• Regression equation: $E(y) = \beta<em>0 + \beta</em>1x$

• $\beta_0$ is the y-intercept of the regression line.

• $\beta_1$ is the slope.

• $E(y)$ is the mean or expected value of y for a given value of x.

• The regression line can be positive, negative, or have no relationship.

Estimated Regression Equation

• Sample statistics $b<em>0$ and $b</em>1$ are estimates of $\beta<em>0$ and $\beta</em>1$.

• Estimated regression equation: $\hat{y} = b<em>0 + b</em>1x$

• $\hat{y}$ is the estimated value of y for a given value of x.

Least Squares Method

• The least squares method is a procedure for using sample data to find the estimated regression equation.

• Minimize the sum of the squares of the deviations between observed values and estimated values.

• Least squares criterion: $\min \sum(y<em>i - \hat{y}</em>i)^2$

• Slope and y-intercept for the estimated regression equation:

  • $b<em>1 = \frac{\sum (x</em>i - \bar{x})(y<em>i - \bar{y})}{\sum (x</em>i - \bar{x})^2}$

  • $b<em>0 = \bar{y} - b</em>1\bar{x}$

• Alternative formula for b1: $b<em>1 = \frac{\sum x</em>i y<em>i - (\sum x</em>i \sum y<em>i)/n}{\sum x</em>i^2 - (\sum x_i)^2/n}$

• Example: Armand’s Pizza Parlors

  • Estimated regression equation: $\hat{y} = 60 + 5x$

  • An increase in the student population of 1000 is associated with an increase of $5000 in expected sales.

Coefficient of Determination

• The coefficient of determination provides a measure of the goodness of fit for the estimated regression equation.

• The ith residual is the difference between the observed value and the estimated value: $y<em>i - \hat{y}</em>i$

• Sum of squares due to error (SSE): $SSE = \sum(y<em>i - \hat{y}</em>i)^2$

• Total sum of squares (SST): $SST = \sum(y_i - \bar{y})^2$

• Sum of squares due to regression (SSR): $SSR = \sum(\hat{y}_i - \bar{y})^2$

• Relationship among SST, SSR, and SSE: $SST = SSR + SSE$

• Coefficient of determination: $r^2 = \frac{SSR}{SST}$

• $r^2$ represents the percentage of the total sum of squares that can be explained by using the estimated regression equation.

Correlation Coefficient

• The correlation coefficient measures the strength of linear association between two variables.

• Values are between -1 and +1.

• Sample correlation coefficient: $r<em>{xy} = (sign of b</em>1)\sqrt{r^2}$

• The sign of the sample correlation coefficient is positive if the estimated regression equation has a positive slope and negative if the estimated regression equation has a negative slope.