Multiple Regression Notes

Multiple Regression

In this chapter, you learn:

• How to develop a multiple regression model
• How to interpret the regression coefficients
• How to determine which independent variables to include in the regression model
• How to determine which independent variables are most important in predicting a dependent variable
• How to use categorical independent variables in a regression model
• How to predict a categorical dependent variable using logistic regression
• How to identify individual observations that may be unduly influencing the multiple regression model

The Multiple Regression Model

• Idea: Examine the linear relationship between 1 dependent (Y) & 2 or more independent variables ($X_i$)

• Multiple Regression Model with k Independent Variables:

  $Y<em>i = \beta</em>0 + \beta<em>1X</em>{1i} + \beta<em>2X</em>{2i} + … + \beta<em>kX</em>{ki}$ (1)

• Multiple Regression Equation

• The coefficients of the multiple regression model are estimated using sample data

• Multiple regression equation with k independent variables:

  $\hat{Y}<em>i = b</em>0 + b<em>1X</em>{1i} + b<em>2X</em>{2i} + … + b<em>kX</em>{ki}$ (2)

  Where:

  • $\hat{Y}_i$ = Estimated or predicted value of Y
  • $b_0$ = Y intercept
  • $b<em>1, b</em>2, b_k$ = slope coefficients

Example: 2 Independent Variables

A distributor of frozen dessert pies wants to evaluate factors thought to influence demand

• Dependent variable:

  • Pie sales (units per week)

• Independent variables:

  • Price (in $)
  • Advertising ($100’s)

• Data are collected for 15 weeks

Excel Multiple Regression Output

The Multiple Regression Equation

$\hat{Sales} = 306.525 − 24.975(Price) + 74.131(Advertising)$ (3)

where:

• Sales is in number of pies per week
• Price is in $
• Advertising is in $100’s.
• $b_1 = −24.975$ : sales will decrease, on average, by 24.975 pies per week for each $1 increase in selling price, net of the effects of changes due to advertising.
• $b_2 = 74.131$: sales will increase, on average, by 74.131 pies per week for each $100 increase in advertising, net of the effects of changes due to price.

Using The Equation to Make Predictions

Predict sales for a week in which the selling price is $5.50 and advertising is $350:

$\hat{Sales} = 306.525 − 24.975(5.50) + 74.131(3.5) = 428.62$ (4)

Note that Advertising is in $100s, so $350 means that $X_2 = 3.5$

• Predicted sales is 428.62 pies

The Coefficient of Multiple Determination, $r^2$

Reports the proportion of total variation in Y explained by all X variables taken together.

$r^2 = \frac{SSR}{SST} = \frac{regression \space sum \space of \space squares}{total \space sum \space of \space squares}$ (5)

Adjusted $r^2$

• $r^2$ never decreases when a new X variable is added to the model

  • This can be a disadvantage when comparing models

• What is the net effect of adding a new variable?

  • We lose a degree of freedom when a new X variable is added
  • Did the new X variable add enough explanatory power to offset the loss of one degree of freedom?

• $r^2$ shows the proportion of variation in Y explained by all X variables adjusted for the number of X variables used

R^2_{adj} = 1 − [\frac{(1 − r^2)(\frac{n − 1}{n − k − 1})] (6)

(where n = sample size, k = number of independent variables)

• Penalizes excessive use of unimportant independent variables
• Smaller than $r^2$
• Useful in comparing among models

Using Dummy Variables

• A dummy variable is a categorical independent variable with two levels:

  • yes or no, on or off, male or female
  • coded as 0 or 1

• Assumes the slopes associated with numerical independent variables do not change with the value for the categorical variable

• If more than two levels, the number of dummy variables needed is (number of levels - 1)

Dummy-Variable Example (with 2 Levels)

$\hat{Y} = b<em>0 + b</em>1X<em>1 + b</em>2X_2$ (7)

Let:

• Y = pie sales

• $X_1$ = price

• $X<em>2$ = holiday ($X</em>2$ = 1 if a holiday occurred during the week)

  ($X_2$ = 0 if there was no holiday that week)

• No Holiday

  $\hat{Y} = b<em>0 + b</em>1X<em>1 + b</em>2(0) = b<em>0 + b</em>1X_1$ (8)

• Holiday

  $\hat{Y} = b<em>0 + b</em>1X<em>1 + b</em>2(1) = b<em>0 + b</em>2 + b<em>1X</em>1$ (9)

Interpreting the Dummy Variable Coefficient (with 2 Levels)

Example:

$\hat{Sales} = 300 − 30(Price) + 15(Holiday)$ (10)

• Sales: number of pies sold per week

• Price: pie price in $

• Holiday: 1 If a holiday occurred during the week

  0 If no holiday occurred

• $b_2$ = 15 on average, sales were 15 pies greater in weeks with a holiday than in weeks without a holiday, given the same price

Dummy-Variable Models (more than 2 Levels)

The number of dummy variables is one less than the number of levels

Example:

• Y = house price ;

• $X_1$ = square feet

• If style of the house is also thought to matter:

  • Style = ranch, split level, colonial

• Three levels, so two dummy variables are needed.

Example:

Let ‘‘colonial’’ be the default category, and let $X<em>2$ and $X</em>3$ be used for the other two categories:

• Y = house price

• $X_1$ = square feet

• $X_2$ = 1 if ranch, 0 otherwise

• $X_3$ = 1 if split level, 0 otherwise

• The multiple regression equation is:

  $\hat{Y} = b<em>0 + b</em>1X<em>1 + b</em>2X<em>2 + b</em>3X_3$ (11)

Table 2: Housing Data

Table 3: Base Data

Table 4: Colonial as Default Data

Interpreting the Dummy Variable Coefficients (with 3 Levels)

Consider the regression equation:

$\hat{Y} = 98.24 + 0.11(X<em>1) + 50(X</em>2) + 100(X_3)$ (12)

• For a colonial: $X<em>2 = X</em>3 = 0$

  $\hat{Y} = 98.24 + 0.11(X_1)$ (13)

• For a ranch: $X<em>2 = 1; X</em>3 = 0$

  $\hat{Y} = 98.24 + 0.11(X_1) + 50.00$ (14)

  With the same square feet, a ranch will have an estimated average price of 50 thousand dollars more than a colonial.

• For a split level: $X<em>2 = 0; X</em>3 = 1$

  $\hat{Y} = 98.24 + 0.11(X_1) + 100$ (15)

  With the same square feet, a split-level will have an estimated average price of 100 thousand dollars more than a colonial.

Logistic Regression

• Used when the dependent variable Y is binary (i.e., Y takes on only two values)

• Examples

  • Customer prefers Brand A or Brand B
  • Employee chooses to work full-time or part-time
  • Loan is delinquent or is not delinquent
  • Person voted in last election or did not

• Logistic regression allows you to predict the probability of a particular categorical response

• Logistic regression is based on the odds ratio, which represents the probability of an event of interest compared with the probability of not an event of interest

$Odds \space Ratio = \frac{probability \space of \space an \space event \space of \space interest}{1 − probability \space of \space an \space event \space of \space interest}$ (16)

• The logistic regression model is based on the natural log of this odds ratio