Multiple Regression Analysis Part II

If there is more than one independent variable available, multiple regression can be used.
Multiple regression lets us explore how several variables influence the dependent variable.

Examine the linear relationship between one dependent (Y) & 2 or more independent variables ( $X_i$ ).
Multiple Regression Model with k Independent Variables: $Y = \beta0 + \beta1X1 + \beta2X2 + \ldots + \betaKX_K + \varepsilon$
- Y-intercept
- Population slopes
- Random Error

The coefficients of the multiple regression model are estimated using sample data.
$\hat{yi} = b0 + b1x{1i} + b2x{2i} + \ldots + bKx{Ki}$
- $\hat{y_i}$ Estimated (or predicted) value of y
- $b_0$ Estimated intercept
- $b1, b2, …, b_K$ Estimated slope coefficients
Multiple regression equation with k independent variables:
A computer is typically used to obtain the regression slope coefficients and other regression summary measures.
The slopes $b1$ through $bk$ show “partial” influences.

Two variable model
- y, x1, x2
- $\hat{y} = b0 + b1x1 + b2x_2$
The best fit equation, $\hat{y}$ , is found by minimizing the sum of squared errors, $\sum e^2$

Three measures to judge how well the sample regression fits the data:
- The Standard Error of the Estimate
- The Coefficient of Determination: $R^2$ and adjusted $R^2$
- Tests of Significance: Individual Significance and Joint Significance

The square root of the MSE is the standard error of the estimate, $s_e$ .
In general, the less variation around the regression line, the smaller the $s_e$ , which implies a better fit to the model.
$se = \sqrt{MSE} = \sqrt{\frac{\sum{i=1}^n (yi - \hat{yi})^2}{n - k - 1}} = \sqrt{\frac{\sum{i=1}^n ei^2}{n - k - 1}}$

More explanatory variables always result in a higher $R^2$ .
Some of these variables may be unimportant and should not be in the model.
The Adjusted $R^2$ tries to balance the raw explanatory power against the desire to include only important predictors.

The Adjusted $R^2$ is computed as:
- Adjusted $R^2 = 1 - (1 - R^2) \frac{n - 1}{n - k - 1}$
The Adjusted $R^2$ penalizes the $R^2$ for adding additional independent variables.

Linear multiple regression model: $y = b0 + b1x1 + b2x2 + \ldots + bkx_k + e$
The most common hypothesis test for an individual slope coefficient is to test whether its value differs from zero.
If a coefficient is equal to zero, then it implies that the explanatory variable is not a significant predictor of the dependent variable:
- $H0: \betaj = 0$
- $H1: \betaj \neq 0$

The test statistic will follow a t-distribution with degrees of freedom df = n – k – 1. It is calculated as:
- $t = \frac{bj - 0}{S{bj}} = \frac{bj}{S{bj}}$
- $S{bj}$ is the standard error of the estimator $b_j$

In addition to conducting tests of individual significance, we also may want to test the joint significance of all k variables at once -> F-Test for Overall Significance of the Model
The competing hypotheses for a test of joint significance are:
- $H0: \beta1 = \beta2 = \ldots = \betaK = 0$
- $H1$ : at least one $\betai \neq 0$ (at least one independent variable affects Y)

The test statistic for a test of joint significance is
- $F = \frac{MSR}{MSE} = \frac{SSR / k}{SSE / (n - k - 1)}$ ,
- where MSR and MSE are, respectively, the mean square regression and the mean square error.
The numerator degrees of freedom is $df1 = k$ , while the denominator degrees of freedom is $df2 = n – k – 1$

These residual plots are used in multiple regression:
- Residuals vs. $\hat{y_i}$
- Residuals vs. $x_{1i}$
- Residuals vs. $x_{2i}$
- Residuals vs. time (if time series data)
Use the residual plots to check for violations of regression assumptions