Week 12 - Multiple Regression Analysis: From Single Predictors to Multi-Variable Models

Learning Objectives and Context for Multiple Regression

• Purpose of Multiple Linear Regression: Multiple linear regression is an extension of simple linear regression used to explain or predict a numerical target (dependent variable) using two or more numerical predictors (independent variables).
• Key Learning Goals:
  • Understand the context and purpose behind the transition from one predictor to many.
  • Master the step-by-step procedure for building a regression model, specifically using the backward elimination method.
  • Learn to interpret regression outputs in software tools like Excel or jamovi to generate actionable predictions.
• Value Proposition of Multiple Regression:
  • Improved Accuracy: Combining multiple predictors typically results in a more precise model than using single predictors in isolation.
  • Isolation of Effects: It allows the analyst to isolate the specific effect of a single predictor on the dependent variable ($Y$) while holding other variables constant.
  • Predictor Significance: It identifies which predictors contribute most significantly to the outcome.

Correlation Tables and the Challenge of Multicollinearity

• The Correlation Matrix: This tool is used to identify relationships between the target variable (e.g., Salary) and various independent variables. For example, if ExperienceYears has a correlation of $0.520$ with Salary, it suggests a clear positive relationship where more experience leads to higher pay.
• Understanding Multicollinearity:
  • Definition: Multicollinearity occurs when two or more independent variables in a regression model are highly related to each other, meaning they provide overlapping information or are too similar.
  • Threshold of Concern: A correlation coefficient where $|r| \ge 0.70$ is generally considered a possible concern for multicollinearity.
  • Impact on Models: Extremely high correlations (e.g., $r \ge 0.80$) can make the regression model unstable, leading to results that are difficult to interpret accurately.

Mathematical Framework of Multiple Regression

• General Regression Model: The relationship is theoretically expressed as: $Y = \beta_0 + \beta_1X_1 + \beta_2X_2 + \dots + \beta_kX_k + \epsilon$
• Regression Equation (Estimated): The equation used for calculation is: $\hat{Y} = b_0 + b_1X_1 + b_2X_2 + \dots + b_kX_k$
• Practical Example (Bookings):$\text{Bookings} = b_0 + b_1(\text{Price}) + b_2(\text{Amenities}) + b_3(\text{Location})$
  • Dependent Variable ($Y$): Number of bookings.
  • $X_1$: Room price (measured in $CHF$).
  • $X_2$: Amenities score (scale of $0-10$).
  • $X_3$: Location rating (scale of $1-5$).

Step-by-Step Procedure for Building a Model

1. Prepare and Explore the Data:
   • Develop a theoretical model by selecting relevant dependent variables.
   • Gather data and conduct descriptive analysis (frequency tables, location/variability indicators, and graphs).
2. Examine Relationships and Multicollinearity:
   • Compute the correlation matrix.
   • Rank variables based on relationship strength.
   • Check scatterplots to visualize potential multicollinearity.
3. Run the Initial Multiple Regression:
   • Include all potential predictors in the first run.
   • Check the Adjusted $R^2$, F-test, and t-tests.
   • Identify and remove outliers if they are present.
4. Refine the Model (Backward Elimination):
   • Identify the predictor with the highest non-significant p-value (where $p > \alpha$).
   • Remove that predictor and rerun the regression.
   • Repeat this process until all remaining predictors are statistically significant.
5. Interpret the Model:
   • Write the final regression equation.
   • Interpret the meaning of individual coefficients ($b_i$).
6. Forecast Future Values:
   • Input new data into the best-fit model to predict the dependent variable ($Y$).
7. Summarize the Analysis:
   • Highlight key findings, model fit, and specific insights.

Statistical Significance: F-Tests and t-Tests

The F-Test (Overall Model Significance)

• Purpose: Determines if the model works as a whole and if the relationship between $Y$ and the set of $X$ variables is statistically significant.
• Null Hypothesis ($H_0$): $\beta_1 = \beta_2 = \dots = \beta_k = 0$ (The model has no predictive power).
• Alternative Hypothesis ($H_1$): At least one $\beta_i \neq 0$.
• Decision Rule: Accept $H_0$ if the p-value (Significance F) is greater than the significance level ($SL$). There is relationship significance when the Significance F ($p\text{-value}$) is less than $\alpha$ (typically $0.05$).

The t-Test (Individual Predictor Significance)

• Purpose: Determines if a specific predictor provides unique value to the model.
• Null Hypothesis ($H_0$): $\beta_i = 0$ (The specific predictor is not significant).
• Alternative Hypothesis ($H_1$): $\beta_i \neq 0$.
• Decision Rule: Accept $H_0$ if the p-value is greater than the significance level ($SL$). A significant relationship exists for that specific variable if the $p\text{-value} < \alpha$.

Measuring Model Variation: R-Squared and Adjusted R-Squared

• Coefficient of Determination ($R^2$): Measures the proportion of the total change in the dependent variable explained by the regression model.
  • Example: If $R^2 = 0.78$ when predicting wages, $78\%$ of the variation in wages is explained by the model's independent variables, while the remaining $22\%$ is unexplained error (residuals).
  • Limitation: $R^2$ always increases or stays the same when new predictors are added, regardless of their actual utility.
• Adjusted $R^2$: This metric penalizes the inclusion of unnecessary predictors. It considers the number of observations ($n$) and the number of independent variables ($k$).
  • Behavior: It only increases if a new predictor improves the model more than would be expected by sheer chance. It is the preferred metric for assessing model quality in multiple regression.

Practical Interpretation of Regression Output

• Example Scenario: Predicting a house starting price based on age, plot size, and living area.
• Model Significance: The F-statistic p-value is $0.0$. Since $0.0 < 0.05$, the model is overall statistically significant (the relationship did not happen by accident).
• Model Fit: The Adjusted $R^2$ is $0.996$. This means approximately $99.6\%$ of the variation in the starting salary/price is explained by the model.
• Variable Assessment:
  • Living Area: Statistically significant ($p < 0.05$). Coefficient = $7,210$. This means each square meter of living area is associated with a price increase of $7,210\,CHF$, holding other variables constant.
  • Plot Size: Statistically significant ($p < 0.05$). Coefficient = $481$. The price per square meter of plot size is $481\,CHF$.
  • Age: Not statistically significant. Its p-value is $93.4\%$. Because $0.934 > 0.05$, the Age variable should be removed from the model during the refinement stage.