LEC 5 Multivariate Regression: Concepts and Application

PUBPOL 5310: Applied Stats - Multivariate Regression

Overview of Regression Analysis

First Part: Focused on univariate and bivariate analysis (one or two variables).
Remainder (Current Topic): Focuses on multivariate analysis.
- Multivariate Data: Involves three or more variables.
- (Earlier material can be considered a special case of multivariate analysis).

Outline and Goals

Goal: To achieve "Controlling for other variables" in regression analysis.
- This means isolating the effect of one independent variable on the dependent variable while holding other relevant factors constant.
Notation: Understanding the symbols used for multivariate data analysis.
Summary of Multivariate Data:
- Graphical summary (with limitations).
- Least squares regression method for multivariate data.
- Interpretation of multivariate regression coefficients.
- Measures of goodness of fit.

"Controlling for Other Variables"

Bivariate Regression Slope ( $b_2$ ): Indicates the change in predicted Y for a one-unit change in X, on average.
- Example: Relationship between health and income.
Confounding Factors: Often, the observed change in Y due to X also includes effects from other factors (e.g., age).
- Example: Age might influence both income and health, making a simple health-income relationship misleading.
Desired Outcome: To understand the relationship between income and health for individuals of the same age (i.e., "for two folks of the same age, how are income and health related?").
Terminology for Control:
- "Control for age."
- "Hold age constant."
- "Hold age fixed."
- "Condition on age."
One Approach (Stata's if modifier):
- Using regress health income if age == 47 to analyze data only for individuals aged 47.
- Using regress health income if (age >= 45 & age <= 49) to analyze data within an age range.
- Limitations of the if approach:
 - Difficulty in choosing specific ages or age ranges.
 - Risk of insufficient data within narrow subsets.
 - Potentially many regressions to run for different age groups.
 - Hard to get an "overall answer" across all ages.
Multivariate Regression as an Alternative: Provides a statistical model to control for age (or other variables) without subsetting the data.

Motivation #2: Optimal Predictions

Utilizing several predictor variables (independent variables) to make the best possible predictions for the dependent variable (Y).

Notation for Multivariate Regression

Variables: Y, and $X2 ext{, } ext{…} XK$
- $X2 ext{, } ext{…} XK$ : Covariates, explanatory variables, independent variables, Right-Hand-Side (RHS) variables, or regressors.
- Y: Dependent variable, outcome, regressand, or Left-Hand-Side (LHS) variable.
$K$ : Represents the total number of variables being analyzed, where $K ext{ >= } 2$ .
- This includes $1$ Y variable and $(K-1)$ X variables.
- $K$ also signifies the total number of regression parameters (coefficients).
  - This includes $1$ intercept (constant term) and $(K-1)$ "slopes" (coefficients for the X variables).

Graphical Description of Multivariate Data

Difficulty: Generally, it is challenging to visualize multivariate data graphically due to its high dimensionality.
One Exception: When there are three variables and one of the X variables is a dummy variable.
"Best practice" examples:
- Edward Tufte: Known for "The Visual Display of Quantitative Information," emphasizing clarity and efficiency in data graphics (e.g., Minard's map of Napoleon's Russian campaign).
- Hans Rosling (animated): Famous for using animated bubble charts to visualize complex socioeconomic data (e.g., Gapminder videos).

The Best-Fit "Line" with Multivariate Data: Definitions and Interpretation

Multivariate Best-Fit "Line" (Equation):
- $\hat{y}i = b1 + b2x{2i} + b3x{3i} + \text{…} + bKx{Ki}$
- $\hat{y}_i$ : The predicted value for Y for individual $i$ .
- $b_1$ : The intercept (predicted Y when all X variables are zero).
- $bk$ (for $k = 2, \text{…}, K$ ): Represents the change in predicted Y for a one-unit increase in $Xk$ , holding constant all other X variables. This is akin to a partial derivative: $\frac{\partial \hat{y}}{\partial xk} = bk$ . (The character in the slide \text{\raise.17em\hbox{\$ }} appears to be a typo for $\partial$ ).
The Fundamental Difference in Interpretation: The key phrase is "holding the other X's constant."

Example: Wage, Education, and Gender

Set-Up:
- Y: Hourly wages (WAGE)
- $X_2$ : Education in years (EDUC)
- $X_3$ : Dummy variable for female (FEMALE, where $1=\text{female}, 0=\text{male})$ .
Best-Fit Line: $\hat{wage}i = b1 + b2educi + b3femalei$
Interpretation of Coefficients:
- $\hat{wage}_i$ : The predicted wage for person $i$ .
- $educi$ and $femalei$ : The specific values of education and female status for person $i$ .
- $b_1$ : The predicted wage of someone with $EDUC=0$ and $FEMALE=0$ (i.e., a male with zero years of education).
- $b_2$ : The average change in WAGE for a one-year increase in EDUC, holding constant FEMALE (i.e., for both males and females, separately).
- $b_3$ : The average change in WAGE when FEMALE changes from $0$ to $1$ (i.e., for females compared to males), holding constant EDUC (i.e., for individuals with the same level of education).
Graphical Interpretation (with FEMALE as a dummy variable):
- Since FEMALE is a dummy, the model generates two parallel lines (assuming no interaction term):
 - For males (FEMALE = 0): $\hat{wage}i = b1 + b2educi$
 - For females (FEMALE = 1): $\hat{wage}i = b1 + b2educi + b3(1) = (b1 + b3) + b2educ_i$
- Visualizing the effect of $b2$ : Holding FEMALE constant (staying on one line), an increase of one unit in EDUC (e.g., from $educ0$ to $educ0 + 1$ ) results in a $b2$ unit change in wage.
- Visualizing the effect of $b3$ : Holding EDUC constant (moving vertically at a given $educ0$ ), changing from FEMALE=0 to FEMALE=1 results in a $b3$ unit change in wage. If $b3 < 0$ , the female line would be below the male line, indicating lower wages for females with the same education.

Least Squares Regression with Multivariate Data

Choosing the "Line": Involves selecting the optimal values for all coefficients: $b1, b2, \text{…}, b_K$ .
Objective: To find $b1, b2, \text{…}, bK$ such that the average observed Y is as close as possible to its prediction ( $\hat{y}i$ ).
Minimization: Minimize the average squared deviation of Y from $\hat{y}$ (the predicted value).
- Deviation (Residual): $ei = yi - \hat{y}_i$
- Squared Deviation: $e_i^2$
- Average Squared Deviation (Mean Squared Error - MSE): $\frac{1}{n} \sum{i=1}^{n} ei^2$
The Problem (Sum of Squared Residuals - SSR): Minimize $\sum{i=1}^{n} (yi - \hat{y}i)^2 = \sum{i=1}^{n} (yi - (b1 + b2x{2i} + \text{…} + bKx{Ki}))^2$
Solving with Calculus (First-Order Conditions): To find the minimum, we take the partial derivative of the SSR with respect to each coefficient ( $b1, b2, \text{…}, b_K$ ) and set them to zero. This results in a system of K linear equations.
- $\frac{\partial}{\partial b1} \left( \frac{1}{n} \sum{i=1}^{n} (yi - b1 - b2x{2i} - \text{…} - bKx{Ki})^2 \right) = 0$
- $\frac{\partial}{\partial b2} \left( \frac{1}{n} \sum{i=1}^{n} (yi - b1 - b2x{2i} - \text{…} - bKx{Ki})^2 \right) = 0$
- … (and so on for all $bk$ terms up to $bK$ )
The Solution: Similar to the bivariate case, there are solution formulas for $b1, \text{…}, bK$ , which are complex functions of the data. In practice, statistical software like Stata is used to compute these coefficients.

Measures of Goodness of Fit

1. Standard Error of the Regression (SER)

Definition: Measures the magnitude of the typical deviation of the observed Y from the predicted Y ( $\hat{y}$ ).
Calculation: Similar to a standard deviation.
- $se^2 = \frac{1}{n-K} \sum{i=1}^{n} ei^2 = \frac{1}{n-K} \sum{i=1}^{n} (yi - \hat{y}i)^2$
- The Standard Error of the Regression is the square root: $s_e$
 - (Note: $n-K$ in the denominator accounts for the degrees of freedom lost in estimating K coefficients).
In Stata Output: Labeled as "Root MSE."
Interpretation:
- Sensitive to the unit of measurement of Y, so it's best interpreted by comparing it across similar regressions.
- A lower Standard Error of the Regression indicates a better fit of the regression model to the data.

2. R-squared ( $R^2$ )

Definition: The fraction of the total variation in Y that is "explained by" the variation in the X variables included in the model.
Formal Formulas:
- ESS (Explained Sum of Squares): $\text{ESS} = \sum{i=1}^{n} (\hat{y}i - \bar{y})^2$
- TSS (Total Sum of Squares): $\text{TSS} = \sum{i=1}^{n} (yi - \bar{y})^2$
- SSR (Sum of Squared Residuals): $\text{SSR} = \sum{i=1}^{n} (yi - \hat{y}_i)^2$
- $R^2 = \frac{\text{ESS}}{\text{TSS}} = 1 - \frac{\text{SSR}}{\text{TSS}}$
In Excel Output: Labeled as "R square" in the "Regression Statistics" table.
Interpretation:
- $R^2 \text{ = } 1$ : A perfect fit; all data points lie exactly on the regression line/plane.
- $0 < R^2 < 1$ : The usual case. A higher $R^2$ indicates a better fit of the regression model.
- $R^2 \text{ = } 0$ : No fit; the model explains none of the variation in Y.
Problem with $R^2$ :
- Adding more explanatory variables (X's) to the model will always increase $R^2$ or leave it unchanged.
- This happens because TSS remains constant, and ESS will either stay the same or go up (SSR will either stay the same or go down).
- Therefore, an increase in $R^2$ might not truly represent an improvement in the model's predictive power or fit, but merely an increase in the number of regressors ( $K$ ).

3. Adjusted R-squared ( $R^2_ ext{adj}$ )

Purpose: An alternative goodness-of-fit measure that "penalizes" models for including too many covariates, addressing the problem of $R^2$ artificially increasing with more variables.
Formula: $\bar{R}^2 = 1 - \frac{(1 - R^2)(n-1)}{n-K-1}$ (The slide presented this as $\bar{R}^2 = 1 - \frac{(1 - R^2)(n-1)}{n-K}$ which is a common variant or simplification, assuming $K-1$ slopes. Using the slide's initial notation of $K$ total parameters: $\bar{R}^2 = 1 - (1-R^2) \frac{n-1}{n-K}$ where $K$ is number of total parameters)
- More precisely from the slide: $\bar{R}^2 = 1 - \left( \frac{n-1}{n-K} \right) (1-R^2)$
Comparison to $R^2$ : Generally, $R^2_ ext{adj} < R^2$ .
Difference between measures is larger when:
- $K$ (the number of parameters/variables) is larger.
- $n$ (the sample size) is smaller.
Behavior: Unlike $R^2$ , Adjusted $R^2$ can either increase or decrease when a new variable is added to the model. It will only increase if the new variable significantly improves the model's fit, outweighing the penalty for adding a new parameter.

Dummy Variables and Multivariate Regression

Recap (Bivariate Dummy Model):
- Model: $Y = b1 + b2D + \epsilon$
- $Y$ : Wages; $D$ : Female (dummy, $1=\text{female}, 0=\text{male})$ .
- Interpretation: $b2$ is the expected change in hourly wages for a respondent who is female, compared to males ( $b2 = E[Y|D=1] - E[Y|D=0]$ ).
Categorical Variables with Many Categories:
- To use a categorical variable (e.g., region, occupation type) with multiple categories in multivariate regression, it must be recoded as a set of dummy variables.
- Create one dummy variable for each category.
- Rule: Include $( \text{number of categories} - 1)$ dummy variables in the regression model.
- Omitted Group: The category for which no dummy variable is explicitly included becomes the "reference group."
- Interpretation: The coefficients for the included dummy variables represent the shift in the intercept relative to the omitted (reference) group, holding other variables constant.
Revisiting the Example: Wage, Education, Female:
- $\hat{wage}i = b1 + b2educi + b3femalei$
- If FEMALE=0 (omitted group for males), then predicted wage is $b1 + b2educ_i$ .
- If FEMALE=1 (included dummy for females), then predicted wage is $(b1 + b3) + b2educi$ .

Dummy Variable Trap

Problem: If you include a dummy variable for every category of a categorical variable (e.g., both male and female dummies, plus an intercept), perfect multicollinearity will occur. This makes the model inestimable.
- Suppose there are $M$ categories for a categorical variable.
Solution: To ensure the model is estimable, always use $(M-1)$ dummy variables.
- The omitted group serves as the baseline, and its line is the one to which the lines for all other groups are compared.
- For the omitted group, all included dummy variables will take a value of zero, making its intercept equivalent to $b_1$ .
- For an included dummy, say for group $Xk$ compared to the omitted group, its line would be $(b1 + bk) + b{\text{other variables}}X$ (if there were other variables). Otherwise, if just using dummy and intercept $(b1 + b3X{dummy})$ . For instance, if original equation $Y = b1 + b3D$ , for the omitted group it's $b1$ , and for the dummy group it's $b1+b3$ . ( $b1 + b3X$ as shown in the slide with $X$ as the dummy would be the intercept for the dummy group).

LEC 5 Multivariate Regression: Concepts and Application

PUBPOL 5310: Applied Stats - Multivariate Regression

Overview of Regression Analysis

Outline and Goals

"Controlling for Other Variables"

Motivation #2: Optimal Predictions

Notation for Multivariate Regression

Graphical Description of Multivariate Data

The Best-Fit "Line" with Multivariate Data: Definitions and Interpretation

Example: Wage, Education, and Gender

Least Squares Regression with Multivariate Data

Measures of Goodness of Fit

1. Standard Error of the Regression (SER)

2. R-squared (R2R^2R2)

3. Adjusted R-squared (Re2xtadjR^2_ ext{adj}Re2​xtadj)

Dummy Variables and Multivariate Regression

Dummy Variable Trap

2. R-squared ( $R^2$ )

3. Adjusted R-squared ( $R^2_ ext{adj}$ )