L3: The multiple regression model

3.1 Multiple Regression Model

• A dependent variable is influenced by multiple explanatory variables (X1, X2, X3, …)

• Example scenarios:

  • Earnings: Determined not only by education but also by experience.

  • Demand: Influenced by price, income, price of substitutes, etc.

  • Output: Relies on various inputs of production and other environmental factors.

Mathematical Representation

• The general formula for the multiple regression model: $Y<em>i = \beta</em>1 + \beta<em>2 X</em>{1i} + \beta<em>3 X</em>{2i} + u_i$

• Where:

  • $\beta_1$ = intercept

  • $\beta<em>2, \beta</em>3$ = partial regression coefficients

  • The coefficients represent the marginal effects on Y of each X variable. The term "partial" is key here: it means we are looking at the effect of one X variable while carefully "holding all other X variables constant" (this is known as ceteris paribus).

  • This isolation allows us to understand the unique contribution of each independent variable to the dependent variable, preventing confusion from intertwined effects.

• If both $X<em>1$ and $X</em>2$ change, then $\delta Y = \beta<em>2 \cdot \delta X</em>1 + \beta<em>3 \cdot \delta X</em>2$

Practical Examples

• Example 1: Family consumption as a function of income and number of children:

  • $C<em>i = \beta</em>1 + \beta<em>2 IN</em>i + \beta<em>3 CH</em>i + u_i$

  • Interpretation of coefficients:

  • $\beta<em>2$ = effect on family consumption of an additional pound of income (e.g., $\beta</em>2 = 0.8$ means an extra £1 increases weekly consumption by 80 pence).

  • $\beta<em>3$ = effect on family consumption of an additional child (e.g., $\beta</em>3 = 12.5$ gives an increase of £12.5).

  • Example: If $C<em>i = 45.98 + 0.8 IN</em>i + 12.5 CH_i$

• Example 2: Manufacturing output as a function of labor and capital inputs:

  • Log-linear function: $Y = 2.89 + 0.37 \ln L<em>i + 0.52 \ln K</em>i$

  • Coefficients represent partial elasticities:

  • A 1% increase in labor input yields a 0.37% increase in output.

  • A 1% increase in capital input yields a 0.52% increase in output.

  • Total returns to scale (TRS):

  • $TRS = 0.37 + 0.52 = 0.89$ (indicates decreasing returns to scale since TRS < 1).

• Example 3: Production function with three inputs:
  

  $Y = 3.44 + 0.32 \ln X<em>1 + 0.41 \ln X</em>2 + 0.38 \ln X*3$

  • Coefficient interpretation:

  • A 1% increase in $X*1$ increases output by 0.32%.
    

  • A 1% increase in $X*2$ increases output by 0.41%.

  • A 1% increase in $X*3$ increases output by 0.38%.
    

  • Total returns to scale:
    

  • $TRS = 0.32 + 0.41 + 0.38 = 1.11$ (indicates increasing returns to scale).

Estimation and Goodness of Fit

The Core Idea: Finding the Best Fit

• The goal of multiple regression is to estimate the relationship between a dependent variable (Y) and multiple independent variables ($X<em>1, X</em>2, \dots$).

• We aim to find a line, or more accurately, a hyperplane (a multi-dimensional line), that best represents the trend in our data. This "best fit" helps us predict Y based on the X variables.

Ordinary Least Squares (OLS) Theory

• The most common method to find this "best-fit" line is called Ordinary Least Squares (OLS).

• What is a "residual" (or error term)? It's the difference between the actual observed value of Y for a data point and the value of Y predicted by our regression model. It tells us how far off our model's prediction is from the reality for each point ($u<em>i = Y</em>i - \hat{Y_i}$).

• Why "Least Squares"? OLS works by choosing the regression coefficients ($\beta$s) that minimize the sum of the squared residuals (RSS).

  • We square the residuals ($u_i^2$) to ensure that positive and negative errors don't cancel each other out, and to penalize larger errors more heavily.

  • Minimizing this sum means finding the line that, on average, is closest to all the data points, ensuring the best possible fit.

• The coefficients calculated through OLS are called OLS estimators ($\hat{\beta}$s). These estimators are unbiased and consistent under certain assumptions, meaning they provide reliable estimates of the true population parameters.

• If error terms are distributed normally, the estimates will also be normally distributed:

  • $\hat{\beta<em>1} \sim N(\beta</em>1, \sigma^2_1)$,

  • $\hat{\beta<em>2} \sim N(\beta</em>2, \sigma^2_2)$,

  • $\hat{\beta<em>3} \sim N(\beta</em>3, \sigma^2_3)$.

• The goodness of fit is measured using the coefficient of determination, $R^2$:

  • Total Sum of Squares (TSS) = Explained Sum of Squares (ESS) + Residual Sum of Squares (RSS)

  • $R^2 = \frac{ESS}{TSS}$

  • This indicates the proportion of total variation in Y explained by the model.

Hypothesis Testing: One Parameter

• Example: Earnings as a function of age and education:
  

  $W<em>i = \beta</em>1 + \beta<em>2 A</em>i + \beta<em>3 E</em>i + u_i$

  • Where:

  • $W_i$ = yearly salary

  • $A_i$ = age

  • $E_i$ = education years

• Testing the significance of coefficients:

  • Null hypothesis for age: $H<em>0: \beta</em>2 = 0$ vs alternate H1: \beta2 > 0.

  • Critical value from the t-distribution used with degrees of freedom $n - k$; in this case, $n = 73$ and $k = 3$.

• Conducting the test:

  • If the calculated t-statistic exceeds critical value, reject $H_0$.

  • In this case:

  • $F^*$ value calculated and compared against critical $F$ value from the F-distribution for multiple parameters.

  • Example Calculation: If $F^* = 2.5$ from $n = 34, RSS = 73$, and ESS = 176.66, can infer if the parameters are significant collectively.

Conclusion and Implications

• For the regression model to be statistically significant, at least one variable must provide explanatory power.

• When conducting tests involving marginal factors, significance provides insight into the relationship between the dependent variable and selected independent variables, aiding in decision-making and comprehension of underlying patterns.