Notes on Multiple Regression Analysis

Multiple Regression Analysis

• Definition of Multiple Regression Analysis: Multiple regression analysis incorporates multiple independent (explanatory) variables to understand their joint effect on a dependent variable.

Key Points

• Sample Regression Coefficients:
    - Sample regression coefficients should not be used to estimate population parameters. This emphasizes the limitations of inference when using sample statistics directly to generalize beyond the sample data.

• Coefficient of Multiple Determination:
    - This statistic is denoted by $R^2$ (also referred to as $ry.12$ when there are two explanatory variables).
    - It measures the proportion of variability in the dependent variable that can be attributed to the independent variables in the model.

• Impact of Additional Independent Variables:
    - Adding independent variables to the regression model will always provide new information, regardless of whether they fundamentally alter the interpretation of the model. However, this will usually increase the value of $R^2$, improving the overall explanatory power of the model.

• F Test in Multiple Regression:
    - The F test is conducted to determine if all regression coefficients in the model are significantly different from zero.
    - This test evaluates whether at least one explanatory variable has a non-zero coefficient, thereby indicating its potential influence on the dependent variable.

• Partial F Criterion:
    - The contribution of an independent variable can be assessed using the partial F criterion. This method elucidates how much a particular independent variable adds to the predictive power of the model after accounting for the other independent variables.

• Relationship Between t Statistic and F Statistic:
    - The relationship is defined mathematically as $t^2_v = F_{1,v}$, where $v$ represents the number of degrees of freedom associated with the respective tests. This shows how the t-test for individual coefficients relates to the overall F-test used in multiple regression.

• Proportion of Variation Explained by Independent Variables:
    - In multiple regression, the coefficient of multiple determination ($R^2$) quantifies the proportion of variation in the dependent variable $y$ that is explained by the joint effects of all the independent variables included in the model.

• Quadratic Regression Model:
    - The F statistic is not applicable when assessing the statistical significance of an overall quadratic regression model. This distinction is crucial for appropriate statistical analysis, highlighting the limitations of the F-test in evaluating quadratic relationships.

• Correlation Among Independent Variables:
    - When two independent variables are correlated, this correlation can be problematic. It may not provide any incremental new information; hence the assessment of their separate effects on the dependent variable becomes challenging. This situation can lead to multicollinearity, which can inflate the variances of the coefficient estimates and make them unstable.