Chapter4-Regression-ModelAdequacyChecking
Chapter 4: Model Adequacy Checking
Major Assumptions
The fitting of linear regression model and related processes are based on key assumptions:
The relationship between the study variable and explanatory variables is linear, at least approximately.
The error term has a zero mean.
The error term exhibits constant variance.
Errors are uncorrelated.
The errors follow a normal distribution.
Importance of Assumptions
Verifying these assumptions is crucial for obtaining meaningful results.
Violating these assumptions can lead to incorrect results and serious consequences.
Minor violations may lead to insignificant changes, while large departures can result in unstable models that draw opposite conclusions with different samples.
Residual diagnostics methods must be employed to check for these assumptions because summary statistics like t-statistics, F-statistics, or coefficient of determination do not provide this information.
Diagnostic Methods
Diagnostic methods to check the validity of regression assumptions involve analyzing model residuals through various graphical techniques.
Checking Linear Relationships
1. One Explanatory Variable
Scatter Diagrams: A scatter plot can be used to visually assess linearity between response (y) and predictor (X). If the scatter plot exhibits a linear trend, the relationship can be considered linear; if not, it may be nonlinear.
2. Multiple Explanatory Variables
Scatterplot Matrix: To assess linearity with multiple predictors, a scatterplot matrix provides a collection of scatter plots that illustrate pairwise relationships among variables.
Each cell contains a scatter plot, while the diagonal typically represents the variables themselves.
Provides more context than pairwise correlation coefficients, as it visually indicates the nature (linear/nonlinear) of the relationships among variables.
Residual Analysis
Definition: Residuals are the differences between observed and predicted values of the study variable and can be analyzed to detect model inadequacies.
Residuals are calculated as:
e_i = y_i - \hat{y}_i
Properties of Residuals:
Residuals should have zero mean and the average variance can be analyzed using:
\text{Var}(e_i) = \frac{\sum (e_i - \bar{e})^2}{n - k}
Residuals’ independence can be examined, keeping in mind degrees of freedom implications.
Residual Scaling Methods
Standardized Residuals: Scaled by subtracting the mean and dividing by the standard deviation. A large value (>3) indicates a potential outlier.
Studentized Residuals: These use the estimated variance of the residuals instead of the average variance, providing better insights into outlier detection.
Special Considerations:
Points with high leverage in x-space (indicating potential outliers influencing the model) should be examined carefully.
PRESS Residuals: Measures prediction accuracy by removing influence of single observations and provides valuable insight by calculating:
e(i) = y_i - \hat{y}(i)
Normal Probability Plots
Used to verify the assumption of normally distributed errors. Significant departures indicate potential issues with the model fit:
Patterns indicating different tails (thick or light) can inform about the robustness of OLS estimations.
Residual vs Fitted Plots
Visual inspections through plots of residuals against fitted values help in assessing model adequacies. Ideal characteristics of the plots include:
Random horizontal bands indicate no evident model defects.
Funnel shapes indicate non-constant variance.
Curved patterns suggest non-linearity or potential transformations needed on the explanatory variables.
Additional Diagnostic Tools
Partial Regression Plots and Partial Residual Plots are useful for examining marginal relationships when evaluating the effect of one explanatory variable in the context of others included in the model.
These plots can suggest the proper functional form (linearity vs. curvilinear) necessary for the model and indicate whether higher-order terms or transformations might be needed.
Outlier Detection and Treatment
Outliers can skew the regression and should be identified using:
Visual methods (residual plots) and scaled residuals (studentized and R-student).
Care must be taken when removing outliers, as they may represent significant data points. Proper justification is necessary to support removal.
Conclusion and Model Adequacy
A comprehensive approach involves constantly evaluating the underlying assumptions of regression analysis, utilizing residual analyses, graphical diagnostics, and sensitivity analyses to ensure the model is robust and reliable.