Linear Models Chaper 2 and 3
Chapter 2: Simple Linear Regression Analysis
2.1 Introduction to Simple Linear Regression
Model Formula: Y = β0 + β1X + ϵ, where ϵ ∼ N(0, σ²).
Goal: Understand and predict Y using X; quantify relationship strength and nature.
2.2 Parameter Estimation with Least Squares
Least Squares Method: Minimizes the residual sum of squares (RSS):
Formula: RSS = Σ (Yi − Ŷi)²
R Code:
model <- lm(Y ~ X, data = dataset) summary(model)
2.3 Hypothesis Testing and P-Values
P-Values: Measure evidence against the null hypothesis (H0: β = 0).
One-Sided: Tests if β is significantly positive or negative.
Two-Sided: Tests if β ≠ 0; more common in regression.
Formula: For t = β̂ / SE(β̂), with degrees of freedom n − 2.
Example R Code:
t_stat <- coef(summary(model))['X', 't value'] p_value <- coef(summary(model))['X', 'Pr(>|t|)']
2.4 Model Assumptions in Regression Analysis
Linearity: Mean response is a linear function of X.
Independence of Observations: No systematic relationship among residuals.
Normality of Error Term:
Definition: Residuals ϵi are normally distributed with mean 0.
Check in R:
qqnorm(residuals(model)); qqline(residuals(model)) Homoscedasticity (Constant Variance):
Definition: Variance of ϵ is constant across values of X.
Check in R:
plot(fitted(model), residuals(model)) 2.5 Binary Dummy Regressors
Purpose: Include categorical predictors by encoding levels as binary (0, 1) variables.
Formula Example: Y = β0 + β1 · Gendermale + ϵ (Gendermale = 1 for males, 0 for females).
Example R Code:
model <- lm(Y ~ Gender, data = dataset) summary(model)
2.6 Observational and Experimental Studies
Observational Study: No manipulation; observe variables as they naturally occur. Potential for confounding.
Experimental Study: Manipulation of predictors; stronger causal conclusions (e.g., randomized control trials).
2.7 Causal Effects
Sharp Causal Effect: Exact effect for each unit.
Average Causal Effect: Expected difference in outcomes if all units were treated vs. untreated.
Average Treatment Effect (ATE): ATE = E[Y(1) − Y(0)], where Y(1) and Y(0) are outcomes with and without treatment.
2.8 Randomized Studies and Causal Inference
Randomized Study: Random assignment to treatment/control groups, reducing confounding.
Conditional Mean Outcome: E[Y | X = x] is the expected outcome given X value.
2.9 Sampling Distribution
Definition: The distribution of a sample statistic (e.g., β̂) over repeated samples.
Example: Sampling distribution of β̂₁ is approximately normal for large samples.
2.10 Predictions and Prediction Intervals
Prediction: Using the model to estimate Y for new values of X.
Prediction Interval: Interval capturing future observations, wider than confidence intervals.
R Code Example:
predict(model, newdata = data.frame(X = 10), interval = "prediction")
Chapter 3: Multiple Linear Regression
3.1 Multiple Linear Regression Model
Model Formula: Y = β0 + β1X1 + β2X2 + ... + βpXp + ϵ
3.2 Model Assumptions (Extended)
Multicollinearity: Predictors shouldn’t be highly correlated (affects interpretability of β coefficients).
Variance Inflation Factor (VIF): Measurement of multicollinearity:
Formula: VIF(Xj) = 1/(1 − R²j) where R²j is from regressing Xj on other predictors.
R Code:
library(car) vif(model)
3.3 Analysis of Variance (ANOVA)
ANOVA Table: Decomposes total variance in Y into parts attributable to regression model (explained) and residuals (unexplained).
Sum of Squares:
SST = Σ(Yi − Ȳ)²: Total variance.
SSR = Σ(Ŷi − Ȳ)²: Explained variance.
SSE = Σ(Yi − Ŷi)²: Residual variance.
Coefficient of Determination (R²): R² = SSR/SST
3.4 Model Types
Additive Normal Linear Model: Linear effects of predictors without interactions.
Non-Additive Model: Includes interaction terms, allowing combined effects of predictors.
3.5 Interaction Effects and Effect Modification
Two-Way Interactions: Effect of X1 on Y depends on X2 (included as X1 × X2).
Example R Code:
model <- lm(Y ~ X1 * X2, data = dataset)
3.6 Leverage and Influence
Leverage: Indicates the influence of each observation on model parameters.
R Code:
hatvalues(model)
3.7 Model Diagnostics and Assumptions
Residual Diagnostics: Evaluate if assumptions hold.
Remedies: Transformations or adding/removing predictors.
Orthogonal Regressors: Predictors that are uncorrelated, simplifying interpretation.
3.8 Model Selection
Selection Methods:
Stepwise Regression: Adds/removes predictors based on significance.
AIC/BIC: Information criteria for model comparison.
Best Practices: Start with theoretically informed predictor set; use diagnostics to refine.
Suggested Approach to Exercises/Problems
Understand the Question: Identify primary goal (e.g., estimating coefficients, testing hypotheses).
Set Up the Model: Specify response and predictors, evaluate transformations or interactions.
Estimate the Model in R:
Fit with lm() and check summaries with summary() and diagnostic plots.
Interpret Coefficients: Examine intercept, slopes, and interaction terms.
Conduct Hypothesis Testing: Perform t-tests or ANOVA for significance.
Validate Assumptions:
Plot residuals, check for normality (Q-Q plots), and assess multicollinearity (VIF).
Refine the Model: Apply transformations, remove insignificant terms, try different selections if assumptions are violated.
Draw Conclusions: Answer posed question, interpreting results in context.