Linear Models Chaper 2 and 3

Chapter 2: Simple Linear Regression Analysis

2.1 Introduction to Simple Linear Regression

  • Model Formula: Y = β0 + β1X + ϵ, where ϵ ∼ N(0, σ²).

    • Goal: Understand and predict Y using X; quantify relationship strength and nature.

2.2 Parameter Estimation with Least Squares

  • Least Squares Method: Minimizes the residual sum of squares (RSS):

    • Formula: RSS = Σ (Yi − Ŷi)²

  • R Code:

    model <- lm(Y ~ X, data = dataset)  
    summary(model)  

2.3 Hypothesis Testing and P-Values

  • P-Values: Measure evidence against the null hypothesis (H0: β = 0).

    • One-Sided: Tests if β is significantly positive or negative.

    • Two-Sided: Tests if β ≠ 0; more common in regression.

    • Formula: For t = β̂ / SE(β̂), with degrees of freedom n − 2.

  • Example R Code:

    t_stat <- coef(summary(model))['X', 't value']  
    p_value <- coef(summary(model))['X', 'Pr(>|t|)']  

2.4 Model Assumptions in Regression Analysis

  1. Linearity: Mean response is a linear function of X.

  2. Independence of Observations: No systematic relationship among residuals.

  3. Normality of Error Term:

  • Definition: Residuals ϵi are normally distributed with mean 0.

  • Check in R:

qqnorm(residuals(model)); qqline(residuals(model))  
  1. Homoscedasticity (Constant Variance):

  • Definition: Variance of ϵ is constant across values of X.

  • Check in R:

plot(fitted(model), residuals(model))  

2.5 Binary Dummy Regressors

  • Purpose: Include categorical predictors by encoding levels as binary (0, 1) variables.

  • Formula Example: Y = β0 + β1 · Gendermale + ϵ (Gendermale = 1 for males, 0 for females).

  • Example R Code:

    model <- lm(Y ~ Gender, data = dataset)  
    summary(model)  

2.6 Observational and Experimental Studies

  • Observational Study: No manipulation; observe variables as they naturally occur. Potential for confounding.

  • Experimental Study: Manipulation of predictors; stronger causal conclusions (e.g., randomized control trials).

2.7 Causal Effects

  • Sharp Causal Effect: Exact effect for each unit.

  • Average Causal Effect: Expected difference in outcomes if all units were treated vs. untreated.

  • Average Treatment Effect (ATE): ATE = E[Y(1) − Y(0)], where Y(1) and Y(0) are outcomes with and without treatment.

2.8 Randomized Studies and Causal Inference

  • Randomized Study: Random assignment to treatment/control groups, reducing confounding.

  • Conditional Mean Outcome: E[Y | X = x] is the expected outcome given X value.

2.9 Sampling Distribution

  • Definition: The distribution of a sample statistic (e.g., β̂) over repeated samples.

  • Example: Sampling distribution of β̂₁ is approximately normal for large samples.

2.10 Predictions and Prediction Intervals

  • Prediction: Using the model to estimate Y for new values of X.

  • Prediction Interval: Interval capturing future observations, wider than confidence intervals.

  • R Code Example:

    predict(model, newdata = data.frame(X = 10), interval = "prediction")  

Chapter 3: Multiple Linear Regression

3.1 Multiple Linear Regression Model

  • Model Formula: Y = β0 + β1X1 + β2X2 + ... + βpXp + ϵ

3.2 Model Assumptions (Extended)

  • Multicollinearity: Predictors shouldn’t be highly correlated (affects interpretability of β coefficients).

  • Variance Inflation Factor (VIF): Measurement of multicollinearity:

    • Formula: VIF(Xj) = 1/(1 − R²j) where R²j is from regressing Xj on other predictors.

    • R Code:

    library(car)  
    vif(model)  

3.3 Analysis of Variance (ANOVA)

  • ANOVA Table: Decomposes total variance in Y into parts attributable to regression model (explained) and residuals (unexplained).

  • Sum of Squares:

    • SST = Σ(Yi − Ȳ)²: Total variance.

    • SSR = Σ(Ŷi − Ȳ)²: Explained variance.

    • SSE = Σ(Yi − Ŷi)²: Residual variance.

  • Coefficient of Determination (R²): R² = SSR/SST

3.4 Model Types

  1. Additive Normal Linear Model: Linear effects of predictors without interactions.

  2. Non-Additive Model: Includes interaction terms, allowing combined effects of predictors.

3.5 Interaction Effects and Effect Modification

  • Two-Way Interactions: Effect of X1 on Y depends on X2 (included as X1 × X2).

  • Example R Code:

    model <- lm(Y ~ X1 * X2, data = dataset)  

3.6 Leverage and Influence

  • Leverage: Indicates the influence of each observation on model parameters.

  • R Code:

    hatvalues(model)  

3.7 Model Diagnostics and Assumptions

  • Residual Diagnostics: Evaluate if assumptions hold.

    • Remedies: Transformations or adding/removing predictors.

  • Orthogonal Regressors: Predictors that are uncorrelated, simplifying interpretation.

3.8 Model Selection

  • Selection Methods:

    • Stepwise Regression: Adds/removes predictors based on significance.

    • AIC/BIC: Information criteria for model comparison.

  • Best Practices: Start with theoretically informed predictor set; use diagnostics to refine.

Suggested Approach to Exercises/Problems

  1. Understand the Question: Identify primary goal (e.g., estimating coefficients, testing hypotheses).

  2. Set Up the Model: Specify response and predictors, evaluate transformations or interactions.

  3. Estimate the Model in R:

  • Fit with lm() and check summaries with summary() and diagnostic plots.

  1. Interpret Coefficients: Examine intercept, slopes, and interaction terms.

  2. Conduct Hypothesis Testing: Perform t-tests or ANOVA for significance.

  3. Validate Assumptions:

  • Plot residuals, check for normality (Q-Q plots), and assess multicollinearity (VIF).

  1. Refine the Model: Apply transformations, remove insignificant terms, try different selections if assumptions are violated.

  2. Draw Conclusions: Answer posed question, interpreting results in context.