Multiple Regression

PUBHLTH 500: Investigating Public Health Issues - Biostatistics

Instructor: Jennifer Daniels
Week 10

Multiple Regression

• Multiple regression introduces the concept of building better statistical models.

• It is feasible to enhance model performance, especially when observing diverse relationships between predictors and the response variable.

• For example, in the BAC (Blood Alcohol Concentration) versus Beer consumption model, the coefficient of determination, denoted as R², approximates 80%. Achieving an R² of 90% or greater is generally regarded as indicating an excellent model.

• Challenges to improve predictors arise due to the common occurrence of multicollinearity, where several predictors exhibit correlation.

Impact of Weight and Sex on BAC

• Intuition suggests that body weight could influence BAC—larger individuals typically can consume more alcohol.

• However, a correlated categorical variable named Sex (coded as 0 for Female and 1 for Male) is utilized in this analysis instead of Weight.

Using Categorical Data

• Categorical data can be effectively handled through Indicator Variables (commonly referred to as "I").

• The linear regression model can be represented as follows:

$\hat{Y} = \beta<em>1 + \beta</em>2 * Beers + \beta_3 * I$

Where:

• $\hat{Y}$ = Blood Alcohol Content

• $I = 0$ if Female and $I = 1$ if Male.

• Models can thus be structured as follows:

  • Female Model: $\hat{Y} = \beta<em>0 + \beta</em>1 * Beers$

  • Male Model: $\hat{Y} = (\beta<em>0 + \beta</em>2) + \beta_1 * Beers$

• The parameter $\beta_2$ illustrates the shift in BAC from Female to Male for the same quantity of beer consumed.

• This approach exemplifies using categorical data in a linear modeling context, acknowledging the correlation between variables such as Sex and Weight.

R Model Summary Output

model <- lm(BAC ~ Beers + Sex, data=ch10alcohol)
summary(model)

Residuals Output

• Residuals analysis indicates:

  • Min: -0.024544

  • 1Q: -0.003570

  • Median: 0.001828

  • 3Q: 0.008125

  • Max: 0.018303

Coefficients Output

• Coefficient estimates:

  • Intercept: 0.028139; Std. Error: 0.009786; t-value: 2.875; p-value: 0.0130 (significant)

  • Beers: 0.010712; Std. Error: 0.001626; t-value: 6.589; p-value: 1.75e-05 (highly significant)

  • SexM: -0.020627; Std. Error: 0.006918; t-value: -2.982; p-value: 0.0106 (significant)

• Significance codes:

  • 0 ‘’ 0.001 ‘’ 0.01 ‘’ 0.05 ‘.’ 0.1 ‘ ’ 1

• The model performance metrics indicate:

  • Residual standard error: 0.01354 on 13 degrees of freedom

  • Multiple R-squared: 0.8291; Adjusted R-squared: 0.8028

  • F-statistic: 31.52 on 2 and 13 DF; p-value: 1.032e-05

Model Improvement Insights

• Observing that R² improved approximately by 3% signals the implored impact of the added variable (Sex).

• It implies that predictors can wield competing influence when they are correlated with one another.

• Confounding can occur when these predictors attempt to explain the same underlying information.

• The goal is to find predictors with differing relationships while minimizing the correlation among them.

Model Interpretation

• The general regression equation derived:

$Y = 0.28139 + 0.010712 * Beers - 0.020627 * I$

• Where:

• $I = 1$ for Males, leading to models:

  • For Females: $Y_{hat} = 0.028139 + 0.010712 * Beers$

  • For Males: $Y_{hat} = 0.007512 + 0.010712 * Beers$

• Interpretation infers that for the same alcohol consumption, a Male's BAC is reduced by 0.020627 compared to a Female’s BAC, indicative of weight-based tolerance differences.

Confounding Issues

• Confounding Identification: Confounding occurs when both predictors are related both to each other and to the outcome variable (BAC).

• Example with Weight proposed as an important predictor; however, its correlation with Sex causes concern about confounding effects when both are applied concurrently in a model.

• Analysis yielded p-values for both Sex and Weight greater than 0.05, indicating that they are competing for relevance and confirmation of their individual effects may be problematic.

Confounding Questions Raised

1. How to demonstrate the presence of confounding?

2. Which predictor should remain in the model?

Likelihood Ratio Test (LRT)

• LRT is a statistical test useful in evaluating model differences, framing as follows:
  $-2(LL<em>{Full} - LL</em>{Reduced}) \sim \chi^2(df<em>{Full} - df</em>{Reduced})$

• Where:

  • LL = Natural Log of the Likelihood Function

  • df = degrees of freedom associated with each model.

• Hypotheses formatted:

  • $H_0$: Models are equivalent

  • $H_a$: Models are different

Implementing LRT in R

• R function employed for LRT diagnosis:

library(lmtest)
lrtest(reduced_model, full_model)

Example Application of LRT

• Full Model:

model2 <- lm(BAC ~ Beers + Sex + Weight, data = ch10alcohol)

• Reduced Model:

model1 <- lm(BAC ~ Beers + Sex, data = ch10alcohol)

AIC - Akaike Information Criterion

• AIC serves as an essential tool in Biostatistics for model evaluation, with the premise that lower (or more negative) AIC values are preferred.

• AIC calculation approach:
  $AIC = -2(LL) + 2k$

Where:

• LL = Natural Log of the Likelihood Function

• $k = $ number of model parameters.

Implementing AIC in R

• To extract the AIC from a model, employ the syntax:

AIC(model)

AIC for Model Selection

• Returning to the ch10alcohol dataset, determining the better model between including Sex or Weight is executed as follows:

modelsex <- lm(BAC ~ Beers + Sex, data = ch10alcohol)
modelwt <- lm(BAC ~ Beers + Weight, data = ch10alcohol)
AIC(modelsex)
AIC(modelwt)

AIC Results

• Model with Sex: AIC = -87.58162

• Model with Weight: AIC = -85.4663

• Thus, the model containing Sex exhibits a lower AIC (indicative of superior fit).

Model Comparison Conclusion

• Following validation through AIC and R²:

  • Model 1 (Beers + Sex): AIC = -87.58162, R² = 82.91%

  • Model 2 (Beers + Weight): AIC = -85.4663, R² = 80.49%

• Preference leans towards Model 1, although the decision minimally impacts practical outcomes, emphasizing the need for researcher preference based on contextual needs (Gender influence versus Weight influence).

Automatic Model Selection

1. In complex biostatistical datasets with multiple predictors, selecting the most appropriate model can be overwhelming for researchers.

2. Model selection algorithms such as Forward Selection, Backward Selection, Stepwise Regression, and Best Subsets are frequently employed.

3. The Stepwise approach will be the focus, aiming to minimize predictor numbers for manageability.

Stepwise Model Selection Syntax

1. A full model formula (utilizing all predictors) is fitted initially:

full_model <- lm(BMI ~ ., data = NH)

2. Apply the Stepwise function:

step(full_model, direction = "both")

3. The algorithm will iterate, adjusting predictor retention along the way.

4. Final models can be refined based on the objective—be it predictions (using AIC) or understanding individual influences (considering low p-values).

Stepwise Model Selection Example

1. Analyzing the NHANES dataset with relevant predictor variables:

NHdata <- NHANES %>%
dplyr::select(BMI, Sex = Gender, Age, MaritalStatus, HomeOwn, Pulse, BPSysAve, BP_DiaAve, TotChol, Diabetes, SleepTrouble, SleepHrsNight, PhysActive, HardDrugs)
NH <- na.omit(NHdata)

2. Employing Stepwise regression gives insights into BMI prediction, illustrating the contributions from the dataset's various characteristics.

fullmodel <- lm(BMI ~ ., data = NH)
stepwise <- step(fullmodel, direction = "both", trace = FALSE)
summary(stepwise)

Interaction (X * I) Terms

1. Interaction signifies that two predictors may impact the outcome differently than independently considered.

2. Gender differences in response to treatment/drug are potential instances:

   • Model framework:

   • Let $I$ denote gender, where $1$ represents male, and $0$ conveys female.

   • $\hat{Y} = b + b<em>1 X + b</em>2 I + b_3 X * I$

3. Specific interpretations can be made accordingly:

   • Females: Could yield a reaction baseline in evaluating $Y$ relative to $X$.

   • Males: Implicates a differential intercept/slope compared to female lines, offering richer modeling insights.

Practice Problem Analysis

• Research indicates a potential correlation between sleep deprivation and BMI increase. An investigation was assigned:

1. Develop a model assessing both Sleep and Sex, including their interaction $Sleep * Sex$.

2. Execute the lm and lrtest functions to decide upon factor inclusion based on statistical significance.

Practice Problem - Solution

1. To analyze the Sleep * Sex relationship:

model1 <- lm(BMI ~ SleepHrsNight + Sex + Sex * SleepHrsNight, data = NH)
model2 <- lm(BMI ~ SleepHrsNight + Sex, data = NH)
lrtest(model1, model2)

Interaction Significance

• The hypothesis test denoting models’ distinction produces a rejection of $H_0$, thereby designating the interaction $SleepHrsNight * Sex$ as significant.

Model Interpretation

• Referring back to the final model collaborative outline:
  $Y = b<em>0 + b</em>1 * SleepHrsNight + b<em>2 * I + b</em>3 * SleepHrsNight * I$

1. For Females ($I=0$):
   $Y = b<em>0 + b</em>1 * SleepHrsNight$

2. For Males ($I=1$):
   $Y = b<em>0 + b</em>1 * SleepHrsNight + b<em>2 + b</em>3 * SleepHrsNight$

Conclusion on Average BMI

• Female average BMI is 31.85674; Male average BMI is 29.01077.

• The additional hour of sleep yields different BMI reductions for Males (0.01052) compared to Females (0.42687).

Another Practice Problem

1. Exercise stepwise algorithm analysis on NHANES data targeting predictions for Total Cholesterol. This includes selecting relevant predictors for the model.

2. Execute the following in R:

NHdata <- NHANES %>%
dplyr::select(TotChol, Age, Sex = Gender, MaritalStatus, HomeOwn, Pulse, BPSysAve, BPDiaAve, TotChol, Diabetes, SleepTrouble, PhysActive, HardDrugs, BMI)
nh <- na.omit(NHdata)
full <- lm(TotChol ~ ., data = nh)

Next Steps in Problem Solving

• Progress to apply stepwise modeling for best predictors aimed at correlating Total Cholesterol with the model outlined.

stepwise <- step(full, direction = "both", trace = FALSE)
summary(stepwise)

Final Insights from Total Cholesterol Analysis

• Output of model predictors confirms significance theories with noted positive or negative influences on Total Cholesterol levels. Their respective estimates, provided in the summary output, offer crucial insights into health-related interventions based on biostatistics findings

• Coefficients highlighted for analysis:

• Intercept: 2.821342, significant with t-value approaching 20

• Age and gender-specific variances provide evidence of multi-faceted influences on Total Cholesterol responses and require further targeted research.

Absolutely — here’s a clear, logical walkthrough of the multiple regression concepts from your slides, organized so that it actually flows and makes sense conceptually (instead of jumping around like lecture slides often do).

🧠 Understanding Multiple Regression (from your slides)

1⃣ Why Use Multiple Regression?

The goal of multiple regression is to build a better model to predict an outcome variable (Y) using two or more predictors (X₁, X₂, … Xₖ).

It extends simple linear regression (which has one predictor) by allowing you to control for other variables and study how each variable contributes while holding the others constant.

Example from the slides:
We want to predict Blood Alcohol Content (BAC) from:

• Beers consumed, and

• Sex (Male/Female).

A heavier person usually tolerates more alcohol, so sex might influence BAC.

2⃣ The Basic Model

The general multiple regression equation is:

[
Y = β_0 + β_1X_1 + β_2X_2 + … + β_kX_k + ε
]

In the example:

[
BAC = β_0 + β_1(\text{Beers}) + β_2(\text{Sex})
]

where:

• (β_0): Intercept (BAC when predictors = 0)

• (β_1): Change in BAC for each additional beer, holding sex constant

• (β_2): Difference in BAC between males and females, holding beers constant

3⃣ Using Categorical Predictors (Indicator Variables)

Since Sex is categorical, we can represent it with a dummy variable (I):

• Female = 0

• Male = 1

Now the equation becomes:

[
\hat{BAC} = 0.0281 + 0.0107(\text{Beers}) - 0.0206(\text{Male})
]

Interpretation:

• For females: ( \hat{BAC} = 0.0281 + 0.0107(\text{Beers}) )

• For males: ( \hat{BAC} = 0.0075 + 0.0107(\text{Beers}) )

→ At any given number of beers, males’ BAC is about 0.02 lower than females’.

That makes intuitive sense since males generally weigh more.

4⃣ Why Add Predictors? (and Why R² Doesn’t Always Jump Much)

Adding predictors can improve the model fit, but not always dramatically.

In the slides:

• Beers-only model: (R² ≈ 0.80)

• Beers + Sex model: (R² ≈ 0.83)

That’s only a small (3%) improvement.

Why? Because Beers and Sex are correlated — one sex tends to drink more beers than the other, so they partly explain the same variation in BAC.

When predictors overlap in what they explain about Y, it’s called multicollinearity or confounding.

5⃣ Confounding in Multiple Regression

Confounding happens when two predictors are related to each other and both related to the outcome, which makes it hard to tell which variable is actually causing the effect.

Example:
We think Weight affects BAC, and Sex is correlated with Weight.

If you include both in the model:

[
BAC = β_0 + β_1(\text{Beers}) + β_2(\text{Sex}) + β_3(\text{Weight})
]

You’ll see that:

• Both Sex and Weight have p > 0.05

• They “compete” to explain the same relationship.

This indicates confounding between Sex and Weight.

6⃣ Proving Confounding Statistically

To formally test whether adding a variable truly improves the model, you can use a Likelihood Ratio Test (LRT):

• Full model: includes all predictors (Beers + Sex + Weight)

• Reduced model: fewer predictors (Beers + Sex)

In R:

lrtest(model_full, model_reduced)

If the p-value > 0.05, the models are statistically equivalent —
meaning the extra variable (Weight) doesn’t add useful information.

In the slides, p = 0.40 → the two models are equivalent → Weight and Sex are confounded.

7⃣ Model Selection Criteria: AIC

To choose between models, we can use Akaike Information Criterion (AIC).
AIC balances goodness of fit with model simplicity:
[
AIC = -2(LL) + 2k
]

• Lower (more negative) AIC = better model.

• AIC only decreases when you add a good predictor.

Example:

→ “Beers + Sex” is slightly better (lower AIC).
So, you’d probably keep Sex in your final model.

8⃣ Automatic Model Selection (Stepwise Regression)

When you have many predictors, manually testing each is inefficient.
Stepwise regression automates the process:

• Starts with all predictors (full model)

• Removes or adds variables based on AIC

• Stops when model improvement is minimal

Example using NHANES data to predict BMI:

fullmodel <- lm(BMI ~ ., data = NH)
stepwise <- step(fullmodel, direction = "both")
summary(stepwise)

Final model keeps only predictors that improve AIC:
Pulse, BP, Diabetes, SleepTrouble, Physical Activity, HardDrugs.

9⃣ Interaction Terms (Effect Modification)

Sometimes the relationship between X and Y changes depending on another variable — that’s an interaction.

Example from slides:
Does the effect of sleep hours on BMI differ by sex?

Model:
[
BMI = β_0 + β_1(\text{SleepHrs}) + β_2(\text{Sex}) + β_3(\text{SleepHrs × Sex})
]

Likelihood Ratio Test shows p = 0.0037 → significant interaction.

Interpretation:

• Females: (BMI = 31.86 - 0.43(\text{SleepHrs}))

• Males: (BMI = 29.01 - 0.01(\text{SleepHrs}))

So, sleep decreases BMI more for females than males.

🔟 Pulling It All Together

✅ Bottom Line
Multiple regression helps us:

• Control for confounding

• Quantify independent effects

• Build predictive models

• Test interactions

We interpret each coefficient as the unique effect of that variable on the outcome, holding all others constant.

Would you like me to turn this into a condensed 1-page “cheat sheet” version (like your epidemiology and biostats ones) for quick studying? It would summarize this flow visually with formulas, examples, and key interpretations.