Lecture 3 -Multiple Regression

What is linear regression?

A statistical method used when there are one or more independent variables and a numeric dependent variable.

What is simple linear regression?

Linear regression with a single predictor variable.

What is multiple regression?

Linear regression with two or more predictor variables.

What are the two main purposes of regression analysis?

1. Test hypotheses about associations between variables. 2. Predict new values of a dependent variable from values of the independent variable(s).

In simple regression, what is the relationship between regression and correlation?

Testing the association between two numeric variables using simple linear regression is equivalent to correlation analysis.

What is the null hypothesis in the lecture's gestation and birthweight example?

There is no linear association between gestation and birthweight; the population slope is zero (H₀: b = 0).

What is the alternative hypothesis in the gestation and birthweight example?

There is a linear association between gestation and birthweight; the population slope is not zero (H₁: b ≠ 0).

What is the GLM equation for simple linear regression?

Yᵢ = a + bXᵢ + eᵢ.

In the simple regression equation, what does Yᵢ represent?

The dependent/outcome variable for case i; in the lecture example, the baby's weight.

In the simple regression equation, what does Xᵢ represent?

The predictor variable for case i; in the lecture example, gestation in weeks.

In the simple regression equation, what does a represent?

The constant or intercept: the predicted value of Y when X = 0.

In the simple regression equation, what does b represent?

The slope: the expected change in Y for a one-unit increase in X.

In the simple regression equation, what does eᵢ represent?

The residual or error: the difference between the observed outcome and the value predicted by the model.

What is a residual?

The difference between an observed outcome and the outcome predicted by the regression model.

How does regression make predictions for a new case?

Use the regression equation but leave out the error term: predicted Y = a + bX.

Why is the error term omitted when making a prediction?

The residual for a new, unobserved case is unknown, so prediction uses the model's fitted component.

What are the two broad hypotheses tested with regression models?

1. Whether the model fits the data overall. 2. Whether a predictor is significantly associated with the outcome.

Which test assesses whether a regression model fits the data overall?

The F-test.

Which test assesses whether an individual regression coefficient differs significantly from zero?

A t-test.

What does the overall F-test assess in regression?

Whether the variance explained by the regression model is greater than the residual (unexplained/error) variance.

Why do the F-test and t-test give the same significance result in simple linear regression?

There is only one predictor, so the overall model test and the test of that single slope are equivalent (F = t²).

Why might we use multiple regression instead of simple regression?

Adding more relevant independent variables may improve the model's ability to explain or predict the dependent variable.

What example is used to motivate multiple regression in the lecture?

Predicting baby's weight using gestation and the mother's pre-pregnancy weight.

What is the GLM equation for multiple regression?

Yᵢ = a + b₁X₁ᵢ + b₂X₂ᵢ + ... + bₖXₖᵢ + eᵢ.

In multiple regression, what does each b coefficient represent?

The expected change in the outcome for a one-unit increase in that predictor, holding the other predictors constant.

What is the model component in multiple regression?

The intercept plus the combined contribution of all predictor variables and their coefficients.

How is the best-fitting regression line or model found in practice?

Through mathematical optimisation: an algorithm searches for the model that minimises the residuals/sum of squared errors.

What is the sum of squared error (SSerror)?

The sum of the squared residuals: the squared differences between observed and predicted outcomes.

What are the steps for conducting the overall F-test in multiple regression?

1. Calculate the model and residual sums of squares. 2. Divide each by its degrees of freedom to obtain mean squares. 3. Divide model mean square by residual mean square to obtain F. 4. Use the F distribution and relevant degrees of freedom to obtain the p-value.

What is the model sum of squares (SSM)?

The variation in the outcome explained by the regression model.

What is the residual sum of squares (SSR) in the lecture's notation?

The variation not explained by the regression model; it is the sum of squared residuals.

What is the total sum of squares (SST)?

The total variation in the outcome around its mean.

How are total, model and residual sums of squares related?

SST = SSM + SSR: total variation is divided into explained model variation and unexplained residual variation.

What does the difference between SST and SSR represent?

The variance explained by the model (SSM).

How do you calculate a mean square in regression?

Divide the relevant sum of squares by its degrees of freedom.

What are the degrees of freedom for the regression model (DFM)?

The number of predictor variables in the model (k).

What are the degrees of freedom for the residuals (DFR)?

N − k − 1, where N is the number of participants and k is the number of predictors; the extra 1 accounts for estimating the intercept.

If a regression has 42 participants and 2 predictors, what are the model and residual degrees of freedom?

DFM = 2; DFR = 42 − 2 − 1 = 39.

What is R-squared (R²)?

The proportion of the total variance in the outcome that is accounted for by the regression model.

How is R² calculated using sums of squares?

R² = SSModel / SSTotal.

What is another name for R²?

The coefficient of determination.

How should an R² of 0.60 be interpreted?

The model accounts for 60% of the observed variance in the dependent variable.

What are the four regression assumptions listed in the lecture?

1. Residuals are normally distributed. 2. Homoscedasticity. 3. Linear relationships between dependent and independent variables. 4. No multicollinearity.

What does normality of residuals mean?

The regression residuals/errors are approximately normally distributed.

What is homoscedasticity?

A similar amount of variation in the outcome/residuals across different values of the predictor(s); residual variance is roughly constant.

What is heteroscedasticity?

Unequal or changing residual variance across values of the predictor(s), meaning the spread is not constant.

What does the linearity assumption mean in regression?

The relationship between the dependent variable and each independent variable is appropriately represented as linear by the model.

What is multicollinearity?

High correlation between two or more independent/predictor variables.

What threshold for high predictor correlation is highlighted in the lecture?

An absolute correlation greater than about .7 (|r| > .7).

When does multicollinearity cause problems?

When interpreting and testing the statistical significance of coefficients for predictors that are highly correlated with each other.

When is multicollinearity less problematic, according to the lecture?

It does not necessarily prevent assessing overall model fit, interpreting coefficients for unrelated predictors, or using the model for prediction, although it can affect p-values.

What is the key difference between simple and multiple regression?

Simple regression has one predictor; multiple regression has two or more predictors.

What is the key difference between the F-test and t-test in multiple regression?

The F-test assesses the overall model; t-tests assess individual coefficients/predictors.

Quick recall: what does the slope mean in simple linear regression?

The expected change in the outcome for a one-unit increase in the predictor.

Quick recall: what does R² tell you?

The proportion of total outcome variance explained by the regression model.

Quick recall: what does DFR = N − k − 1 account for?

The number of observations minus the number of predictors and minus one for the estimated intercept.