lecture 6 - multiple regression

Introduction to Multiple Regression

  • Multiple Regression is a statistical method used to model the relationship between one dependent variable and multiple independent variables.

Basics of a Line

  • A straight line can be defined by:

    • Slope (gradient b1): Indicates the steepness of the line.

    • Intercept (b0): The value of Y when X = 0.

Bivariate Regression

  • Components:

    • Dependent Variable (Y): The variable being predicted (e.g., latency).

    • Independent Variable (X): The predictor variable (e.g., garlic).

    • Two model coefficients:

      • β0 (intercept)

      • β1 (slope)

  • Method of Least Squares: Minimizes the sum of squared differences between observed and predicted values.

Regression Line Explained

  • The equation of the regression line can be denoted as:

    • b0 = predicted value when predictor is 0.

    • Example: If b0 = 15.62, then when X = 0, Y is predicted to be 15.62.

Practical Example - Zombies and Garlic

  • Scenario: Addressing a misconception that garlic repels zombies.

  • Two models for prediction:

    • rTMS frequency and Taser voltage relate to zombie immobility.

  • Following the hypothesis with data, various studies suggest different interventions (rTMS or Taser).

Regression Plane Basics

  • Regression Plane represents values of the dependent variable based on multiple predictors.

  • Intercept: Value of Y when both predictors (e.g., rTMS, Taser) are zero.

Extending to Multiple Predictors

  • We can include theoretically unlimited predictors in a regression model.

  • General formula can be expressed as:

    • Y = β0 + β1X1 + β2X2 + ... + βkXk

Key Assumptions of Regression

  1. Linearity: The relationship between X and Y is linear.

  2. Homoscedasticity: Constant variance of Y across all levels of X.

  3. Independent Errors: No correlation among error terms for different observations.

  4. Normality: The residuals (errors) of the model should be normally distributed.

  5. Outliers: Extreme scores that could unduly influence the model.

Checking Assumptions - Linearity and Homoscedasticity

  • Linearity Check: Plot data using scatterplots to observe relationships.

  • Homoscedasticity Check: Analyze variance of Y across different X values graphically.

Assessing Residuals

  • Check residual plots to confirm linearity and homoscedasticity; ideally, residuals should appear random and show no patterns.

  • Focus on whether residuals maintain constant variance across all levels of predicted Y.

Independent Errors

  • Each individual's score prediction error should be independent of the others.

  • Situational dependencies (e.g., similar knowledge traits) may violate independence assumptions.

Normality of Residuals

  • Verify residuals for normal distribution through histograms or Q-Q plots.

  • A residual of zero indicates a perfect prediction by the model.

Identifying Outliers

  • Monitor extreme values. Use Cook’s distance to assess influence of outliers in your dataset.

  • A Cook’s distance exceeding 1 should be investigated further for possible effects on the model.

Multicollinearity in Multiple Regression

  • Definition: Occurs when predictors in a regression model are highly correlated.

    • Can complicate interpretation and lead to inflated standard errors.

  • Use Variance Inflation Factor (VIF) to diagnose multicollinearity issues. Values above 10 are concerning.

Developing Models: Approach to Adding Predictors

  • Hierarchical Regression: Select predictors based on past literature; decision on their entry sequence is specified.

  • Forced Entry: All selected predictors are included in the model at once.

  • Stepwise Regression: Entry order of predictors is determined mathematically, less frequently used in practice.

Implementing Multiple Regression in R

  • Example code snippet:

    • ggplot(data, aes(x = rTMS, y = Immobility)) + geom_point() + theme_light()

  • Generate scatter plots to visualize relationships between predictors and the outcome.

Model Fit and Predictive Improvement

  • Evaluate overall fit of the regression model compared to predictions.

    • Consider remaining prediction errors to assess improvement.

Understanding Model Coefficients

  • Coefficients indicate the degree of impact for each predictor on the dependent variable while controlling for the others.

    • Example: An increase in rTMS frequency by 1 Hz results in added time of zombie immobility (controlling for Taser voltage).

  • Coefficients and their respective significance levels can be evaluated using hypothesis tests.

Summary of Learning Outcomes

  • Key topics to master include:

    • Understanding slope/intercept and their interpretations.

    • Familiarity with regression assumptions.

    • Different methodologies for regression analysis.

    • Output interpretation from R, including significance and confidence intervals.

    • Predictive capabilities in multiple regression settings.

Introduction to Multiple Regression

Multiple regression is a comprehensive statistical method utilized for modeling the complex relationships between one dependent variable and multiple independent variables. This analytical approach is widely used across various fields, including economics, social sciences, and biomedical research, to evaluate how several factors simultaneously affect an outcome.

Basics of a Line

A straight line can be characterized by two primary components:

  • Slope (gradient b1): This reflects the steepness and direction of the line, indicating how much the dependent variable is expected to increase or decrease for each unit increase in the independent variable.

  • Intercept (b0): This represents the expected value of the dependent variable when all independent variables equal zero, effectively anchoring the regression line on the y-axis.

Bivariate Regression

This foundational regression analysis involves two components:

  • Dependent Variable (Y): The outcome being predicted, such as latency in response times.

  • Independent Variable (X): The predictor variable, for instance, garlic intake.

Two key model coefficients in this model are:

  • β0 (intercept)

  • β1 (slope)

To find the best fitting line, the Method of Least Squares is employed, which minimizes the sum of the squared differences between the observed values and the values predicted by the model.

Regression Line Explained

The equation of the regression line can be summarized as follows:

  • Y = β0 + β1X

    • For example, if β0 = 15.62, this indicates that when X (the independent variable) equals 0, the predicted value of Y (the dependent variable) is 15.62.

Practical Example - Zombies and Garlic

In a hypothetical scenario addressing the misconception of garlic repelling zombies, two predictive models can be applied:

  • The relationship between rTMS frequency and Taser voltage concerning zombie immobility can be explored.

  • By following the hypothesis with empirical data, various studies suggest multiple interventions (either rTMS or electric shocks) to achieve desired outcomes, impacting the mobility of zombies in science fiction narratives.

Regression Plane Basics

The regression plane extends the concept of the regression line to scenarios involving multiple predictors, visualizing the interaction of several independent variables:

  • Intercept: Refers to the value of Y when all predictors (e.g., rTMS and Taser voltage) are set to zero.

Extending to Multiple Predictors

In multiple regression, one can theoretically incorporate an unlimited number of predictors into the model. The general formula is:

  • Y = β0 + β1X1 + β2X2 + ... + βkXk This format allows researchers to estimate the dependent variable based on several independent variables simultaneously.

Key Assumptions of Regression

To ensure valid results, multiple regression analysis adheres to several critical assumptions:

  1. Linearity: Assumes a straight-line relationship between predictors and the outcome.

  2. Homoscedasticity: Indicates that the residuals (prediction errors) exhibit constant variance across levels of the independent variables.

  3. Independent Errors: Assumes that the errors for different observations are uncorrelated.

  4. Normality: The distribution of residuals should be normal, which is essential for accurate hypothesis testing.

  5. Outliers: Extreme score values should be monitored as they can potentially distort model results.

Checking Assumptions - Linearity and Homoscedasticity

  • Linearity Check: Scatterplots are utilized to visualize the relationship between independent and dependent variables.

  • Homoscedasticity Check: The variance of Y is analyzed across different values of X graphically, ensuring that spreads remain consistent.

Assessing Residuals

Residual plots should be checked to confirm both linearity and homoscedasticity, with the expectation that residuals appear randomly distributed without discernible patterns.

Independent Errors

Assumption of independence implies that an individual's score prediction error does not relate to those of another. Dependencies, such as similar knowledge traits, can violate this assumption.

Normality of Residuals

Residuals are evaluated for normal distribution using methods like histograms or Q-Q plots, where a residual of zero denotes a perfect prediction.

Identifying Outliers

It's crucial to detect extreme values which could influence model results. Cook’s distance is a measurement used to assess outliers, where a value exceeding 1 warrants further investigation for potential effects on the model.

Multicollinearity in Multiple Regression

  • Definition: This condition occurs when independent variables within a regression model are highly correlated, complicating interpretations and potentially inflating standard errors.

  • The Variance Inflation Factor (VIF) is deployed to diagnose multicollinearity, with values above 10 indicating problematic levels.

Developing Models: Approach to Adding Predictors

Several strategies can be employed in model development:

  1. Hierarchical Regression: Selects predictors based on previous literature, specifying the sequence in which the predictors enter the model.

  2. Forced Entry: All chosen predictors are included in the model simultaneously, regardless of their significance.

  3. Stepwise Regression: Employs a mathematical approach to determine the order of predictor entry based on their contribution to the model, though its use has declined in practice due to potential pitfalls.

Implementing Multiple Regression in R

An example code snippet for visualizing relationships involves:

ggplot(data, aes(x = rTMS, y = Immobility)) + geom_point() + theme_light()

This generates scatter plots that assist in understanding the correlation between predictors and the outcome variable.

Model Fit and Predictive Improvement

Assessment of the regression model's overall fit is crucial compared to its predictions, alongside consideration of remaining prediction errors to evaluate improvement efficacy.

Understanding Model Coefficients

Model coefficients represent the degree of impact that each predictor has on the dependent variable whilst controlling for the presence of other variables. For instance, an increase in rTMS frequency by 1 Hz might lead to an additional duration of zombie immobility when controlling for Taser voltage. Significance levels of coefficients can be determined through statistical hypothesis testing.

Summary of Learning Outcomes

Key concepts to master within multiple regression include:

  • Understanding the interpretations of slope and intercept coefficients.

  • Familiarity with the fundamental regression assumptions and their significance.

  • Different methodological approaches to conduct regression analysis.

  • Comprehensive output interpretation from R, emphasizing significance and confidence intervals.

  • The predictive capabilities inherent in multiple regression settings and their implications in research and practical applications.