Bivariate and Multivariate Regression Analysis

Foundations of Univariate, Bivariate, and Linear Statistics

  • Univariate Statistics Overview:

    • Univariate statistics involve the analysis of a single variable in isolation.

    • Common univariate measures include measures of central tendency (mean, median, mode), measures of dispersion (range, standard deviation), and frequency distributions.

  • Transition to Multivariable Analyses:

    • Real-world phenomena typically involve complex relationships where an outcome is influenced by more than one factor.

    • When research interests shift from describing a single distribution to investigating how variables relate to and change with one another, bivariate and multivariate statistics are required.

  • The Linear Equation as a Model of Change:

    • The classical algebraic slope-intercept formula is expressed as:     Y=MX+BY = MX + B

    • This formula serves as a mathematical representation of change and underpins structural and physical modeling throughout modern engineering and science.

    • For instance, structural engineers use variations of this linear slope equation to calculate the structural integrity and expected lifespan of walls and buildings, determining how different materials and construction methods predict the threshold before structural collapse occurs.

  • The Statistical Adaptation of the Linear Formula:

    • In statistical regression analysis, the classical formula is rewritten with standard regression notation:     Y=Constant+B×XY = \text{Constant} + B \times X

    • Components of the regression formula:

    • YY: The estimated or predicted value of the dependent variable (the outcome measure).

    • XX: The value of the independent variable (the predictor measure).

    • Constant\text{Constant}: The YY-intercept, defining the baseline expected value of YY when X=0X = 0.

    • BB: The regression coefficient, representing the slope of the line.

    • Interpretation of the regression coefficient (BB):

    • BB is the numeric multiplier preceding the independent variable.

    • It quantifies the expected amount of change (rise) produced in the dependent variable YY for each single-unit increase (run) in the independent variable XX.

The Bivariate Regression Model

  • Definition and Function:

    • Bivariate regression estimates the linear relationship between exactly one independent variable (XX) and one dependent variable (YY).

    • It calculates a single slope coefficient indicating how much YY increases or decreases per one-unit increase in XX.

  • Inherent Assumptions and Critical Limitations:

    • Running separate bivariate regressions for multiple potential predictors implicitly assumes that the predictors are entirely independent of one another.

    • It assumes that no independent variable mediates, confounds, or influences any other independent variable.

    • When predictors are intercorrelated (e.g., family wealth and educational attainment), separate bivariate analyses produce biased estimates because they cannot isolate which variable truly accounts for the observed variance in the outcome.

Bivariate Analysis Example: Predicting Occupational Prestige

  • Context of the Investigation:

    • The research objective is to identify predictors of occupational prestige.

    • In occupational prestige rankings within the United States, the single most prestigious occupation is held by only nine individuals (referring to Justices of the Supreme Court of the United States).

  • Model 1: Education Predicting Occupational Prestige:

    • Predictor variable: Highest Year of School Completed.

    • Outcome variable (YY): Occupational prestige.

Bivariate regression model predicting occupational prestige from highest year of school completed
  • Regression output statistics:

    • Constant (Intercept)\text{Constant (Intercept)}: B=9.417B = 9.417, Std. Error=1.421\text{Std. Error} = 1.421, t=6.625t = 6.625, Sig.=.000\text{Sig.} = .000.

    • Highest Year of School Completed\text{Highest Year of School Completed}: B=2.488B = 2.488, Std. Error=.108\text{Std. Error} = .108, Beta=.520\text{Beta} = .520, t=23.056t = 23.056, Sig.=.000\text{Sig.} = .000.

  • Interpretation of Model 1 values:

    • Intercept (9.4179.417): An individual with zero years of completed schooling (X=0X = 0) has a predicted occupational prestige score of approximately 9.4179.417.

    • Slope (2.4882.488): For each additional year of schooling completed, occupational prestige is predicted to increase by approximately 2.4882.488 points (or roughly 2.52.5 points).

    • Predictive calculation example: For an individual with 33 years of education, the predicted prestige is:       Y^=9.417+(2.5×3)=9.417+7.5=16.917\hat{Y} = 9.417 + (2.5 \times 3) = 9.417 + 7.5 = 16.917

    • Statistical significance: The significance column indicates the pp-value. By standard decision rules, if the pp-value is low, reject the null hypothesis (H0H_0). Because p=.000p = .000 (p<.05p < .05), the positive relationship between education and prestige is statistically significant and attributable to systematic factors rather than sampling chance.

    • Model 2: Family Income Predicting Occupational Prestige:

  • Predictor variable: Respondent family income when 16 years old.

  • Outcome variable (YY): Occupational prestige.

Bivariate regression model predicting occupational prestige from family income when 16 years old
  • Regression output statistics:

    • Constant (Intercept)\text{Constant (Intercept)}: B=35.608B = 35.608, Std. Error=1.290\text{Std. Error} = 1.290, t=27.611t = 27.611, Sig.=.000\text{Sig.} = .000.

    • RS Family Income When 16 Years Old\text{RS Family Income When 16 Years Old}: B=2.075B = 2.075, Std. Error=.446\text{Std. Error} = .446, Beta=.122\text{Beta} = .122, t=4.652t = 4.652, Sig.=.000\text{Sig.} = .000.

  • Interpretation of Model 2 values:

    • Individuals raised in families with higher income levels systematically achieve higher occupational prestige in adulthood.

    • The slope is positive (B=2.075B = 2.075) and statistically significant (p=.000p = .000, t=4.652t = 4.652).

Principles and Mechanics of Multiple Regression

  • Conceptual Framework:

    • Multiple regression models evaluate multiple independent variables simultaneously within a single equation.

    • It calculates partial relationships, isolating the unique contribution of each predictor while holding constant (controlling for) all other variables in the model.

    • This approach rules out alternative explanations, identifies confounding or mediating mechanisms, and enhances the overall predictive power for the dependent variable.

  • Empirical Interdependence Example:

    • Family background and educational attainment are strongly intertwined. In the contemporary United States, an individual with a 2.02.0 grade point average (GPA) from a high-income area is statistically more likely to attend college than an individual with a perfect 4.04.0 GPA from an economically disadvantaged background.

    • Bivariate regression cannot untangle whether family income directly influences occupational prestige or operates indirectly through educational access.

  • Structural Requirements for Multiple Regression:

    • Dependent Variable: Must be a continuous variable measured at the interval or ratio level.

    • Independent Variables: Must include two or more variables, which may be continuous, ordinal, or dummy-coded nominal variables.

    • Model Specification: To prevent omitted-variable bias, models should incorporate all relevant predictors hypothesized to influence the outcome.

  • General Equation for Multiple Regression:   Y=Constant+B1X1+B2X2+⋯+BkXkY = \text{Constant} + B_1 X_1 + B_2 X_2 + \dots + B_k X_k

    • YY: Estimated value of the continuous dependent variable.

    • Constant\text{Constant}: Baseline value of YY when all predictors (X1,X2,…,XkX_1, X_2, \dots, X_k) equal zero.

    • B1,B2,…,BkB_1, B_2, \dots, B_k: Partial unstandardized regression coefficients representing the unique slope of each corresponding independent variable (X1,X2,…,XkX_1, X_2, \dots, X_k).

  • Representative Research Applications:

    • Domestic Labor: Testing whether educational level, family size, and the number of hours a husband works predict the volume of domestic household labor he performs.

    • Parenting Quality: Testing whether personality traits, household income, and marital satisfaction predict parental nurturance.

Multivariate Regression Model: Occupational Prestige and Mediation

  • Simultaneous Model Results:

    • When education and family income at age 16 are entered simultaneously into a multiple regression predicting occupational prestige, the coefficients shift:

    • Education coefficient: Remains essentially unchanged from its bivariate estimate (B≈2.49B \approx 2.49) and retains strong statistical significance (p<.05p < .05).

    • Family income coefficient: Decreases substantially relative to its bivariate slope (b=2.075b = 2.075) and loses statistical significance (t<1.96t < 1.96, p≥.05p \ge .05).

  • Substantive Conclusion on Mediation:

    • The disappearance of family income's significant effect demonstrates that the relationship between family income and occupational prestige is mediated through educational attainment.

    • Family income provides access to higher levels of education, but once education is held constant, family income provides no independent, unique predictive power over adult occupational prestige.

Interpreting Multivariable Output: Predictors of Family Size

  • Model Context:

    • Dependent variable (YY): Total family size, quantified as the number of children (\text{# of children}, continuous metric).

    • Independent variables: Income, Education, and Race/Ethnicity (categorized using dummy indicators for Black and Hispanic respondents).


Predictors of family size regression table showing unstandardized and standardized coefficientsPredictors of family size table continuing with education, race/ethnicity, and constant
  • Model Data Table:

    • Income\text{Income}: b=0.08b = 0.08, Standard Error=0.04\text{Standard Error} = 0.04, t=2.04t = 2.04, p=0.04p = 0.04, Beta=0.05\text{Beta} = 0.05.

    • Education\text{Education}: b=−0.08b = -0.08, Standard Error=0.01\text{Standard Error} = 0.01, t=−8.51t = -8.51, p=0.00p = 0.00, Beta=−0.19\text{Beta} = -0.19.

    • Black (compared to White)\text{Black (compared to White)}: b=0.22b = 0.22, Standard Error=0.06\text{Standard Error} = 0.06, t=3.79t = 3.79, p=0.00p = 0.00, Beta=0.08\text{Beta} = 0.08.

    • Hispanic (compared to White)\text{Hispanic (compared to White)}: b=0.35b = 0.35, Standard Error=0.06\text{Standard Error} = 0.06, t=5.86t = 5.86, p=0.00p = 0.00, Beta=0.12\text{Beta} = 0.12.

    • _cons (Constant)\text{\_cons (Constant)}: b=2.22b = 2.22, Standard Error=0.39\text{Standard Error} = 0.39, t=5.67t = 5.67, p=0p = 0, Beta=.\text{Beta} = .

  • Evaluation of Statistical Significance:

    • The pp-value (or Sig.\text{Sig.} column) determines if an independent variable is a statistically significant predictor of the outcome.

    • All four predictors are statistically significant at p<.05p < .05:

    • Income: p=0.04p = 0.04

    • Education: p=0.00p = 0.00

    • Black: p=0.00p = 0.00

    • Hispanic: p=0.00p = 0.00

    • The tt-statistic directly evaluates whether the regression slope (bb) is significantly different from zero (H0:b=0H_0: b = 0).

  • Handling Nominal Predictors via Reference Categories:

    • Nominal variables lack inherent mathematical order and must be split into dummy variables evaluated against an omitted baseline or reference category.

    • In this model, White serves as the reference category.

    • Interpretation of race coefficients:

    • Black (b=0.22b = 0.22): Holding income and education constant, Black individuals are predicted to have on average 0.220.22 more children than White individuals.

    • Hispanic (b=0.35b = 0.35): Holding income and education constant, Hispanic individuals are predicted to have on average 0.350.35 more children than White individuals.

Standardized versus Unstandardized Coefficients (Beta Weights)

  • The Metric Scaling Problem:

    • Unstandardized coefficients (bb or BB) reflect the natural measurement units of each variable.

    • In the family size model, income and education share identical unstandardized magnitudes (∣b∣=0.08|b| = 0.08), where income has b=0.08b = 0.08 and education has b=−0.08b = -0.08.

    • Direct comparison of unstandardized coefficients across different variables is misleading because their measurement scales differ drastically:

    • Education is bounded by a restricted range of potential single-unit annual changes (typically spanning 00 to 2020 or 2525 years maximum).

    • Income contains an effectively infinite number of single-unit currency steps.

  • Function of Standardized Coefficients (Beta\text{Beta}, β\beta):

    • Standardized regression coefficients eliminate raw units of measurement by converting all variables into a shared standard deviation (z-score) scale.

    • A standardized beta represents the expected standard deviation change in YY produced by a one standard deviation increase in XX.

  • Relative Strength Comparison:

    • Standardized beta for income: β=0.05\beta = 0.05.

    • Standardized beta for education: β=−0.19\beta = -0.19.

    • Assessing predictor strength requires examining absolute distance from zero (∣β∣|\beta|), regardless of whether the sign is positive or negative.

    • Because ∣−0.19∣>∣0.05∣|-0.19| > |0.05|, educational attainment exerts a substantially stronger relative impact on family size than household income.

  • Guidelines for Coefficient Selection:

    • Unstandardized coefficients (bb): Used when comparing groups (such as dummy-coded nominal indicators) and when calculating real-world unit predictions.

    • Standardized coefficients (β\beta): Used when comparing the relative predictive importance of different continuous variables measured on disparate scales.

Evaluating Model Quality: Coefficient of Multiple Determination (R2R^2)

  • Definition and Function:

    • The coefficient of multiple determination, denoted as R2R^2 (or R-squared), serves as the index of predictive accuracy for a regression model.

    • It quantifies the proportion of total variance in the dependent variable explained by the entire collection of independent variables in the equation.

  • Mathematical Range and Scale:

    • R2R^2 values fall strictly within the theoretical boundaries of 00 and 11:     0≤R2≤10 \le R^2 \le 1

    • R2=0R^2 = 0: The model accounts for none of the variation in the dependent variable (0%0\%   explained).

    • R2=1R^2 = 1: The model accounts for all variation in the dependent variable, representing perfect mathematical prediction (100%100\%   explained).

  • Interpretation in Applied Research:

    • In research reports, R2R^2 is multiplied by 100100 and communicated as a percentage of explained variance.

    • Conceptual analogy: If all possible social, biological, and environmental factors determining family size equaled 100%100\%   of the reasons people vary in family size, an R2R^2 of 0.200.20 would indicate that the included predictors account for exactly 20%20\%   of that variation.

Model Presentation and Significance Notations: Predicting Missed Classes

  • Analysis Context:

    • Dependent variable (YY): Total number of missed academic classes.

    • Independent variables: Time spent studying, Frequency of alcohol consumption, Hours working, Gender, Age, Parents' income, and GPA.


Regression table displaying unstandardized coefficients and significance markers for missed classesContinuation of missed classes regression table showing demographic variables, GPA, and R-squared
  • Regression Table Summary:

    • Time studying\text{Time studying}: b=−1.03b = -1.03, p=∗p = *

    • Freq. of alcohol consumption\text{Freq. of alcohol consumption}: b=1.72b = 1.72, p=∗p = *

    • Hours working\text{Hours working}: b=.05b = .05, (not significant)

    • Gender\text{Gender}: b=5.01b = 5.01, p=∗p = *

    • Age\text{Age}: b=−.20b = -.20, (not significant)

    • Parents’ income\text{Parents' income}: b=.67b = .67, (not significant)

    • GPA\text{GPA}: b=−2.85b = -2.85, p=∗p = *

    • Model R2\text{Model } R^2: R2=.41R^2 = .41

    • Significance footnote: ∗p<.05*p < .05

  • Interpreting Asterisks and Slopes:

    • Asterisks (∗*) replace exact numeric pp-values in academic tables to flag coefficients that achieve statistical significance at specified alpha levels (p<.05p < .05).

    • Time studying (b=−1.03∗b = -1.03^*): Achieving significance, the unstandardized slope indicates that for each additional hour spent studying per week, an individual is predicted to miss approximately one fewer class (1.031.03 classes missed).

    • Model R2=.41R^2 = .41: The entire set of predictors in this model accounts for 41%41\%   of the total variance in the number of classes missed by students.

Questions & Discussion

  • Sample Test Question on Correlation and Causation:

    • Question Prompt: "A correlation coefficient between hours spent studying and GPA of .80∗.80^* indicates that ________."

    • a. there is no association between hours spent studying and GPA

    • b. hours spent studying probably causes GPA

    • c. GPA probably causes hours spent studying

    • d. there is a relationship between hours spent studying and GPA

    • Correct Selection: d. there is a relationship between hours spent studying and GPA

    • Rationale: The asterisk denotes a statistically significant association (p<.05p < .05), and the coefficient value of .80.80 demonstrates a strong positive relationship. However, correlation and regression coefficients establish mathematical association, not causal directionality; therefore, options asserting that one variable causes the other (b and c) are methodologically invalid.