Advanced Linear Regression: Assumptions, Significance Testing, and Standardized Coefficients

Assumptions of Linear Regression

• Regression analysis requires the fulfillment of specific assumptions to ensure the statistical tools work effectively and the results are reliable.

• Assumption 1: Randomization and Random Sampling     - In the social sciences, this assumption is often violated due to practical limitations.     - Research is disproportionately conducted on college students because they are a convenience sample: cheap, easy to access, and often provide participation for extra credit.     - College student populations are described by the acronym WEIRD: Western, Educated, Industrialized, Rich, and Democratic. This group is not representative of the generalized US population.     - Random assignment is often impossible for variables like race, gender, sexuality, or trauma (e.g., Adverse Childhood Experiences or ACE scores).

• Assumption 2: Normal Distribution of Y at every level of X     - This implies that for every value of the independent variable ($x$), the dependent variable ($y$) should follow a normal distribution (forming a sideways bell curve at every point of $x$).     - This is the least important assumption in the presence of a relatively large sample size ($N$) because large samples approximate normality if the variable is normally distributed in real life.     - Example: In a sample of $2,700$ people comparing education and occupational prestige, lower levels of education (0–5 years) may not show a perfect bell curve because fewer people exist at those levels, and alternate vocational tracks (trade jobs like unionized electricians, plumbers, and carpenters) provide prestige without academic schooling.

• Assumption 3: Homoskedasticity (Homoscedasticity)     - The variance of the prediction errors (residuals) must be identical across all values of the independent variable.     - Homoskedasticity: The error spread is the same throughout ($\text{Homo} = \text{same}$, $\text{schedasticity} = \text{bias/error}$).     - Heteroskedasticity: The error terms fluctuate significantly (e.g., errors getting larger as income increases or following a non-linear U-shape); this violates the assumption and makes results less robust.

Understanding Residuals and Ordinary Least Squares (OLS)

• Definition of Residual: The difference between the observed value and the predicted value ($y - \hat{y}$).

• Ordinary Least Squares (OLS): A regression method that defines the best-fit line by minimizing the total sum of squared errors ($SS_{\text{error}}$).

• Residual Plots:     - Plotting residuals for every data point helps identify patterns.     - If residuals fluctuate randomly with no obvious trend, the assumption of homoskedasticity is met.     - A fan-shaped pattern (residuals getting larger at the higher end of the distribution) indicates heteroskedasticity.     - A U-shaped or curved pattern indicates a non-linear relationship.

Null Hypothesis Significance Testing

• Significance testing requires a clear understanding of the null hypothesis ($H_0$).

• Null Hypothesis ($H_0$): Postulates that the population mean of the dependent variable ($y$) is the same at every level of the independent variable ($x$). In regression terms, the slope ($\beta$) equals zero ($\beta = 0$), appearing as a horizontal line. This suggests $x$ has no effect on $y$.

• Alternate Hypothesis ($H_a$): Postulates that at different levels of $x$, there is a different level of $y$, meaning the slope is not zero.

• Terminological Equivalents: "Coefficient," "Beta ($\beta$", and "Slope" are often used interchangeably in the context of the regression output line.

Calculating Statistics and P-Values in Stata

• T-Statistic (T-Value): Measures how many standard errors an estimated coefficient is away from zero. A higher absolute T-value provides stronger evidence against the null hypothesis.

• Degrees of Freedom ($df$): Represent the number of independent pieces of information in the data available to estimate variability.

• P-Value Manual Calculation (Synthetic Data Example):     - To create a dummy dataset and compute a p-value:         - clear         - set obs 1         - gen t_value = 30.49         - gen df = 2666         - gen p_value = 2 * ttail(df, abs(t_value))         - list t_value df p_value

Confidence Intervals for Regression Slopes

• Confidence intervals can be applied to slopes ($\beta$) just as they are to means, providing a range for the expected impact of a population.

• Formula for 95% Confidence Interval:     - $\beta \pm (1.96 \times SE_{\beta})$     - Example: If $\beta = 2.38$ and $SE_{\beta} = 0.078$:         - Lower End: $2.38 - (1.96 \times 0.078) \approx 2.23$         - Higher End: $2.38 + (1.96 \times 0.078) \approx 2.53$

• Interpretation: We are $95\%$ confident that in the population, for every additional year of school completed, occupational prestige increases between $2.23$ and $2.53$ points.

• Visualizing in Stata: Adding lfitci to a scatter plot command (e.g., twoway (scatter prestige educ) (lfitci prestige educ)) draws the regression line with a shaded confidence interval region.

Advanced Visualization: Coefficient Plots

• The cove plot Command: A specialized Stata tool used to visualize coefficients and their confidence intervals across different models.

• Installation: ssc install cove plot, replace (Note: sometimes referred to as coefplot in standard Stata documentation).

• Usage Example (Automotive Data):     - sysuse auto, clear     - regress price mpg trunk length turn if foreign == 0     - estimates store domestic     - regress price mpg trunk length turn if foreign == 1     - estimates store foreign     - cove plot domestic foreign, drop(_cons)

• Interpretation: If a coefficient's interval bar crosses the zero mark, it is not significant because there is a possibility the effect is either positive or negative. For the foreign car model, length was significant and positive, while turn radius was non-significant due to its wide range crossing zero.

Case Study: Predicting Support for White Supremacy

• In research predicting support for white supremacist groups, several variables were analyzed using coefficient plots:     - Significant Variables:         - Political Conservatism: Positive relationship (more conservative means higher likelihood of support).         - Gender (Women): Positive relationship compared to men in this specific dataset.         - Education: Negative relationship (higher education predicts decreased endorsement of white supremacy).     - Non-Significant Variables:         - Race (White): Interestingly, being white did not strongly predict support when controlling for emotions like pride, anger, shame, and guilt, as well as racial ideologies.

• Implicit Findings: Traits like masculinity are often associated with the far-right (e.g., "boys with tiki torches"), but the data suggests significant movement among "trad-wife" far-right femininity.

Standardized Beta Coefficients

• Standardized Beta ($\beta$): Used to compare the strength of effects across variables measured in different units (e.g., dollars vs. years).

• Calculation: Multiply the raw coefficient ($b$) by the ratio of the standard deviation of $x$ to the standard deviation of $y$:     - $\text{Standardized } \beta = b \times (\frac{SD_x}{SD_y})$

• Interpretation: A one standard deviation increase in $x$ results in a specified standard deviation change in $y$.

• Stata command: regress y1 x1, beta.

• By looking at standardized betas, a researcher can identify which variable has the "strongest impact" regardless of the original unit scale.

Questions & Discussion

• Q: Does a normal distribution indicate the strength of the relationship?     - A: No, but meeting the assumption allows for less biased and more robust statistical tools, improving the reliability of the result.

• Q: What happens if the residuals are non-linear (U-shaped)?     - A: This suggests the data requires a non-linear or interaction term (like a quadratic equation) rather than a simple linear regression.

• Examples of U-shaped relationships:     - Dunning-Kruger Effect: Confidence is high for those with the least knowledge, plummets for those with some knowledge, and increases again for experts.     - Happiness and Age: Happiness tends to plummet around age $30$ (parenting/career stress) and increases again as people age (e.g., as children turn $20$).

• Discussion on Interpretation of Confidence Intervals:     - Israel asked about the range of $0.07$ to $3.262$.     - Correct interpretation: We are confident that in the population, the relationship is positive and the slope falls within that range. Inferential statistics are about the population, not the sample itself.

• Discussion Contributors: Amanda, Adriana, Miles, Andrew, Lindsay, Paula, Kathy, Vincent (marker provider), and Israel.