NHST Shortcomings, Meta Analysis, and Testing Proportion

NHST Shortcomings and Meta-Analysis

Issues with NHST

• The Problem: NHST (Null Hypothesis Significance Testing) is widely used but has inherent problems.
• Example Scenario 1:
  • A between-subjects experiment yields:
    • Experimental condition: M = 55.2, SD = 11.6, N = 40
    • Control condition: M = 48.9, SD = 10.7, N = 40
    • t(78) = 2.54, p = .013
  • Replication yields:
    • Experimental condition: M = 53.8, SD = 9.6, N = 40
    • Control condition: M = 51.2, SD = 8.9, N = 40
    • t(78) = 1.30, p = .199
  • Question: How do we reconcile these conflicting results?
• Cohen’s d and Confidence Intervals:
  • Even when p-values differ significantly, confidence intervals (CIs) can overlap substantially. Overlap of more than 50% in CIs suggests agreement, despite differing significance levels.
• Example Scenario 2:
  • Two studies on the same topic produce similar p-values but different effect sizes.
    • Study 1: p = .016, N = 28
    • Study 2: p = .015, N = 420
  • The point estimate for the effect size in Study 1 is five times larger than in Study 2.
  • There is very little overlap in the CIs, indicating disagreement between the studies.
  • Focusing solely on p-values obscures these critical differences.
• Distraction from Deeper Questions:
  • NHST focuses on a binary 'Is there an effect?' question rather than the magnitude or nature of the effect.
  • Science advances by challenging hypotheses, but rejecting H0 doesn't directly challenge the research hypothesis.
• Better Questions:
  • Instead of just asking if an effect exists, researchers should ask:
    • What is the effect and how large is it? (regression slope, effect size).
    • How precise is our estimate? (confidence intervals).
    • Are there other explanations for the data that are simpler or better than mine?

NHST and Bias

• Publication Bias: The pressure to achieve 'significant' results leads to:
  • Overestimation of effect sizes in published literature.
  • Increased likelihood of publishing Type I errors (false positives).
  • Limited understanding due to the failure to publish non-significant results.
• P-Hacking: Manipulating data analysis to achieve statistical significance:
  • Stopping rules: Halting data collection once a significant result is obtained.
  • Data trimming: Selectively excluding outliers until a significant result is found.
  • Variable manipulation: Subgrouping or subsetting data to produce significance.
  • Selective reporting: Only reporting significant results and ignoring non-significant outcomes.
• Overcoming Bias:
  • Pre-registering studies: Publicly specifying the research plan (hypotheses, methods, and statistical analysis plan) before conducting the study.
  • Sharing collected data in a public repository.

Remedies for NHST Issues

• Understanding P-values: Recognize that p-values do not directly indicate the truthfulness of the research hypothesis.
• Interpreting Non-Significant Results: A non-significant result doesn't provide much information on its own; consider sample size and direction of the potential effect.
• Estimation Thinking:
  • Focus on the magnitude of the effect, precision of estimate and combined estimate from available studies

Meta-Analysis

• Definition: A statistical technique to combine effect sizes from multiple studies investigating the same effect to obtain a better estimate of the population effect size.
• Purpose:
  • To quantitatively combine data from multiple independent studies on a specific topic.
  • To derive more robust and precise findings.
  • To quantify the overall effect size of an intervention, treatment, or phenomenon, providing a more accurate estimate of its true effect in the population.
• Hierarchy of Scientific Evidence: Meta-analyses and systematic reviews are considered the strongest forms of scientific evidence.
• Weighting Studies:
  • Meta-analyses focus on effect sizes.
  • Larger studies (with larger sample sizes) are given higher weight, as they provide more accurate effect size estimates.
  • Integrating findings from multiple studies increases statistical power to detect true effects.
• Example: A meta-analysis exploring whether physical activity during pregnancy prevents postpartum depression.
  • Included 17 studies with 93,676 participants.
  • Found a significant reduction in postpartum depression among physically active pregnant individuals compared to inactive ones.
• Forest Plots:
  • Used to visually represent meta-analysis results.
  • List each included study, along with its point estimate and 95% CI.
  • A diamond shape represents the meta-analysis point estimate and 95% CI.
  • Can separate studies into subgroups (e.g., intervention vs. observational) to show varying effects.
• What a meta- analysis estimates:
  • Mean population effect size

Key Takeaways

• Understand shortcomings of NHST.
• Know how a meta-analysis works.
• Know what a meta-analysis estimates (mean population effect size).

Chi-Square Test for Proportions

Testing Proportions

• Analyzing nominal data when the dependent variable (DV) is on a nominal scale.

Previous Weeks

• Outcome variable (DV) was always a continuous variable (interval or ratio scale)
• Independent variable (IV) was either continuous (e.g. age) or on a nominal scale (e.g. experimental or control group)

This Week

• DV is about category membership (nominal scale)
  • E.g. Did participant solve the task? (Yes/No)
  • Which do you prefer? (A/B/C/D)
• Need to analyze nominal data

Chi-Square (χ2) - One Nominal Variable

• Used when there is one nominal variable with two or more categories (e.g., Yes/No, A/B/C).
• Compares observed frequencies in each category with expected frequencies (usually 50/50).
• Asks: Does the data convincingly show inconsistency between the observed and expected frequencies?

Hypotheses

• H0: There is no inconsistency between observed and expected frequencies; the observed frequencies follow the same distribution as expected frequencies.
• H1: There is an inconsistency between observed and expected frequencies; the observed frequencies do not follow the same distribution as expected frequencies.

Evaluation

• Quantify how different observed frequencies are from expected.
• Large deviations from chance provide evidence for the research hypothesis.

Simple Proportions Example

• Does expensive wine taste better?
  • 21 blinded wine experts try very expensive vs. cheap wine.
  • 8 prefer expensive, 13 prefer cheaper wine.

Effect Size

• Effect Size is the Proportions – this is your point estimate
• Divide the count by the total number
  • $\frac{count}{total \, number}$
• Times by 100 (to make a percentage)
  • $\frac{count}{total \, number} * 100$

Wine Example

• Eight chose expensive, out of 21:
  • $(8/21) \times 100 = 38\%$
• Thirteen chose cheaper, out of 21:
  • $(13/21) \times 100 = 62\%$

Confidence Intervals

• Used to show how accurately our sample data reflect preference in our population
• 38% expensive wine choices, 95% CI [21%, 59%]

Significance Testing

• NHST: p – value (p data | H0)
• If there was no overall preference in the population (H0 is true), how likely would this observed (or larger) deviation be from 50/50?
• SPSS output for wine data using χ2:
  • Participants in each group
  • Expected participants in each group (50/50 split)
  • Degrees of freedom: Number of groups - 1
  • P value

Example

• 'The pattern of observed preferences (38% old expensive wine, 62% new cheaper wine) did not differ significantly from chance, χ2(1) = 1.19, p = .275’

CI for Significance Testing

• For d and r, zero represents H0.
• For simple proportions 50% represents H0 (due to chance).
• 95% CI [21%, 59%], includes the value that represents H0 (50%). Therefore χ2 test not statistically significant (p = .275).

Chi-Square (χ2) - Two Nominal Variables

• Two nominal variables, each with two or more categories.
• Determines whether proportions for one variable differ across categories of the other.
• Hypotheses:
  • H0: Proportions of one variable are independent of the second variable.
  • H1: Proportions of one variable are not independent of the second variable.

Comparing Proportions

• Modified wine study: Does knowledge affect preference?
  • Between subject's design, control group are blind to the wine they are drinking, experimental group are told which is the expensive and which is the cheaper wine (nominal two groups: blind/unblind)
  • DV – Which wine do you prefer (nominal two groups: expensive/cheap)

Results

Proportions

Estimation Perspective

• Proportion of choices for expensive is 55 percentage points higher in experiment group

Significance Testing

• Blind condition : 9/20
• Non-Blind condition: 21/21
• Is chance a viable explanation for the observed difference in choice patterns?
• If the manipulation has no effect on choice pattern (H0 is true) how likely are we to observe this difference (or larger)?

Example

• The patterns of preferences (blind: 45% expensive; non-blind: 100% expensive) differed significantly from each other χ2(1) = 15.8, p < .001’.

Testing Proportions and Power

• Sample size and effect size show positive relationship with statistical power.
• Stronger manipulation / larger bias is easier to detect than a small bias
• Larger N more precise estimate and more likely to detect bias between groups

Caution

• χ2 yields unreliable result is any expected frequency is less than 5 (especially if the number of categories is small)
• So should not be used
• Using a larger sample size reduces this chance

Key Takeaways

• … be able to identify when the chi2 test is appropriate.
• … be able to correctly report the chi2 test results.
• … be able to draw appropriate conclusions from these results.
• … be able to report and correctly interpret confidence intervals.