Meta Analysis in Psychological Research

Introduction to Meta Analysis

• Course context: Psych '4000 01/2020 two Applied Research Methods two.

• Acknowledgment of traditional custodians of the land.

• Today's focus: Meta analysis and its importance in psychological research.

Overview of Lecture Contents

• Definition and purpose of meta analysis.

• Brief history of the development of meta analysis.

• Review of effect sizes crucial for meta analysis.

• Steps involved in conducting a meta analysis:

  • Formulating the problem.

  • Identifying studies.

  • Coding and extraction of data.

  • Utilizing fixed or random effects models.

  • Analyzing the data.

  • Interpreting outcomes.

• Hands-on example using SPSS version 28 for practical application.

• Evaluation of meta analysis, focusing on concerns and strengths.

Defining Meta Analysis

• Definition: Statistical analysis of a large collection of analysis results from completed studies to integrate findings (DeCoster, 2009).

  • Importance in evidence-based practice.

  • Used extensively in clinical decision-making and policy formulation.

Historical Development of Meta Analysis

• Introduced by Carl Pearson in 1904.

• Gained traction in the 1970s within psychological research:

  • Glass's 1976 publication pioneered its application.

  • Rosenthal and Rubin's paper in 1978 expanded its reach.

  • Hedges and Olkin refined the methodology in 1985.

• Growth trajectory peaked in the mid-2000s (referencing Andy Field, 2003).

Hierarchy of Evidence

• Classification of evidence in scientific research from weak to strong:

  • Weakest: Case reports, opinion papers, letters.

  • Intermediate: Animal trials, in vitro studies, cross-sectional studies, case-control studies, cohort studies, randomised controlled trials.

  • Strongest: Meta analyses and systematic reviews at the pinnacle.

  • Non-scientific evidence includes personal anecdotes and unreliable web sources (e.g., YouTube, Natural News).

Understanding Effect Sizes

• Effect Sizes:

  • Measure of the magnitude of effects in, e.g., regression and analysis of variance.

  • In regression: measured as $R^2$ (the proportion of variance explained by predictors).

  • In ANOVA: measured as Eta squared (or has been referred to as Eta squared).

  • Correlation coefficients, odds ratios, and risk rates are also measures of effect size but correlation coefficients are predominant in psychological research.

Types of Effect Sizes:

1. Regression Analysis: $R^2$ indicates proportion of variance.

2. ANOVA: Eta squared captures variance amongst groups.

3. Correlation: Measures strength and direction of linear relationships.

4. Specifics of Effect Size Calculation:

   • Effect Size (4): $ext{Delta} = rac{ ext{Mean}<em>1 - ext{Mean}</em>2}{ ext{Standard Deviation}}$.

   • Sample context: $ext{Delta}<em>D = rac{ ext{Mean}</em>1 - ext{Mean}_2}{ ext{Standard Error}}$.

Key Effect Size Statistics:

• Glass's Delta: Mean difference across control group.

• Cohen's D: Mean difference across pooled standard deviations.

  • $ext{Pooled SD} = ext{sqrt} rac{(N<em>1-1)S</em>1^2 + (N<em>2-1)S</em>2^2}{N<em>1 + N</em>2 - 2}$.

• Hedges' G: Similar calculation; adjusted standard deviation.

• General Effect Size Considerations:

  • Recommend Cohen’s D for interpretability; Hedges G as more accurate in small samples.

Process of Conducting Meta Analysis

Steps to Follow:

1. Formulate the Problem:

   • Define experimental group differences and treatment effects.

   • Consider variability across study outcomes.

2. Identify Studies:

   • Broad literature search for relevant studies using effective search terms.

   • Consider unpublished studies to mitigate file drawer problem.

   • Methods:

     • Contact authors for unpublished data.

     • Reference searching or citation analysis.

3. Coding and Data Extraction:

   • Collect key information such as:

     • Sample sizes, study design, intervention details, dependent variables, p-values, and effect sizes.

   • Introduce potential moderating variables that account for variance.

4. Analyzing the Data:

   • Examine effect size distributions and outliers.

   • Report significance of main effects.

5. Interpreting Outcomes:

   • Use results to draw conclusions on effect sizes and differences.

Importance of Moderator Analysis

• Analyze whether variables significantly influence effect size results.

• Heterogeneity tests indicate variability, suggesting moderators may exist (Hedges & Olkin, 1985):

  • $QT = ext{sum}(Wi imes (Di - MD)^2)$.

• Addressing variability leads to more robust and meaningful insights.

Meta Analysis Concerns:

1. Overgeneralization risk from diverse study inclusion.

2. Quality and reliability of incorporated studies.

3. Publication bias, especially towards positive results.

4. Possibility of ignoring interactions; must evaluate moderators.

5. Subjective decision-making inherent; necessitate transparency in process.

Strengths of Meta Analysis:

• Allows synthesis from numerous studies.

• Uncovers relationships missed by traditional testing methods.

• Promotes objective reviews and scrutiny of decisions recorded for peer review.

Evaluating Meta Analysis Efficiency

• Criteria for Judging Quality (DeCoster, 2009):

  1. Internal Validity:

  • Check validity of included primary studies.

  • Ensure sufficient studies (recommendation: at least 30).

  1. External Validity:

  • Is literature search extensive? Inclusion/exclusion justifications warranted.

  • Are effect sizes derived from consistent theoretical constructs?

  1. Theoretical Contribution:

  • Type A: Effect strength summarization.

  • Type B: Moderator exploration.

  • Type C: New evidence or moderator introduction.

Conclusion and Next Steps

• Covered foundational elements of meta analysis.

• Mini worked example illustrated calculations and considerations.

• Acknowledged both challenges and advantages of conducting meta analyses.

• Next workshop includes practical SPSS application for meta analysis executions.