Weeks 1-5 Advanced Research Methods

General Linear Model (GLM) and Comparing Means

• General Linear Model (GLM):

  • A framework for comparing group means and testing relationships between variables.

  • Includes techniques like ANOVA and ANCOVA, which extend regression analyses.

  • Used in experimental and non-experimental research to analyse continuous dependent variables (DVs) and categorical or continuous independent variables (IVs).

Comparing Several Means (ANOVA)

• Definition:

  • ANOVA (Analysis of Variance) is a statistical test used to compare means from three or more groups.

  • Tests the null hypothesis (H0) that group means are equal.

• Purpose:

  • Determines whether observed differences between group means are likely due to chance.

  • Prevents Type I error inflation caused by conducting multiple t-tests.

• Key Concepts:

  • Omnibus Test:

    • Tests for overall group differences.

    • Does not specify which groups differ; follow-up tests (planned contrasts or post-hoc tests) are required.

  • F-Ratio:

    • Ratio of model variance (explained variance) to error variance (unexplained variance).

    • If the F-ratio > 1, it suggests the model explains more variance than random error.

Types of ANOVA

1. Between-Groups ANOVA:

   • Different participants in each group (independent groups).

   • Tests whether means differ across unrelated groups.

   • Example: Comparing weight loss between three diet groups.

   • Assumption: Homogeneity of variance (Levene’s test).

2. Within-Groups ANOVA:

   • Same participants measured across multiple conditions (repeated measures).

   • Example: Measuring cognitive performance at different times of the day.

   • Requires sphericity—equal variances of the differences between levels (Mauchly’s test).

3. Factorial ANOVA:

   • Includes two or more IVs (categorical).

   • Tests main effects (individual IVs) and interactions (combined effects of IVs).

   • Example: Testing effects of diet and exercise on weight loss.

4. Mixed ANOVA:

   • Combines between- and within-subjects designs.

   • Example: Testing the effect of a therapy intervention (between-groups) over time (within-groups).

5. ANCOVA (Analysis of Covariance):

   • Extends ANOVA by including covariates (continuous variables that may influence the DV).

   • Reduces error variance, improving statistical power.

   • Example: Comparing test scores across schools while controlling for prior knowledge.

Assumptions of ANOVA

1. Normality:

   • Data should follow a normal distribution.

   • Check using histograms, skewness (≈0), and kurtosis (≈0).

2. Homogeneity of Variance:

   • Variances across groups should be similar.

   • Tested with Levene’s test (p > .05 indicates assumption met).

3. Independence of Observations:

   • Observations within each group must be independent.

4. Sphericity (for Within-Groups Designs):

   • Variances of the differences between all combinations of conditions should be equal.

   • Tested using Mauchly’s test (p > .05 indicates sphericity).

   • If violated:

     • Greenhouse-Geisser correction (if sphericity estimate < 0.75).

     • Huynh-Feldt correction (if sphericity estimate > 0.75).

t-Tests

• Definition:

  • Statistical tests comparing the means of two groups.

• Types:

• Independent Samples t-Test:

  • Compares means of two unrelated groups.

  • Example: Comparing test scores of males and females.

• Paired Samples t-Test:

  • Compares means of related groups (e.g., pre- and post-test).

  • Example: Measuring weight before and after a diet.

• Assumptions:

  • Normally distributed DV within groups.

  • Homogeneity of variance (tested with Levene’s test).

Effect Sizes

• Definition:

  • Quantifies the magnitude of a relationship or difference.

  • Provides context beyond p-values.

• Common Metrics:

  • Cohen’s d:

    • Measures the standardised mean difference.

    • Small (0.2), Medium (0.5), Large (0.8).

  • r² (Coefficient of Determination):

    • Proportion of variance explained by the model.

    • Small (0.04), Medium (0.16), Large (0.25).

  • η² (Eta Squared):

    • Proportion of total variance explained by an IV.

    • Small (0.01), Medium (0.06), Large (0.14).

Correlation

• Definition:

  • Measures the strength and direction of a relationship between two continuous variables.

• Types:

• Pearson’s Correlation Coefficient (r):

  • Parametric; assumes linearity and normality.

• Spearman’s Rho:

  • Non-parametric; ranks data.

• Kendall’s Tau:

  • Non-parametric; better for small samples.

• Assumptions:

  • Linearity, normality, and homoscedasticity.

• Key Concepts:

  • Positive Correlation: As one variable increases, so does the other.

  • Negative Correlation: As one variable increases, the other decreases.

  • No Correlation: No relationship exists.

Post-Hoc and Planned Comparisons

1. Planned Comparisons:

   • Pre-specified hypotheses.

   • Conducted only for predicted differences.

2. Post-Hoc Tests:

   • Exploratory; examines all group differences.

   • Requires correction for multiple comparisons (e.g., Bonferroni correction).

Choosing the Right Test

1. Identify Variables:

   • DV: Continuous or categorical?

   • IV: Number and type (categorical/continuous)?

2. Reference Table:

   • Continuous DV + 2 Categorical Levels (IV): t-Test.

   • Continuous DV + 3+ Categorical Levels (IV): ANOVA.

   • Continuous DV + Covariate: ANCOVA.

Sampling and Representativeness

(Based on More than Just Sample Size PDF)

• Why Sample Size Matters:

  • Larger sample sizes improve the reliability of results by reducing the variability of estimates.

  • Power: Larger samples increase the ability to detect true effects (statistical power).

  • Risks of Oversampling: While large samples may identify statistically significant differences, these differences might be trivial or irrelevant in practice. Example: A tiny effect size might become significant due to excessive power, but its practical implications might be negligible.

• Beyond Size:

  • Sampling isn't only about size; representativeness is key.

  • A sample must reflect the diversity of the population to generalise findings effectively.

• Generalisation Challenges:

  • Psychology research often uses WEIRD samples: Western, Educated, Industrialised, Rich, and Democratic populations.

  • These samples may not represent other cultural, economic, or social groups, leading to skewed findings. Examples: Differences in moral reasoning, cognitive styles, or decision-making.

• Improving Representativeness:

  • Stratified Sampling: Ensures subgroups (e.g., gender, socio-economic status) are proportionally represented in the sample.

  • Example: If a class of 180 students has 90 chemistry students and 90 psychology students, a stratified sample of 40 participants will reflect these proportions (20 chemistry and 20 psychology).

Observational vs Experimental Designs

(Based on Observational vs Experimental Designs PDF)

• Observational Designs:

  • Definition: Researchers examine natural relationships between variables without manipulation.

  • Example: Studying the relationship between stress levels and junk food consumption.

  • Advantages: Ethical and practical for real-world settings; does not require intervention.

  • Limitations: Cannot infer causation due to lack of control over confounding variables.

• Experimental Designs:

  • Definition: Researchers manipulate one or more independent variables (IVs) to observe effects on dependent variables (DVs).

  • Example: Assigning participants to stress or relaxation conditions and measuring cortisol levels.

  • Advantages: Allows causal inferences by controlling for extraneous factors.

  • Limitations: May lack external validity (real-world applicability).

• Key Conditions for Causation:

• Co-Variation: The IV and DV must change together.

• Temporal Order: Changes in the IV must precede changes in the DV.

• Control of Confounds: Extraneous variables must be controlled to avoid spurious results.

Reliability and Validity

(Based on Reliability and Validity PDF)

• Reliability:

  • Inter-Rater Reliability: Consistency among multiple observers. Example: Different judges independently coding aggression in children.

  • Internal Consistency: Measures how well test items assess the same construct. Example: Survey questions on anxiety should correlate well.

  • Test-Retest Reliability: Stability of results over time. Example: Repeated IQ tests yielding similar scores.

• Validity:

  • Internal Validity: Confidence that changes in the DV are caused by the IV, not confounds.

  • External Validity: Generalisability of findings to other populations or settings.

  • Construct Validity: How well a measure represents the theoretical construct. Example: Using a validated scale to measure depression.

• Threats to Validity:

  • Selection Bias: Non-equivalent groups.

  • Placebo Effects: Participants’ expectations influence outcomes.

  • Maturation: Natural changes in participants over time.

  • Attrition: Dropouts that systematically differ from those who remain.

Factorial ANOVA

(Based on Factorial Designs PDF & Main and Interaction Effects PDF)

• Definition:

  • Used to examine the effects of two or more independent variables (factors) on a dependent variable.

• Key Terminology:

  • Main Effect: The independent impact of one IV on the DV.

  • Interaction Effect: When the effect of one IV depends on the level of another.

• Design Types:

  • Two-Way ANOVA: Two IVs. Example: Testing how diet type (low-carb, high-carb) and exercise intensity (low, high) affect weight loss.

  • Three-Way ANOVA: Three IVs. Example: Adding time of day to the above example.

• Advantages:

  • Increases statistical power by analysing multiple factors simultaneously.

  • Provides insights into complex interactions between variables.

• Interpreting Results:

  • Interaction effects are often more interesting and meaningful than main effects. Example: A high-fat diet might only lead to weight gain when combined with low exercise intensity.

Mixed ANOVA

(Based on Mixed ANOVA Theory PDF & Field, 2017: Chapter 16)

• Definition:

  • Combines between-subjects and within-subjects factors in a single design.

• Purpose:

  • Measures the effect of repeated conditions (within-subjects) across groups (between-subjects).

• Example:

  • Testing whether a new drug (control vs experimental group) reduces anxiety over three time points (pre-treatment, mid-treatment, post-treatment).

• Assumptions:

• Normality within each group/condition.

• Homogeneity of variance for between-subjects factors.

• Sphericity for within-subjects factors (corrected with Greenhouse-Geisser if violated).

Regression and the Method of Least Squares

(Based on Regression PDFs & Navarro & Foxcroft, 2022: Chapter 14)

• Regression Basics:

  • Models the relationship between variables using a straight line.

  • Equation: Y=b0+b1X+ϵY=b0​+b1​X+ϵ, where:

    • YY: Predicted DV.

    • b0b0​: Intercept.

    • b1b1​: Slope (change in YY per unit change in XX).

    • ϵϵ: Error term.

• Variance Explained:

  • r2r2: Proportion of variance in the DV explained by the IV. Example: r2=0.64r2=0.64 means 64% of variance in sales is explained by advertising budget.

• Method of Least Squares:

  • Minimises the sum of squared residuals (differences between observed and predicted values).

  • SST: Total variance in the DV.

  • SSM: Variance explained by the model.

  • SSR: Unexplained (residual) variance.

• Model Fit:

  • A good model has a high SSMSSM relative to SSRSSR, resulting in a significant F-ratio.