Multi- Group Designs

Multi-group design

An experimental design with more than two conditions; strictly, multiple conditions represent multiple levels of one IV.

Reasons to use multi-group designs

Test multiple hypotheses efficiently, investigate confounds, compare multiple controls, or assess relationships across levels of an IV.

Efficiency of multi-group designs

Conditions can be reused across comparisons, allowing several hypotheses to be tested with fewer groups than separate two-group studies.

Number of pairwise comparisons

With k conditions, the number of unique pairs is k(k − 1)/2; three conditions allow three comparisons and four allow six.

Costs of multi-group designs

They require more participants than a single two-group study and involve more complicated analyses.

Multiple control conditions

Comparing one experimental condition with different standards to clarify what produces its effect.

True control condition

A condition receiving no manipulation.

Conceptual control condition

A condition receiving a neutral manipulation that accounts for some aspect of the experimental experience.

Nonspecific treatment effects

Effects resulting from features such as interacting with a therapist rather than the specific therapeutic technique.

Multiple-control example

Compare CBT with no treatment to assess whether it helps, and with nonspecific talk therapy to assess whether its specific approach adds benefit.

Controlling for multiple confounds

Include experimental conditions that differ on potential confounds while keeping the rest of the manipulation the same.

Multiple experimental conditions with one control

Compare each experimental condition with the control and with the other experimental conditions to assess possible confounds.

Multiple levels of one IV

Manipulating one IV at several amounts to investigate the direction and potentially the magnitude or pattern of its relationship with the DV.

Designing multi-group manipulations

Manipulations should be comparable across conditions and capable of producing the intended levels of the IV; pilot tests and manipulation checks remain important.

Assigning levels

The researcher selects particular IV values for conditions, such as 0, 6, and 12 months of therapy.

Limitation of assigning levels

Chosen values may introduce confounds, and equal numerical spacing may not produce equal changes in the psychological construct.

Sampling levels

Define a range of possible IV values and randomly sample levels from that range.

Benefit of sampling levels

Reduces reliance on a few arbitrarily chosen levels when estimating the IV–DV relationship.

Limitation of sampling levels

Usually requires many conditions and a very large sample; often impractical in psychological research.

Levels of measurement in multi-group designs

Meaningfully ordered or quantified IV levels may permit ordinal, interval, or ratio interpretations rather than only nominal condition labels.

Absence of manipulation versus absence of construct

Receiving none of a manipulation does not necessarily mean having none of the underlying psychological construct.

Error of inference

An incorrect decision about whether to reject the null hypothesis.

Significance threshold (α)

The criterion used to decide whether a p-value is sufficiently small to reject the null hypothesis.

Rejecting the null hypothesis

When p < α, reject the null; the result provides evidence against the null but does not automatically establish the proposed explanation.

Failing to reject the null hypothesis

When p ≥ α, evidence is insufficient to reject the null; this does not prove the null is true.

Type I error

Rejecting a true null hypothesis; a false positive.

Type II error

Failing to reject a false null hypothesis; a false negative.

Alpha and Type I error

An α of .05 allows a 5% long-run probability of rejecting a true null for a test under its assumptions.

Reducing alpha

A stricter significance threshold reduces Type I error risk but can reduce power and increase Type II error risk.

Statistical power

The probability of detecting an effect by correctly rejecting a false null hypothesis.

Power notation

Conventionally, power = 1 − β, where β is the probability of a Type II error; the lecture slide reverses these labels.

Factors affecting power

True effect size, sample size, and the significance threshold.

Effect size and power

Larger true effects are easier to detect.

Sample size and power

Larger samples generally increase the ability to detect real effects.

Significance threshold and power

A less restrictive threshold increases power but also increases Type I error risk.

Multiple comparisons

Conducting several statistical tests; without appropriate control, the chance of false-positive findings across the set can increase.

Pairwise comparisons

Comparisons involving two groups at a time.

Correcting for multiple comparisons

Adjusting testing procedures or significance thresholds to control false positives across multiple tests.

Familywise error rate

The probability of making at least one Type I error within a defined family of tests.

False discovery

A statistically significant result that rejects a true null hypothesis.

False discovery rate (FDR)

The expected proportion of false discoveries among the results declared significant.

Bonferroni correction

Divide the desired familywise significance level by the number of tests in the family.

Bonferroni example

For two comparisons with a familywise α of .05, use .025 as the threshold for each test.

Bonferroni tradeoff

It controls familywise error but can be conservative, reducing power.

Benjamini–Hochberg correction

A procedure controlling FDR by ranking p-values and comparing them with rank-based thresholds.

Benjamini–Hochberg threshold

For rank i among m tests and target FDR q, the threshold is (i/m) × q; find the largest qualifying rank and reject through that rank.

Omnibus test

A single test assessing whether there is evidence of at least one difference among the groups.

Limitation of an omnibus test

A significant result indicates that some difference exists but does not identify which groups differ.

Analysis of variance (ANOVA)

An omnibus test comparing variation between conditions with variation within conditions.

When to use a one-way ANOVA

To compare independent groups defined by one categorical IV on an interval- or ratio-level DV.

Significant ANOVA

Provides evidence that at least one group mean differs from another; follow-up comparisons locate the differences.

Kruskal–Wallis test

A rank-based alternative discussed for comparing independent groups when the outcome is ordinal.

Chi-squared test of independence

A test assessing whether categorical variables are associated by comparing observed category counts with counts expected under independence.

Chi-squared in multi-group designs

Used when the DV is nominal to assess whether outcome-category frequencies differ across conditions.

Chi-squared goodness-of-fit test

Tests whether the category frequencies of one variable match a specified expected distribution.

Degrees of freedom

The amount of independently varying information available for a particular statistical test.

Between-groups degrees of freedom

For a one-way ANOVA with k groups, the between-groups degrees of freedom are k − 1; three groups give two.

Planned contrasts

Specific comparisons chosen before examining the results, often comparing meaningful sets of conditions.

Planned-contrast example

Compare treatment plus placebo with no treatment, then compare treatment with placebo.

Post-hoc tests

Follow-up comparisons selected after the initial analysis to determine where group differences occur.

Planned contrasts versus post-hoc tests

Planned contrasts are specified in advance; post-hoc comparisons are chosen after the initial analysis.

Fisher's Least Significant Difference (LSD)

A follow-up procedure named in the slides that uses pairwise t-tests after a significant omnibus F-test; its protection against multiple-testing error is limited.

Tukey's Honestly Significant Difference (HSD)

A post-hoc procedure for comparing all pairs of group means while controlling familywise error.