Experimental Design & Post Hoc Tests

Practice flashcards covering principles of experimental design, completely randomized designs (CRD), randomized complete block designs (RCBD), ANOVA assumptions, and post hoc tests (Tukey's HSD and Scott-Knott).

Confounding Risk (Observational Self-Selection)

A bias occurring when experimental units self-select into treatment groups rather than being randomly assigned, allowing underlying participant characteristics (such as motivation or health consciousness) to independently influence recovery and distort the apparent treatment effect.

Type I Error ($\alpha$)

The error that occurs when the null hypothesis ($H_0$) of equal treatment means is true, but is rejected, occurring with probability $\alpha$.

Type II Error ($\beta$)

The error that occurs when the null hypothesis ($H_0$) is false (means truly differ), but fails to be rejected, occurring with probability $\beta$.

Statistical Power

The probability of correctly rejecting a false null hypothesis ($H_0$), equal to $1 - \beta$.

Balanced Design

An experimental layout with an equal number of replicates per treatment group, which maximizes statistical power for a fixed total sample size and ensures sums of squares are independent of the order variables enter the model.

Degrees of Freedom for Error ($\text{SSE}$)

In a balanced one-way ANOVA with $k$ treatments and total sample size $N$, the degrees of freedom given by $N - k$, representing the remaining pieces of information after estimating $k$ group means.

Blocking Factor (RCBD)

A nuisance variable (such as location, batch, or time period) included in an experiment specifically to account for and isolate a known source of variability from the error term, thereby improving the precision of treatment comparisons.

Treatment-by-Block Interaction

A condition in an RCBD where the relative effectiveness of treatments changes depending on the specific block, violating the standard additive model ($Y_{ij} = \mu + \tau_i + \beta_j + \varepsilon_{ij}$) and inflating the error sum of squares ($\text{SSE}$).

Treatment-by-Block Interaction Plot

A diagnostic plot showing non-parallel, crossing lines across blocks, visually indicating that treatment effects depend on the block and that the additivity assumption is violated.

Per-Comparison Error Rate

The probability of committing a Type I error for a single individual pairwise comparison, typically set at $\alpha = 0.05$.

Familywise Error Rate

The probability of making at least one Type I error across an entire set or family of pairwise comparisons performed together.

Bonferroni Adjustment

A post hoc correction method that controls the familywise error rate by dividing the nominal significance level $\alpha$ by the total number of pairwise comparisons ($\binom{k}{2}$), establishing a stricter threshold for individual comparisons.

Multiple Comparisons Problem

The rapid inflation of the overall familywise Type I error rate that occurs when conducting multiple independent two-sample tests at $\alpha = 0.05$ without post hoc corrections.

Tukey's HSD Test

A post hoc procedure specifically designed to control the familywise error rate at $\alpha$ while performing all pairwise comparisons between treatment means.

Scott-Knott Method

A hierarchical clustering post hoc algorithm that partitions ordered treatment means into distinct, non-overlapping groups by maximizing the between-group sum of squares at each split.

Overlapping Groups (Tukey's HSD)

A letter-grouping pattern (such as "ab") in post hoc tests where a treatment is not significantly different from two other treatments that are significantly different from one another, reflecting statistical ambiguity.

Cohen's $f$

An effect size measure for ANOVA representing the standardized dispersion of treatment means relative to the within-group error standard deviation, defined as $f = \frac{\sigma_{\text{means}}}{\sigma}$.

Cost of Ignoring Blocks

The consequence of omitting an influential blocking factor, which causes block variability to be absorbed into $\text{MSE}$, deflating the $F$-statistic ($F = \frac{\text{MSTr}}{\text{MSE}}$) and increasing Type II error risk.

Three ANOVA Assumptions

The required statistical assumptions for ANOVA: normality of errors (checked via Q-Q plots and the Shapiro-Wilk test), homogeneity of variance (checked via Bartlett's or Breusch-Pagan tests), and independence of errors (checked via residual-vs-order plots, lag plots, or the Durbin-Watson test).

Incomplete RCBD

An RCBD where at least one treatment-block cell is missing, breaking the balance and making sums of squares order-dependent (Type I vs Type II vs Type III).

Relative Efficiency ($\text{RE} < 1$)

An outcome where the relative efficiency of an RCBD is below $1$, indicating that blocking on an uninformative or noise factor was less efficient than a CRD would have been due to lost degrees of freedom without a compensating reduction in $\text{MSE}$.