Experimental Research Methods, Validity Threats, and ANOVA

Comprehensive review flashcards covering experimental design, threats to internal validity, one-way ANOVA (between and repeated measures), factorial ANOVA, post-hoc procedures, and graphical interpretation.

What distinguishes a quasi-experiment from a true experiment?

In a true experiment, participants are randomly assigned to the levels of the independent variable; in a quasi-experiment, groups cannot be randomly assigned because the independent variable is a subject variable that participants bring with them.

Which threat to internal validity applies exclusively to independent-groups (between-subjects) designs?

Selection effect (when the two independent variable groups have systematically different kinds of participants in them).

Which threat to internal validity applies exclusively to repeated-measures (within-subjects) designs?

Order effect (when the effect of the independent variable is confounded with carryover from one level to another, or with practice, fatigue, or boredom).

What experimental design solution is used to prevent maturation, history, and regression to the mean threats?

Adding an adequate comparison or control group.

How can a researcher prevent attrition from compromising internal validity?

Remove the dropouts' initial scores from the pre-test group data.

What are the three essential criteria required to establish a causal claim?

Covariance (the variables are associated), temporal precedence (the causal variable comes first), and internal validity (elimination of plausible alternative explanations or confounds).

What is the difference between a control variable (constant variable) and a confound variable?

A control variable is a potential extraneous factor that the researcher recognizes and holds constant experimentally or statistically, whereas a confound is an unrecognized or uncontrolled factor that varies systematically with the independent variable and provides an alternative explanation for changes in the dependent variable.

In experimental designs, how do comparison, control, and placebo groups differ?

A comparison group provides a baseline contrast (another treatment or level); a control group receives no treatment or a neutral condition; and a placebo group receives an inert treatment while believing they received the active intervention.

What is the distinction between complete counterbalancing and partial counterbalancing using a Latin square?

Complete counterbalancing presents every possible order of conditions to different participant subgroups (e.g., $6$ orders for $3$ conditions), whereas a Latin square selects a formal subset of orders such that every condition appears at each ordinal position and follows every other condition without duplicating in any row or column.

How many possible presentation orders exist in a fully counterbalanced repeated-measures experiment with $3$ conditions?

$6$ possible presentation orders: $1-2-3$, $2-3-1$, $3-1-2$, $3-2-1$, $2-1-3$, and $1-3-2$.

What is the primary function of a manipulation check in experimental research?

An extra measure administered to assess whether the experimental manipulation of the independent variable actually worked and was perceived or detected as intended by the participants.

What two major structural design problems can cause a null effect (failing to find a significant difference)?

Between-groups differences that are too small (due to weak manipulations, insensitive measures, ceiling/floor effects, or counteracting confounds) or within-groups variability that is too large (due to measurement error, individual differences, or situation noise).

What are three primary sources of noise that increase within-groups variability?

Measurement error, high levels of individual differences, and situation noise (environmental distractions).

What is statistical power, and what are three ways to increase it?

Power is the probability of finding an effect when one truly exists in the population; it is increased by adding more participants (larger sample size), using stronger manipulations, and improving experimental control to reduce variability.

What is the difference between a Type I error and a Type II error?

A Type I error is a false positive (rejecting a true null hypothesis), whereas a Type II error is a false negative (failing to reject a false null hypothesis).

What formula calculates the overall experimentwise probability of committing at least one Type I error across $m$ independent comparisons at an alpha level of $\alpha$?

$$p(\text{Type I error}) = 1 - (1 - \alpha)^m$$

Who is the British statistician, pictured here, credited with developing the Analysis of Variance (ANOVA)?

Sir Ronald A. Fisher

Why is ANOVA used instead of conducting multiple independent t-tests when comparing three or more group means?

ANOVA tests differences among three or more groups simultaneously while holding the experimentwise alpha level constant (e.g., $\alpha = .05$), preventing alpha inflation and excess Type I error.

What are three distinct distributional properties of the F-ratio statistic in ANOVA?

The F-ratio is always positive (cannot be less than zero), peaks around $1$, and has a positively skewed distribution.

What are the formulas for the degrees of freedom in a one-way between-subjects ANOVA with $k$ groups and $N$ total observations?

$df_{\text{Between}} = k - 1$, $df_{\text{Within}} = N - k$, and $df_{\text{Total}} = N - 1$.

What are the benchmark values for interpreting partial eta squared ($\eta_p^2$) effect sizes in ANOVA?

Small = $.01$, Medium = $.06$, and Large = $.14$.

In Conti and Musty's (1984) study on THC dose and rat exploratory activity, what were the omnibus ANOVA results?

$F(2, 26) = 10.35, p < .05$, with an effect size of $\eta^2 = .443$, where the $0.5\,\mu\text{g}$ group showed significantly more activity ($M = 60.33$) than the placebo ($M = 34.00$) and $2\,\mu\text{g}$ ($M = 38.10$) groups.

Under what specific circumstance should post-hoc tests be conducted following an omnibus ANOVA?

Post-hoc tests are only conducted if the omnibus F-statistic is statistically significant; if the omnibus F is not significant, the researcher must stop and disregard post-hoc comparisons.

What post-hoc test is specifically recommended for repeated-measures ANOVA to protect against Type I error?

The Bonferroni procedure.

What are the four primary statistical assumptions of a one-way within-subjects (repeated-measures) ANOVA?

A continuous DV (interval or ratio), normal sampling distribution, homogeneity of variance across condition differences (sphericity), and dependent observations.

How is the sum of squares for error ($\text{SS}_{\text{Error}}$) calculated in a one-way repeated-measures ANOVA?

$$\text{SS}_{\text{Error}} = \text{SS}_{\text{Within}} - \text{SS}_{\text{Subject}}$$

What are the degrees of freedom formulas for a one-way repeated-measures ANOVA with $n$ participants and $k$ levels of the independent variable?

$df_{\text{Between}} = k - 1$, $df_{\text{Subjects}} = n - 1$, $df_{\text{Error}} = (n - 1)(k - 1)$, and $df_{\text{Total}} = N - 1$.

What formula defines the Bonferroni corrected alpha level ($\alpha'$) for $c$ pairwise comparisons?

$$\alpha' = \frac{\alpha}{c}$$

In the tortilla chip brand repeated-measures study with $23$ participants evaluating Mission, Tostitos, and Kroger, what were the ANOVA findings?

$F(2, 44) = 3.82, p = .03, \eta_p^2 = .15$, with Mission ($M = 78.57$) rating significantly higher than Kroger ($M = 62.00$), while Tostitos ($M = 71.61$) did not differ significantly from either.

How do you determine the total number of conditions from factorial notation such as $2 \times 2 \times 3$?

Multiply the numbers representing the levels of each factor together ($2 \times 2 \times 3 = 12$ conditions across $3$ independent variables).

In a factorial matrix of means, how are main effects identified?

The main effect of Factor A is assessed by comparing the column marginal means, and the main effect of Factor B is assessed by comparing the row marginal means.

In the Papa's Perfect Pizza study, what was the factorial design notation, and how many conditions were evaluated?

A $3 \times 5$ factorial design ($3$ crust types: hand tossed, thick, thin; $5$ topping types: cheese, pepperoni, sausage, veggie, everything) resulting in $15$ conditions.

In a two-way factorial between-subjects ANOVA with $k_A$ levels of Factor A, $k_B$ levels of Factor B, and $n$ participants per cell, what is the formula for $df_{\text{Error}}$?

$$df_{\text{Error}} = k_A \times k_B \times (n - 1)$$

In the provided line graph testing word recall by gender and music condition, what statistical effect is demonstrated by the crossing lines?

A significant $A \times B$ interaction (music improves word recall for men from $6$ to $12$, but decreases recall for women from $20$ to $10$).

In the $2 \times 2$ factorial analysis of weight loss as a function of diet (Normal vs. Low Cal) and exercise (No vs. Yes), what was the interaction effect?

The effect of exercise was greater for those on a low-calorie diet ($15\,\text{lbs}$ lost from exercise) than for those on a normal diet ($10\,\text{lbs}$ lost from exercise).

What constitutes a three-way interaction in a factorial design?

A three-way interaction occurs when the nature or pattern of the two-way interaction between two independent variables depends on the level of a third independent variable.

What is the general graphing rule of thumb for selecting between a bar graph and a line graph?

Use a bar graph when the independent variable is nominal or discrete (discontinuous categorical data), and use a line graph when the independent variable is ordinal or continuous (e.g., changes over time, dosage, age progression).

In the one-way between-subjects study examining study hours ($1\text{--}3$ hours, $3\text{--}6$ hours, and $>6$ hours per week) and college GPA ($N = 64$), what did the ANOVA reveal?

$F(2, 61) = 6.25, p = .003$, with post-hoc tests indicating that students studying $>6$ hours ($M = 3.39$) and $3\text{--}6$ hours ($M = 3.17$) had significantly higher GPAs than those studying $1\text{--}3$ hours ($M = 2.74$).