Experimental Design, Hypothesis Testing, and Single-Factor ANOVA

Experimental Design Naming Conventions and Core Principles

• Rigorous Reporting Standards:

  • Experimental reporting operates under strict grading criteria comparable to a formal laboratory write-up, requiring exact components and standard terminology.

  • Full grammatical sentences are strictly mandatory for conclusions and hypothesis statements.

  • Full sentences are not required for simple listing tasks on exams, such as identifying independent variables.

• Three-Phrase Design Naming Structure (Single-Factor Framework):

  • Prior to covering factorial designs, all experimental design labels require a three-phrase formula:

    • Phrase 1 (Factor Count): Must be stated as $\text{Single factor}$, indicating the presence of exactly one independent variable.

    • Phrase 2 (Condition Count): Must be specified as either $\text{Two level}$ (two experimental conditions) or $\text{Multilevel}$ (three or more experimental conditions).

    • Phrase 3 (Design Classification): Must designate one of four foundational design types:

      • Independent groups

      • Matched groups

      • Ex post facto

      • Repeated measures

• Hypothesis Construction Requirements:

  • Null Hypothesis ($H_0$): Must state that there is no difference on the dependent variable between the specific levels or conditions of the independent variable.

  • Alternative Hypothesis ($H_1$): Must state that there is a difference on the dependent variable between the specific levels or conditions of the independent variable.

  • Mandatory Elements:

    • Both the dependent variable and every condition of the independent variable must be explicitly identified.

    • Omission of baseline or control conditions (e.g., stating only that a measure changes after an intervention without referencing the pre-intervention state) invalidates the hypothesis.

• Homogeneity of Variance Principle:

  • Definition: The assumption that the variability, spread, or dispersion of scores around the mean is equivalent across all treatment groups (characterized as equally distributed variability).

  • Within-Subjects Exemption: Homogeneity of variance is not tested in within-subjects (repeated measures) designs. Because the identical individuals participate across all conditions, extraneous participant variables (such as fatigue, hunger, attention span, and intrinsic traits) are held constant, ensuring identical baseline variability across conditions by definition.

  • Between-Subjects Requirement: Homogeneity of variance must be evaluated in between-subjects designs because different individuals constitute each group, introducing individual differences that may distort variance equality.

Experiment 3: Stair Climbing and Heart Rate (Within-Subjects Design)

• Variable Identification and Operationalization:

  • Independent Variable: Physical exertion via stair climbing, manipulated within-subjects across two conditions: before climbing stairs (baseline control) and after climbing stairs.

  • Manipulation Status: Inherently manipulated because the investigator assigns the order and exposure of the conditions to the participants.

  • Dependent Variable: Heart rate, measured on a ratio scale of measurement because it possesses a true, absolute theoretical zero (representing an absence of heart rate, even if fatal).

• Design Classification:

  • $\text{Single factor, two level repeated measures design}$.

• Hypotheses Formulation:

  • Null Hypothesis ($H_0$): There is no difference in heart rate between before climbing stairs and after climbing stairs.

  • Alternative Hypothesis ($H_1$): There is a difference in heart rate before climbing stairs as compared to after climbing stairs.

• Statistical Analysis:

  • A repeated measures $t$-test (paired-samples $t$-test), selected because the layout consists of a single factor with two repeated conditions.

• Empirical Data and Output Interpretation:

  • Baseline (Before Climbing): Mean ($M$) = $91$, Standard Deviation ($SD$) = $3.97$ (or $3.9$).

  • Post-Intervention (After Climbing): Mean ($M$) = $106$.

  • Test Statistics: $t(8) = 4.17$, two-tailed significance $p = 0.003$.

  • Two-Tailed Standard: Statistical testing defaults to a non-directional (two-tailed) test to remain sensitive to both increases and decreases; bidirectional evaluation represents standard scientific practice.

  • Significance Evaluation:

    • The criterion alpha level is $\alpha = 0.05$.

    • Because $p = 0.003 < 0.05$, the null hypothesis is rejected.

    • A $p$-value of $0.003$ signifies that there is only a $0.3\%$ probability that the observed difference occurred by chance or natural physiological fluctuation in the absence of an experimental effect.

• Conclusion Writing Requirements:

  • Conclusions must specify the direction of the effect rather than merely stating that a difference exists.

  • Must report descriptive statistics ($M$ and $SD$) formatted in APA style alongside the test statistic and exact probability.

  • Formal Statement: Heart rate significantly increased after climbing stairs ($M = 106$) compared to before climbing stairs ($M = 91$, $SD = 3.97$), $t(8) = 4.17$, $p = 0.003$.

Experiment 2: GRE Study Sessions (Two-Level Independent Groups Design)

• Variable Identification:

  • Independent Variable: Study session format, consisting of two levels: new study session versus old study session.

  • Manipulation Status: Manipulated between-subjects via random assignment of different participants to each session condition.

  • Dependent Variable: GRE score, classified on a ratio scale of measurement because a score of zero theoretically represents the absence of correct responses.

• Design Classification:

  • $\text{Single factor, two level independent groups design}$.

• Hypotheses Formulation:

  • Null Hypothesis ($H_0$): There is no difference in GRE scores between the new study session and the old study session.

  • Alternative Hypothesis ($H_1$): There is a difference in GRE scores between the new study session and the old study session.

• Statistical Analysis:

  • An independent groups $t$-test (independent samples $t$-test).

• Evaluation of Homogeneity of Variance:

  • Evaluated via Levene's Test for Equality of Variances.

  • Output Statistic: $F$ statistic with a significance value of $p = 0.253$.

  • Decision Criterion:

    • Unlike standard hypothesis testing where $p < 0.05$ is desired, evaluating assumptions requires $p > 0.05$.

    • A significance level of $p = 0.253 > 0.05$ demonstrates that the variances between the groups are not significantly different.

    • The assumption of homogeneity of variance is not violated, allowing the researcher to assume equal variances and proceed to interpret the standard $t$-test output.

    • If $p < 0.05$, the assumption would be violated, precluding standard interpretation.

  • Formal Statement: The data do not violate the assumption of homogeneity of variances, $F$, $p = 0.253$; the variances across the groups do not differ.

• Empirical Conclusion:

  • Inspection of the top row ("Equal variances assumed") indicates a two-sided significance value of $p < 0.001$.

  • Because $p < 0.001 < 0.05$, the null hypothesis is rejected.

  • Formal Statement: GRE scores after the new study session were significantly higher than GRE scores after the old study session, with results reported via the appropriate $t$-statistic, degrees of freedom, and $p < 0.001$.

Decision Errors: Type I and Type II Errors

• Nature of Inferential Errors:

  • Because empirical conclusions are drawn from sample data in a single experiment, researchers risk reaching an erroneous inference.

• Type I Error (False Positive):

  • Definition: Concluding that an effect, relationship, or difference exists when, in reality, there is no true difference.

  • Probability: The nominal probability of making a Type I error is defined by the a priori alpha level, set at $\alpha = 0.05$ ($5\%$).

  • Application to Findings: Whenever an experiment yields a statistically significant difference ($p < 0.05$), the researcher risks committing a Type I error.

  • Required Explanatory Phrasing: The researcher concluded that there was a significant increase in heart rate (or higher GRE scores); however, it may be the case that no true difference exists and the outcome was an artifact of chance.

• Type II Error (False Negative):

  • Definition: Concluding that there is no difference or effect when a true difference actually exists.

  • Application to Findings: Whenever an experiment fails to reach statistical significance ($p \ge 0.05$), the researcher risks committing a Type II error.

Multilevel Experimental Designs and Analysis of Variance (ANOVA)

• Definition and Architecture:

  • Multilevel designs incorporate three or more levels or conditions of a single independent variable.

  • Can be arranged across any foundational design type: independent groups, matched groups, ex post facto, or repeated measures.

• Primary Benefits of Multilevel Designs over Two-Level Designs:

  • Ability to Discover Nonlinear Effects:

    • Two data points can only define a straight line, forcing the interpretation of an effect to be linear.

    • Caffeine and Response Time Example:

      • Testing $0\text{ mg}$ versus $100\text{ mg}$ of caffeine yields two connected points, implying a continuous linear rule: more caffeine produces faster response times.

      • Expanding to four conditions ($0\text{ mg}$, $50\text{ mg}$, $100\text{ mg}$, and $500\text{ mg}$) reveals a nonlinear, curvilinear function:

        • $0\text{ mg}$ to $50\text{ mg}$: No observable increase in response speed (a threshold of approximately $100\text{ mg}$, or one full cup of coffee, is required for speed benefits).

        • $100\text{ mg}$ to $500\text{ mg}$: Extreme doses cause physiological distress (jitteriness), increasing reaction time and slowing performance down.

  • Ability to Rule Out Alternative Explanations:

    • Context in Reading Comprehension Study (Bransford & Johnson Style):

      • Target text contains confusing, ambiguous statements: "If the balloons popped, the sound wouldn't be able to carry since everything would be too far away from the correct floor."

      • Two-level structure: Reading the passage with no context versus seeing a clarifying visual context picture (a man playing guitar connected by a microphone and speaker suspended by balloons outside an upper-floor window) prior to reading. The visual context group shows superior recall.

      • Alternative hypotheses tested via additional conditions:

        • Partial context: Providing half the illustration (showing balloons and speakers without the full scene) fails to aid memory.

        • Delayed context: Showing the full visual context after reading the passage fails to aid memory.

        • Repetition control: Rereading the passage twice without illustrations fails to reproduce the comprehension benefits.

      • Outcome: Multilevel testing isolates the necessity of prior integrated contextual framing rather than mere partial hinting, delayed exposure, or repeated reading.

• Statistical Logic of Single-Factor ANOVA:

  • Running multiple pairwise $t$-tests directly across several groups inflates the familywise (experiment-wise) Type I error rate.

  • Omnibus $F$-Test: A single-factor (one-way) Analysis of Variance evaluates whether an overall difference exists somewhere among the group means without specifying which specific pairs differ.

  • Post-Hoc Pairwise Tests: If the omnibus ANOVA yields a statistically significant $F$-statistic ($p < 0.05$), independent pairwise comparisons (follow-up $t$-tests) are conducted across all group permutations.

  • Combinatorial Expansion: For four conditions ($0\text{ mg}$, $50\text{ mg}$, $100\text{ mg}$, $500\text{ mg}$), a total of six pairwise comparisons are executed:

    • $0\text{ mg}$ vs. $50\text{ mg}$

    • $0\text{ mg}$ vs. $100\text{ mg}$

    • $0\text{ mg}$ vs. $500\text{ mg}$

    • $50\text{ mg}$ vs. $100\text{ mg}$

    • $50\text{ mg}$ vs. $500\text{ mg}$

    • $100\text{ mg}$ vs. $500\text{ mg}$

  • Each comparison possesses an individual $p$-value and requires an independent directional conclusion sentence.

Practice Experiment: Boredom and Temporal Estimation

• Experimental Design and Protocol:

  • Total sample size of $N = 21$ students randomly assigned across three equal groups ($n = 7$ per condition).

  • All experimental tasks lasted an identical, objective duration of $10\text{ minutes}$.

  • Conditions (Independent Variable Levels):

    • Condition 1: Sat quietly in an empty room and watched wet paint dry.

    • Condition 2: Listened to an audio recording of an academic lecture on paint drying.

    • Condition 3: Listened to an audio lecture on pandas.

  • Dependent Variable: Perceived time passage (temporal duration estimate), measured on a ratio scale.

  • Design Classification: $\text{Single factor, multilevel, independent groups design}$.

  • Selected Statistical Analysis: Single-factor analysis of variance (ANOVA) with follow-up $t$-tests.

• Formal Hypotheses:

  • Null Hypothesis ($H_0$): There is no difference in time estimates comparing watching paint dry, listening to a recording of paint drying, and listening to a recording of pandas.

  • Alternative Hypothesis ($H_1$): There is a difference somewhere in time estimates between watching paint dry, listening to a paint lecture, and listening to a panda lecture.

• SPSS Execution and Descriptive Statistics:

  • Descriptive statistics tables must be verified by selecting "Descriptives" within the options menu under $\text{Analyze} > \text{Compare Means} > \text{One-Way ANOVA}$.

  • Observed Group Means:

    • Group 1 (Watching Paint Dry): $M = 15.00\text{ minutes}$.

    • Group 2 (Paint Lecture Audio): $M = 7.57\text{ minutes}$.

    • Group 3 (Panda Lecture Audio): $M = 7.57\text{ minutes}$.

• Homogeneity of Variance Assessment:

  • Levene's Test significance value: $p = 0.974$.

  • Because $p = 0.974 > 0.05$, there is no significant difference among group variances.

  • The assumption of homogeneity of variance is preserved, permitting standard ANOVA interpretation.

• Omnibus ANOVA Table Interpretation:

  • ANOVA features two distinct degrees of freedom: between-groups degrees of freedom ($df_1 = 2$) and within-groups (error) degrees of freedom ($df_2 = 18$), formatted as $F(2, 18)$.

  • Statistical significance: $p < 0.001$.

  • Omnibus Result: There is a significant difference in time perceived between the watching paint group, paint lecture group, and panda lecture group, $F(2, 18)$, $p < 0.001$.

• Multiple Comparisons (Post-Hoc Pairwise Comparisons):

  • Post-hoc output presents a pairwise matrix containing redundant reciprocal comparisons (e.g., Group 1 vs. Group 2 and Group 2 vs. Group 1).

  • Pairwise Comparisons Breakdown:

    • Group 1 vs. Group 2 (Watching Paint vs. Paint Lecture): Significant difference, $p < 0.001$. For the watching paint task, participants perceived significantly more time passing ($M = 15.00$) than for the paint lecture task ($M = 7.57$).

    • Group 1 vs. Group 3 (Watching Paint vs. Panda Lecture): Significant difference, $p < 0.001$. For the watching paint task, participants perceived significantly more time passing than for the panda lecture task ($M = 7.57$).

    • Group 2 vs. Group 3 (Paint Lecture vs. Panda Lecture): No significant difference, $p = 1.000$. There was not a significant difference in perceived time between the paint lecture and the panda lecture.

  • APA Style In-Text Rule: When writing a continuous paragraph, reporting the mean and standard deviation the first time a group is introduced satisfies reporting standards; repeating them in subsequent sentences within the same paragraph is optional.

Questions & Discussion

• Confidence Levels and Error Probabilities on $t$-Tests:

  • Student Prompt: When the observed $p$-value falls substantially below the standard cutoff (e.g., $p = 0.003$ versus $0.05$), does the researcher still state that there is a $5\%$ chance of making a Type I error?

  • Clarification: Yes. The nominal risk of committing a Type I error is fixed a priori by the chosen alpha criterion ($\alpha = 0.05$). While researchers often casually associate smaller observed $p$-values with lower risk of being wrong, the $p$-value strictly represents the probability of observing sample data that extreme given a true null hypothesis, not the post-experiment probability that the conclusion itself is false.

  • Manipulating sample sizes or testing parameters can readily adjust observed $p$-values downward without altering the foundational familywise error parameters.

In-Class Lab Exercise: M&M Candy Variety and Count

• Empirical Investigation:

  • Evaluating whether the physical characteristics and size across different candy varieties systematically affect the quantity packed into standardized bag weights.

  • Varieties Evaluated: Plain, Peanut, Pretzel, Peanut Butter, and Caramel.

  • Underlying Hypothesis: Because candy bags are packaged according to weight rather than unit count, larger or denser candies (e.g., Peanut or Pretzel) will yield significantly lower counts per bag compared to smaller varieties (e.g., Plain).

  • Task Execution: Students tally total units within individual bags across varieties to assemble empirical data for a single-factor multilevel ANOVA.