Comprehensive Study Guide: Experimental Variables, Levels of Processing, and Analysis of Variance (ANOVA)

Classification of Variables in Experimental Design

• Independent Variables (Fully Manipulated Variables):

  • Definition: Variables that are completely regulated and manipulated by the experimenter.
  • Essential Criterion: Requires random assignment of participants to experimental conditions or groups.
  • Example: Assigning individuals who enter a study randomly to either a drug condition or a no-drug condition.
  • Purpose: Ensures that pre-existing differences between participants do not systematically bias group comparisons.

• Quasi-Independent Variables:

  • Definition: Variables that resemble independent variables in structural comparisons but lack random assignment.
  • Characteristics: Used when measuring events, behaviors, or groupings that have already occurred naturally; the experimenter cannot dictate or manipulate condition assignment.
  • Example: Investigating seat belt usage in traffic fatalities. Researchers cannot ethically or practically assign individuals to wear seat belts or to die in a vehicle crash; they can only measure post-hoc whether a seat belt was worn and whether the individual survived.
  • Statistical Treatment: Generally treated identically to fully manipulated independent variables in statistical analysis. However, the distinction becomes relevant in specific modeling decisions, such as selecting fixed versus random model nodes.
  • Methodological Distinction: In academic experimental psychology, researchers strictly differentiate between true independent variables (fully manipulated with random assignment) and quasi-independent variables (pre-existing groups).

• Subject Variables:

  • Definition: Measured individual characteristics that participants naturally possess when entering a study.
  • Characteristics: Traits that cannot be assigned or manipulated by the experimenter ("they come to you that way").
  • Example: Classifying participants into groups based on height, age, or pre-existing education levels.
  • Application: Examined to evaluate group differences across non-manipulated human attributes.

• Summary of Variable Distinctions:

  • The presence or absence of random assignment of participants to conditions serves as the definitive boundary between a true independent variable and a quasi-independent variable or subject variable.

Model Experimental Study: Level of Processing and Memory Recall

• Research Question and Variable Identification:

  • Research Question: Does the level of mental processing during exposure affect an individual's ability to recall information?
  • Independent Variable ($IV$): Level of processing (manipulated across distinct operational levels).
  • Dependent Variable ($DV$): Memory recall, operationally measured as the total number of words recalled out of a $27$-word list.

• Experimental Procedure:

  • Task Structure: Every participant completes three total passes through a list of $27$ words. Upon reaching the end of the list, the participant returns to the top and repeats the procedure until three full exposures are complete.
  • Total Word Exposures: All participants across all conditions are exposed to the exact same $27$ words exactly three times ($27 \times 3$ total exposures).
  • Experimental Design: Between-groups (between-subjects) design. Each participant is randomly assigned to only one processing condition.

• Sample Size Notation and Parameters:

  • Grand Total Sample Size ($N$): $N = 50$ total participants across the entire study.
  • Subgroup Sample Size ($n$): $n = 10$ participants per experimental condition.
  • Group Breakdown: $5 \text{ groups} \times 10 \text{ participants per group} = 50 \text{ total participants}$.
  • Notation Standard: Capital $N$ represents the overall sample size across all conditions; lowercase $n$ represents the sample size within an individual subgroup. Subgroup sizes do not strictly need to be equal, although equal sample sizes are ideal.

• Operational Description of the Five Processing Conditions (Lowest to Highest Level):

  • 1. Counting Condition (Lowest Processing Level):
  • Task: Participants read through the list of $27$ words three times and count the number of letters in each word.
  • Processing Depth: Minimal cognitive processing; participants do not process word meaning or semantic content, only structural letter counts.
  • 2. Rhyming Condition:
  • Task: Participants pass through the list three times and generate a rhyming word for each word on the list.
  • Processing Depth: Phonological processing; requires evaluation of word acoustics.
  • 3. Adjective Condition:
  • Task: Participants generate an appropriate adjective to modify each word on the list across three exposures.
  • Processing Depth: Semantic and grammatical processing.
  • 4. Imagery Condition:
  • Task: Participants form a distinct visual image or mental picture for each of the $27$ words across three passes.
  • Processing Depth: Deep visual and semantic encoding.
  • 5. Intentional Imagery Condition (Highest Processing Level):
  • Task: Participants create visual images while explicitly and intentionally attempting to memorize and remember each word for a subsequent memory test.
  • Processing Depth: Deepest encoding, combining visual imagery with explicit intentional rehearsal.

• Empirical Results from the Model Study:

  • Counting Condition Mean: $\bar{X}_{1} = 7.0$ (Variance $s_1^2 = 3.334$)
  • Rhyming Condition Mean: $\bar{X}_{2} = 6.9$
  • Adjective Condition Mean: $\bar{X}_{3} = 11.0$
  • Imagery Condition Mean: $\bar{X}_{4} = 13.4$
  • Intentional Imagery Condition Mean: $\bar{X}_{5} = 12.0$
  • Grand Overall Mean ($\bar{\bar{X}}$): $\bar{\bar{X}} = 10.06$
  • Grand Overall Variance: $s_{\text{grand}}^2 = 16.06$

Foundations and Mechanics of Analysis of Variance (ANOVA)

• Purpose and Scope of ANOVA:

  • Core Function: Evaluates whether statistically significant differences exist among the means of multiple groups simultaneously.
  • Group Capacity:
  • Formal Definition: Designed to test differences across two or more group means ($k \ge 2$).
  • Practical Application: Primarily utilized for three or more groups ($k \ge 3$). For two groups ($k = 2$), an independent samples $t$-test is typically preferred because it exhibits greater robustness against assumption violations.

• Relationship Between the $t$-Test and the $F$-Test:

  • When comparing exactly two groups, the square of the calculated $t$ statistic equals the calculated $F$ statistic:     $t^2 = F$
  • The resulting $p$-values for a two-group $t$-test and a two-group one-way ANOVA on identical data are mathematically identical.

• Control of Type I Error Rate:

  • Type I Error Definition: Rejecting a true null hypothesis ($H_0$); committing a "false alarm" by claiming a true effect or group difference exists when it does not.
  • Inflation of Type I Error via Multiple $t$-Tests: Running separate pairwise $t$-tests across five experimental groups requires up to $10$ to $15$ individual comparisons. If each test uses an alpha level of $\alpha = 0.05$, the cumulative probability of committing at least one Type I error across the set of tests inflates dramatically above $0.05$
  • The ANOVA Advantage: ANOVA calculates a single omnibus test statistic that holds the overall family-wise error rate constant at $\alpha = 0.05$ across all simultaneous group comparisons.

• Hypothesis Formulation in One-Way ANOVA:

  • Null Hypothesis ($H_0$):$H_0: \mu_1 = \mu_2 = \mu_3 = \mu_4 = \mu_5$
  • States that all population group means are equal, indicating that the independent variable (level of processing) exerts no effect on the dependent variable (recall).
  • Alternative Hypothesis ($H_1$):$H_1: \text{At least one population mean } \mu_j \text{ is significantly different from the others.}$
  • Interpretation of Rejection: Rejecting $H_0$ ($p \le 0.05$) confirms that a statistically significant difference exists somewhere among the group means. However, the omnibus $F$-test does not reveal which specific groups differ, how many pairs differ, or the direction of the differences.
  • Necessity of Follow-up Tests: To locate specific group differences following a significant omnibus ANOVA, post-hoc follow-up tests must be conducted. Each follow-up test produces its own individual $p$-value.

• Statistical Decision Rules for the $F$-Test:

  • Test Statistic: The $F$ ratio ($F$).
  • Directionality: The $F$-test is non-directional (it does not possess separate one-tailed or two-tailed rejection regions) because it simultaneously evaluates variance across multiple group means.
  • Critical Value Decision Rule: Reject $H_0$ if the obtained statistic exceeds the critical value:     $\text{Obtained } F > \text{Critical } F$
  • $p$-Value Decision Rule: Reject $H_0$ if the calculated $p$-value is less than or equal to the significance threshold:     $p \le 0.05$
  • Inverse Relationship: As the calculated $F$ statistic increases in magnitude, the corresponding $p$-value becomes smaller.
  • Standard Reporting Format: Report the $F$ statistic accompanied by its associated between-groups and within-groups degrees of freedom and the exact $p$-value:     $F(df_{\text{between}}, df_{\text{within}}) = \text{value}, \, p = \text{value}$

Partitioning Sum of Squares and Variance Components

• Review of Variance Fundamentals:

  • Sample Mean Formula:     $\bar{X} = \frac{\sum X}{N}$
  • Deviation Score Formula:     $\text{Deviation} = X - \bar{X}$
  • Sum of Squared Deviations ($SS$):     $SS = \sum (X - \bar{X})^2$
  • Mean Square ($MS$):     $MS = \frac{SS}{df}$

• Logic of Partitioning Variance:

  • ANOVA partitions total variation ($SS_{\text{total}}$) into two independent, additive components:
  1. Variation due to the independent variable (Treatment / Between-Groups variation).
  2. Unexplained variation due to individual differences and random noise (Error / Within-Groups variation).

• Mathematical Breakdown of Sum of Squares Components:

  • Total Sum of Squares ($SS_{\text{total}}$):
  • Conceptual Meaning: Measures the total variation of all individual scores around the grand mean across the entire dataset.
  • Formula (Dot Notation):       $SS_{\text{total}} = \sum_{i} \sum_{j} (X_{ij} - \bar{X}_{\dots})^2$
  • Treatment / Between-Groups Sum of Squares ($SS_{\text{treatment}}$ or $SS_{\text{between}}$):
  • Conceptual Meaning: Measures the variability attributable directly to the independent variable by comparing each group mean to the grand mean.
  • Formula:       $SS_{\text{treatment}} = \sum_{j} n_j (\bar{X}_j - \bar{X}_{\dots})^2$
  • Error / Within-Groups Sum of Squares ($SS_{\text{error}}$ or $SS_{\text{within}}$):
  • Conceptual Meaning: Measures variability within each experimental group by comparing individual scores to their specific group mean. Reflects random individual differences unrelated to the independent variable.
  • Formula:       $SS_{\text{error}} = \sum_{i} \sum_{j} (X_{ij} - \bar{X}_j)^2$
  • Additive Partitioning Identity:$SS_{\text{total}} = SS_{\text{treatment}} + SS_{\text{error}}$

• Terminology Equivalencies in Variance Partitioning:

  • $SS_{\text{treatment}} = SS_{\text{between}} = \text{Explained Variance} = \text{Systematic Variance}$
  • $SS_{\text{error}} = SS_{\text{within}} = \text{Unexplained Variance} = \text{Unsystematic Variance}$

• Conceptual Derivation of the $F$-Ratio:

  • Conceptual Formula:     $F = \frac{MS_{\text{treatment}} + MS_{\text{error}}}{MS_{\text{error}}} = \frac{MS_{\text{between}}}{MS_{\text{within}}}$
  • Baseline Value Under Null Conditions: If the independent variable has zero effect ($MS_{\text{treatment}} = 0$), the formula reduces to:     $F = \frac{0 + MS_{\text{error}}}{MS_{\text{error}}} = 1.0$
  • Decision Threshold: An $F$ ratio equal to $1.0$ indicates that observed group differences are entirely consistent with random chance. To achieve statistical significance, $F$ must substantially exceed $1.0$. If an obtained $F$ statistic is less than or equal to $1.0$, it is declared non-significant ($F < 1$) without needing further evaluation.

Parametric Assumptions of One-Way ANOVA

• 1. Homogeneity of Variance:

  • Definition: The population variances of the dependent variable must be equal across all experimental groups.
  • Evaluation Method: Assessed using Levene's Test of Equality of Variances.
  • Scope: Evaluates individual score spread and variance within each group independently of the group means.

• 2. Normality:

  • Definition: The scores within each population/group must be normally distributed around their respective group means.

• 3. Independence of Observations:

  • Definition: Each participant's response must be completely independent of every other participant's response.
  • Violation Consequences: Dependent observations invalidate the calculated probability distribution of the $F$ statistic.

Construction and Mechanics of the ANOVA Summary Table

• Standard ANOVA Summary Table Structure:

  • Columns: Source of Variation, Sum of Squares ($SS$), Degrees of Freedom ($df$), Mean Square ($MS$), $F$ Statistic, $p$-value.

• Degrees of Freedom Calculations (For Sample Size $N = 50$, $k = 5$ Groups, $n = 10$ per Group):

  • Total Degrees of Freedom ($df_{\text{total}}$):     $df_{\text{total}} = N - 1 = 50 - 1 = 49$
  • Treatment / Between-Groups Degrees of Freedom ($df_{\text{treatment}}$):     $df_{\text{treatment}} = k - 1 = 5 - 1 = 4$
  • Error / Within-Groups Degrees of Freedom ($df_{\text{error}}$):     $df_{\text{error}} = N - k = 50 - 5 = 45$
  • Alternative Formula: $df_{\text{error}} = k \times (n - 1) = 5 \times (10 - 1) = 45$
  • Additive Verification Check:     $df_{\text{treatment}} + df_{\text{error}} = df_{\text{total}} \implies 4 + 45 = 49$

• Mathematical Relationships Across Table Entries:

  • Mean Square Treatment:     $MS_{\text{treatment}} = \frac{SS_{\text{treatment}}}{df_{\text{treatment}}}$
  • Mean Square Error:     $MS_{\text{error}} = \frac{SS_{\text{error}}}{df_{\text{error}}}$
  • $F$ Statistic Derivation:     $F = \frac{MS_{\text{treatment}}}{MS_{\text{error}}}$

• Reporting Standard for Degrees of Freedom:

  • Researchers report both the between-groups degrees of freedom ($df_{\text{treatment}} = 4$) and the within-groups degrees of freedom ($df_{\text{error}} = 45$).
  • Total degrees of freedom ($df_{\text{total}} = 49$) is omitted from standard statistical reports because readers can compute it directly by adding the two reported degrees of freedom ($4 + 45 = 49$).

Questions & Statistical Troubleshooting

• Conducting Multiple $t$-Tests vs. ANOVA:

  • Question: Can multiple $t$-tests be performed on a dataset containing three or more groups instead of an ANOVA?
  • Clarification: Pairwise $t$-tests can physically be performed on the data, but doing so inflates the overall Type I error rate. If an experiment contains only two groups, an independent samples $t$-test should be used instead of an ANOVA because the $t$-test is slightly more robust against minor violations of assumptions.

• Levene's Test and Group Variances:

  • Question: What does Levene's test indicate if all participant scores in a group are identical?
  • Clarification: Levene's test evaluates the equality of score variances across groups, not group means. If groups have identical means, their variances can still differ widely.

• Troubleshooting Degrees of Freedom Discrepancies:

  • Question: What does it mean if statistical software displays total degrees of freedom as $47$ when $50$ participants were entered into the study?
  • Clarification: A discrepancy in total degrees of freedom (e.g., $df_{\text{total}} = 47$ instead of $49$) signals user coding error or missing data. Common causes include incorrectly specified group filter codes, failing to drag across all participant rows during selection, or misclassifying continuous numeric variables as nominal variables.