Ch. 15: Type I Error Correction

Discusses various cognitive components tested against one another in a study context.
Introduces several continuous variables, focusing on connections between them.

Vocabulary
- Represents language proficiency and verbal understanding.
Sex
- Factor influencing cognitive performance; potential biological underpinnings.
Reading
- Assesses comprehension and information processing.
Processing Speed
- Measures how quickly an individual can complete tasks.
Inhibition
- Involves the ability to suppress responses or distractions.
Episodic Memory
- Refers to the ability to recall personal experiences.
Cognitive Flexibility
- Reflects the ability to adapt cognitive processing strategies.
Attention
- Concentration and focus in performing tasks.
General Cognitive Ability
- Broad measure of overall cognitive function.
Age
- Considered as a factor potentially affecting cognitive abilities.
Working Memory
- Short-term memory system emphasizing manipulation of information.

Conducting multiple tests on 11 continuous variables leads to significant results via statistical analysis.
- Derivation of the number of tests using the formula: $n(n-1)/2$ .
- Derives into 55 tests from 11 variables.

Alpha (α) is the probability that results are due to chance.
As the number of tests increases, so does the probability of obtaining significant results:
- Equation: $1 - (1 - \bar{α})^{comparisons}$
- Calculated example: $1 - (0.95)^{55} = 0.94$
- Result indicates a 94% chance of a significant result being due to chance.

Type I error occurs when a true null hypothesis is incorrectly rejected (a false positive).
Critical to determine if these corrections are necessary when conducting multiple statistical tests.

Type I error correction is warranted when multiple comparisons are made to reveal discoveries from the same dataset.
Specifically emphasized in repeated tests addressing similar hypotheses or datasets.

Various disciplines have different standards for significance:
- Social Sciences: typically set at 0.05.
- Pharmaceutical Research: often tighter range between 0.01 and 0.001.
- Physics: requires a stringent threshold of '5 sigma' corresponding to a probability of 1 in 3.5 million.

Type I error is typically evaluated through the lens of the family-wise error rate.
- Defined as a collection of inferences drawn from the same data or a related question.
Acknowledges that Type I error probability rises with increasing test numbers, hence needing control mechanisms.

Applied in post-hoc analyses, such as ANOVA or split contingency tables.
Less overt cases include tests addressing similar questions across various test types, e.g. using data in both ANOVA and linear models.

Two primary methods to guard against Type I errors:
- Altering the significance threshold—modifying the original alpha level.
- Adjusting the p-value through a correction factor—both methods effectively achieve similar results.

Tukey's Test
- Best used for all pairwise comparisons post-ANOVA.
Scheffé Test
- Suitable for multiple comparisons but not as powerful as Tukey.
Bonferroni Correction
- First recommended for Type I error correction, particularly for pre-selected contrasts.
- Formula: $α^* = α / k$
- Example: For 4 comparisons, $0.05 / 4 = 0.0125$ .
Holm-Bonferroni (Holm)
- More powerful than Bonferroni, involves ordering p-values and adjusting thresholds sequentially.
Newman-Keuls
- A sequential Tukey version but less accepted.
Dunnett’s Test
- Most powerful for comparing control against treatment groups.

Avoid Bonferroni when dealing with several planned contrasts.
Consider using Holm-Bonferroni for pairwise comparisons after ANOVA.
If comparing only control to treatment conditions, opt for Dunnett’s test.
If willing to accept occasional Type I errors for increased power, use Benjamini-Hochberg.

Understanding the necessity and methods of Type I error correction is crucial for ensuring the integrity and reliability of statistical findings within cognitive research and related fields.