Topic 8: Valid and Invalid Comparison Procedures

What are valid procedures?

Wherein a MCP has STPSs and SCIs are compatible procedures that provide analysis that are coherent, and are such that a non-significant overall test implies all follow up tests are non-significant.
STPs and SCIs use the same critical values for all tests and therefore control alpha at FWER.

Compare Tukey, Bonferroni and ANOVA F STP (and Scheffe SCI procedures

Tukey - is the single more powerful MCP for analysis restricted to inferences on all pairwise comparisons.
Contrast –
Bonferroni 1 (or F) procedures) –
are more powerful than the F STP for inferences on planned contrasts when K is less than J-1
When K is more than J-1, choose the smaller contrast Fc between Scheffe and Bonferroni.

ANOVA F STP and Scheffe SCI
Is the only single stage test (technically you don’t have to test the overall test) that compatible with the ANOVA F test
Only procedure valid for post-analysis of contrast.

Contrast the size of confidence intervals for the Tukey, Bonferroni and Scheffe SCI.

WHEN THE ANALYSIS IS BEING DONE ON ALL COMPARISONS
Tukey - has the narrowest SCI

with Scheffe the widest and Bonferroni being the middle.
That is → Tukey 95% SCIs are more precise than either Bonferroni-t SCIs (for planned analysis of k = 6 comparisons) or Scheffé post-hoc 95% SCIs for all comparisons.
This reflects the magnitude differences between each method's critical value.

If you restrict comparisons, then Bonferroni because smaller as F critical is impacted by k.

Scheffe is most conservative

because it is controlling alpha across a BIGGER SET OF COMPARISONS — TUKEY IS LESS AS IT IS ONLY ONE TECH - AS IN MAXIMAL COMPARISON IS THE ONLY ONE — BONFERRONI DEPENDS ON WHICH CC YOU USE BECAUSE IF J IS MORE THAN N-1 THEN MAYBE SCHEFE IS MORE POWERFUL. → ALSO PLANNED.

Why q*?

Allows for direct comparison as it keeps the standard error the same across the bonferroni, scheffe and Tukey → it is adjusted .

What is an invalid multiple comparison procedure?

Invalid data analysis methods are ones that do not provide adequate familywise Type 1 error.

When does an invalid multiple comparison procedure occur?

normally a sequentially multi-stage test procedure where
 a different critical value is used at different stages of the analysis (e.g., one type of statistic for the overall test, a different statistic for follow-up tests) and this can lead to incoherent analyses please note that the order of tests is crucial, they usually do not allow for CI inference and allow for an inflation of FWER above α that can be substantial.

Why do invalid multiple comparison procedures still occur?

These methods have been around for a long time
Some textbooks still promote them as acceptable
Statistical packages (such as SPSS) encourage their use.
The false belief that these methods afford greater statistical power than other MCPs deemed ‘too conservative’ (such as Scheffé, Bonferroni or Tukey). (They have to be conservative in order to control FWER)

Describe the Fisher’s least significant difference test?

Stage 1 begins with a .05-level ANOVA F test of the homogeneity hypothesis and continues to Stage 2 if the overall H0 is rejected, otherwise the analysis is terminated.
In Stage 2 .05-level t-tests are conducted for all comparisons.
Argues that it is protected as the ANOVA F test occurs.

Describe what happens to the FWER in the Fisher Least Significant Difference test when the homogeneity hypothesis is true?

The ANOVA F test allows for control of the Type 1 error rate, if the homogeneity hypothesis is true, then when the procedure proceeds to the stage two only on no more than 5% of replications, therefore the ANOVA F-test allows for the protection of inflation above α.

Describe what happens to the FWER in the Fisher Least Significant Difference test when the homogeneity hypothesis is false?

The ANOVA F test does not allow for control of the type 1 error rate. In this case, the probability of proceeding stage 2 depends on the power of the ANOVA t-test.
If the power of the ANOVA test is high, then when you pass into stage 2, multiple t-tests will inflate the type 1 error rate.