Comprehensive Guide to Statistical Tests and Research Validity

Statistical Test Selection

Identify type of data (numerical, nominal, ordinal), number of groups, and whether samples are independent or dependent.

Parametric Tests

Tests that require normal distribution and interval/ratio data (e.g., t-tests).

Nonparametric Tests

Tests that do NOT require normal distribution; used for ordinal or nominal data and small samples.

Independent Samples t-Test

Used to compare the means of two different (independent) groups.

When to Use Independent t-Test

Two separate groups, numerical data, normal distribution, equal variances assumed.

Dependent Samples t-Test

Used to compare means from the same participants (paired data).

When to Use Dependent t-Test

Same participants measured twice (before/after or matched pairs).

Null Hypothesis (t-tests)

No difference between groups (μ1 = μ2 or μD = 0).

Alternative Hypothesis (t-tests)

There is a difference between groups (μ1 ≠ μ2 or μD ≠ 0).

Degrees of Freedom (Independent t-Test)

df = (N1 - 1) + (N2 - 1)

Degrees of Freedom (Dependent t-Test)

df = N - 1

One-Tailed Test

Tests for a specific direction (greater than or less than).

Two-Tailed Test

Tests for any difference (≠).

Reject Null Hypothesis

There is a statistically significant difference or relationship.

Fail to Reject Null Hypothesis

No significant evidence; results may be due to chance.

Effect Size (Cohen's d)

Measures how large the difference is between groups.

Confidence Interval Interpretation

If CI includes 0 → not significant; if it does not include 0 → significant.

Chi-Square Test

Used for categorical (nominal) data to compare observed vs expected frequencies.

One-Way Chi-Square (Goodness of Fit)

Tests whether observed frequencies match expected frequencies for one variable.

When to Use One-Way Chi-Square

One categorical variable, comparing observed counts to expected counts.

Two-Way Chi-Square (Test of Independence)

Tests whether two categorical variables are related.

When to Use Two-Way Chi-Square

Two categorical variables, testing for a relationship.

Null Hypothesis (Chi-Square)

No relationship; variables are independent OR frequencies match expected.

Alternative Hypothesis (Chi-Square)

There is a relationship OR frequencies differ from expected.

Degrees of Freedom (One-Way Chi-Square)

df = k - 1 (k = number of categories)

Degrees of Freedom (Two-Way Chi-Square)

df = (r - 1)(c - 1)

Chi-Square Assumption

Expected frequency in each category must be at least 5.

When to Use Nonparametric Tests

Ordinal data, small sample sizes, or when normality assumption is violated.

Mann-Whitney U Test

Used for two independent groups with ordinal data when N < 20 per group.

Rank Sums Test

Used for two independent groups with ordinal data when N ≥ 20 per group.

Wilcoxon T-Test

Used for two dependent groups with ordinal (paired) data.

Kruskal-Wallis H Test

Used for 3+ independent groups with ordinal data; requires N ≥ 5 per group.

Friedman Test

Used for 3+ dependent groups with ordinal data.

Friedman Sample Size Rule

If 3 groups → N ≥ 10; if 4+ groups → N ≥ 5.

Spearman's Rho

Correlation for ranked (ordinal) data.

McNemar Test

Used for nominal data with two related conditions (same participants).

Cochran Test

Used for nominal data with three or more related conditions.

Independent Groups Design

Different participants in each group.

Repeated Measures Design

Same participants in all conditions.

Internal Validity

Whether the study truly shows cause and effect.

Threat: History

External events influence results.

Threat: Maturation

Participants naturally change over time.

Threat: Testing Effect

Practice or familiarity affects results.

Threat: Instrumentation

Measurement tools change over time.

Threat: Selection

Groups differ at the start.

Threat: Attrition

Participants drop out of study.