Simulation-Based Hypothesis Testing

Null hypothesis (H0)

The default or status quo claim about a population parameter; assumed true for the test.

Alternative hypothesis (HA)

The claim we suspect may be true instead of H0; the direction of the test.

Purpose of hypothesis testing

To evaluate whether data provide evidence against H0 in favor of HA.

Logic of testing

Assume H0 is true and check if the observed data would be unlikely under that assumption.

Point estimate

A single number from the sample used to estimate a population parameter.

Interval estimate

An interval that likely contains the true parameter and expresses uncertainty.

Test statistic

A sample statistic (such as a mean or difference in means) used to evaluate H0.

Null distribution

The distribution of the test statistic assuming H0 is true; created by simulation or theory.

p-value

The probability, under the null distribution, of observing a result as extreme or more extreme than the observed one in the direction of HA.

Decision rule: p-value ≤ alpha

Reject H0 (statistically significant).

Decision rule: p-value > alpha

Fail to reject H0 (not statistically significant).

Significance level (alpha)

The maximum acceptable probability of committing a Type I error; usually 0.01, 0.05, or 0.10.

Meaning of alpha 0.05

If H0 is true, repeating the study many times would lead to incorrect rejections about 5 percent of the time.

Type I error

Rejecting H0 when it is actually true; probability equals alpha.

Type II error

Failing to reject H0 when HA is true; probability equals beta.

Example research question

Is the mean age of first marriage greater than 23 years?

Example hypotheses

H0: mean = 23; HA: mean > 23 (right-tailed test).

Constructing the null model

Shift sample data so the mean equals the hypothesized value and resample to create the null distribution.

Comparing two means (H0)

H0: mean1 − mean2 = 0.

Comparing two means (HA)

HA: mean1 − mean2 < 0, > 0, or ≠ 0 depending on the research question.