11 Statistical testing background

How do standard deviation and standard error differ?

SD describes variability among observations. SE describes variability of an estimator across repeated samples; collecting more observations does not itself make individual observations less variable.

What is the estimated standard error of an independent sample mean?

SE(mean) = s/sqrt(n), where s is the sample SD. Multiplying sample size by four halves this SE, all else equal.

What do the null and alternative hypotheses represent?

H0 specifies the reference claim being tested. The alternative specifies the relevant departure. Its direction should be chosen before observing the result.

What is a p-value?

Under H0 and the test’s assumptions, it is the probability of a result at least as extreme as the observed one in the relevant direction. It is not P(H0 is true).

What does failing to reject H0 establish?

The chosen test did not supply enough evidence to reject at that level. It does not prove equality or that the null is true.

What are Type I error, Type II error and power?

Type I: reject a true H0. Type II: fail to reject a false H0. Power = 1 - β, where β is the Type II error probability for a specified alternative.

What does a 95% frequentist confidence-interval procedure mean?

Under its assumptions, the procedure covers the fixed true parameter in 95% of repeated samples. It does not assign a 95% probability to the fixed parameter lying in the one realized interval.

What does a chi-square goodness-of-fit test compare?

Observed category counts with counts expected from specified category probabilities. With k categories and fully specified probabilities, the usual degrees of freedom are k-1.

What is Pearson’s chi-square count-discrepancy statistic?

χ² = ∑ (O-E)²/E, with observed counts O and positive expected counts E. Use an appropriate count model and check the approximation conditions.

What are expected counts and degrees of freedom for a chi-square independence test?

E_ij = (row_total_i*column_total_j)/N. For an r-by-c table, df = (r-1)*(c-1), under the usual assumptions.

What is a one-sample t-test statistic for a mean?

t = (mean(x)-μ0)/(s/sqrt(n)), with df=n-1. Compare the estimated mean departure to its sampling uncertainty under the test assumptions.

When should a paired t test be considered instead of an independent-groups t test?

For linked before/after or matched numerical observations. Analyze the within-pair differences as one sample, rather than pretending paired values are independent.

How does Welch’s t test differ from the pooled two-sample t test?

Welch does not impose equal population variances and uses an adjusted sampling-uncertainty/df calculation. Pooling requires the equal-variance model in addition to other assumptions.

What question does a proportion test address?

A binary success rate versus a specified rate, or a difference between appropriate independent binary rates. Paired binary observations require a paired-data method.

What does one-way ANOVA’s F statistic compare?

F = [SSB/(G-1)]/[SSW/(N-G)]. SSB is between-group sum of squares, SSW within-group sum of squares, G group count and N total observations.

Does a significant one-way ANOVA identify which group means differ?

No. It indicates evidence against equality of all means under the model. Specific pair comparisons require appropriate follow-up analysis.

Why do many unadjusted hypothesis tests change the evidence?

More opportunities make at least one false rejection more likely. Bonferroni addresses familywise error; Benjamini-Hochberg targets false discovery rate under its conditions.

Why is a t test on features that created clusters not independent confirmation of those clusters?

The grouping was selected using the same measurements being tested. Descriptive differences are useful, but ordinary post-selection p-values do not account for that selection.