Lecture 23 - Advanced Hypothesis Testing: Assumptions, Errors, and Significance

Overview of Hypothesis Testing

• Conceptual Framework: Hypothesis testing is an inferential procedure in which sample statistics (summary measures from a sample of data) are used to make formal statements or claims about population parameters (fixed values that describe an entire population).

• The Core Hypothesis: Researchers hypothesize a specific value for a population parameter, compute a corresponding sample statistic, and then use the test to determine the validity of the hypothesized value.

• Scope of Parameters: The primary focus in this context is on two specific parameters:

  • Population Mean: Denoted by $\mu$.

  • Population Proportion: Denoted by $\pi$.

• Alternative Procedures: While hypothesis testing is a central tool, other inferential procedures exist, notably confidence intervals, which provide a range of plausible values for a parameter.

The Five-Step Hypothesis Testing Process

Every hypothesis test, regardless of the parameter involved, follows a rigorous five-step methodology:

1. Hypothesize: State the null hypothesis ($H_0$) and the alternative hypothesis ($H_a$ or $H_1$) regarding the population parameter.

2. Collect and Summarize Data: Gather the sample data and calculate the appropriate test statistic.

3. Determine the P-value: Use the test statistic to calculate the probability of observing the result (or something more extreme) assuming the null hypothesis is true.

4. Make a Conclusion: Use the p-value to decide whether to reject or fail to reject the null hypothesis.

   • If the p-value is less than the significance level ($\alpha = 0.05$), the result is "statistically significant," and the null hypothesis is rejected.

5. Contextualize: Relate the statistical findings back to the original real-world problem to ensure the results are meaningful and practical.

Directives for One-sided and Two-sided Tests

• Two-sided Tests: Used when the researcher is interested in any difference from the hypothesized value (not equal to). The alternative hypothesis is stated as $\text{parameter} \neq \text{value}$. These tests examine probabilities on both tails of the distribution.

• One-sided Tests: Used when the researcher is interested in a specific direction (greater than or less than).

  • Upper-tail: Alternative is \mu > \text{value}.

  • Lower-tail: Alternative is \mu < \text{value}.

• Notation Styles: The null hypothesis always contains an equality. It can be written as $\text{parameter} = \text{value}$, or in some styles, as the logical opposite of the alternative (e.g., $\mu \text{ (parameter)} \times \text{ (value)} \ge$ or $\le$).

• Symmetry and P-values: Due to the symmetric nature of the normal and t-distributions, the p-value for a two-sided test is exactly twice ($2 \times$) the p-value of a one-sided test for the same data.

Test Statistic Definitions and Interpretations

• Verbatim Definition of Test Statistic: A test statistic is a transformed sample statistic or standardized score (like a z-score) that is used to find probability. It represents the number of standard errors the sample statistic lies from the hypothesized value.

• Formulaic Interpretation: The general structure of a test statistic is:     $\text{Test Statistic} = \frac{\text{Observed Value} - \text{Hypothesized Value}}{\text{Standard Error}}$

• Verbatim Definition of P-value: The p-value stands for probability; it is the probability of observing the test statistic actually seen (or something more extreme), assuming the null hypothesis is true.

• Bayesian Distinction: While p-values describe the probability of the data given the hypothesis, Bayesian methods specifically address the "probability that the null hypothesis is true."

Formulas for Proportions and Means

Population Proportion ($\pi$)

• Sample Statistic: $\hat{p}$

• Test Statistic Formula (z-score):     $z = \frac{\hat{p} - \pi_0}{\sqrt{\frac{\pi_0(1 - \pi_0)}{n}}}$

• Distribution: P-values are determined using the Standard Normal Distribution table.

Population Mean ($\mu$)

• Sample Statistic: $\bar{x}$ (sample mean)

• Test Statistic Formula (t-score):     $t = \frac{\bar{x} - \mu_0}{\frac{s}{\sqrt{n}}}$

• Distribution: P-values are determined using the t-distribution because the population standard deviation is usually unknown and replaced by the sample standard deviation ($s$).

Underlying Assumptions for Validity

For hypothesis tests to be valid and robust, certain assumptions must be checked.

Assumptions for Proportions

1. Representativeness: The sample must accurately represent the population. Methodology and sampling frames must be critiqued.

2. Independence: Samples must be collected independently, usually achieved through probability-based methods like Simple Random Sampling.

3. Sample Size ("Large Enough"): Two conditions must be satisfied:

   • n \times \pi_0 > 10

   • n \times (1 - \pi_0) > 10

   • Note: If $\pi_0$ is near 0 or 1, a much larger sample size is required.

Assumptions for Means

1. Representativeness and Independence: Identical to proportion assumptions.

2. Normality or Large Sample Size: Either the population must be normally distributed, or the sample size must be large (n > 30) to satisfy the Central Limit Theorem.

3. Outliers: If outliers are present, a sample size even larger than 30 may be necessary.

Case Study: ACT Party Political Polling (July 2020)

• Objective: Test if the ACT party could receive more than $5\%$ of the party vote (the threshold for parliament).

• Methodology Critique (Colmar Brunton Poll):

  • Sample Size ($n$): $1,004$ eligible New Zealand voters.

  • Frame: Households with landline telephones ($403$ participants) or mobile phones ($601$ participants).

  • Critique: $40\%$ weighting on landlines may be high for a modern population, but historically accurately mirrors the voting population.

  • Method: Computer-assisted telephone interviewing (CATI), which may reduce interviewer bias.

• Hypothesis Setup:

  • $H_0: \pi \le 0.05$

  • H_a: \pi > 0.05 (One-sided test)

• Assumption Check for ACT Poll:

  • n \times \pi_0 = 1,004 \times 0.05 = 50.2 > 10

  • n \times (1 - \pi_0) = 1,004 \times 0.95 = 953.8 > 10

  • Conclusion: Sample size is large enough.

• Result: Among $1,004$ people, $48$ said they would vote ACT ($\hat{p} \approx 0.0478$).

  • Test Statistic: $-0.31$

  • P-value: $0.62497$

• Conclusion: Fail to reject $H_0$. At that time (July 2020), there was no evidence ACT would exceed $5\%$. (Note: While they eventually got $~6\%$ in the November election, a statistic is a "static" snapshot of that specific time).

Error Types and Statistical Power

Type I Error

• Definition: Occurs when the null hypothesis is true, but we reject it (a "false positive").

• Probability: The probability of a Type I error is exactly equal to the level of significance ($\alpha$). At $\alpha = 0.05$, there is a $5\%$ chance of this error.

Type II Error

• Definition: Occurs when the alternative hypothesis is true, but we fail to reject the null hypothesis (a "false negative").

• Influencing Factors:

  • Sample Size ($n$): A larger $n$ reduces the probability of Type II error.

  • Significance Level ($\alpha$): Increasing $\alpha$ (e.g., from $0.01$ to $0.05$) reduces Type II error but increases Type I error. Alpha ($0.05$) is used as a standard balance.

  • Parameter Value: The actual value of the population parameter relative to the hypothesized value.

• Power of the Test: The probability of making the correct decision (rejecting $H_0$ when it is false).

Statistical vs. Practical Significance

• The Power of Sample Size: Large sample sizes can make even tiny, unimportant differences "statistically significant."

• Standard Error Effect: As $n$ increases, the denominator of the test statistic ($\frac{s}{\sqrt{n}}$) decreases. This makes the overall test statistic ($t$ or $z$) larger, which in turn makes the p-value smaller.

• Example: Taste Test Comparison:

  • Sample proportion ($\hat{p}$) = $55\%$ for both tests.

  • Study X ($n=60$): P-value = $0.439$ (Not significant).

  • Study Y ($n=960$): P-value = $0.02$ (Statistically significant).

• Example: Birthday and Height Study:

  • An Austrian study of over $500,000$ military recruits found men born in spring were statistically significantly taller than those born in autumn.

  • The Magnitude: The difference was only $0.6\,cm$ ($6\,mm$).

  • Analysis: While statistically significant due to the massive sample size, a $6\,mm$ difference is practically insignificant for most applications.

• Conclusion: Always contextualize results. A p-value does not provide information about the magnitude of the effect; it only indicates the strength of evidence against the null hypothesis.