Week 9: Hypothesis Testing Errors and Confidence Intervals

Learning Outcomes

• LO1: Defining and Explaining Errors: Understand how to define and explain Type I and Type II errors within the context of hypothesis testing.

• LO2: Decision Outcomes and Errors: Distinguish between rejecting and failing to reject the null hypothesis and identify how these decisions relate to Type I and Type II errors.

• LO3: Confidence Interval Interpretation: Define a confidence interval and explain its interpretation as an interval estimator for a population parameter.

• LO4: Construction and Interpretation: Construct and interpret confidence intervals for various population parameters.

Overview of Statistical Inference and Review

• Sampling Distribution Context: In previous sessions (Lecture 6), the sampling distribution for the sample mean ($\bar{X}$) was explored. This was based on simple random samples of $n = 10$ students drawn from a population of $1,368$ students based on test marks (out of $20$ points).

• Population Parameters: The true population mean ($\mu$) for these test marks was known to be $15.92$.

• Sample Statistics: For each sample, the sample mean ($\bar{X}$), sample variance ($s^2$), and standard deviation ($s$) were calculated.

• Probability of Deviance: While most sample means fell close to the true population mean ($15.92$), there is a small but finite probability of obtaining a sample mean that is very different from the true mean.

Hypothesis Testing: Decision Scenarios

Scenario A: Correct Decision (Fail to Reject True Null)

• Hypotheses:

  • $H_0: \mu = 15.92$

  • $H_A: \mu \neq 15.92$

• Sample Data: Sample Mean ($\bar{X} = 15.9$), Sample Variance ($s^2 = 9.4333$), Sample Size ($n = 10$).

• Critical Value: $t_{0.05/2, 9} = 2.262$.

• Decision Rule: Reject $H_0$ if the $t$-statistic is greater than $2.262$ or less than $-2.262$.

• Test Statistic Calculation:

  • $t = \frac{\bar{X} - \mu}{s / \sqrt{n}} = \frac{15.9 - 15.92}{\sqrt{9.4333 / 10}} = -0.0206$

• Outcome: Since $t = -0.0206$ is between $-2.262$ and $2.262$, and the $p$-value ($0.9789$) is greater than $\alpha$ ($0.05$), the decision is to not reject the null hypothesis.

• Conclusion: At a $5\%$ significance level, there is insufficient evidence to conclude the population mean is different from $15.92$. This is the correct decision because $\mu$ is indeed $15.92$.

Scenario B: Type I Error (Rejecting a True Null)

• Context: Sampling variability means we might occasionally draw a sample with an unusually high or low mean.

• Sample Data: Sample Mean ($\bar{X} = 18.1$), Sample Variance ($s^2 = 3.6556$), Sample Size ($n = 10$).

• Test Statistic Calculation:

  • $t = \frac{18.1 - 15.92}{\sqrt{3.6556 / 10}} = \frac{2.18}{0.6046} = 3.6056$

• Outcome: Since $t = 3.6056 > 2.262$, and the $p$-value ($0.0057$) is less than $\alpha$ ($0.05$), the decision is to reject the null hypothesis.

• Conclusion: This decision is incorrect because the true population mean is $15.92$. We have rejected a true null hypothesis, which is a Type I error.

Scenario C: Type II Error (Failing to Reject a False Null)

• Context: Testing a claim that the average mark ($\mu$) is $14.5$. We know this claim is false because the true mean is $15.92$.

• Hypotheses:

  • $H_0: \mu = 14.5$

  • $H_A: \mu \neq 14.5$

• Sample Data: $n = 10$, $\bar{X} = 15.9$, $s^2 = 9.4333$.

• Test Statistic Calculation:

  • $t = \frac{15.9 - 14.5}{\sqrt{9.4333 / 10}} = 1.4414$

• Outcome: Since $t = 1.4414 < 2.262$, the decision is to not reject the null hypothesis.

• Conclusion: This is an incorrect decision (Type II error) because the null hypothesis was false. This illustrates why we say "fail to reject" rather than "accept"; failing to reject means we lack sufficient evidence against the null, not that the null is definitely true.

Types of Errors in Hypothesis Testing

Type I Error

• Verbatim Definition: A Type I error is the incorrect decision to reject $H_0$ when $H_0$ is true.

• Probability: The probability of committing a Type I error is exactly equal to the significance level, denoted by $\alpha$.

• $P(\text{Type I Error}) = \alpha = P(\text{Reject } H_0 | H_0 \text{ is True})$

Type II Error

• Verbatim Definition: A Type II error is the decision NOT to reject $H_0$ when it is false.

• Probability: The probability of committing a Type II error is denoted by $\beta$.

• $\beta = P(\text{Fail to reject } H_0 | H_0 \text{ is False})$

• Challenges: Calculating $\beta$ is difficult because it requires knowing the true population parameter, which is typically unknown in practice.

Factors Influencing Errors and The Trade-off

• Significance Level ( $\alpha$ ) Influence:

  • Decreasing $\alpha$: Reducing $\alpha$ (e.g., from $0.05$ to $0.01$) makes the rejection region smaller. This reduces the risk of a Type I error but makes it harder to reject the null, thereby increasing the risk of a Type II error ($\beta$).

  • Increasing $\alpha$: Increasing $\alpha$ makes it easier to reject the null. This reduces $\beta$ but increases the risk of a Type I error ($\alpha$).

• Sample Size ($n$) Influence:

  • Larger samples reduce variability and improve precision, resulting in a smaller $\beta$ (lower risk of Type II error).

  • Smaller samples have more variability, resulting in a higher $\beta$.

• Data Variability Influence:

  • Higher variability ($s$) leads to less precise estimates and a higher $\beta$.

• Power of a Test:

  • Definition: The probability of correctly rejecting a false null hypothesis (avoiding a Type II error).

  • $\text{Power} = 1 - P(\text{Type II Error}) = 1 - \beta$

  • A high-power test is desirable as it indicates a strong ability to detect when the null hypothesis is false.

Confidence Intervals (CIs)

Conceptual Basis

• Point Estimate: A single value (like $\bar{X}$) used to estimate a population parameter. It is limited by sampling variability.

• Interval Estimator: A range of values constructed from sample data that is likely to contain the true population parameter. It is more informative than a point estimate because it quantifies uncertainty.

Derivation from Hypothesis Testing

• A confidence interval represents the range of all hypothesised values ($\mu_0$) for which the null hypothesis $H_0: \mu = \mu_0$ would not be rejected in a two-tailed test.

• Non-Rejection Condition: $-t_{\alpha/2, n-1} \leq \frac{\bar{X} - \mu_0}{s / \sqrt{n}} \leq t_{\alpha/2, n-1}$

• The Resulting Interval: $\bar{X} - t_{\alpha/2, n-1} \times \frac{s}{\sqrt{n}} \leq \mu_0 \leq \bar{X} + t_{\alpha/2, n-1} \times \frac{s}{\sqrt{n}}$

Confidence Interval for a Mean ( $\mu$ )

• Formula: $\bar{X} \pm t_{\alpha/2, n-1} \times \frac{s}{\sqrt{n}}$

• Properties:

  • The sample mean $\bar{X}$ is an efficient estimator of $\mu$.

  • The distribution is Student-t distributed with $n - 1$ degrees of freedom.

  • The interval is symmetric around $\bar{X}$.

• Interpretation: A $(1 - \alpha) \times 100\%$ confidence interval means that if we were to repeat the sampling process many times, approximately $(1 - \alpha) \times 100\%$ of the resulting intervals would contain the true population mean.

• Important Caveat: We are confident in the method used to construct the interval. We do not say the mean is "certainly" in a specific calculated interval.

Worked Example: Test Marks ($n=25$)

• Data: $\bar{X} = 15.68$, $s^2 = 5.7267$, $n = 25$, $\alpha = 0.05$.

• Critical Value: $t_{0.025, 24} = 2.064$.

• Standard Error: $\frac{s}{\sqrt{n}} = \sqrt{\frac{5.7267}{25}} = 0.4786$.

• CI Calculation:

  • $15.68 \pm 2.064 \times 0.4786$

  • $15.68 \pm 0.988$

  • Interval: $[14.749, 16.531]$

• Hypothesis Link: Any $H_0: \mu = \mu_0$ where $\mu_0$ is outside this range (e.g., $17$ or $14$) would be rejected at the $5\%$ level. Values inside (e.g., $16$ or $15$) would not be rejected.

Precision and Length of Confidence Intervals

• Length Formula: $L = 2 \times t_{\alpha / 2, n - 1} \times \frac{s}{\sqrt{n}}$

• Informativeness:

  • Short Interval: Indicates the sample is highly informative about the parameter.

  • Wide Interval: Indicates higher uncertainty; the sample is less informative.

• Ways to achieve a shorter (more precise) interval:

  1. Smaller population variability ($s$).

  2. Larger sample size ($n$).

  3. Lower confidence level (e.g., $90\%$ instead of $99\%$, which reduces the critical value $t$).

Confidence Intervals for Other Parameters

Population Proportion ($P$)

• Use Case: Constructing intervals for proportions, such as urban dwellers in Victoria.

• Formula: $\hat{p} \pm t_{\alpha/2, n-1} \times \sqrt{\frac{\hat{p}(1 - \hat{p})}{n}}$

• Example Calculations: If $\hat{p} = 0.270$, $n = 1000$, $s.e.(\hat{p}) = 0.012$, and $t = 2.581$ (for $99\%$ confidence):

  • $0.270 \pm 2.581 \times 0.014 = [0.234, 0.306]$

Difference Between Two Independent Means ($\mu_1 - \mu_2$)

• Use Case: Comparing average wages between urban and regional dwellers.

• Formula: $(\bar{X}_2 - \bar{X}_1) \pm t_{\alpha/2, n_1+n_2-2} \times \sqrt{\frac{s_1^2}{n_1} + \frac{s_2^2}{n_2}}$

• Example Calculation: $\bar{X}_2 - \bar{X}_1 = 9,510.46$, $t = 1.962$, combined Standard Error = $1,689.66$.

  • $9,510.46 \pm 1.962 \times 1,689.66 = [6,195.36, 12,825.57]$

• Note: A wide interval here reflects high uncertainty in the estimated difference in earnings.

Difference Between Two Independent Proportions ($P_1 - P_2$)

• Formula: $(\hat{p}_1 - \hat{p}_2) \pm t_{\alpha/2, n_1+n_2-2} \times \sqrt{\frac{\hat{p}_1(1 - \hat{p}_1)}{n_1} + \frac{\hat{p}_2(1 - \hat{p}_2)}{n_2}}$

• Note: For large sample sizes, the t-distribution provides a good approximation of the standard normal distribution.

Mean of the Differences (Matched Pairs) ($\mu_D$)

• Formula: $\bar{X}_D \pm t_{\alpha/2, n-1} \times \frac{s_D}{\sqrt{n}}$

Summary Keywords

• Error Types: Type I error, Type II error, Probability of errors ($\alpha$, $\beta$).

• Test Performance: Power of a test ($1 - \beta$), Error trade-offs.

• Estimation: Point estimate, Interval estimator, Confidence Level ($1 - \alpha$), Confidence Limits (Upper and Lower).