Significance Tests

The Idea of a Significance Test

• A significance test is a procedure for using observed data to decide between two competing claims, called hypotheses.

• The hypotheses are usually statements about a parameter, like the population proportion $p$ or the population mean $\mu$.

• The claim we weigh evidence against in a statistical test is called the null hypothesis $(H_o)$. The null hypothesis often has the form: parameter = null value.

• The claim about the population that we are trying to find evidence for is the alternative hypothesis $(H_a)$.

• A one-sided alternative hypothesis has the form: parameter < null value or parameter > null value.

• A two-sided alternative hypothesis has the form: parameter $\neq$ null value.

• A standardized test statistic measures how far a sample statistic is from what we would expect if the null hypothesis $(H_o)$ were true, in standardized units.

  • Standardized test statistic = $\frac{statistic - null \ value}{standard \ deviation \ (error) \ of \ statistic}$

• The P-value of a test is the probability of getting evidence for the alternative hypothesis as strong or stronger than the observed evidence by chance alone when the null hypothesis $(H_o)$ is true.

• Small P-values are evidence against the null hypothesis and for the alternative hypothesis because they say that the observed result is unlikely to occur when $(H_o)$ is true. To determine if a P-value should be considered small, we compare it to the significance level $\alpha$.

Significance Test Conclusions and Errors

• We make a conclusion in a significance test based on the P-value.

• If P-value < $\alpha$: Reject $(H<em>o)$ and conclude there is convincing evidence for $(H</em>a)$ (in context).

• If P-value > $\alpha$: Fail to reject $(H<em>o)$ and conclude there is not convincing evidence for $(H</em>a)$ (in context).

• When we make a conclusion in a significance test, there are two kinds of mistakes we can make.

• A Type I error occurs if we find convincing evidence that $(H_a)$ is true when it really isn’t.

• A Type II error occurs if we do not find convincing evidence that $(H_a)$ is true when it really is.

• The probability of making a Type I error is equal to the significance level $\alpha$. There is a tradeoff between P(Type I error) and P(Type II error): as one decreases, the other increases, assuming everything else remains the same. So it is important to consider the possible consequences of each type of error before choosing a significance level.

How Significance Tests Work

• When you perform a significance test, follow the four-step process:

  • STATE: State hypotheses, parameter(s), significance level, and evidence for $(H_a)$.

  • PLAN: Identify the appropriate inference method and check conditions.

  • DO: If the conditions are met, perform calculations:

    • Identify the sample statistic(s).

    • Calculate the standardized test statistic.

    • Find the P-value.

  • CONCLUDE: Make a conclusion about the hypotheses in the context of the problem.

Significance Tests for a Proportion

• The conditions for performing a significance test of $H<em>o : p = p</em>o$ are:

  • Random: The data come from a random sample from the population of interest.

  • Large Counts: Both $np<em>o$ and $n(1 - p</em>o)$ are at least 10.

• The standardized test statistic for a one-sample z test for p is

  • $z = \frac{\hat{p} - p<em>o}{\sqrt{\frac{p</em>o(1 - p_o)}{n}}}$

• When the Large Counts condition is met, the standardized test statistic has approximately a standard normal distribution. You can use Table A or technology to find the P-value.

• Confidence intervals provide additional information that significance tests do not—namely, a set of plausible values for the population proportion p based on sample data. A 95% confidence interval for p gives information about the parameter that is generally consistent with a two-sided test of $H<em>o : p = p</em>o$ at the $\alpha = 0.05$ significance level.

Significance Tests for a Mean

• The conditions for performing a significance test of $H<em>o : \mu = \mu</em>o$ are:

  • Random: The data come from a random sample from the population of interest.

  • Normal/Large Sample: The data come from an approximately normally distributed population or the sample size is large $(n \ge 30)$. When the sample size is small and the shape of the population distribution is unknown, a graph of the sample data shows no strong skewness or outliers.

• The standardized test statistic for a one-sample t test for $\mu$ is

  • $t = \frac{\bar{x} - \mu_o}{S / \sqrt{n}}$

• When the Normal/Large Sample condition is met, the standardized test statistic can be modeled by a t distribution with n - 1 degrees of freedom (df). You can use Table B or technology to find the P-value.

• Confidence intervals provide additional information that significance tests do not—namely, a set of plausible values for the population mean $\mu$ based on sample data. A 95% confidence interval for $\mu$ gives information about the parameter that is consistent with a two-sided test of $H<em>o : \mu = \mu</em>o$ at the significance level $\alpha = 0.05$.

Using Tests Wisely

• Very small deviations from the null hypothesis can be highly significant (small P-value) when a test is based on a large sample. A statistically significant result may not be practically important.

• Many tests that are run at once will likely produce some significant results by chance alone, even if all the null hypotheses are true. Beware of P-hacking.