Statistical Significance and the Role of Probability in Statistics

The Role of Probability in Statistics: Statistical Significance

• The fundamental learning goal is to understand the concept of statistical significance and the essential role that probability plays in defining it within a statistical study.

• Definition of Statistical Significance: A set of measurements or observations in a statistical study is said to be statistically significant if it is unlikely to have occurred by chance.

Qualitative Examples of Statistical Significance

• Criminal Investigation in Detroit:

  • A detective finds that $25$ of the $62$ guns used in crimes during the past week were sold by the same gun shop.

  • This finding is considered statistically significant because there are many gun shops in the Detroit area. Consequently, having $25$ out of $62$ guns originate from a single shop is deemed unlikely to have occurred by chance alone.

• Global Temperature Records:

  • Data Observation 1: In terms of the global average temperature, $5$ of the years between $1990$ and $1999$ were the five hottest years in the $20\text{th}$ century. This finding is statistically significant.

  • Data Observation 2: Having the $8$ hottest years on record occur all in a row in a data set that goes back approximately $140$ years is statistically significant.

  • Conclusion: Such a streak of hot years is very unlikely to have occurred by chance alone and therefore provides strong empirical evidence of a warming world.

• Basketball Win-Loss Records:

  • Scenario: The team with the worst win-loss record in a basketball league wins a single game against the defending league champions.

  • Assessment: A single win in this context is not statistically significant. Although a team with a poor record is expected to lose most of its games, it is also expected to win occasionally, even against high-performing teams, due to normal variation in performance. This event is reasonably likely to occur by chance.

Statistical Significance in Experimental Research

• Case Study: Herbal Formula for Preventing Colds

  • Methodology: A researcher conducts a double-blind experiment using a treatment group and a control group.

  • Participants: $100$ randomly selected people in the treatment group (given the herbal formula) and $100$ randomly selected people in the control group (given a placebo).

  • Duration: The study lasted for a three-month period.

  • Results:

    • Treatment Group: $30$ people developed colds.

    • Control Group: $32$ people developed colds.

  • Analysis: Whether a person contracts a cold during a three-month period depends on many unpredictable factors. Small differences between groups should be expected.

  • Conclusion: The difference between $30$ individuals and $32$ individuals is small enough to be explained by chance. Therefore, the difference is not statistically significant, and the researcher cannot conclude that the herbal formula is effective.

Quantifying Statistical Significance

• Necessity of Quantification: Qualitative definitions of significance are too vague for rigorous science. Probability is used to quantify the likelihood that an observed result occurred by chance.

• The Significance Threshold Question: Is the probability that the observed difference occurred by chance less than or equal to $0.05$ (also expressed as $1$ in $20$)?

• Decision Rules:

  • If the probability is less than or equal to $0.05$, the difference is said to be statistically significant at the $0.05$ level.

  • If the probability is greater than $0.05$, the observed difference is considered reasonably likely to have occurred by chance and is therefore not statistically significant.

• Common Probability Levels:

  • Statistical significance at the $0.05$ level means the probability of the result occurring by chance is $\leq 0.05$ ($1$ in $20$ or less).

  • Statistical significance at the $0.01$ level means the probability of the result occurring by chance is $\leq 0.01$ ($1$ in $100$ or less).

  • Although $0.05$ is a common choice, it is somewhat arbitrary; statisticians may also use other probabilities such as $0.1$ or $0.01$ depending on the study requirements.

Case Study: Salk Polio Vaccine Significance

• Study Parameters:

  • Treatment Group: $200,000$ children received the Salk polio vaccine.

  • Control Group: $200,000$ children received a placebo.

• Observed Data:

  • Treatment Group Cases: $33$ children developed paralytic polio.

  • Control Group Cases: $115$ children developed paralytic polio.

• Probability Calculation:

  • Researchers calculated that the probability of this specific difference occurring by chance was approximately $0.00000000002$ (or $0.000000002\%$).

• Assessment of Significance:

  • Because this probability is significantly lower than both $0.05$ and $0.01$, the results are considered statistically significant at both the $0.05$ and $0.01$ levels.

  • This extremely low probability—later referred to in statistics as a "P-value"—gave researchers high confidence that the vaccine was truly responsible for the reduction in polio cases rather than random chance.

Questions & Discussion

• Think About It Question: Suppose an experiment finds that people taking a new herbal remedy get fewer colds than people taking a placebo, and the results are statistically significant at the $0.01$ level. Has the experiment proven that the herbal remedy works? Explain.

  • Implicit Explanation: Statistical significance at the $0.01$ level indicates a very low probability (1 in 100 or less) that the results were due to chance, suggesting the remedy is likely effective. However, in statistics, "proof" is rarely absolute; rather, it indicates strong evidence where the likelihood of being wrong is quantified (e.g., a $1\%$ chance the result still happened by luck).