AGRI 2400 Lectures 23-24 - Error, Power, Significance

Error

• every time you make a decision to reject or retain a null hypothesis, based on a statistical test, there is a risk that the decision you make is wrong

  • there are 2 sorts of mistakes you can make, false positives, or false negatives

Type 1 Error - Man is Pregnant

• false positives

• if you choose to use an alpha of 0.05, there is a 5% chance that you find a significant difference when 1 does not exist

• you would therefore incorrectly reject the null hypothesis when it is correct

Type 2 Error - Say Pregnant Woman not Pregnant

• false negatives

• when you retain a null hypothesis even though it is false

• the real effect of your treatment was not detected due to chance or a lack of power to detect the effect size

• the probability of type 2 error is symbolized as beta

Power of a Test

• the power of a test is the probability of making the correct decision and rejecting the null hypothesis when its false

Determinants of the Power of a Statistical Test

• uncontrollable factors

  • effect size

  • natural variation of populations

• controllable factors

  • sample size

  • chosen alpha value

Effect Size

• the change in your measured variable as a result of the experimental treatment that was applied

  • i.e. the ‘strength’ or ‘size’ of the phenomenon being studied

    • e.g. difference in means between treatment groups in t test or ANOVA

• generally unknown without prior experience or pilot studies

Natural Variation

• the more naturally variable the experimental units are, the harder it is to detect differences as the result of applying an experimental treatment (increase in risk of type 2 error)

Alpha Value

• there is a direct trade-off between type 1 (alpha) and type 2 (beta) errors

  • the harder you make it to reject your null (by reducing alpha), the more likely you are to also retain false null hypotheses

How to Calculate the Power of a Statistical Test?

• all stats software packages provide the ability to determine the power of various stats test

• the most important question to ask before performing an experiment is what min sample size is required to provide sufficient power

“Significance”

• statistical significance and real (biological) significance are too often assumed to be the same thing

• the real (biological) significance of research results is up to the researcher to establish based on their knowledge of the study system, it should never be assumed

Beware, Snake Oil!

• products with extraordinary claims have a low probability of success have been prevalent in ag for centuries

• farmers and consumers have had to use a “buyer beware” strategy to sort out fact from fiction for many “innovative” products