Statistical Logic and Significance Testing Notes: Null Hypothesis Significance Testing (NHST)

The Logic of Statistical Testing and Null Hypothesis Significance Testing

• General Overview: The lecture outlines the logic behind statistical testing, specifically focusing on Null Hypothesis Significance Testing (NHST). This is the process of running a mathematical test to determine whether to retain or reject a null hypothesis based on data.

• Foundational Concepts: Concepts introduced include p-values, alpha levels, effect sizes, and power analysis. These will be revisited in future discussions on specific tests such as t-tests and ANOVA.

• Bridging Research and Statistics: Quantitative research involves turning vague, theoretical research questions into measurable variables (lists of numbers). This is done through operationalization.

  • Example 1: Age and Creativity:

    • Age: Easily operationalized as the number of years old.

    • Creativity: A nebulous concept. Operationalized here using a lateral or divergent thinking task (e.g., "How many uses for a paperclip can you think of in two minutes?").

    • Variables: The predictor (independent) variable is age (split into younger adults and older adults). The outcome (dependent) variable is the creativity score.

    • Statistical Question: Is there a significant difference between the two groups?

  • Example 2: Education and Happiness:

    • Predictor Variable: Years or months of formal education.

    • Outcome Variable: Self-reported happiness on a questionnaire scale.

    • Statistical Question: Is there a significant correlation (association) between these two continuous variables? A positive diagonal relationship on a scatter plot would indicate that more education correlates with higher happiness.

• The Problem of Noise: Psychology data is inherently noisy due to individual variability and the difficulty of pinning down psychological constructs. Visual inspection ("eyeballing") of data is unreliable for determining if an effect is real. Mathematical thresholds for "significance" are required to make trustworthy decisions.

Hypotheses: Lay vs. Statistical

• Definitions: In statistics, a hypothesis is not just a hunch or expectation; it is a specific mathematical relationship between variables.

• Alternative Hypothesis ($H_1$): Captures the researcher's hunch or expected effect.

  • Creativity Example: The mean creativity score in young adults is higher than in older adults.

  • Education/Happiness Example: If a researcher believes education makes people less happy, they expect a negative correlation ($r < 0$).

  • Telepathy Example: A psychic guesses cards with an accuracy greater than $50\%$.

• Null Hypothesis ($H_0$): The default case representing the absence of the effect of interest. It is the position you provisionally believe until evidence suggests otherwise.

  • Creativity Example: Mean creativity in young participants is less than or equal to the mean in older participants.

  • Education/Happiness Example: There is either no association (totally flat) or a positive correlation.

  • Telepathy Example: Accuracy is exactly $50\%$ or less.

• Complementarity: The null and alternative hypotheses must be complementary, together containing all possible mathematical cases.

Null Hypothesis Significance Testing (NHST) Framework

• The Judicial Analogy: NHST acts as a "trial" for the null hypothesis. The null is assumed "innocent" (true) until proven "guilty" (false) beyond a reasonable doubt (the significance threshold).

• The Workflow:

  1. State the research hypothesis in lay language.

  2. Formulate statistical null and alternative hypotheses.

  3. Determine the level of significance ($\alpha$).

  4. Determine and compute the test statistic from the data.

  5. Calculate the p-value ($p$).

  6. Compare the p-value with the level of significance.

  7. Decide whether to reject or retain the null hypothesis.

Possible Outcomes of a Statistical Test

• The researcher never knows the "ground truth" state of the world ($H_0$ is actually true vs. $H_0$ is actually false); they only know the decision they make.

• True Negative: The null hypothesis is true (no effect exists), the test is non-significant, and the researcher correctly retains the null.

• True Positive: The null hypothesis is false (an effect exists), the test is significant, and the researcher correctly rejects the null.

• False Positive (Type I Error): The null hypothesis is true (no effect exists), but the data happens to be extreme by chance, leading the researcher to incorrectly reject the null. This is the positive result that is "false."

• False Negative (Type II Error): The null hypothesis is false (an effect exists), but the test is non-significant, leading the researcher to incorrectly retain the null. This is "missing" an effect that was there.

Significance Levels and P-Values

• Alpha ($\alpha$): The level of significance, which serves as the threshold for how strict the evidence must be to reject the null. In psychology, it is almost always set at $0.05$ ($5\%$).

  • This means the researcher accepts a $5\%$ risk of making a false positive (Type I error).

  • Specialized fields like clinical medical research may use lower alpha levels (e.g., $0.01$ or $1\%$) because the stakes of a false positive (e.g., a drug with side effects that doesn't work) are higher.

• P-value ($p$): The probability value. It represents the probability of obtaining a test statistic as extreme as, or more extreme than, the one observed, assuming the null hypothesis is true.

  • If $p < \alpha$: The result is statistically significant; reject the null.

  • If $p \ge \alpha$: The result is not significant; retain the null.

• Test Statistic: A number calculated from the data that summarizes the strength of the effect (e.g., a proportion of correct guesses, a correlation coefficient, or a t-value).

Experimental Example: The Telepathic Magician

• Scenario: A magician claims to be psychic. We test them using a two-choice task (black or white cards).

• Chance Level: $50\%$

• Distribution of Data: If the null hypothesis (guessing at chance) is true, the outcomes follow a binomial distribution.

  • In a simulation of $100$ trials, getting $50$ correct is the most likely outcome.

  • The Critical Region: This is the extreme tail of the distribution representing the top $5\%$ of outcomes. If the magician guesses in this region (e.g., $62$ out of $100$ correct), it is deemed too unlikely to have occurred by chance alone.

• One-tailed vs. Two-tailed Tests:

  • One-tailed (One-sided): Used when the researcher is only interested in one specific direction (e.g., ONLY guessing better than chance).

  • Two-tailed: Used when any unlikely effect (much higher or much lower than expected) is of interest. The $5\%$ significance level is split into $2.5\%$ at the top tail and $2.5\%$ at the bottom tail.

• Sample Calculation: In an experiment with $100$ trials, the magician gets $62$ correct. R software computes a p-value of $0.021$. Since $0.021 < 0.05$, the result is significant, and we reject the null hypothesis.

Effect Sizes

• Significance vs. Magnitude: A p-value tells you if there is evidence of an effect, but it does not tell you if the effect is large or important.

  • Example: Testing a psychic for $10,000$ trials. An accuracy of $51\%$ might be statistically significant ($p < 0.05$), but the actual ability is very small (a small effect size).

• Cohen's D ($d$): A standardized measure of the difference between two means, calculated by dividing the difference in means by the standard deviation:

  • $d = \frac{\text{Mean}_1 - \text{Mean}_2}{SD}$

• Rules of Thumb for Cohen's D:

  • $0.2$: Small effect.

  • $0.5$: Medium effect.

  • $0.8$: Large effect.

• New Zealand Height Example:

  • Mean height for men: $176\,cm$.

  • Mean height for women: $163\,cm$.

  • Standard deviation ($SD$): $8\,cm$.

  • $d = \frac{176 - 163}{8} = 1.625$

  • Interpretation: This is a "massive" effect (much larger than typical psychology findings) because the difference between the groups is $1.6$ times larger than the variation within the groups.

Power Analysis

• Power ($1 - \beta$): The probability of correctly detecting an effect (rejecting the null) if an effect truly exists. It represents the sensitivity of the experiment.

• Determinants of Power:

  • Effect Size: Larger effects are easier to detect and thus increase power.

  • Sample Size ($N$): Larger samples produce more precise measurements and less noise, increasing power.

• Trade-offs: Researchers use power analysis to determine how many participants ($N$) are needed before starting an experiment.

  • A common standard is aiming for $80\%$ power ($0.80$ chance of detecting the effect).

  • Example: To detect a medium effect size ($d = 0.5$) at $\alpha = 0.05$ with $80\%$ power, a power analysis in R or Jamovi might reveal that $64$ people are needed in each group.

Reporting Statistics

• P-value precision: Standards (like APA) generally require reporting p-values to three decimal places (e.g., $p = 0.021$).

• Placement: P-values are usually placed at the end of the sentence reporting the finding.

Questions & Discussion

• Question: What are Type I and Type II errors?

• Response: These are historical terms for false positives and false negatives. Type I is a false positive (rejecting a true null), and Type II is a false negative (retaining a false null). While textbooks use these terms, focusing on "false positive" and "false negative" is often clearer for conceptual understanding.

• Question: If I get a significant result, what is the chance it is a false positive?

• Response: By definition, if the null hypothesis is true, the chance of getting a significant result (a false positive) is equal to your significance level ($\alpha$). If $\alpha = 0.05$, there is a $5\%$ chance of a false positive.