Null Hypothesis Significance Testing, Sampling Distributions, and the Replication Crisis

Course Framework and Administrative Overview

  • Territorial Acknowledgement:

    • Recognition and respect are paid to the Kaurna, Boondik, and Bhangala people as the traditional custodians of the ancestral lands.
    • Deep feelings of attachment, relationship to country, and past, present, and ongoing connections to the land and cultural beliefs are formally acknowledged.
  • Instructional Staff and Module Progression:

    • Course Coordinator and Instructor (Weeks 1 to 3): Doctor Matt Dry (covers foundational quantitative testing, probability, and group comparisons).
    • Subsequent Instructor: Doctor John Bartholomeus (covers multiple regression analyses).
    • Final Qualitative Instructor: Doctor Clemmie Dewey (covers qualitative research methodologies).
    • Administrative Communication: Inquiries concerning extensions or personal learning circumstances must be directed to Doctor Matt Dry via email rather than posted on public course discussion boards.
  • Assessment Schedule and Deadlines:

    • Quantitative Assignment:
    • Available on MyLearning; completion requires foundational material from online lectures covering correlation (Week 1), between-groups comparisons (Week 2), and within-groups comparisons (Week 3).
    • Practical coding and R script instruction are provided in the online lectures, with the Week 4 tutorial specifically dedicated to coding for the assignment.
    • Official Due Date: September 14 (not December 14).
    • Weekly Mini Quizzes:
    • Total of 10 mini quizzes administered across the semester.
    • Format: Two questions per quiz drawn directly from corresponding lecture content.
    • Week 2 Mini Quiz Deadline: Friday at 05:00 (5:00 PM5:00\,\text{PM}); strict hard deadline with no reopenings granted under any circumstances.
    • Final Examination Preparation:
    • A guaranteed short-answer question (valued at 5 to 10 points; requiring a two- to three-paragraph or half-page explanation) requires defining Null Hypothesis Significance Testing (NHST), detailing what the associated pp-value represents, and explaining its relationship to sampling error.

Classroom Interaction

  • In-Class Exchange:
    • An interruption occurred regarding audience noise: "Please don't talk. Hello. Yes. You there? Yes. Hello. Hi. Hi. You there? Don't talk. Please. Don't talk. Thank you."

Foundations of Quantitative Psychological Research

  • Quantitative versus Qualitative Methodologies:

    • Quantitative psychological science relies on the collection and statistical analysis of numerical data to identify patterns, characterize relationships, evaluate differences between groups, and formally test hypotheses or mathematical models.
    • Research inquiries target psychological functioning, cognitive processes, and observable behaviors.
  • The Four-Stage Quantitative Research Process:

    • 1. Research Question Formulation: Defining a specific empirical query (e.g., investigating the relationship between perfectionistic attitudes and problematic eating behaviors, or evaluating whether distinct study approaches yield differential academic outcomes).
    • 2. Data Collection: Utilizing surveys, laboratory experiments, or observational instruments to gather structured numerical data.
    • 3. Statistical Test Selection and Computation: Applying analytical procedures appropriate for the research design, including bivariate correlation, multiple regression, independent or paired tt-tests, Analysis of Variance (ANOVA), or non-parametric alternatives.
    • 4. Outcome Interpretation: Evaluating statistical output to determine whether expectations were supported, whether contrary patterns emerged, or if findings remain inconclusive.

Statistical Indicators and Effect Sizes

  • Function of Effect Sizes:

    • Effect sizes quantify the magnitude of an observed phenomenon, representing either the strength of association between variables or the absolute distance between group means relative to variance.
  • Common Effect Size Metrics:

    • Pearson's correlation coefficient (rr): Denotes the direction and strength of a linear relationship between two continuous variables.
    • Coefficient of determination (r2r^2): Represents the proportion of variance in a dependent variable accounted for by a predictor or a linear regression/multiple regression model.
    • Eta (η\eta) / Eta-squared: Indicates the proportion of variance accounted for within an Analysis of Variance (ANOVA) framework.
    • Cohen's dd: Expresses the difference between two group means scaled relative to their pooled standard deviation.
  • Cohen's (1988) Interpretive Benchmarks for Correlation Coefficients (rr):

    • r=0r = 0: Complete absence of a linear relationship.
    • r=.10r = .10: Weak correlation; typically imperceptible upon visual inspection of a scatterplot data cloud.
    • r=.30r = .30 to r=.40r = .40: Moderate correlation; discernible directional trajectory in a scatterplot.
    • r=.50r = .50: Strong correlation; substantial linear clustering.
    • r=.80r = .80: Extremely strong correlation; exceptionally rare in empirical psychological investigations.
    • r=1.0r = 1.0 or r=−1.0r = -1.0: Perfect positive or negative linear relationship forming an exact straight line.

Probability Values (pp-Values) and APA Reporting Conventions

  • Mathematical Nature of the pp-Value:

    • The pp-value is a continuous probability bounded strictly between 00 and 11.
    • Extreme values:
    • p=1p = 1: Absolute certainty (100%100\% probability) that the event occurs.
    • p=0p = 0: Absolute impossibility (0%0\% probability) that the event occurs.
    • Benchmark values:
    • p=.50p = .50: A 1 in 2 chance (50%50\% probability; equivalent to an unbiased coin flip yielding heads).
    • p=.05p = .05: A 1 in 20 chance (5%5\% probability of occurrence).
    • p=.01p = .01: A 1 in 100 chance (1%1\% probability of occurrence).
  • Statistical Significance and Alpha (α\alpha):

    • Statistical significance is established by comparing the calculated pp-value against a predetermined decision threshold termed the alpha level (α\alpha).
    • Standard psychological convention establishes α=.05\alpha = .05.
    • A test is classified as statistically significant if and only if p<.05p < .05.
  • American Psychological Association (APA) Formatting Standards:

    • Decimal Notation: Omit the leading zero before the decimal point for statistics that cannot mathematically exceed 11 (write p<.05p < .05 or p=.02p = .02, never 0.050.05 or 0.020.02).
    • Typography: All statistical symbols (pp, rr, tt, FF) must be italicized.
    • Precision: Report exact pp-values to two or three decimal places.
    • Lower Bound Threshold: When a calculated probability falls below .001.001, it must be reported as p<.001p < .001 rather than as zero.
    • Tables: Shorthand asterisk notations (e.g., ∗* for p<.05p < .05, ∗∗** for p<.01p < .01) are permissible to conserve space, whereas statistical blocks within running text require explicit reporting.

Null Hypothesis Significance Testing (NHST) Framework

  • Prevalence in Psychological Science:

    • NHST constitutes the dominant inferential paradigm, utilized in approximately 80%80\% to 90%90\% of quantitative psychological analyses (contrasting with Bayesian methods, which reject the NHST framework).
  • Dual Hypothesis Structure:

    • Null Hypothesis (H0H_0): Asserts the complete absence of an effect, difference, or association in the underlying population (rpop=0r_{\text{pop}} = 0).
    • Alternative Hypothesis (H1H_1): Asserts the presence of a real effect, difference, or association in the underlying population (rpop≠0r_{\text{pop}} \neq 0).
  • Empirical Illustration (Conscientiousness and Academic Performance):

    • Research question: Does the Big Five personality trait of conscientiousness correlate with performance on a Psychology 1 examination?
    • Hypotheses:
    • H0H_0: There is no correlation between conscientiousness and exam performance (r=0r = 0).
    • H1H_1: There is a non-zero correlation between conscientiousness and exam performance (r≠0r \neq 0).
    • Sample Data: A random sample of n=50n = 50 undergraduate psychology students yields an observed correlation of r=.32r = .32 with an associated p=.02p = .02.
    • Decision Rule: Because p=.02p = .02 is less than α=.05\alpha = .05, the test is statistically significant, and the researcher rejects the null hypothesis (H0H_0).
  • Precise Definition of the pp-Value:

    • Common Fallacy 1: The pp-value is not the probability that the null hypothesis is true (e.g., p=.02p = .02 does not mean there is a 2%2\% chance that H0H_0 is true).
    • Common Fallacy 2: The pp-value is not the probability that the alternative hypothesis is false (e.g., p=.02p = .02 does not mean there is a 2%2\% chance that H1H_1 is false).
    • True Definition: The pp-value represents the exact probability of obtaining a test statistic or sample effect at least as extreme as the one observed (r=.32r = .32), assuming that the null hypothesis is strictly true in the population (rpop=0r_{\text{pop}} = 0).

Populations, Samples, and Sampling Error

  • Populations versus Samples:

    • Population: The comprehensive totality of all individuals or entities sharing a defined characteristic of interest (e.g., all undergraduate psychology students globally; all individuals diagnosed with major depressive disorder). Populations are almost never directly accessible to researchers.
    • Sample: An accessible, finite subset extracted from the target population (e.g., an enrolled cohort of psychology students at Adelaide University). Samples are studied to draw inferences and generalize findings back to the broader population.
  • Mechanisms of Sampling Error:

    • Definition: Sampling error is the unavoidable, natural discrepancy between an observed sample statistic (e.g., sample mean xˉ\bar{x}, sample correlation rr) and the true, unobserved population parameter (e.g., population mean μ\mu, population correlation ρ\rho).
    • Causation: It does not arise from experimental mistake, procedural flaw, or researcher negligence; it is an intrinsic consequence of measuring a subset rather than the entire population.
    • Physical Metaphor: Measuring the mean height of every person in a lecture hall provides the exact population parameter. Randomly drawing repeated sub-samples of 1010 individuals will generate varying sample means (some lower if shorter individuals are chosen, some higher if taller individuals are chosen). None will precisely equal the population parameter.
  • First-Year Adelaide University Psychology Model:

    • Consider a defined population of approximately 900 to 1,000 first-year psychology students enrolled at Adelaide University.
    • If all 900 students could be evaluated, the hypothetical true population parameter might be r=.27r = .27.
    • Drawing a single sample of n=50n = 50 students might yield an empirical statistic of r=.32r = .32.
    • The divergence between r=.27r = .27 (parameter) and r=.32r = .32 (statistic) is directly attributable to sampling error.

Sampling Distributions and Theoretical Properties

  • Conceptualizing a Sampling Distribution:

    • If an infinite series of independent random samples of identical size (e.g., n=50n = 50) is repeatedly drawn from a single population of 900 students and the correlation is calculated for each sample, plotting these correlations produces a sampling distribution.
    • While sample statistics cluster densely around the true population parameter (r=.27r = .27), extreme variations occur purely by chance:
    • A fortuitous selection of 50 specific students could produce an observed correlation as high as r=.70r = .70.
    • Another distinct sample of 50 students could produce an observed correlation of r=−.31r = -.31 (an inverted directional relationship relative to the population).
    • In a simulated distribution centered at r=.27r = .27, only 5%5\% of all sample correlations fall beyond the bounds of r<.01r < .01 or r>.51r > .51. The probability of sampling an extreme correlation (r≥.70r \ge .70 or r≤−.31r \le -.31) purely by chance is less than 5%5\% (p<.05p < .05).
  • The Null Population Sampling Model (rpop=0r_{\text{pop}} = 0):

    • Consider a hypothetical population of 930 students where the true correlation between conscientiousness and exam performance is strictly zero (r=0r = 0).
    • Repeatedly drawing samples of size n=50n = 50 generates a null sampling distribution centered exactly at zero.
    • Empirical dispersion: Exactly 95%95\% of all sample correlations drawn from this null population fall within the range of r=−.30r = -.30 to r=.30r = .30.
    • Only 5%5\% of samples yield correlations of r≥.32r \ge .32 or r≤−.32r \le -.32.
    • Therefore, obtaining an observed sample correlation of r=.32r = .32 from this null population has a probability of less than 5%5\% (p<.05p < .05), providing the mathematical justification to reject the null hypothesis.
  • Mathematical Foundations and Properties of Sampling Distributions:

    • Researchers do not physically collect repeated samples from a population; over 150 years of mathematical theory allows the derivation of theoretical probability distributions directly from a single sample.
    • Property 1 (Unbiased Expected Value): The mean of a sampling distribution is an exact, unbiased estimate of the true population parameter (if rpop=0r_{\text{pop}} = 0, the sampling distribution mean is 00; if rpop=.30r_{\text{pop}} = .30, the sampling distribution mean is .30.30).
    • Property 2 (Central Limit Tendency): Irrespective of the underlying distribution of individual scores in the population, the sampling distribution of the statistic approximates a normal distribution as sample size increases.
    • Property 3 (Sample Size and Sampling Error): Sampling error is inversely related to sample size:
    • At n=10n = 10, dispersion is wide and sampling error is substantial.
    • As sample size increases across n=20n = 20, n=50n = 50, and n=100n = 100, the sampling distribution narrows progressively, converging tightly around the population parameter.
    • Property 4 (Exact Probability Mapping): Because the mathematical functions of these distributions are known, exact probabilities can be computed for any obtained test statistic (tt, FF, rr).
  • Historical Tables versus Modern Computational Packages:

    • Historically, researchers hand-calculated statistics and compared them against printed tables of critical values across specific degrees of freedom:
    • For n=20n = 20, a correlation must reach r=.423r = .423 to attain statistical significance at α=.05\alpha = .05 in a two-tailed test.
    • For n=30n = 30, a correlation must reach r=.349r = .349 to attain statistical significance at α=.05\alpha = .05.
    • For 1010 degrees of freedom, an observed r=.576r = .576 exceeds the critical threshold (p<.05p < .05).
    • Modern analysis relies on software environments such as RStudio to compute exact probability densities instantaneously.

Decision Matrix, Inferential Errors, and the Replication Crisis

  • The Four Inferential Outcomes in NHST:

    • A contingency analysis between the unobserved true state of the population and the researcher's statistical decision yields four possibilities:
    • True Negative (Correct Inference): The null hypothesis is true in the population (H0H_0 true), and the researcher fails to reject H0H_0.
    • True Positive (Correct Inference): The alternative hypothesis is true in the population (H0H_0 false), and the researcher rejects H0H_0.
    • Type I Error (False Positive): The null hypothesis is true in the population (H0H_0 true), but the researcher rejects H0H_0.
    • Type II Error (False Negative): An effect exists in the population (H0H_0 false), but the researcher fails to reject H0H_0.
  • The Inherent Hazard of Setting α=.05\alpha = .05:

    • Establishing an alpha criterion of α=.05\alpha = .05 explicitly accepts that 5%5\% of all tests conducted on populations where there is zero true effect will yield a statistically significant result purely due to sampling error.
    • Approximately 1 in every 20 significant findings in the absence of a true effect is a false positive.
  • Publication Bias and the File Drawer Phenomenon:

    • Academic publishing exhibits systematic bias toward printing exclusively statistically significant outcomes (p<.05p < .05).
    • Studies yielding non-significant results (p>.05p > .05) are discarded into researchers' "file drawers" and remain unpublished.
    • Consequently, the published literature becomes overrepresented by chance Type I errors that cannot be replicated.
  • The Replication Crisis in Psychological Science:

    • Spurious Findings Example: A hypothetical published study might report that holding a pencil between one's nose and upper lip while walking forwards generates significantly more positive thoughts regarding the future than holding the pencil in one's mouth while walking backwards (p<.05p < .05). Subsequent researchers attempting the exact methodology fail to reproduce the effect because the original finding was an artifact of sampling error rather than a genuine population phenomenon.
    • Empirical Investigation: The seminal initiative Estimating the Reproducibility of Psychological Science coordinated systemic replications of published psychology experiments. A substantial proportion of studies failed to replicate the original findings or demonstrated drastically diminished effect sizes.
    • Epistemic Demarcation: The presence of non-replicable studies does not render psychological science invalid. Foundational phenomena remain robustly replicable across decades, such as the standard Digit Span Task assessing working memory capacity, which consistently replicates an immediate recall limit of 7±27 \pm 2 items.

Core Synthesis Checklist for Exam Preparation

  • Essential Concepts to Master:
    • Population: The complete collection of all possible entities or individuals under investigation.
    • Sample: A measured subset drawn from the population to make inductive inferences.
    • Sampling Error: The natural, unavoidable numerical discrepancy between a sample statistic and the true population parameter.
    • Sampling Distribution: The theoretical or empirical frequency distribution of a statistic obtained from an infinite series of random samples of equal size drawn from a given population.
    • Properties of Sampling Distributions: Mean equals the population parameter; distribution approaches normality with increasing sample size; dispersion narrows as sample size increases; each value possesses an exact calculable probability density.
    • NHST Hypotheses: Null (H0H_0: effect equals zero) versus Alternative (H1H_1: effect does not equal zero).
    • pp-Value: The precise probability of obtaining a sample result at least as extreme as the observed statistic, conditional on the assumption that the null hypothesis (H0H_0) is true in the population.