Comprehensive Guide to T-Tests, Statistical Assumptions, and Result Interpretation

Statistical Foundations: Significance, Effect Size, and Variable Associations

Core Metrics of Statistical Reporting:
- Statistical Significance: Assessed through hypothesis testing to determine whether an observed effect is likely due to chance under the null hypothesis.
- Clinical Significance: Reflects the practical or clinical importance and real-world relevance of a finding, distinct from mere mathematical significance.
- Effect Size: Quantifies the magnitude or strength of an experimental effect or relationship, independent of sample size.
Associations Between Different Types of Variables (Reference: https://effectsize.science):
- Discrete – Continuous Variables:
- Raw Mean Difference: The simple arithmetic difference between the means of groups, expressed in original measurement units.
- Standardized Mean Difference (SMD): Expresses the difference between group means relative to the standard deviation of the variable, facilitating comparisons across different measures:
  - Cohen's $d$ : Measures the distance between two group means in standard deviation units.
  - Hedge’s $g$ : A variation of Cohen's $d$ that applies a correction factor for small sample sizes.
- Discrete – Discrete Variables:
- Risk Difference (RD): The absolute difference in the probability (risk) of an outcome between two distinct groups.
- Risk Ratio (RR): The ratio of the probability of an outcome occurring in an exposed group to the probability in an unexposed group (relative risk).
- Odds Ratio (OR): The ratio of the odds of an outcome occurring in one group compared to the odds of it occurring in another group.
- Continuous – Continuous Variables:
- Correlation ( $r$ ): Pearson's product-moment correlation coefficient, measuring the linear strength and direction of the association between two continuous variables.

Fundamentals of Hypothesis Testing with the T-Test

The Two-Step T-Test Calculation Framework:
- Step 1 — Finding the $t$ -value: Calculate the empirical $t$ -statistic from the sample data using the sample mean, hypothesized population mean, standard deviation, and sample size.
- Step 2 — Comparing to the Critical Value: Compare the calculated $t$ -value against the theoretical critical value from the Student's $t$ -distribution table.
- The critical value threshold depends on two parameters:
  - Directionality of the Test: Whether the hypothesis is formulated as a one-tailed test (directional) or a two-tailed test (non-directional).
  - Alpha Level ( $\alpha$ ): The pre-established Type-I error rate (the probability of incorrectly rejecting a true null hypothesis, conventionally set at $\alpha = 0.05$ or $0.01$ ).

Standardized Empirical Workflow in JASP: Weight Gain Investigation

Sequential Six-Step Framework for Hypothesis Testing:
- Step 1 — Verbalize the Hypothesis: Articulate the substantive scientific question in plain language:
- Example: Determine if the average weight before intervention is larger than the known population average of $75\,kg$ .
- Step 2 — Create Null Hypothesis Significance Testing (NHST):
- Define the Null Hypothesis ( $H_0$ ): The mean weight before intervention is equal to (or less than or equal to) the population average ( $\mu \le 75\,kg$ ).
- Define the Alternative Hypothesis ( $H_1$ ): The mean weight before intervention is strictly greater than the population average ( $\mu > 75\,kg$ ).
- Step 3 — Conduct the Appropriate Statistical Test: Execute the test suited to the experimental design (in this scenario, a one-sample $t$ -test comparing sample weight data to the test value of $75\,kg$ ).
- Step 4 — Evaluate Statistical Significance: Inspect the computed $p$ -value and compare it directly to the designated alpha threshold ( $\alpha$ ) to decide whether to reject $H_0$ .
- Step 5 — Quantify the Effect Size: Calculate and report the standardized effect size (such as Cohen's $d$ ) to ascertain practical magnitude.
- Step 6 — Verbalize the Results: Translate statistical conclusions, confidence intervals, and effect metrics into clear, substantive domain-level findings.

Taxonomy of T-Tests and Methodological Alternatives

Classification of T-Test Procedures:
- One-Sample T-Test: Compares the mean of a single sample against a known, pre-specified population mean or theoretical benchmark value.
- Independent-Samples T-Test: Compares the means of two distinct, unrelated groups to evaluate whether their population means differ significantly.
- Paired-Samples T-Test: Evaluates the mean difference between two matched, related groups or repeated measurements from the same participants (e.g., pre-test versus post-test scores).
Extended T-Tests and Non-Parametric Alternatives:
- One-Sample Designs:
- Standard Parametric Test: Student's $t$ -test.
- Alternative Method: Wilcoxon signed-rank test (employed when the assumption of normality is violated).
- Independent-Samples Designs:
- Standard Parametric Test: Student's $t$ -test.
- Alternative for Heterogeneity of Variance: Welch's test (Welch's $t$ -test; utilized when the assumption of equal population variances is violated).
- Alternative for Non-Normality: Mann-Whitney $U$ test (employed when the assumption of normal distribution within groups is violated).
- Paired-Samples Designs:
- Standard Parametric Test: Student's $t$ -test.
- Alternative Method: Wilcoxon signed-rank test (employed when the assumption of normality among paired difference scores is violated).

Core Assumptions of T-Tests and Diagnostic Protocols

Nature and Definition of Assumptions:
- Statistical tests rely on theoretical conditions regarding data distribution, measurement scales, and experimental designs.
- Violating these assumptions invalidates error rates, test statistics, and substantive inferences.
- Specific assumptions vary according to the structure and execution of each statistical test.

Summary table of statistical assumptions across one-sample, independent-samples, and paired-samples t-tests

Assumption 1: Variable Type of the Outcome
- Requirements Across Tests:
- One-sample $t$ -test: Outcome variable must be continuous.
- Independent-samples $t$ -test: Outcome variable must be continuous.
- Paired-samples $t$ -test: Outcome variable must be continuous.
- How to Check: Inspect the operational measurement level of the outcome variable (confirming interval or ratio scale rather than nominal or ordinal).
- Response to Violation: Change the statistical method to an alternative suited for categorical, count, or ranked data.
Assumption 2: Independence of Observations
- Requirements Across Tests:
- One-sample $t$ -test: All observations must be independent of one another.
- Independent-samples $t$ -test: Observations must be independent both within each individual group and between the two groups.
- Paired-samples $t$ -test: Pairs of observations must be independent of each other, although the two measurements within any given pair are dependent.
- How to Check: Review the sampling procedure, experimental design, participant recruitment, and study execution.
- Response to Violation: Change to a statistical method that explicitly accommodates dependent or clustered data (e.g., repeated-measures analysis, multilevel/mixed-effects models).
Assumption 3: Normality
- Requirements Across Tests:
- One-sample $t$ -test: The outcome distribution must be normally distributed in the population.
- Independent-samples $t$ -test: The outcome must be normally distributed within each of the two respective groups.
- Paired-samples $t$ -test: The differences between paired observations must be normally distributed.
- How to Check:
- Graphical visualization: Histograms and Quantile-Quantile plots (Q-Q plots).
- Statistical testing: Formal goodness-of-fit testing via the Shapiro-Wilk test.
- Response to Violation:
- Mild deviations from normality are generally acceptable without modifications, especially with larger sample sizes due to the Central Limit Theorem.
- Substantial deviations require adopting non-parametric tests (Mann-Whitney $U$ test or Wilcoxon signed-rank test), robust statistical procedures, or bootstrapping techniques.
Assumption 4: Homogeneity of Variance
- Requirements Across Tests:
- One-sample $t$ -test: Not applicable (no comparison between sample variances).
- Independent-samples $t$ -test: Equal population variance between the two comparison groups ( $\sigma_1^2 = \sigma_2^2$ ).
- Paired-samples $t$ -test: Not applicable (the analysis reduces to a single vector of difference scores).
- How to Check: Inferential testing via Levene's test for equality of variances.
- Response to Violation: Employ Welch's independent-samples $t$ -test, which adjusts the test degrees of freedom to account for unequal variances.

Scatter plot of birdies versus number of events played demonstrating heteroscedasticity across predictor values

Assumption 5: Absence of Outliers
- Requirements Across Tests:
- One-sample $t$ -test: No extreme outliers present in the distribution.
- Independent-samples $t$ -test: No extreme outliers present within each separate group.
- Paired-samples $t$ -test: No extreme outliers present within the distribution of paired differences.
- How to Check: Visual evaluation of distributions utilizing boxplots and histograms.
- Response to Violation: Inspect the dataset for recording or data entry errors, investigate influential observations to evaluate their impact and veracity, or consider robust statistical methods resistant to outlier contamination.

The Statistical Decision Tree Framework

Sequential Five-Stage Analytical Strategy:
- 1. Research Question: Formulate the primary scientific inquiry and state the directional or non-directional hypothesis clearly.
- 2. Variables: Determine and categorize all study variables according to measurement level (discrete vs. continuous) and functional role (independent/grouping vs. dependent/outcome).
- 3. Statistical Test: Select the candidate statistical test aligned with the research design and variable classifications (e.g., one-sample, independent-samples, or paired-samples Student's $t$ -test).
- 4. Assumptions: Systematically verify whether the mathematical and design assumptions of the chosen test are met using graphical plots and formal diagnostic tests.
- 5. Response to Assumptions: When diagnostic testing reveals assumption violations, systematically select the corresponding corrective action (e.g., transition from Student's $t$ -test to Welch's $t$ -test, apply the Mann-Whitney $U$ test, or use the Wilcoxon signed-rank test).