Dependent Samples t-Tests, Variability, and Matched Designs

Assumptions of the Dependent Samples t-Test

• Normality of Difference Scores:
  • The primary parametric assumption for a repeated measures or matched samples $t$-test concerns the difference scores ($D$), not the raw baseline or post-test scores.
  • The population of difference scores is assumed to be normally distributed.
  • In introductory and classroom computations, small sample sizes are frequently utilized for computational tractability, requiring an explicit assumption that normality is met.
  • In professional research settings, sufficiently large sample sizes frequently mitigate violations of normality via the Central Limit Theorem.
  • Normality can be formally screened and verified using statistical software packages such as SPSS before running the test.

Role and Impact of Variability in Hypothesis Testing

• General Effect of Variability:
  • Variability measures the dispersion or spread of data, quantified through statistics such as the range, variance, and standard deviation.
  • High variability within data is generally detrimental in hypothesis testing because it can obscure, mask, or dilute true treatment effects.
• Consistency of Treatment Effects:
  • When an independent variable exerts an effect, the ideal scenario is a consistent effect across all experimental units.
  • Low Variability Scenario: If difference scores cluster tightly (e.g., values such as $10$, $8$, $9$, $11$, $10$, and $11$ points, centered around approximately $9$ points), the minimal variability makes identifying a statistically significant difference straightforward.
  • High Variability Scenario: If treatment effects are inconsistent across individuals (e.g., difference scores of $0$, $+20$, $+2$, $-2$, $+10$, and $-10$), the dispersion makes it exceedingly difficult to reject the null hypothesis ($H_0$), despite the presence of large individual score swings.
• Standard Deviation Comparison Example:
  • Consider a mean difference score of $\bar{D} = 4$:
    • If the standard deviation is large ($s = 8$), scores are broadly dispersed. This high variability inflates the standard error, making it nearly impossible to detect an effect, typically leading to a failure to reject $H_0$.
    • If the standard deviation is small ($s = 2$) for the exact same mean change of $\bar{D} = 4$, the scores cluster tightly around the mean, resulting in a smaller standard error and a high likelihood of rejecting $H_0$.

Methodological Comparison of Experimental Designs

• Independent Groups Design:
  • Participants in each condition are completely distinct, independent individuals with no overlap or systematic connection.
  • Does not track individual longitudinal change over time, comparing distinct groups instead.
• Repeated Measures Design:
  • The same participants are evaluated across multiple points in time (e.g., pre-test and post-test).
  • Advantages:
    • Requires half the total number of participants relative to independent designs.
    • Facilitates tracking changes within individuals across time.
    • Eliminates between-subject variability between conditions because individuals serve as their own controls.
    • Reduced error variability enhances statistical power, which is the probability of correctly rejecting the null hypothesis when it is false.
  • Disadvantages:
    • Confounding external factors (e.g., significant life events, external stressors) occurring between measurements can account for score changes rather than the independent variable.
    • Testing effects, practice effects, or carryover effects can arise from measuring subjects multiple times, potentially distorting post-test results.
• Matched Samples Design:
  • Operates as a hybrid compromise between independent groups and repeated measures designs.
  • Employs distinct individuals who are paired based on specific pre-existing baseline characteristics.
  • Advantages:
    • Captures the variability reduction and power benefits of repeated measures by pairing similar units.
    • Avoids testing, practice, and carryover effects because each participant is measured only once.

Case Study: Anxiety Medication and Cholesterol Levels

• Research Scenario:
  • A researcher examines whether a novel pharmacological treatment for anxiety produces secondary side effects on cholesterol levels.
  • Design: Matched samples design involving $6$ matched pairs ($n = 6$ pairs; $12$ individual participants total).
  • Matching Criterion: Participants are paired based on their baseline, pre-existing cholesterol level prior to treatment.
  • Within each pair, one member is randomly assigned to receive the active anxiety drug while the other receives an inactive placebo.
  • Different pairs can begin at different initial baseline levels (e.g., one pair may begin at a baseline of $175$), but both individuals within a given pair must share an identical baseline.
  • Cholesterol concentrations are measured after a treatment period of $18$ weeks.
• Observed Post-Treatment Data ($18$ Weeks):
  • Pair A: Placebo = $180$, Drug = $188$
  • Pair B: Placebo = $200$, Drug = $201$
  • Pair C: Placebo = $190$, Drug = $197$
  • Pair D: Placebo = $170$, Drug = $174$
  • Pair E: Placebo = $210$, Drug = $215$
  • Pair F: Placebo = $195$, Drug = $194$

Step-by-Step Matched Samples t-Test Execution

• Step 1: Stating the Hypotheses:
  • A two-tailed non-directional hypothesis is tested.
  • Null Hypothesis ($H_0$): $\mu_D = 0$
    • States that the anxiety drug has no effect on cholesterol levels; the population mean difference score equals zero.
  • Alternative Hypothesis ($H_1$): $\mu_D \neq 0$
    • States that the drug changes cholesterol levels (either an increase or decrease); the population mean difference score is not equal to zero.
• Step 2: Determining Degrees of Freedom and Critical Values:
  • Sample size represents the number of pairs: $n = 6$.
  • Degrees of freedom formula: $df = n - 1$.
  • $df = 6 - 1 = 5$.
  • Significance level: $\alpha = 0.05$ (two-tailed).
  • Critical $t$-value: $t_{\text{crit}} = \pm 2.571$.
• Step 3: Calculating Difference Scores ($D$):
  • Subtraction order: $\text{Difference} = \text{Drug} - \text{Placebo}$.
  • Pair A: $188 - 180 = +8$
  • Pair B: $201 - 200 = +1$
  • Pair C: $197 - 190 = +7$
  • Pair D: $174 - 170 = +4$
  • Pair E: $215 - 210 = +5$
  • Pair F: $194 - 195 = -1$
• Step 4: Computing the Sum and Mean of Difference Scores:
  • Sum of differences:         $\sum D = 8 + 1 + 7 + 4 + 5 + (-1) = 24$
  • Sample mean of difference scores ($\bar{D}$):         $\bar{D} = \frac{\sum D}{n} = \frac{24}{6} = 4$
• Step 5: Computing the Sum of Squared Differences and Standard Deviation ($s_D$):
  • Squared difference scores ($D^2$):
    • Pair A: $8^2 = 64$
    • Pair B: $1^2 = 1$
    • Pair C: $7^2 = 49$
    • Pair D: $4^2 = 16$
    • Pair E: $5^2 = 25$
    • Pair F: $(-1)^2 = 1$
  • Sum of squared differences:         $\sum D^2 = 64 + 1 + 49 + 16 + 25 + 1 = 156$
  • Sample variance and standard deviation of difference scores:         $s_D = \sqrt{\frac{\sum D^2 - \frac{(\sum D)^2}{n}}{n - 1}}$$s_D = \sqrt{\frac{156 - \frac{24^2}{6}}{5}} = \sqrt{\frac{156 - 96}{5}} = \sqrt{\frac{60}{5}} = \sqrt{12} \approx 3.46$
• Step 6: Calculating the Standard Error and the $t$-Statistic:
  • Standard error of the mean differences ($s_{\bar{D}}$, the denominator of the $t$-test):         $s_{\bar{D}} = \frac{s_D}{\sqrt{n}} = \frac{3.46}{\sqrt{6}} \approx 1.41$
  • Calculation of obtained $t$:         $t = \frac{\bar{D} - \mu_D}{s_{\bar{D}}} = \frac{4 - 0}{1.41} \approx 2.83$
• Step 7: Statistical Decision:
  • Compare $t_{\text{obtained}} = 2.83$ against $t_{\text{crit}} = \pm 2.571$.
  • Because $2.83 > 2.571$, the test statistic falls in the critical region beyond the threshold.
  • Decision: Reject the null hypothesis ($H_0$).
  • Conclusion: The anxiety medication has a statistically significant effect on cholesterol levels.

Effect Size and Confidence Intervals

• Effect Size Quantification (Cohen's $d$):
  • Quantifies the magnitude of the change standardized in units of standard deviation.
  • Formula:         $d = \frac{\bar{D}}{s_D}$
  • Computation:         $d = \frac{4}{3.46} \approx 1.16$
  • Interpretation: An effect size of $1.16$ is categorized as large ($d > 0.80$). The drug elevates cholesterol by more than one full standard deviation.
• Confidence Interval (CI) Construction:
  • Provides a range of values likely to contain the true population mean difference ($\mu_D$).
  • General Structure:         $\text{CI} = \text{Point Estimate} \pm \text{Margin of Error}$
  • Point Estimate: The sample mean of difference scores ($\bar{D} = 4$), representing the best single estimate of expected change.
  • Margin of Error: The two-tailed critical value multiplied by the denominator of the $t$-test (the standard error):         $\text{Margin of Error} = t_{\text{crit}} \times s_{\bar{D}} = 2.571 \times 1.41 \approx 3.63$
  • Interval Bounds:
    • Lower Bound: $4 - 3.63 = 0.37$
    • Upper Bound: $4 + 3.63 = 7.63$ (also referenced as $7.863$ depending on rounding precision)
• Clinical and Practical Application:
  • For a patient with severe anxiety who already suffers from elevated cholesterol, hypothesis test significance alone is insufficient.
  • The effect size and confidence interval indicate that the drug is estimated to increase cholesterol levels by approximately $4$ points on average, with the plausible average increase falling within the range of approximately $0.37$ to $7.63$ points. This informs clinical risk-benefit assessments.