Two Sample T-Tests and Power

Central Limit Theorem

  • Conditions:
    • Random and independently selected sample.
    • Large sample size (more than 30).
    • Sample size less than 10% of the population.
  • Theorem: Distribution of the sample mean (\bar{x}) is approximately Normal.
    • Mean equals the population mean \mu.
    • Standard deviation equals \frac{\sigma}{\sqrt{n}}, where \sigma is the population standard deviation and n is the sample size.
  • Sample mean is distributed as N(\mu, \frac{\sigma}{\sqrt{n}}).

Two Sample T-Tests

  • Used to compare two groups to see if their means are different.

Paired vs. Unpaired Data

  • Paired Data
    • Observations in each dataset are paired.
    • Samples have the same size.
    • Examples:
      • Comparing student performance from midterm to final.
      • Comparing textbook prices at the bookstore and on Amazon.
  • Unpaired Data
    • Observations in each dataset don’t have a natural pairing.
    • Sample sizes don’t need to be the same.
    • Examples:
      • Comparing student scores from Fall and Spring semesters.
      • Comparing overall average prices on Amazon and at the university bookstore.

Inference for Paired Data

  • Take the difference of paired values and treat that as one sample.
  • Run a one-sample t-test on the differences.

Hypothesis Testing for Two Sample Means

  • Steps are similar to one-sample tests.
    • Formulate hypotheses.
    • Prepare (check conditions, set alpha level).
    • Calculate the t-statistic.
    • Calculate the p-value and interpret results.

Hypotheses

  • Null Hypothesis: H0: \mu1 = \mu2 or H0: \mu1 - \mu2 = 0
  • Alternative Hypothesis: HA: \mu1 \neq \mu2 or HA: \mu1 - \mu2 \neq 0

Calculating the Test Statistic

  • Point estimate: Difference in sample means ( \bar{x}1 - \bar{x}2 ).
  • Degrees of freedom: Smaller of (n1 - 1) and (n2 - 1).
  • Standard error: SE = \sqrt{\frac{s1^2}{n1} + \frac{s2^2}{n2}}, where s1 and s2 are the sample standard deviations, and n1 and n2 are the sample sizes.
  • Test statistic: t = \frac{\bar{x}1 - \bar{x}2}{SE}

Example: Song Length by Genre

  • Question: Are folk songs on average longer than classic pop and rock songs?
  • Data:
    • 226 classic pop and rock songs: mean duration = 222.45 seconds, standard deviation = 91.37 seconds.
    • 122 folk songs: mean duration = 232.2 seconds, standard deviation = 73.33 seconds.
  • Steps:
    • Formulate Hypotheses.
    • Check Conditions (random sample, less than 10% of songs).
    • Compute Test Statistic.

Confidence Intervals

  • Formula: \bar{X}1 - \bar{X}2 \pm t{df} * \sqrt{\frac{s1^2}{n1} + \frac{s2^2}{n_2}}

Example: Find a 95% confidence interval for the difference in song lengths.

  • Conditions are met.
  • Calculate the interval.
  • Interpretation: We are 95% confident that the difference in song length between folk songs and classic pop and rock songs is between -8.07 and 27.57.

Power

  • Framework to test if there is a difference between sample means.

Illustrative Example: Pharmaceutical Company Testing New Drug

  • Pharmaceutical company develops a new drug for lowering blood pressure.
  • They conduct a clinical trial.
    • Recruit people taking a standard blood pressure medication.
    • Control group continues current medication (with generic-looking pills).
  • Researchers want to run the trial on patients with systolic blood pressures between 140 and 180 mmHg.
  • Previous studies suggest:
    • Standard deviation of patients’ blood pressures will be about 12 mmHg.
    • The distribution of patient blood pressures will be approximately symmetric.
  • If we had 100 patients per group, the approximate standard error would be: SE = \sqrt{\frac{12^2}{100} + \frac{12^2}{100}} = 1.70.

Detecting a Difference

  • Determine values of \bar{x}{treatment} - \bar{x}{control} that would lead to rejecting the null hypothesis.
  • Assume \alpha = 0.05 (two-sided test).
  • Reject if the difference is in the lower 2.5% or upper 2.5%.
  • Assuming a Normal distribution, any difference below -1.96 * 1.70 = -3.332 or above 1.96 * 1.70 = 3.332 would be in the rejection region.
  • Suppose the new drug reduces blood pressure by 3 mmHg relative to the standard medication.

Finding the Probability of Detecting a Difference

  • Called the "power of a test."
  • Depends on the size of a difference we want to detect, sample size, and standard deviation.
  • Effect size is the difference we are looking for.

Connection with Type II Error

  • Type I error: Reject the null hypothesis when it’s actually true.
  • Type II error: Fail to reject the null hypothesis when it’s not true.
  • \alpha = probability of making Type I error.
  • \beta = probability of making a Type II error.
  • Power of a test = 1 - \beta
  • We can set the probability of making a Type I error using the alpha level.
  • We have less control over the probability of making a Type II error, but we can measure it and account for it using the power.

Using Power Calculations

  • Determine the power of a test to find a sample size that gives enough power to detect a minimum effect size.

Power in Blood Pressure Medication Test

  • With a sample size of 100 in each group to detect an effect size of 3 mmHg, the power of the test was 0.42.
  • Increase sample size to be able to detect it.
Finding Sample Sizes for Blood Pressure Medication
  • Find the sample size that gives a power of 80% with an effect size of 3 mmHg.
  • Find the z-score in the true sampling distribution that gives us 80% below.
    We need a standard error such that a z-score of 0.84 in the true sampling distribution is the same as a z-score of -1.96 in the null sampling distribution