Sampling, Bias, and Inference: T-Tests and Confidence Intervals

Sampling Methods and Bias

• Sample of Convenience vs. Simple Random Sample (SRS)

  • A sample of convenience, such as sampling students in the early morning or only those living in a specific dorm, often lacks representativeness. For example, it might yield more students who live on campus than the actual average.
  • A simple random sample involve taking a random selection from the population (e.g., $n = 30$ students).
  • While an individual SRS may not match the population proportion exactly, the average of sample proportions from repeated simple random sampling will exactly equal the population proportion.

• Unbiased Sampling Methods

  • A sampling method is unbiased when there is no tendency to overestimate or underestimate the parameter being estimated.
  • Statistics computed using a simple random sample tend to cluster around the true parameter.
  • Whether a method is biased is determined by the sampling method and the statistic used. The sample mean ($\bar{x}$) and the sample proportion ($\hat{p}$) are unbiased estimates of population parameters when derived from an SRS.

• Sample Size and Bias

  • Increasing the sample size ($n$) does not fix bias if the sampling method itself is flawed. For example, selecting 75 students instead of 30 using the same biased early-morning sampling method does not reduce bias.
  • It is generally better to have a smaller, truly random, and representative sample than a large, biased sample.
  • The advantage of a larger sample size is the reduction of sample-to-sample variability, which leads to higher statistical power.

Challenges in Survey Design

• Question Volume and Response Bias
  • A common error in non-statistician survey design is asking too many questions, often because the researcher is unsure of what they will analyze later.
  • Long surveys can "turn off" potential respondents, leading to wasted resources and, more importantly, biased results.
  • Non-response bias occurs because only certain types of people choose to complete a long survey—typically those with extreme positions (either very positive or very negative).
  • Ambivalent individuals are more likely to abandon a survey with 50 or more questions, leaving only the responses from those with strong feelings.
  • Designing surveys with as few questions as possible reduces bias and increases the effective sample size by encouraging more respondents.

Perception of Time and Data Metrics

• The Jackson 5 "ABC" Experiment

  • In a study on time perception, subjects were played a 10-second snippet of the song "ABC" by the Jackson 5 and asked to estimate its length.
  • Example parameters:
    • Null Hypothesis ($H_0$): $\mu = 10$ seconds (the population average estimate is accurate).
    • Alternative Hypothesis ($H_a$): $\mu \neq 10$ seconds (two-sided test) or $\mu > 10$ seconds (if testing specifically for overestimation).
  • Sample Data ($n = 48$ students):
    • Sample Mean ($\bar{x}$): $13.71\,\text{seconds}$.
    • Median: $12\,\text{seconds}$.
    • Standard Deviation ($s$): $6.5\,\text{seconds}$.

• Skewness and Outliers

  • Right Skewed: Data is right-skewed if several outliers appear on the high end, pulling the mean to the right (greater than the median).
  • Left Skewed: Data is left-skewed if there are more values to the left, pulling the mean lower than the median.
  • Symmetry: When the mean and median are equal or very close, the data is consistent with a symmetric distribution.
  • Resistance:
    • The mean ($\bar{x}$) and standard deviation ($s$) are sensitive to outliers. Changing a single high value (e.g., from 30 seconds to 300 seconds) would drastically increase both.
    • The median is resistant to outliers; shifting an extreme value does not change the middle value in a sorted list.

The Theory-Based T-Test

• Origins and the Student T Distribution

  • Introduced in the early 1900s by a PhD student named Gossett, who published under the pseudonym "Student."
  • The $t$ distribution is similar in shape to the normal distribution but has wider ("fatter") tails, particularly when sample sizes are small.
  • As sample size increases, the $t$ distribution approaches the normal distribution.

• Assumptions for a T-Test

  • Observations must be independent (often assumed if using an SRS).
  • The population from which the sample is drawn should be normally distributed.
  • Mathematics dictates that if data are binary (zeros and ones), they can never be normally distributed; therefore, a $t$-test is technically inappropriate for binary data, though it is sometimes incorrectly done in published research.
  • There is a common rule of thumb suggesting $n \geq 20$, but mathematically, the $t$-test remains valid for smaller samples provided the population is normal.

• T-Test Calculations

  • Test Statistic Formula: $t = \frac{\bar{x} - \mu}{SE}$.
  • Standard Error ($SE$) for quantitative data: $SE = \frac{s}{\sqrt{n}}$.
  • Calculation for song snippet example: $t = \frac{13.71 - 10}{6.5 / \sqrt{48}} = 3.95$.
  • Degrees of Freedom ($df$): For a single variable test, $df = n - 1$. In the snippet example, $n = 48$, so $df = 47$.
  • P-value Result: The probability of getting a result as extreme as 3.95 is $0.000261$ (or $0.0261\%$), leading to the rejection of the null hypothesis.

Significance Levels and Errors

• Significance Level ($\alpha$)

  • The cutoff value used to determine if a p-value is small enough to reject the null hypothesis. The standard is typically $0.05$.
  • There is no "magical" property to $0.05$; it is a tradition. Some fields use different levels.
  • Genomics: Often uses lower levels (e.g., $0.01$ or less) because testing thousands of genes simultaneously increases the likelihood that 5% will appear significant purely by chance (false positives).
  • Lowering $\alpha$ (e.g., from $0.05$ to $0.01$) makes it harder to reject the null hypothesis.

• Type I and Type II Errors

  • Type I Error (False Positive): Rejecting the null hypothesis when it is actually true. The probability of this occurring is equal to the significance level ($\alpha$).
  • Type II Error (False Negative): Failing to reject the null hypothesis when it is actually false.

• Statistical Power

  • Power is defined as $1 - P(\text{Type II error})$.
  • It represents the probability of correctly rejecting a false null hypothesis.
  • Factors that increase power:
    • Larger effect size (the difference between truth and the null).
    • Larger sample size ($n$).
    • Smaller standard deviation ($s$).
    • Higher significance level (e.g., using $0.05$ instead of $0.01$ increases power but also increases Type I risk).

Confidence Intervals

• Understanding Plausibility

  • A confidence interval (CI) provides a range of plausible values for a parameter rather than a single "reject/fail to reject" binary.
  • It consists of all values of a parameter that would not be rejected in a two-sided test at a specified significance level.

• Case Study: Marine the Cancer-Sniffing Dog

  • Marine was tested on her ability to identify colorectal cancer in breath samples. In 33 trials involving five bags (one cancer, four healthy), Marine was correct 30 times.
  • Sample Proportion ($\hat{p}$): $30 / 33 = 0.909$ ($90.9\%$).
  • Null Hypothesis: $\pi = 0.20$ (random chance).
  • Confidence Interval Construction:
    • Testing $\pi = 0.70$: P-value is $0.009$. Reject (not plausible).
    • Testing $\pi = 0.80$: P-value is $0.147$. Fail to reject (plausible).
  • The 95% Confidence Interval for Marine's long-term success rate is $[0.761, 0.974]$. Values between these two bounds are considered compatible with the observed data.

• Width and Confidence Levels

  • Increasing the confidence level (e.g., from 95% to 99%) makes the interval wider. A wider range is required to be more certain that the true parameter is captured within it.
  • A 99% CI uses a significance level of $0.01$ for the underlying tests, whereas a 95% CI uses $0.05$.

• Technical Interpretation Note

  • It is considered technically incorrect to say there is a "95% probability" that the true parameter is in a specific calculated interval. The parameter is fixed.
  • Instead, the "95% confidence" refers to the process: 95% of all possible random confidence intervals generated from the population will contain the true parameter.

Questions & Discussion

Question (Sofia): Why do we conclude the students guessed "longer" than the snippet if we performed a two-tailed test? Response: While the null hypothesis ($\mu = 10$) and alternative ($\mu \neq 10$) are for a two-sided test, once the null is rejected, we look at the direction of the data. Since the sample mean ($13.71$) was significantly higher than $10$, it is acceptable to conclude that they significantly overestimated the snippet length.

Question (Ursai): For homework problems, should we do a one-sided test instead? Response: It depends on the phrasing of the question. If the prompt asks to test whether guesses were "significantly longer" than the mean, that suggests a one-sided test. If it asks if they were "significantly different," use a two-sided test.

Question (Chat): Do we use the t-test instead of the z-test when hypotheses aren't a percentage? Response: Generally yes. The $t$-test is for quantitative data where the population is assumed normal. Z-tests are for binary data (proportions/percentages). Because binary data (0s and 1s) can never be normal, a $t$-test is technically not appropriate for that data type.

Question (Chat): Is the degree of freedom always one less than the sample size? Response: For a $t$-test on a single variable, yes ($df = n - 1$). Later in the course, when dealing with two variables (e.g., seeing if $X$ causes $Y$), the degrees of freedom may be $n - 2$.