Statistical Simulations, P-values, and Hypothesis Testing

Introduction to Statistical Simulations

• Understanding simulations provides a foundation for grasping complex statistical concepts such as $z$     statistics, $p$-values, and general hypothesis testing.
• Simulations allow for the estimation of the standard error and distribution of a test statistic without solely relying on complex formulas.

Hyperemesis Gravidarum (HG) and Preterm Delivery

• Condition Overview: Hyperemesis Gravidarum (HG) is a rare pregnancy condition affecting approximately $1\%$ of pregnant women. It is characterized by severe nausea and vomiting, leading to a loss of at least $5\%$ of body weight instead of the expected weight gain.
• Background Statistics: In the general population, the rate of preterm delivery is $10\%$.
• Sample Data: In a specific study on women with HG, the sample size was $n = 254$. The observed preterm delivery rate in this sample was $15.34\%$.
• The Statistical Question: Is the observed rate of $15.34\%$ statistically significantly higher than the population rate of $10\%$?

Key Statistical Parameters and Statistics

• Statistic ($p_{hat}$): Any property of a sample is considered a statistic or random variable. It is often denoted by a Roman letter with a "hat" to signifies it is an estimate.
  • In this study, $p_{hat} = 0.1534$.
  • This is the best estimate of the proportion of all women with HG who deliver preterm based on the sample.
• Parameter ($\pi$): A parameter represents the true value within the entire population.
  • Under the null hypothesis ($H_0$), it is assumed there is no relationship between HG and preterm delivery, meaning the population proportion ($\pi$) for women with HG is the same as the general population: $\pi = 0.10$.
• Quantity of Interest: The difference between the observed statistic and the expected parameter.
  • $0.1534 - 0.10 = 0.0534$ (or $5.34\%$).
  • The goal is to determine if this difference of $5.34\%$ is larger than what would be expected to occur by chance alone under the null hypothesis.

Standard Deviation (SD) vs. Standard Error (SE)

• Standard Deviation (SD): A measure of how much individual numbers in a dataset typically differ from their mean. It describes the spread of observations.
• Standard Error (SE): Specifically refers to the standard deviation of a statistic (e.g., the standard deviation of the difference $5.34\%$).
  • The SE measures how much a statistic would typically vary from one sample to another if the study were repeated many times.
  • While SD is an operation applied to any set of numbers, SE is a special case of SD applied to a distribution of summary statistics.

Simulating the Null Hypothesis in R

• To estimate the SE and visualize the distribution under the null hypothesis, a computer can simulate thousands of samples of size $n = 254$ where each person has a $10\%$ chance of preterm delivery.
• The RUNIF Function: Generates pseudo-random numbers uniformly distributed between 0 and 1.
• Process:
  1. Generate 254 random numbers: runif(254).
  2. Evaluate if numbers are $< 0.1$ (True if preterm, False if not).
  3. Store as a vector where True = 1 and False = 0.
  4. The mean of this vector represents the proportion ($p_{hat}$) of preterm births for that specific simulation.
• Large-Scale Simulation: Performing this $10,000$ times creates a distribution of simulated proportions centered at $10\%$.
• Calculating SE via Formula:     $SE = \sqrt{\frac{\pi \times (1 - \pi)}{n}}$
  • Using the HG data: $\sqrt{\frac{0.10 \times 0.90}{254}} \approx 0.01882$.
  • Simulations of $10,000$ runs yielded an estimated SE of $0.01885$, demonstrating that simulations and formulas provide nearly identical results.

Understanding the $p$-value

• Definition: The $p$-value is the probability, assuming the null hypothesis is true, of observing a test statistic as extreme or more extreme than the one actually observed.
• The "As Extreme or More Extreme" Convention: Statistics does not calculate the probability of the exact observed value because that probability is often very low even for typical results.
  • Example: In $400$ coin flips, getting exactly $201$ heads has a probability of only $3.97\%$, even though it is a very typical result for a fair coin.
  • By looking at the "tail" (everything as extreme or more), we get a meaningful measure of compatibility with the null hypothesis.
• Guidelines for Interpretation:
  • $p > 0.10$: Weak or no evidence against the null hypothesis.
  • $0.05 < p < 0.10$: Moderate evidence (gray area).
  • $0.01 < p < 0.05$: Strong evidence against the null; result is statistically significant; reject the null.
  • $p < 0.01$: Very strong evidence; highly significant; reject the null.
• HG Example Result: In the simulations, only $51$ out of $10,000$ samples were as extreme as the observed $15.34\%$. The $p$-value is approximately $0.51\%$, indicating the result is highly significant.

Case Study: St. George’s Hospital Heart Transplants

• The Incident: In 2004, mortality spiked at St. George’s Hospital in London. Out of the last $10$ heart transplants, $8$ patients died within 30 days.
• Comparison: The national mortality rate for this procedure is $15\%$.
• Variables:
  • Observational Unit: Each individual heart transplant.
  • Variable: Survival vs. Death (binary).
  • Parameter ($\pi$): The long-term frequency of death at St. George's.
  • Statistic ($p_{hat}$): $0.80$ ($8/10$).
• Hypotheses:
  • $H_0$: $\pi = 0.15$ (St. George's is identical to the national average).
  • $H_a$: $\pi > 0.15$ (One-sided: mortality is significantly higher).
• Simulation Results: In $1,000$ simulations where $\pi = 0.15$, the maximum simulated death rate was $60\%$. The probability of observing $80\%$ was $0$. This is a $p$-value of $0$, providing overwhelming evidence to reject the null hypothesis.
• Extended Dataset: Researchers looked at the previous $361$ transplants and found $71$ deaths ($19.67\%$).
  • Comparing $19.67\%$ to the national $15\%$.
  • SE for this larger sample was $0.018$.
  • $p$-value: $0.003$ ($0.3\%$). Even though the effect size decreased, the large sample size maintained strong evidence against the null.

Standardized Statistics ($z$-Statistics)

• Definition: The number of standard errors an observed statistic is above or below the mean of the null distribution.
• Formula:     $z = \frac{\text{Observed Statistic} - \text{Mean of Null Distribution}}{\text{Standard Error}}$
• Null Mean: Usually the value hypothesized in the null (e.g., $0.15$ or $0.10$).
• Cut-off Value: Traditionally, a $z$-statistic outside of the range $[-2, 2]$ indicates strong evidence against the null.
  • Note: The precise mathematical cut-off for a $5\%$ significance level is actually $1.96$.
• Comparison of Heart Transplant Data:
  • Small sample ($8/10$): $z = 5.75$.
  • Large sample ($71/361$): $z = 2.61$.
  • Both are greater than $2$, but the first dataset shows even stronger evidentiary weight.

Case Study: Predicting Elections from Faces

• Research Premise: Can voters predict election results based purely on candidate appearance?
• Methodology: Participants were shown photos of two candidates and asked who looked more "competent."
• Data: In the 2004 Senate races, the panel correctly identified the winner in $23$ out of $32$ races ($71.9\%$).
• Testing Parameters:
  • $H_0$: $\pi = 0.50$ (Random guessing).
  • $H_a$: $\pi > 0.50$ (Faces have predictive power).
• Results:
  • SE from simulation: $0.09$.
  • $z$-statistic: $2.43$.
  • $p$-value: $0.009$ (below $1\%$).
  • Conclusion: Reject the null hypothesis; appearance has a statistically significant relationship with election outcomes.

Factors Influencing Strength of Evidence

1. Effect Size: The magnitude of the difference between the observed statistic and the null hypothesis. Larger differences (e.g., $80\%$ vs $15\%$) provide stronger evidence.
2. Sample Size ($n$): As sample size increases, the variability (Standard Error) decreases.
   • A smaller spread makes it harder for high differences to occur by chance.
   • With a larger $n$, even a small effect size can be statistically significant.
3. One-sided vs. Two-sided Tests:
   • One-sided tests only look for a difference in one direction.
   • Two-sided tests are more conservative and standard in research.
   • Switching from a one-sided to a two-sided test generally doubles the $p$-value.

Experimental Design: Fluoxetine and Fibromyalgia

• Terminology:
  • Placebo: An inactive substance (e.g., sugar pill) given to a control group.
  • Blinded: Participants do not know if they received the drug or the placebo.
  • Double-Blind: Neither the patients nor the clinicians evaluating the outcome know which group a subject belongs to. This prevents observer bias.
• Study Overview: $42$ women were randomized to receive either $20\,mg$ of Fluoxetine (Prozac) or a placebo.
• Findings:
  • Placebo group showed improved self-rated anxiety at three weeks (possibly due to the placebo effect).
  • Fluoxetine group: Improvement in Beck Depression Scale was not significant ($p = 0.34$).
  • At six weeks, sleep quality improvement was noted with a $p$-value of $0.03$.

Questions & Discussion

• Direct Message (DM): What does the runif code do?
  • runif(254) generates $254$ independent random numbers between $0$ and $1$ that are uniformly spread.
• DM: Is $\pi$ the same as $p$-hat?
  • No. $p_{hat}$ is the proportion in the specific sample ($15.34\%$). $\pi$ is the hypothetical population proportion under the null ($10\%$). The $p$-value is the probability of the result given $\pi$.
• DM: Will we use R in exams?
  • No. Exam questions will not include R code. Simulation data and results will be provided where necessary.
• DM: What is the difference between SD and SE?
  • SD is a general measure of spread for any list of numbers. SE is a special case of SD specifically applied to the distribution of a test statistic across repeated samples or simulations.
• DM: What do the y-axis values on a histogram represent?
  • They represent frequency (how many times a result occurred) or the proportion of times a result occurred within the total number of simulations.
• Direct Message (DM): Is the simulation repeating the sample or testing one sample of 1,000?
  • It is repeating the 10-observation (or whatever sample size) study 1,000 different times to see the variety of possible outcomes under the null hypothesis.