PSTAT 5LS – Data Collection: Random Sampling vs Random Assignment

Course Logistics and Timeline

• Today’s Agenda
  • Topic 2: Introduction to Data Collection ($\text{beginning at slide 9}$)
  • Topic 3: Simulation-Based Inference for $p$
• Tomorrow’s Topic
  • Topic 4: Normal Distributions
• Upcoming Deadlines
  • Homework 2 due Friday, July 4, 11:59 PM
  • Homework 3 due Tuesday, July 8, 11:59 PM
• Office Hours
  • Instructor office hours: Tuesdays & Thursdays, 2–3 PM (via Zoom)

Conceptual Framework: Sampling vs Assignment

• Sampling answers “Who are we collecting data from?”
  • Focuses on how the sample is selected from the population.
  • When selection is done randomly, every member of the population has a known, non-zero probability of inclusion ($P(\text{selected}) = \frac{n}{N}$ in simple random sampling).
  • Result: the sample is representative → findings can be generalized to the entire population.
• Random Assignment answers “Who receives which treatment?”
  • Occurs only in experimental designs.
  • Subjects already in the study are allocated to treatments by chance.
  • Balances both measured & unmeasured confounders across groups, supporting causal conclusions.
• Why the Distinction Matters
  • Random Sampling ⇢ External Validity (generalizability).
  • Random Assignment ⇢ Internal Validity (cause-and-effect).

Detailed Definitions & Consequences

• Random Sampling
  • Guarantees each subset of size $n$ in a population of size $N$ is equally likely.
  • Minimizes selection bias.
  • Enables computation of precise sampling distributions (e.g., $\hat p \sim \text{Normal}(p,\sqrt{\tfrac{p(1-p)}{n}})$ for large $n$).
• Random Assignment
  • Creates treatment groups whose expected covariate distributions are identical: $E[X\,|\,\text{Treatment}] = E[X\,|\,\text{Control}]$.
  • Justifies use of permutation tests or two-sample $t$-tests under the assumption of independence generated by design.
• No Random Sampling but Yes Random Assignment
  • Causal statements are permitted within the study sample but cannot automatically be generalized.
• Yes Random Sampling but No Random Assignment
  • Descriptive & associative findings can be generalized; causal claims remain speculative.

Visual Summary (Slide 11 – “Scope of Inference”)

• Matrix of possibilities:
  • Neither ⇢ No causal inference, no generalization.
  • Sampling only ⇢ Generalization without causality.
  • Assignment only ⇢ Causality within sample only.
  • Both ⇢ Strongest design: causal & generalizable.

Example Analyses

• Smoking & Lung Cancer (Historical Case–Control Study)
  • Design: Compared hospitalized lung-cancer patients to non-cancer controls; recorded smoking status.
  • No random sampling: Patients were already hospitalized, not a random draw from the population.
  • No random assignment: Researchers could not assign smoking status.
  • Correct multiple-choice answer: c. Neither random sampling nor random assignment.
• Memory & “Saving” Belief (Trivia-Typing Experiment)
  • Procedure: Volunteers typed 40 trivia facts; half were told data would be saved, half told it would be erased.
  • Random Assignment present (participants randomly allocated to “saved” vs “erased” belief).
  • Random Sampling absent (volunteers ≠ random sample of population).
  • Scope-of-inference question: b. cannot (generalize), can (infer causality).

Practical Take-Aways for Research Design

• To claim $\text{Treatment} \Rightarrow \text{Outcome}$ and apply that claim broadly, design must incorporate:
  • $\textbf{Random Sampling} + \textbf{Random Assignment}$.
• When only one form of randomness is feasible, state limitations explicitly in any publication or presentation.
• Remember that statistical significance does not overcome design flaws; inference rests on both data quality and mathematical analysis.

Study Tips & Connections

• Relate today’s content to upcoming Topic 3 (simulation-based inference):
  • Random assignment justifies randomization tests.
  • Random sampling underpins validity of bootstrap confidence intervals.
• Preview of Topic 4 (Normal Distributions):
  • Many sampling distributions converge to Normal form (Central Limit Theorem) when random sampling assumptions hold.
• Ethical Reminder:
  • Mislabeling a convenience sample as “representative” is scientifically misleading and can have real-world policy consequences.