Sampling Concepts from Ryzin Ch 5 (Video)

Generalizability and External Validity

• Generalizability: the extent to which the findings of a study can be projected and generalized to other people in other situations.

• Purpose: to interpret what the study tells us about the larger world, not just the sample.

• Katrina example (Page 1): CBS poll of 725 adults nationwide found 77% felt government response was inadequate and 80% said the government did not act as fast as it could have. A Red Cross shelter study questioned 132 residents about their experiences.

  • On average, survivors evacuated after 4 full days.

  • 63% had sustained injuries; 81% were separated from family; 63% reported direct exposure to corpses.

  • Despite the small samples, these studies illustrate generalizability: what happened to the 132 shelter participants sheds light on what happened to millions of others.

• Generalizability is linked to external validity: the extent findings hold true outside the context of the study.

• Population of interest, sampling, and generalizability: larger true characteristics of the population improve generalizability across time, places, and groups.

• Katrina evacuees did not only go to Red Cross shelters; studying more places increases generalizability; a small random sample (e.g., CBS poll with millions of viewers calling in) can still inform broader inferences about a population.

• Experiential takeaway: generalizability focuses on what the sample implies about the population as a whole, not just the specific sample size.

Population, Sampling Frames, and Generalizability

• Population of interest: the group researchers aim to investigate (the target of inference).

• Parameters: characteristics or features of the population of interest that researchers study.

• The CBS poll example: measuring the proportion of the country that saw the government response as inadequate.

• True characteristics of the population make findings more generalizable across areas, times, and groups.

• Sampling frame: a list or map representing the population from which the sample is drawn (e.g., membership lists, registered voters, property tax records).

• Population → sampling frame → sample: the flow from the broader group to the reachable subset.

• Exit polls: based on established polling plus visual tallies of people leaving polling places.

Generalizability in Practice: Types of Samples

• Random sample (probability sampling): participants are chosen randomly from the population, yielding high generalizability.

• Voluntary sample: participants volunteer; often less representative and reliable.

• Convenience sample: participants available to the researcher; often less generalizable.

• Red Cross shelter study (example): less generalizable due to nonrandom selection and convenience of shelter-based sampling.

What Is the Population for a Given Poll? (AP Tulsa Poll Example)

• Poll question: pay raises for teachers with tradeoffs between salary and tax cuts.

• Population of interest: Oklahoma voters.

• Data in example: 757 likely voters; 54% support raising salaries to regional average instead of merit pay; 38% support merit-based plan; 57% want a teacher pay raise rather than a tax cut (text truncated in transcript).

• Notes: population is the set of likely voters in Oklahoma; among 3.7M residents, 2.6M were eligible to vote; 1.5M actually voted.

How to Select a Sample: Sampling Frames and Steps

• Steps in sampling:

  • Define the population of interest.

  • Identify a sampling frame representing this population.

  • Select a subset of units from the frame to be included in the sample.

  • Contact sampled individuals and request participation.

  • Record responses or observe.

  • Summarize findings and draw conclusions about the population.

• Sampling frame examples: membership mailing lists for a volunteer organization; lists of registered voters; property tax records.

• Exit polls combine polling with an intercept of voters at polling places on Election Day.

How Large Does the Sample Have to Be? Precision and Sample Size

• Statistical precision: the amount of random error in statistics computed from a sample.

• Larger sample → less random fluctuation and more precise statistics.

• Subgroup analysis matters: size of subgroups (e.g., men vs women) drives precision, not just overall sample size.

• Problem of coverage and nonresponse bias can offset the benefits of a larger sample.

Problems and Biases in Sampling

• Coverage bias: sampling frame members differ systematically from the target population in ways that affect results.

• Nonresponse bias: respondents differ systematically from nonrespondents; affects estimates when nonresponse is related to what’s being measured.

• Response rate = contact rate × cooperation rate; both influence the overall ability to generalize.

• Propensity to respond: people’s likelihood to participate can be related to the study’s topic (e.g., environmentalists more likely to respond to recycling surveys).

• Common cause model vs separate causes model: other variables (Z) may drive both participation (P) and outcome (Y) or may influence loosely linked factors; understanding causal structure is key to diagnosing bias.

• On-campus vs off-campus college life example: on-campus students may have higher propensity to respond to campus surveys, biasing results.

• Coverage problems occur when younger, unmarried people are underrepresented in a landline sampling frame, and these people may also differ on key outcomes (e.g., sexual behavior).

Ethics of Nonresponse

• When measuring production quality, researchers can weight responses, but when sampling people, consent and voluntary participation are essential.

• Researchers must inform participants about the study and protect voluntary participation; Institutional Review Boards (IRBs) oversee ethical considerations.

Nonprobability vs Probability Sampling

• Nonprobability sampling: no basis for calculating a known probability of selection; common forms include voluntary, convenience, snowball, quota, and purposive sampling.

  • Voluntary sampling: volunteers may differ from population; volunteer bias.

  • Convenience sampling: easy access; bias due to nonrepresentativeness.

  • Snowball sampling: initial respondents recruit others; useful for hard-to-reach groups (e.g., drug users, sex workers, gang members).

  • Quota sampling: divide population into groups and fill quotas with nonprobability methods.

  • Purposive sampling: select individuals with unique perspectives or roles; aim for theoretical representation rather than statistical representativeness.

• Qualitative sampling vs quantitative sampling:

  • Qualitative sampling aims for causal understanding and deep contextual insight; representativeness is not the primary goal.

  • The crucial question is identifying what to study and in what sequence to build theory.

Random (Probability) Sampling vs Randomized Experiments

• Random sampling: elements are selected from the population to make inferences about the population; primary goal is representativeness and generalizability.

• Randomized experiments: assign units to treatments to test causal effects; goal is making treatment groups equivalent, not necessarily representative of the population.

• Random samples are observational; they provide limited evidence of causal relationships.

Simple Random Sampling: Concept, Process, and Variability

• Simple random sampling (SRS): each individual has an equal chance of selection; ensures representation and nonbias in entry.

• Process: assign random numbers to units, sort by random number, select first n units.

• Example: unemployment study with a simple random sample of n = 400.

  • Let p denote the sample proportion of unemployment.

  • If the sample yields p̂ = 0.22 (22% unemployed in sample of 400), then the transcript notes a conflicting statement p = 0.055 (5.5% unemployment); this appears to be a transcription error in the source text.

• Sampling variability: different random samples from the same population yield different estimates of the population parameter.

Sampling Distributions, Standard Errors, and Confidence Intervals

• Sampling distributions: with many repeated samples, the distribution of a statistic (e.g., the sample mean or sample proportion) centers around the population parameter and tends toward a normal distribution.

• Standard Error (SE): the standard deviation of the sampling distribution; measures how much a statistic from a sample is expected to vary from sample to sample.

• For a proportion, SE is $SE = rac{ ext{sd}}{\u221a n} ext{ or more specifically } SE = \sqrt{\frac{P(1-P)}{n}}.$ (common form)

• For a mean, SE is $SE = \frac{\sigma}{\sqrt{n}}.$ (common form)

• The goal is to have SE as small as possible to improve precision.

• The size of SE depends on two factors:

  • The variability of the population (higher variability → larger SE).

  • The sample size (larger n → smaller SE).

• Confidence intervals (margins of error): provide a range where the population parameter is believed to lie with a specified probability (commonly 95%).

  • With a normal approximation, a 95% CI uses approximately ±2 standard errors: $\hat{P} \pm 2\cdot SE.$ (transcript uses this convention)

• Example for unemployment with n = 400 and p̂ = 0.22:

  • $SE = \sqrt{\frac{0.22(1-0.22)}{400}} \approx \sqrt{\frac{0.1716}{400}} \approx \sqrt{0.000429} \approx 0.0207.$

  • 95% CI: $0.22 \pm 2(0.0207) \approx 0.22 \pm 0.0414, \text{ i.e., } [0.1786, 0.2614].$

  • Note: the transcript’s other figures (e.g., p = 0.055) appear inconsistent with this calculation and may be transcription errors.

• How large a sample is needed for a desired margin of error (ME):

  • General formula (for proportions with z-critical value): $n \approx \frac{Z^2 P(1-P)}{ME^2}.$

  • A simplified, rough rule from the transcript: $n = \left(\frac{1}{ME}\right)^2.$ (this ignores P(1-P) and Z, used for quick yardstick in the notes)

  • Example from transcript for ME = 0.03 (3 percentage points) using the simplified rule: $n = \left(\frac{1}{0.03}\right)^2 = \left(33.333\right)^2 \approx 1{,}111.$

• Subgroup analysis: when breaking the sample into subgroups (e.g., ethnicity, age), you need adequate sizes in each subgroup to achieve desired precision.

Sampling in Practice: Methods

• Systematic sampling: use a sampling frame, pick a random start, then select every k-th unit.

  • Example: exit polling by sampling every 20th person (every Nth interval).

• Stratified sampling: random samples drawn separately from each stratum (group) of the population.

  • Strata must cover the entire population; every individual belongs to one stratum.

• Disproportionate sampling (oversampling): oversample underrepresented groups to ensure adequate representation in analysis; analysis adjusts for oversampling.

• Multistage sampling: study the same group over time or contexts; involves multiple stages of sampling units.

• Cluster sampling: sample clusters (e.g., households) rather than individuals; used when a natural grouping exists.

External Validity and Generalizability in Qualitative Studies

• Generalizability of qualitative studies focuses on how and why groups behave as observed, not on statistical representativeness.

• The primary goal is to understand causal relationships and use logic to determine what subjects or cases to study next.

Replication and Meta-Analysis

• Replication: repeating a study with a different sample, in a different location, time period, or with a different design.

• Replication helps establish patterns, even if a single study has limited generalizability.

• Meta-analysis: method of pooling together multiple smaller studies to obtain a larger combined study, providing a more generalizable estimate of an effect.

• Role in generalizing findings: meta-analyses often show effects that are more generalizable than descriptive percentages from a single study.

Generalizability of Relationships vs Descriptive Findings

• Relationships (e.g., health and happiness) may generalize more than simple descriptive statistics like percentages.

• Example: a positive association between health and happiness may persist across contexts even if GDP varies.

Quick Reference: Key Concepts and Terms

• Population of interest: group researchers want to draw conclusions about.

• Sampling frame: list or map representing the population from which the sample is drawn.

• Sample: subset of units selected from the sampling frame.

• Parameter: true characteristic of the population.

• Statistic: characteristic computed from the sample (e.g., sample mean, sample proportion).

• Coverage bias: when the sampling frame misses segments of the population or includes nonmembers.

• Nonresponse bias: differences between those who respond and those who do not respond.

• Propensity to respond: likelihood that a unit responds, which can bias results if related to study outcomes.

• Nonprobability sampling: sampling methods without known probabilities of selection (voluntary, convenience, snowball, quota, purposive).

• Probability sampling (random sampling): sampling methods with known probabilities of selection (e.g., simple random, systematic, stratified, cluster).

• Randomized experiments: assign units to treatments, focus on causal inference rather than representativeness alone.

• Confidence interval: a range around a statistic that would capture the population parameter a specified proportion of the time in repeated sampling.

• Margin of error: the half-width of a confidence interval.

• Sampling distribution: the probability distribution of a statistic over repeated samples from the population.

• Sampling variability: natural variation from one sample to another.

• Ethics: informed consent, voluntary participation, IRB oversight when conducting human subjects research.

Summary Takeaway

• Generalizability depends on population definitions, sampling frames, and sampling design.

• Probability (random) sampling improves generalizability and allows for estimation of population parameters with quantified precision.

• Nonprobability sampling can be useful but requires caution regarding bias and limits on generalizability.

• Replication and meta-analysis strengthen evidence by combining multiple studies to reveal broader patterns and generalizable effects.

• Always consider ethics, nonresponse, and coverage biases when planning and interpreting sampling-based research.