Lecture 7 – External Validity & Sampling (PSC 41, Week 3)

30 question-and-answer flashcards covering bar-graph interpretation, external validity, probability vs. non-probability sampling, types of sampling methods, random sampling vs. assignment, WEIRD samples, and practical evaluation of research headlines and claims.

What does external validity assess in a study?

The degree to which results can be generalized beyond the current study to other people, situations, or time periods.

In research, what is the difference between a population and a sample?

The population is the entire group of interest to the researcher, while the sample is the subset of that population actually studied.

Why is a representative sample important for external validity?

Because a representative sample mirrors the population, allowing findings to generalize accurately to the larger group.

Define probability sampling.

A sampling approach that uses random selection so every member of the population has an equal chance of being chosen.

Define non-probability sampling.

Sampling that does not rely on random selection; some members of the population have different or zero chances of being selected.

Name four common types of probability sampling.

Simple random sampling, systematic sampling, stratified random sampling, and cluster sampling.

Give two examples of non-probability sampling methods.

Convenience sampling and snowball sampling (others include quota and purposive/judgmental sampling).

Is selecting every 10th customer on a website an example of probability or non-probability sampling? What kind?

Probability sampling; specifically systematic sampling.

Standing outside a library and surveying passers-by is what type of sampling?

Convenience sampling (a form of non-probability sampling).

What is the main risk of convenience (self-selected) samples?

They often suffer from sampling bias, reducing external validity because volunteers may differ systematically from non-volunteers.

What does a bar graph’s error bar typically represent?

A margin of error or confidence interval showing expected variability if the study were repeated.

If two error bars on a bar graph do not overlap, what tentative conclusion is suggested?

There likely is a statistically significant difference between the groups (p < .05).

If error bars overlap on a bar graph, what does this suggest?

There may be no real difference between groups; the result may not be statistically significant (p ≥ .05).

What is the key question to ask when evaluating the external validity of a headline claiming “95 % of workers are considering quitting”?

Who exactly was sampled and does that sample represent all workers (e.g., was it only users of a job site like Monster.com)?

What four ‘big validities’ do researchers consider when judging a study?

Construct validity, external validity, statistical validity, and internal validity.

Which type of validity is most critical for frequency claims?

External validity, because frequency claims generalize a percentage or rate to a population.

How does random sampling differ from random assignment?

Random sampling selects participants from the population (enhancing external validity); random assignment allocates sampled participants to experimental conditions (enhancing internal validity).

What acronym describes many psychology study samples that limit external validity?

WEIRD – Western, Educated, Industrialized, Rich, and Democratic.

Why might a researcher knowingly accept low external validity?

Because the study’s primary goal may be to test causal mechanisms (internal validity) rather than to generalize widely.

Which probability sampling technique did the ABCD Study use to recruit schools?

Cluster sampling of schools within geographic catchment areas to achieve a nationally representative sample of children.

What is stratified random sampling?

Sampling in which the population is divided into meaningful strata (e.g., gender, race), and random samples are drawn from each stratum proportionally.

What is systematic sampling?

Selecting every k-th element from an ordered list after a random start.

Describe snowball sampling.

A non-probability method where existing participants recruit acquaintances, often used for hard-to-reach populations.

Give one advantage and one disadvantage of cluster sampling.

Advantage: Practical and cost-effective for widely dispersed populations. Disadvantage: Higher sampling error if clusters differ greatly from each other.

What question helps judge whether research findings generalize to other times?

Under what conditions (times, contexts) would we expect to see the same results?

Why is weighting sometimes applied to survey data?

To adjust the sample so it matches population parameters (e.g., census data) and corrects for unequal selection probabilities or nonresponse.

In the Pew study on factual vs. opinion statements, why is the sample considered nationally representative?

Because participants were recruited via probability methods (RDD and ABS) and weighted to match U.S. census demographics.

How can overlapping error bars be misleading?

Overlaps do not guarantee nonsignificance—formal statistical tests are needed; likewise, slight non-overlap may exaggerate significance.

What is a quota sample?

A non-probability sample where researchers set target numbers for subgroups and recruit until quotas are filled, without random selection.

Explain the term ‘sampling bias.’

A systematic error in selecting participants that makes the sample unrepresentative of the population, threatening external validity.