Inferential Statistics: Sampling Methods and External Validity Notes
Populations, Samples, and External Validity
Population vs. Sample vs. Census
Population: The entire group of people or things that a researcher is interested in studying.
Sample: A smaller set of individuals or instances taken from the larger population.
Census: An investigation or test that involves testing the entire whole population.
External Validity
Refers to whether a study's sample accurately represents the larger population that was not directly studied.
Good External Validity: Present when the sample is a good representative of the larger population.
Unknown External Validity: Occurs when a sample is biased, making it unclear whether results generalize.
Population of Interest
Researchers typically do not study the entire population.
Population of Interest: The specific population that researchers choose and want their results to apply to.
Examples: Lab mice, college women, or men with dementia.
Prerequisite Step: Before determining if a sample represents a population, researchers must first explicitly decide which population they are interested in.
Coming from a Population vs. Generalizing to a Population
Simply coming from a population does not guarantee that a sample represents that population.
Biased Sample: Some individuals in the population have a higher chance of being selected than others.
Unbiased (Representative) Sample: Every individual in the population has an equal and known chance of being selected.
Biased Sampling Practices
Mechanisms of Sampling Bias
Bias occurs when researchers only select people who are easy to reach or individuals who volunteer to participate.
Sampling bias degrades external validity because easy-to-reach or eager participants may hold systematically different opinions or traits compared to non-participants.
Sampling Only Those Easy to Contact
Convenience Sampling: A nonprobability technique involving the study of people who are easy to reach and readily available.
Frequently utilized when researchers are unable to reach certain specific groups.
Sampling Only Those Who Volunteer
Self-Selection: A situation where a sample consists exclusively of people who voluntarily choose to participate.
Creates substantial problems for external validity.
Everyday Examples: Individuals choosing to rate a product, complete an online quiz, or evaluate a professor online.
Behavioral Distinction: Researchers cannot fully determine how individuals who choose to rate items differ from those who do not; voluntary raters typically possess stronger opinions or a higher willingness to share them.
Exception / Caveat: Self-selection bias is significantly minimized in online surveys if participants are randomly invited to participate rather than self-initiating.
Probability Sampling Techniques
Core Principles of Probability Sampling
The optimal choice when a representative sample of a population is required.
Always relies on some form of random selection.
Ensures every member of the population has an equal and known probability of selection.
Delivers strong external validity and genuine population representation.
Nonprobability Sampling: Lacks random selection, resulting in potentially biased samples.
Simple Random Sampling
The most basic form of probability sampling.
Assigns a number to every individual in the population and randomly selects numbers using computer software.
Systematic Sampling
A method where the researcher randomly selects a starting number and subsequently chooses every person until the desired sample size is attained.
Research Example: Mehl and colleagues sampled personal conversations every , functioning analogously to random conversation selection.
Cluster Sampling vs. Multistage Sampling
Simple random and systematic sampling can be excessively time-consuming and difficult due to the challenge of listing an entire population.
Cluster Sampling: Used when people are pre-divided into existing groups. Researchers randomly select specific clusters and include all individuals within those chosen clusters.
Multistage Sampling: Consists of two sequential random steps: randomly selecting clusters/groups first, and then randomly selecting individual participants from within those chosen groups.
Stratified Random Sampling
Involves categorizing the population into specific meaningful groups (strata) and then randomly selecting individuals from each group in exact proportion to their occurrence in the broader population.
Example: In a study of Canadians, researchers ensure the sample contains the exact percentage of South Asian individuals as present in the national Canadian population.
Distinction from Cluster Sampling: Strata represent meaningful demographic groups, whereas clusters are more random aggregations. Stratified samples explicitly match population proportions, whereas cluster samples do not.
Oversampling
A variation of probability sampling where researchers intentionally recruit a higher percentage of a specific group than exists in the actual population.
Example: South Asian individuals may constitute of a national population, but researchers intentionally sample them at to ensure precise analysis.
Combining Techniques and Weighting
Researchers frequently combine sampling strategies, such as pairing multistage sampling with oversampling.
Statistical Weighting: Applied to correct sample imbalances. Responses from underrepresented groups are weighted to count more, while responses from overrepresented groups are weighted to count less.
A true representative sample avoids systematic exclusion that leads to bias.
Methodological Distinction: Random Sampling vs. Random Assignment
Definition of Randomness: Refers to a process occurring entirely without a specific order, bias, or pattern.
Random Sampling
Randomly selecting participants from a population so each member has an equal opportunity for selection.
Primary Purpose: Establishes generalizability to the broader population (External Validity).
Random Assignment
Used exclusively in experimental research designs.
Involves randomly placing recruited participants into different treatment or control conditions.
Primary Purpose: Ensures groups are baseline-equivalent at the start of an experiment (Internal Validity).
Example: In an exercise intervention study, random assignment ensures baseline happiness levels are balanced equally across experimental groups prior to testing.
Nonprobability Sampling Techniques
Settling for Unrepresentative Samples
Randomly selected probability samples provide good external validity but can be difficult or resource-prohibitive to gather.
Gold Standard Example: Gallup surveys populations globally via random phone contact or random selection of geographic villages.
Nonprobability sampling is utilized whenever external validity is not the primary focus of the study.
Types of Nonprobability Sampling
Convenience Sampling: The most widely used sampling method; relies on easily accessible participants.
Purposive Sampling: Non-random selection where researchers intentionally target a specific demographic or group.
Example: Evaluating a smoking cessation protocol by intentionally selecting only active smokers.
Snowball Sampling: A targeted purposive technique where existing participants recruit or recommend additional acquaintances with specific characteristics.
Example: Studying individuals with Crohn's disease by starting with or patients and asking them to identify others with the condition.
Quota Sampling: Dividing a target population into categories and assigning quantitative target quotas for each category. Participants are recruited non-randomly until each specified target quota is filled.
External Validity in Frequency Claims and Real-World Evaluation
Priority of External Validity in Frequency Claims
Frequency Claim: A claim asserting a specific rate, percentage, or frequency of occurrence within a population.
External validity is a critical priority when evaluating frequency claims.
Direct Verification Limits: Researchers rarely test entire populations to verify sample accuracy directly.
Hypothetical Verification Example: Surveying every single citizen in Afghanistan regarding their well-being to verify the exact percentage of people suffering.
Because direct verification across an entire population is rarely feasible, evaluating a frequency claim depends on assessing the rigorousness of the sampling methodology used.
Contexts Where External Validity Is a Lower Priority
When financial or material resources prevent probability sampling, researchers prioritize other aspects of internal validity or experimental design.
In controlled experiments, researchers consistently prioritize random assignment over random sampling.
Evaluating Nonprobability Samples
When evaluating an unrepresentative sample in empirical research or real-world reports, analyze the specific degree to which the lack of representativeness limits or alters the interpretation of results.
Generalizability Principle: A frequency claim achieves optimal accuracy when derived from an unbiased, representative sample that safely generalizes to the broader population.