1/99
Comprehensive vocabulary flashcards covering sampling designs, probability vs non-probability sampling, questionnaire design, administration modes, data quality checks, nonresponse weighting, and total survey error.
Survey
A set of questions that is designed and administered to a group of people in a systematic and standardized way.
Systematic Data Collection
A feature of surveys where data collection adheres to formal, pre-determined procedures designed to ensure the collection of quality data.
Standardized Data Collection
A feature of surveys where every respondent receives the same questions, asked the same way, so that responses are comparable across individuals.
Target Population
The entire group of people or units that a researcher aims to learn about and draw conclusions on through a survey.
Sample
A selected subset of people or units drawn from a target population to participate in a survey.
Sampling
The operational process of selecting a sample from a target population.
Census
A study design in which the entire target population is surveyed, such that the sample is equal to the population (sample=population).
Non-probability Sampling
Any sampling technique in which the probability of any individual in the target population being selected into the sample is unknown.
Convenience Sampling
A non-probability sampling method where respondents are chosen based on ease of accessibility or proximity (e.g., asking anyone near you).
Purposive Sampling
A non-probability sampling method in which the researcher deliberately selects sample units based on specific characteristics or subjective judgment.
Quota Sampling
A non-probability sampling method in which researchers sample individuals until a predetermined number or proportion of specific subgroups is achieved.
Snowball Sampling
A non-probability sampling technique where existing survey participants recruit future participants from among their acquaintances.
Sample Selection Bias
A systematic bias that occurs when the outcome of interest is related to an individual's likelihood of being included in the sample.
Probability Sampling
A sampling technique in which every unit in the target population has a known, nonzero chance of being included in the sample through random selection.
Random Selection
A sample selection mechanism where inclusion into the sample is determined purely by random chance.
Nonzero Probability of Selection
The probability sampling condition requiring that every unit in the target population has a mathematical chance greater than zero (P>0) of being sampled.
Sampling Frame
The list or procedure used to identify and select units from the target population.
Ideal Sampling Frame
A list containing a single record for every individual in the target population (and no one outside it), containing contact information and unit attributes.
Simple Random Sampling (SRS)
A probability sampling method where sample units (e.g., individuals, businesses, addresses) are picked one at a time based purely on random chance, analogous to a lottery.
SRS Selection Probability
The uniform selection probability in simple random sampling, calculated as population sizesample size (e.g., 30,000100=3001).
Non-representative Sample by Random Chance
The risk in simple random sampling where random chance inadvertently yields an imbalanced sample (e.g., selecting too few freshmen).
Systematic Random Sampling
A sampling method involving picking a random number to start with and then selecting every kth unit on a list, where k=sample sizepopulation size.
Sampling Interval (k)
The fixed selection step size in systematic sampling, calculated by dividing the population size by the sample size (e.g., 10030,000=300).
Random Start (Systematic Sampling)
The randomly selected initial number within the first interval (1 to k) designating the first unit included in a systematic sample (e.g., row 179).
Sorted Sampling Frame (Systematic Sampling)
Arranging a sampling frame by a relevant variable (e.g., class year) prior to systematic sampling to ensure every group is represented in proportion to its size.
Periodicity
A vulnerability in systematic sampling where a repeating cyclical pattern in the list matches or divides evenly into interval k, skewing the sample (e.g., every 10th house is a corner house).
List Order Dependency
A drawback of systematic sampling where results depend on list order, making it impossible to know if the arrangement helps or hurts if the ordering is unknown.
Stratified Random Sampling
A probability sampling method where the population is divided into subgroups (strata) by known characteristics (e.g., age, sex, grade level), followed by independent random sampling within each stratum.
Stratum
A homogenous subpopulation defined by shared known characteristics from which sample units are randomly selected in stratified sampling.
Proportionate Stratified Sampling
A stratified sampling method where units are drawn from each stratum at the exact same rate (e.g., 100/30,000=1/300≈0.0033333), preserving population proportions.
Disproportionate Stratified Sampling
A stratified sampling design where different strata have unequal selection probabilities, producing subgroup sample proportions that differ from the target population.
Oversampling Subgroups
A form of disproportionate stratified sampling where one or more groups have a selection probability greater than their representation in the target population.
Selection Weight
A statistical weight equal to the inverse of a respondent's selection probability (1÷probability) used to restore groups to their true population share.
Multistage Cluster Sampling
A probability sampling design that involves selecting separate samples from multiple groups or clusters of units across successive hierarchical stages.
Cluster (Sampling Unit)
A naturally occurring grouping or collective unit of the population (e.g., high schools, city blocks, or counties) sampled in cluster sampling.
Cluster Heterogeneity vs. Stratum Homogeneity
The structural distinction where strata in stratified sampling are designed to be homogenous, while clusters in cluster sampling should ideally be heterogeneous.
Loss of Precision in Cluster Sampling
The increase in standard errors and reduction in statistical precision that occurs with cluster sampling compared to simple or stratified random sampling.
National Crime Victimization Survey (NCVS)
A nationally representative survey measuring criminal victimizations experienced in the past 6 months using a stratified, multi-stage cluster design.
NCVS Target Population
People age 12 and older living in U.S. households.
NCVS Reference Period
The past 6 months prior to the interview, during which victimizations reported by respondents occurred.
NCVS Sampling Unit
The physical residential address, rather than specific families or persons.
NCVS Multi-Stage Cluster Stages
The clustering hierarchy of the NCVS: Stage 1 = counties/metro areas; Stage 2 = areas of 750–1,500 people; Stage 3 = clusters of housing units.
NCVS Household Interview Protocol
The interview rule in the NCVS requiring that all household members aged 12 and older residing at the selected address are interviewed.
Panel of Addresses
A longitudinal survey structure (used in the NCVS) where physical housing units remain in the sample over time, rather than tracking specific persons or families.
Questionnaire Design Process
A three-step development sequence: 1) choosing a preliminary set of questions, 2) evaluating the questions, and 3) pretesting the questionnaire in the field.
Focus Group Discussions (Questionnaire Design)
Facilitated discussions conducted with target population members during preliminary question development to ensure questions are simple to understand and answer.
Cognitive Interviews
An evaluation technique where respondents think aloud while answering draft questions to help researchers detect flaws and comprehension difficulties.
Field Pretesting
Administering a draft survey under field conditions to assess interviewer delivery, respondent clarification requests, and answer adequacy.
Social Desirability Bias
The tendency of respondents to answer survey questions inaccurately in order to look good in front of the person administering the survey.
Self-Administered Survey
A survey format completed directly by the participant without an interviewer present, recommended for sensitive questions to minimize social desirability.
Aural vs. Visual Survey Mode Design
The design requirement that questions must match the mode: simpler questions for aural modes (listening), and layout design features for visual modes (reading).
Web Survey Mode (Strengths)
Low cost per complete, fast deployment, capability for complex skip logic and visuals, and privacy for sensitive questions.
Web Survey Mode (Limitations)
Lack of a general email sampling frame, lower response rates, exclusion of offline populations, and vulnerability to bots and fraud.
Mail Survey Mode (Strengths)
Address lists cover nearly all households, offers privacy, and allows respondents to set their own completion pace.
Mail Survey Mode (Limitations)
Slow turnaround times, printing and postage costs, limited skip logic capacity, and inability to confirm who completed the questionnaire.
Phone Survey Mode (Strengths)
Interviewers can clarify questions and probe answers, and surveys can be fielded quickly.
Phone Survey Mode (Limitations)
Very low response rates that drive up costs, necessity for short and simple questions, and increased vulnerability to social desirability bias.
In-Person Survey Mode (Strengths)
Highest response rates, capacity to handle long and complex interviews, and option for respondents to enter sensitive answers privately on a laptop.
In-Person Survey Mode (Limitations)
The most expensive and slowest survey administration mode, along with vulnerability to interviewer effects.
Interviewer Effects
Distortions introduced into survey responses caused by the presence, demeanor, tone, or characteristics of an interviewer.
Mixed-Mode Survey Design
The practice of using multiple contact modes, response modes, or both to leverage the strengths of certain modes and overcome weaknesses of others.
Contact Mode vs. Response Mode
The distinction between how a respondent is reached (e.g., an email invitation) versus the medium used to complete the survey (e.g., online web form).
Measurement Problem (Mixed-Mode Surveys)
The methodological complication where the exact same question produces different answers across different modes, confounding mode effects with true values.
McMaster et al. (2017)
A study evaluating survey response performance by comparing web-push surveys against paper-only surveys.
Unit Nonresponse
The failure to obtain any survey participation from an individual or unit selected into the sample.
Item Nonresponse
When an active survey participant fails or refuses to answer a specific question within the questionnaire.
Nonresponse Rate
The percentage of sampled units that did not participate in or complete the survey.
Gaining Access to Sampled Individuals
Methods to reach sampled units by making repeated contact attempts during different days and times, and employing interviewers with flexible schedules.
Getting Sampled Individuals to Agree
Methods to encourage participation, including advance letters, small cash incentives, selling survey value, limiting burden, and professional presentation.
Advance Letter
An informative letter sent prior to survey administration to establish credibility, explain the survey's value, and improve agreement to participate.
Cash Incentive (Advance)
A small financial token provided upfront with the invitation, recognized as particularly effective for boosting response rates.
Cognitive Burden
The mental effort and time required by a respondent to interpret, process, and complete a survey, which should be minimized.
Imputation
A statistical procedure used to estimate and fill in missing values resulting from item nonresponse.
Nix et al. (2019)
An empirical study focusing on police research, officer surveys, and methodologies for improving response rates.
Data Quality Check (DQC)
A procedure used to detect, prevent, or correct errors, inattentive responses, or invalid submissions in survey datasets.
Real-Time DQC
Quality control procedures built directly into digital surveys that prompt respondents to rectify errors or inconsistencies before submission.
CAPTCHA Test
A challenge-response security verification used in web surveys to block automated software and bots from submitting responses.
Single-Use Survey Link
An individualized survey URL that restricts completion to one submission, preventing repeat responses and preventing links from being shared.
Survey Farmers
Human respondents who complete surveys for which they are not eligible solely to collect financial incentives.
Westwood (2025)
Research showing that AI agents can bypass most automated quality checks, concluding that sample design (who gets invited) is the strongest defense.
Range Check
A data quality check that flags numerical responses falling outside possible or plausible bounds (e.g., reporting an age of 500).
Consistency Check
A data quality check comparing an answer to an earlier response to detect logical contradictions (e.g., prisoners by race not summing to total prisoners).
Straightlining Flag
A data quality check identifying inattentive respondents who select the same response across a set of items, including reverse-worded questions.
Speeding Flag
A data quality check that flags respondents who complete pages too quickly (e.g., completing a 5-minute page in <1 minute).
Disengagement Flag
A quality indicator identifying respondents who appear uninvested based on high numbers of skipped items or 'don't know' responses.
Honeypot Question
A hidden survey question invisible to human respondents but filled out by automated bots, used for bot identification.
Duplicate Indicator
Tracking metrics such as recorded IP addresses and device fingerprints used to identify multiple submissions from the same respondent.
Douglas et al. (2023)
A scholarly study examining data quality maintenance and threats in online human-subjects research.
Differential Nonresponse
The condition where specific demographic or behavioral subgroups in a sample fail to participate to a greater extent than other subgroups.
Poststratification
A weighting technique used to adjust a post-survey sample to make it match known population characteristics on selected variables.
Poststratification Assumption
The assumption that respondents from a given subgroup are systematically similar to nonrespondents from that same subgroup.
Poststratification Weight Calculation Example
In a 50% male / 50% female population where a sample of 400 is 40% male (N=160) and 60% female (N=240), males are weighted by 0.5/0.4=1.25 (160×1.25=200) and females by 0.5/0.6≈0.833 (240×0.833≈200).
Total Survey Error (TSE)
The conceptual framework stating that the total difference between a survey estimate and the true population value is the sum of representation errors and measurement errors.
Coverage Error
An error of representation that occurs when the sampling frame misses eligible population members or includes people outside the target population.
Sampling Error
An error of representation reflecting chance variation that arises from surveying a sample rather than the entire population.
Nonresponse Error
An error of representation that occurs when participating survey respondents differ systematically from nonrespondents.
Adjustment Error
An error of representation that occurs when post-survey adjustments, such as weighting, fail to fully correct differences between sample and population.
Validity (Survey Measurement)
An error of measurement that occurs when a survey question fails to accurately capture or reflect the intended theoretical concept.
Measurement Error
An error of measurement caused by question wording, administration mode, interviewer characteristics, social desirability bias, or recall error.
Processing Error
An error of measurement resulting from mistakes in data entry, coding, editing, programming, or data cleaning.