Sampling Methods, Bias, and Survey Design

Sampling Foundations and Statistical Bias

  • Convenience Sampling:

    • Definition: The practice of selecting individuals from the population who are easiest to reach.

    • Statistical Reliability: Convenience samples almost guaranteed to exhibit bias, rendering them ineffective for generalizing to a broader population.

    • Library Homework Study Scenario:

    • Research Objective: Estimate the average time spent on homework during the previous week by students at a large high school.

    • Sampling Method: The researcher enters the school library and surveys the first 3030 students encountered.

    • Resulting Flaw: The sample is unrepresentative of the entire student body because students congregating in a library are typically more studious than the general school population.

    • Direction of Bias: Repeated trials of this sampling strategy will systematically overestimate the true population parameter for average weekly homework time.

  • Statistical Bias:

    • Formal Definition: A statistical study exhibits bias if its design produces a sample statistic that systematically overestimates or systematically underestimates the true population parameter due to an inherent flaw in the data collection process.

    • Essential Rule for Describing Bias: Any complete statistical description of bias must identify both the underlying mechanism causing the error and the explicit direction of the bias (whether the statistic is an overestimate or an underestimate of the true population parameter).

Voluntary Response and Self-Selection Bias

  • Voluntary Response Sample:

    • Definition: A sample composed of individuals who choose to participate in response to a general invitation (frequently termed a self-selected sample).

    • Mechanism of Bias: These samples systematically overrepresent individuals with strong, entrenched opinions—most often individuals with extreme viewpoints in a single direction.

    • Call-In Media Polls: Talk radio shows and televised broadcasts soliciting call-in votes on polarizing social controversies (such as abortion, affirmative action, or gun control) systematically capture respondents with extreme feelings, yielding data that cannot represent the broader public.

  • Case Study: The Ann Landers Parenthood Survey:

    • Background: Syndicated advice columnist Ann Landers asked her readership whether they would choose to have children if they could live their lives over again.

    • Outcome: An overwhelming 70%70\% of respondents stated that they would not have children again.

    • Comparative Contrast: A concurrent survey conducted by Good Housekeeping in the same year revealed that 95%95\% of respondents would choose to have children again.

    • Methodological Failure: Both surveys relied on voluntary responses. Disgruntled, overwhelmed parents disproportionately wrote to Ann Landers, whereas satisfied parents disproportionately answered Good Housekeeping. The resulting 25%25\% divergence demonstrates why voluntary response data cannot be trusted.

  • Case Study: Parade Magazine Cell Phone Survey:

    • Background: In June 20082008, Parade magazine prompted readers with the question: "Should drivers be banned from using all cell phones?" by directing them to cast a vote online at parade.com.

    • Published Findings: The July 1313, 20082008, issue reported a total response count showing that 24072407 participants (85%85\%) voted "Yes," while 410410 participants (15%15\%) voted "No."

    • Sample Classification: Voluntary response sample.

    • Direction and Source of Bias: The reported 85%85\% statistic is substantially higher than the true population parameter of all adults favoring a complete ban. Individuals who have personal experience with vehicular collisions caused by distracted driving hold intense emotional stakes and are significantly more motivated to visit the website and cast a ballot in favor of the ban.

Undercoverage and Nonresponse Errors

  • Undercoverage:

    • Definition: Occurs when specific groups within the target population are systematically excluded from the sampling frame or possess a lower probability of being selected.

    • Real-World Undercoverage Examples:

    • Household Surveys: Standard residential sampling frames regularly omit homeless individuals, institutionalized prison inmates, and university students residing in dormitories.

    • Landline Phone Polls: Calling landline phone numbers excludes households that rely entirely on mobile devices as well as low-income households lacking any phone service.

    • Consequences of Undercoverage: If the excluded demographic groups hold attitudes, economic statuses, or behaviors distinct from the sampled segment, the final sample statistic will be biased.

  • Nonresponse:

    • Definition: Occurs when an individual who has been properly selected for a probability sample cannot be reached or refuses to cooperate.

    • Impact on Study Validity: As nonresponse rates rise, the risk of nonresponse bias increases, which can undermine the representative properties of an otherwise sound probability sampling design.

    • Differentiating Nonresponse from Voluntary Response:

    • Nonresponse can only occur after an individual has been selected into the sample through a formal sampling procedure.

    • Voluntary response samples involve participants selecting themselves from the outset, which means nonresponse does not occur in that design.

  • Case Study: High School Bus Arrival Survey:

    • Scenario: School administrators want to determine the average duration of time before the start of the school day that students get out of bed. They select a random sample of students riding each school bus on a single morning.

    • Primary Flaw: Undercoverage. The administrators sample only students using public school bus transit, entirely omitting students who walk, bicycle, drive personal vehicles, or receive private rides from parents.

    • Direction of Bias: The resulting estimate will be too large (an overestimate of average wake-up time). Bus riders are required to wake up significantly earlier to meet rigid bus pickup timetables compared to peers traveling by private transportation.

Response Bias and Question Design

  • Response Bias:

    • Definition: A systematic pattern of inaccurate or incorrect responses supplied by participants in a sample survey.

    • Drivers of Response Bias:

    • Leading or confusing question wording.

    • Influence or demeanor of the interviewer.

    • Hesitancy to disclose sensitive, embarrassing, or illegal behaviors.

  • Impact of Question Wording:

    • Dental Hygiene Case:

    • Biased/Leading Formulation: "The American Dental Association recommends brushing your teeth three times a day. How often do you brush your teeth on a typical day?"

    • Neutral Formulation: "How often do you brush your teeth on a typical day?"

    • Effect: Invoking professional authority creates psychological pressure on respondents to inflate their reported tooth-brushing frequencies.

    • Space Exploration Funding Case:

    • Neutral/Aspirational Formulation: "Do you favor or oppose a new US space program that would send astronauts to the moon?"

    • Cost-Framed Formulation: "Do you favor or oppose the US government spending billions of dollars to send astronauts to the moon?"

    • Effect: Highlighting the extreme financial burden ("spending billions of dollars") prompts participants to oppose the program at much higher rates.

  • Interviewer and Social Desirability Effects:

    • Demographic Characteristics: The visible race, perceived gender, or age of an interviewer can distort responses, particularly when survey items address racial relations, civil rights, or feminist issues.

    • Compliance and Social Stigma: In face-to-face or telephone interviews, respondents frequently conceal behaviors that violate social norms, community expectations, or statutory laws out of embarrassment.

  • Case Study: El Paso Seat Belt Observational Study:

    • Methodology: Researchers in El Paso, Texas, observed drivers parking at randomly selected convenience stores to document physical seat belt compliance. Upon exiting their vehicles, the drivers were invited to complete a survey that included questions about their personal seat belt usage habits.

    • Empirical Data:

    • Self-Reported Compliance: 75%75\% of drivers claimed that they "always" wear their seat belts.

    • Observed Compliance: Only 61.5%61.5\% of those same drivers were observed wearing a seat belt when driving into the parking lot.

    • Cause of Bias: Participants knew seat belt usage is socially expected and legally mandated; they were ashamed or embarrassed to admit non-compliance to the interviewer.

    • Generalizability: This upward bias in self-reported compliance can be anticipated in virtually all direct surveys assessing seat belt usage and similar public health safety practices.

AP Free-Response Analysis: University Dining Hall Survey

  • Problem Context:

    • Environment: A university housing complex where dormitory residents eat at a common dining facility.

    • Context: Residents voiced complaints regarding food quality.

    • Management Goal: The dining hall manager sought to estimate the proportion of dormitory residents who believe dining hall food quality should be improved.

    • Sampling Strategy: The manager positioned themselves at the entrance one evening and surveyed the first 100100 students who walked inside.


Survey question administered to dormitory students regarding dining hall food quality

  • Survey Question Text:

    • "Many students believe that the food served in the dining hall needs improvement. Do you think that the quality of food served here needs improvement, even though that would increase the cost of the meal plan?"

    • Response Options: \_\_\_ Yes, \_\_\_ No, \_\_\_ No opinion.

  • Part (a): Sampling Procedure Bias and Correction:

    • Flaw Identification: The manager used a convenience sample consisting exclusively of the first 100100 students entering the cafeteria.

    • Mechanism of Bias: Students arriving early to the dining hall may have schedules, dining habits, or food quality perspectives that systematically differ from the rest of the dormitory population (for example, individuals eager to eat early may have different culinary expectations, specific meal schedules, or distinct class routines).

    • Methodological Remedy: Replace the convenience approach with a simple random sample of 100100 students selected from a complete registry of all dormitory residents.

  • Part (b): Question Wording Bias and Correction:

    • Dual Wording Flaws:

    • First Clause Bias: The introductory statement ("Many students believe that the food served in the dining hall needs improvement") is leading; it establishes a prevailing social norm that encourages respondents to conform by answering "Yes."

    • Second Clause Bias: The conditional consequence ("even though that would increase the cost of the meal plan") introduces a financial deterrent, compelling students who want better food to answer "No" simply because they wish to avoid higher fees.

    • Unbiased Rewording: Remove all leading statements and financial qualifiers, framing the query neutrally:

    • "Do you think that the quality of the food served in the cafeteria needs improvement?"