Sampling in Research: Understanding Biased and Representative Samples
Live Assignment 1 Feedback
Overall Performance: The class performed exceptionally well on Live Assignment 1, demonstrating a strong understanding of the material.
Grading Status: Grades for online submissions will be entered on Moodle today or tomorrow. Hard copies were not printed for online submissions.
Key Area for Point Loss: If significant points (more than a couple) were lost, it was likely due to a misunderstanding in Question 1D.
Misinterpretation of Question 1D: The assignment focused on the measurement of cell phone checking, specifically its inter-rater reliability, not its correlation with course grades.
Video Recording Scenario: Students observed a video recording to count how many times other students checked their phones during a lecture.
Measure of Interest: The "cell phone checking measure" was the primary focus, and its reliability needed to be assessed.
Common Error: Some students anticipated the conclusion of a future study, which would link cell phone checking to course scores (e.g., a negative correlation: more phone checking $\implies$ lower grades). This would be shown on a scatterplot with phone checking on one axis and grades on the other, displaying a downward trend.
Correct Focus: The assignment asked about the inter-rater reliability of the phone checking measure itself. This involves assessing the correlation between coders (e.g., Coder 1's counts vs. Coder 2's counts). A nice, tight positive correlation was expected here, indicating both coders consistently counted similar numbers of phone checks for the same student.
Self-Correction: Some students initially went down the wrong path but managed to self-correct during the assignment.
Introduction to Sampling
Context: The course has covered measurement (Live Assignment 1) and now moves to sampling. Sampling decisions are made before data collection (surveys, polls, observation, physiological methods).
Purpose of Sampling: Deciding how to select participants for a study.
Example 1: USA Today Poll (Crispy Survey)
Claim: " of people are looking to liven up their meal routines."
Type of Claim: Frequency claim.
Sample Size: adults nationwide (in the US).
Generalizability Question: Can we generalize this finding from adults to the entire US population?
Answer: Yes, people can be enough as long as it's a random sample.
Definition of Random Sample: All Americans (or individuals in the population of interest) had an equal likelihood or probability of being selected into the study. This contrasts with a biased sample where only certain people are selected (e.g., only website visitors).
Feasibility: Obtaining a random sample of people from the US population is feasible, though achieving true randomness is challenging.
Example 2: BuzzFeed Poll
Question: "How often do you have sex in a typical week?"
Respondents: Approximately people.
Generalizability Question: Can we generalize from BuzzFeed respondents?
Answer: Probably not. While is a large number, it's highly unlikely to be a random sample. It's a convenience sample (people who visit BuzzFeed) and a self-selected sample (people who choose to answer an intimate poll).
Bias: The method of sampling (BuzzFeed users volunteering) introduces significant bias, making generalization problematic.
Counter-intuitive Point: More people in a survey isn't automatically better if the sampling method is biased. A truly random sample of is often more generalizable than a non-random sample of .
Populations vs. Samples
Population: The entire set of people, places, things, objects, or animals that a researcher is interested in studying. (e.g., Americans, Clarkson University students, all people in the world).
Population of Interest: The specific group of individuals to whom researchers want their study results to generalize.
Sample: A smaller set of individuals drawn from the population.
Census: When every single member of the population is sampled (e.g., a national census, or surveying every student at Clarkson University).
Goal: To draw a sample whose results can generalize to the population of interest. This relates directly to external validity.
Biased Samples (Non-Representative)
Definition: A sample is biased if it does not accurately represent the population it intends to, meaning not all members of the population had an equal probability of being included.
Causes: Often due to researchers selecting only those who are easy to contact or those who volunteer.
Types of Biased Samples:
Convenience Sampling: Sampling only individuals who are easily accessible, available, or readily at hand.
Example 1: University Research: Most psychological research at universities often relies on undergraduate students (e.g., Intro Psych students) as participants. This is a convenience sample because they are readily available.
External Validity Issue: Findings from this sample might generalize to other university students but likely not to the broader non-university population or older adults.
Example 2: Prolific Academic & Teenage Girls: Prolific Academic, a crowdsourcing platform for research, experienced a surge of teenage girl participants after a popular TikTok video. Researchers using the platform suddenly found their samples heavily skewed towards 18-year-old women, limiting generalizability beyond that specific demographic.
Example 3: Landline-Only Surveys (Historical/Partial Issue): Historically, polling agencies used landlines, which biased samples towards older individuals and potentially higher-income households (people who maintained landlines alongside cell phones).
Health Behavior Discrepancies: Data from showed significant differences in health behaviors (smoking, drinking, flu shots) between landline-only households and wireless-only households, demonstrating the generalization problem.
Age Discrepancy: In , only of older adults (65+) were wireless-only, compared to of young adults (25-29), highlighting how landline-only sampling misses younger demographics.
Self-Selection: Sampling only individuals who volunteer to participate in a study.
Example: American Academy of Pediatrics (AAP) Car Seat Poll: An online poll on the AAP website asked parents if they would keep their child rear-facing until age 2 or 3. said yes.
Bias: This figure is likely an overestimation because parents visiting the AAP website are typically more invested in pediatric guidelines, care more about safety, and are more likely to follow such advice. Those who don't follow the guidelines are less likely to visit the site and answer the poll.
External Validity Issue: The results do not generalize to the broader population of parents.
When is a Biased Sample Acceptable/Okay?
While representative samples are ideal, biased samples can be acceptable in certain contexts, requiring a judgment call based on the research question.
Political Poll Predicting Governor's Race: Representative sample essential. A biased sample would lead to inaccurate predictions.
Online Readers' Ratings of a Film: Biased sample acceptable. People who review films online tend to have strong feelings (positive or negative), and these biases often balance out, leading to star ratings that can still broadly reflect public sentiment. The type of person reviewing is not expected to objectively alter the quality assessment of the film.
Estimating School Achievement from Test Scores: Representative sample essential. To accurately evaluate and compare schools, the sample of test scores must represent the student bodies of each school, not just a convenient subset.
Evaluating Surfing Conditions by Asking Surfers on the Beach: Biased sample acceptable. Surfers who have just been in the water are the most direct source of information. Their personal characteristics (beyond being surfers) are unlikely to bias their objective assessment of wave quality. The conditions are what they are, regardless of the surfer's background.
Yelp Reviews Example: Illustrates how biased samples can create validity threats.
Threat to External Validity (Self-Selection/Convenience): Yelp users tend to be younger and more budget-conscious. This means top-rated restaurants on Yelp often skew towards