Sampling from a Population: Sampling Bias, Randomization, and Data Collection Distortions

Foundations of Sampling and Statistical Inference

• Core Terminology and Definitions:

  • Population: Includes all individuals, entities, or objects of interest in a study.

  • Sample: A subset of the population comprising all the cases on which data are collected.

  • Statistical Inference: The formal process of using data collected from a sample to draw conclusions, gain insights, and make generalizations about the larger population.

  • Cases: The individual observational units or subjects on which data are gathered (for example, individual humans in a public opinion survey).

  • Variables: Characteristics recorded for each case, which can be categorical (placing an individual into one of several groups or categories) or quantitative (taking numerical values for which arithmetic operations make sense).

  • Example (Dietary Habits): To estimate the proportion of people who follow a vegetarian diet, each individual case is classified as either a vegetarian or not a vegetarian. Because the response falls into discrete descriptive categories rather than measurements, the variable measured is categorical.

• The Data Inference Cycle:

  • Statistical investigation operates as a continuous inferential cycle between the broader world and collected data points.

  • Sampling Step: The researcher transitions from the comprehensive Population to a manageable Sample by selecting specific cases.

  • Statistical Inference Step: After collecting and analyzing data from the sample, the researcher projects findings back onto the unobserved Population.

• The Soup Analogy for Representative Sampling:

  • Consider tasting a pot of soup to judge its overall flavor profile.

  • The Population represents the entire bowl or pot of soup.

  • The Sample represents the specific contents contained within tasting spoonfuls.

  • If bites are taken non-randomly—such as deliberately stabbing with a fork or selectively skimming only noodles while avoiding vegetables—the sample fails to represent the true composition of the soup.

  • If the soup is thoroughly stirred and tasted at random, only a few small spoonfuls are necessary to obtain an accurate assessment of the entire pot.

Sampling Bias and Historical Case Studies

• Sampling Bias Mechanics:

  • Sampling bias occurs when the method utilized to select a sample systematically causes that sample to differ from the population of interest in a relevant way.

  • The primary goal of any sampling design is to obtain a sample that mirrors the population in all relevant traits, only smaller in size.

  • When sampling bias is present in a study design, generalizations from the sample to the population are fundamentally compromised and cannot be trusted.

• Case Study: The $1948$ Presidential Election ("Dewey Defeats Truman"):

  • Prior to the conclusion of the $1948$ United States presidential election between Thomas E. Dewey and Harry S. Truman, the Chicago Daily Tribune printed its early morning edition featuring the headline "DEWEY DEFEATS TRUMAN".

  • The premature headline relied heavily on a large-scale telephone survey conducted during the final phases of the campaign, which projected a sweeping victory for Dewey.

  • In the actual election, Harry S. Truman won the presidency.

  • Source of Systematic Error: In the year $1948$, telephone ownership was not universally distributed across the American population; telephones were disproportionately owned by wealthier households. Wealthier citizens were systematically more likely to vote Republican (for Dewey) compared to the general electorate. Consequently, the sampling frame differed systematically from the target voting population, generating substantial sampling bias.

Human Intuition versus Random Selection

• The Human Inability to Select Representative Samples:

  • Human beings are inherently flawed at subjectively choosing representative samples, even when explicitly trying to avoid bias.

  • When instructed to pick items that look typical of a group, individuals systematically gravitate toward larger, more prominent, or more noticeable units.

• The Gettysburg Address Sampling Experiment:

  • Population: All $268$ words making up Abraham Lincoln's Gettysburg Address.

  • Population Parameter of Interest: The true average word length (number of letters per word), which equals exactly $4.29$ letters.

  • Non-Random Trial Procedure: Participants visually examine the text and deliberately select a sample of $10$ words intended to resemble the overall address in word length, calculating the sample mean for those $10$ words.

  • Full Text of Abraham Lincoln's Gettysburg Address: "Four score and seven years ago our fathers brought forth, on this continent, a new nation, conceived in Liberty, and dedicated to the proposition that all men are created equal. Now we are engaged in a great civil war, testing whether that nation, or any nation so conceived and so dedicated, can long endure. We are met on a great battle-field of that war. We have come to dedicate a portion of that field, as a final resting place for those who here gave their lives that that nation might live. It is altogether fitting and proper that we should do this. But, in a larger sense, we can not dedicate—we can not consecrate—we can not hallow—this ground. The brave men, living and dead, who struggled here, have consecrated it, far above our poor power to add or detract. The world will little note, nor long remember what we say here, but it can never forget what they did here. It is for us the living, rather, to be dedicated here to the unfinished work which they who fought here have thus far so nobly advanced. It is rather for us to be here dedicated to the great task remaining before us—that from these honored dead we take increased devotion to that cause for which they here gave the last full measure of devotion—that we here highly resolve that these dead shall not have died in vain—that this nation, under God, shall have a new birth of freedom—and that government of the people, by the people, for the people, shall not perish from the earth."

  • Random Trial Procedure: Each of the $268$ words is cataloged and assigned an identification number from $1$ to $268$. Participants draw $10$ numbers completely at random (using random number tables or computational random number generators) and compute the average length of the corresponding words.

• Analysis of Distribution Results:

  • Non-Random Distribution: Sample means selected by subjective choice cluster significantly above the truth, typically centering around $5$ to $6$ letters per word. Humans overlook short, ubiquitous words (such as "a", "to", "in", "we", and "that") and disproportionately pick long, meaningful words (such as "consecrated", "proposition", or "unfinished").

  • Random Distribution: Sample means selected via random numbers scatter evenly above and below the true population parameter of $4.29$ letters per word, centering directly on the truth.

  • Inferential Takeaway: Random samples produce statistics that center directly on the true population parameter. Non-random samples systematically drift away from truth due to intrinsic cognitive bias. Only randomized samples permit trustworthy generalizations to a population.

• Empirical Demonstration of Random Sampling Power (Gallup Poll $2008$):

  • Prior to the $2008$ United States presidential election, the Gallup Poll surveyed a random sample of $2{,}847$ American adults.

  • In this random sample, $52\%$ of respondents indicated support for candidate Barack Obama.

  • In the national election involving tens of millions of actual voters, Obama received $53\%$ of the vote.

  • A relatively small, purely random sample of fewer than $3{,}000$ individuals accurately estimated the behavior of a population exceeding $100{,}000{,}000$ voters within a single percentage point.

Sampling Methodology and Practical Implementation

• Simple Random Sampling (SRS):

  • In a Simple Random Sample, every individual unit of the population has an equal chance of being selected, and every group of $n$ units has the same chance of being selected, completely independent of all other units chosen.

  • A physical conceptualization of an SRS consists of placing every unit's name or identifier into a bowl, thoroughly mixing the container, and drawing identifiers blindly.

  • In modern practice, random selection is carried out via pseudo-random number algorithms in statistical software or specialized technology.

• Real-World Constraints and Target Populations:

  • Obtaining a complete and flawless Simple Random Sample is frequently impossible or cost-prohibitive in practical research.

  • Common real-world barriers include:

  • A comprehensive registry or list of every single population unit may not exist.

  • Contacting, tracking down, or securing responses from every randomly selected individual is often unfeasible.

  • When practical constraints arise, researchers must frequently redefine their population of interest to match the subpopulation that can realistically be sampled.

  • Scope of Inference Rule: Any statistical conclusions or inferences derived from sample data are strictly limited to the specific population that was actually sampled from, never to an unsampled broader population.

• Flawed Non-Random Sampling Approaches:

  • Sampling Based on Correlated Explanatory Variables:

  • Occurs when selection into the sample depends on a mechanism inherently tied to the variable being measured.

  • Example (Study Time): Surveying solely individuals found inside a university library or students enrolled in an advanced statistics course to assess average student weekly study hours. Students in these environments study systematically more than the broader student body.

  • Example (Personal Trainer Usage): An online poll published on fitnessmagazine.com asked: "Have you ever hired a personal trainer?", yielding $27\%$ affirmative responses. It is statistically invalid to generalize this finding to infer that $27\%$ of all humans have hired a personal trainer, as visitors to a dedicated fitness website are significantly more health-conscious and inclined to hire trainers than the general population.

  • Volunteer Bias (Voluntary Response Sampling):

  • Occurs when the sample is formed exclusively of individuals who self-select into the study.

  • Individuals who take the initiative to respond or volunteer almost always harbor unusually strong opinions, extreme experiences, or heightened emotional investment compared to the broader population.

  • Example (Airline Satisfaction): An airline transmits an email survey to all recent passengers asking them to evaluate flight satisfaction. Passengers with smooth, uneventful flights generally ignore the prompt, whereas passengers who experienced severe delays or lost luggage are highly motivated to respond, generating an artificially negative evaluation.

Nonsampling Bias and Data Collection Distortions

• Overview of Data Collection Bias:

  • Obtaining an entirely random sample does not guarantee unbiased data.

  • Bias can be introduced during the data collection and measurement stage after subjects have been selected.

• Question Wording Bias:

  • The subtle phrasing, emotional tone, or loaded vocabulary used in a question can dramatically alter respondent choices.

  • Empirical Study (Rugg, $1941$ - Freedom of Speech):

  • A controlled experiment tested wording effects regarding democratic opposition (Rugg, D., "Experiments in wording questions," Public Opinion Quarterly, $1941$, $5$, $91$-$92$):

  • When asked: "Do you think the US should allow public speeches against democracy?", only $21\%$ stated speeches should be allowed.

  • When asked: "Do you think the US should not forbid public speeches against democracy?", $39\%$ stated speeches should not be forbidden.

  • While logically asking whether anti-democratic speech should be permitted, the psychological weight of the word "forbid" compared to "allow" nearly doubled the level of tolerance.

  • Empirical Study (Tax Policy Wording):

  • When a random sample was asked: "Should there be a tax cut, or should money be used to fund new government programs?", preferences divided as:

    • Tax Cut: $60\%$

    • Programs: $40\%$

  • When a matched random sample was presented with specific program categories: "Should there be a tax cut, or should money be spent on programs for education, the environment, health care, crime-fighting, and military defense?", responses shifted drastically:

    • Tax Cut: $22\%$

    • Programs: $78\%$

• Context and Priming Bias:

  • The background material, surrounding narrative, or emotional context preceding a query can drastically bias responses.

  • Empirical Study (Ann Landers Parenting Survey):

  • The syndicated advice columnist Ann Landers asked readers: "If you had it to do over again, would you have children?"

  • Iteration $1$: The initial query appeared directly alongside a letter from a distressed young couple detailing the emotional hardships, expenses, and worries of parenting, highlighting reasons not to have children. Among self-selected respondents, only $30\%$ answered "yes".

  • Iteration $2$: In response to the initial results, Ann Landers wrote a passionate column stating she was "stunned, disturbed, and just plain flummoxed" by the feedback and reiterated the prompt. Following this editorial defense of parenthood, $95\%$ of respondents answered "yes".

• Inaccurate Responses and Social Desirability Bias:

  • Respondents frequently provide dishonest, exaggerated, or socially acceptable answers rather than truthful disclosures, particularly regarding self-image or illegal conduct.

  • Superiority Illusion in Driving (Svenson, $1981$):

  • In a landmark study examining risk perception and skill assessment among American students (Svenson, O., "Are we all less risky and more skillful than our fellow drivers?" Acta Psychologica, $1981$, $47$(2), $143$-$148$), $93\%$ of respondents rated themselves in the top $50\%$ of the sample regarding driving capability.

  • Statistically, it is impossible for $93\%$ of an entire cohort to occupy the upper $50\%$ of an ability distribution.

  • Illicit Substance Reporting (SAMHSA, $2009$/$2010$):

  • In a national probability survey conducted by the Substance Abuse and Mental Health Services Administration (Results from the 2009 National Survey on Drug Use and Health: Volume 1. Summary of National Findings, HHS Publication No. SMA 10-4856Findings), $22.7\%$ of college students reported using illicit drugs.

  • Due to privacy concerns, fear of penalization, and social stigma associated with illegal drug use, self-reported survey statistics on sensitive illegal behaviors routinely underestimate the true population proportion.

Questions & Discussion

• Student Life Importance Discussion:

  • Prompt: A classroom is surveyed on which facet of college is most important: Athletics, Academics, Social Life, Community Service, or Other. Suppose researchers studying overall student life use these responses to investigate what students find important. What is the sample, what is the population, and can the results be generalized?

  • Response: The sample comprises the specific students present in that single statistics classroom who answered the question. The intended population is the entire student body (or all college students nationwide). The data cannot be generalized to the broader student population because students enrolled in a single particular class are not a random cross-section of the university; their academic disciplines, schedules, and priorities introduce sampling bias.

• Evaluation of Voluntary Newspaper Polls:

  • Question: If a newspaper publishes a question about parenting satisfaction entirely by itself without leading context, asking readers to write or call in with their responses, can the resulting percentages be trusted to represent all parents?

  • Answer: No. Even without loaded language, voluntary response polls suffer from volunteer bias. Only individuals motivated by strong positive or negative feelings will take the effort to reply. A valid inference requires an active random sampling method.

• Evaluation of Unbiased Probability Sampling (Newsday Investigation):

  • Question: The newspaper Newsday commissioned a nationwide random sample of American adults asking whether, if they had to do it over again, they would have children (without leading text). In that study, $91\%$ responded "yes". Is the true proportion of American parents who are glad they had children close to $91\%$?

  • Answer: Yes. Because Newsday utilized a genuine random sample where participants were selected probabilistically rather than volunteering, the sample proportion serves as an unbiased estimate of the true population parameter, meaning the true population proportion should be near $91\%$.

• Method Comparison for Weekly Study Hours:

  • Scenario: A researcher wishes to estimate the average hours students spend studying per week across a university campus.

  • Evaluation of Alternative Sampling Strategies:

  • Option A (Library): Surveying all students inside the campus library selects cases based on an environment correlated with heavy studying, producing severe upward bias.

  • Option B (University-wide Email): Emailing every student and analyzing returned responses generates voluntary response bias; students with strong work ethics or excess free time respond at different rates than those struggling or disengaged.

  • Option C (Mandatory Class Response): Forcing responses from all students in a single statistics course eliminates volunteer bias within the room, but the room itself is an unrepresentative cluster of the university as a whole.

  • Option D (Student Center Entrance): Standing outside the main student center and intercepting every person entering captures a broader cross-section of majors, but remains a convenience sample subject to the times of day chosen and the types of students who utilize physical student center facilities.