Types of Error in Statistics

Learning Objectives

  • Understand how errors can be introduced during sampling.

  • Identify and differentiate between sampling errors and non-sampling errors.

  • Categorize non-sampling errors due to:

    • Sampling bias

    • Response bias

    • Non-response bias

  • Identify common sources of non-sampling errors such as:

    • Incomplete frame

    • Misrepresented answers

    • Pollster pressure

    • Interviewer error

    • Wording of questions

Overview of Statistical Errors

  • Sampling Error:

    • Definition: This is the natural error that occurs when using samples to represent a larger population. Variation between samples will always produce small differences between sample results. Sampling error is expected and acceptable in any study using samples. It does not indicate a serious flaw. Results obtained can be trusted.

    • Example: Suppose the average height of all adults in a city is 170 cm170\,\text{cm}. A sample of 100100 adults is taken, and the measured average height is 168 cm168\,\text{cm}. The difference of 2 cm2\,\text{cm} is a natural sampling error.

  • Non-sampling Error:

    • Definition: This is a serious error that occurs due to mistakes collecting or recording the data. This error does not occur due to natural variations between samples. It means that the data do not represent the population, and the results are meaningless, or worse, misleading. Results obtained cannot be trusted!

Categories of Non-Sampling Errors

Non-sampling errors fall into three main categories:

  • Sampling Bias:

    • Definition: Occurs when the methods used to collect the sample data are flawed, or when the method of selecting the sample systematically favors certain groups.

    • Example: A university wants to know students' opinions on campus food and only surveys students eating in the cafeteria at lunchtime. Students who bring their own food or skip lunch are left out. The sample is biased because it only represents cafeteria diners rather than all students.

  • Non-Response Bias:

    • Definition: Occurs when individuals who fail to respond or complete a survey differ in opinion or key characteristics from those who do respond.

    • Example: A mail survey asks individuals about their income. Individuals with very high or very low incomes frequently choose not to respond, causing responses to come predominantly from middle-income individuals.

  • Response Bias:

    • Definition: Occurs when the answers on the survey do not accurately reflect the true feelings or beliefs of the respondent (due to question wording, embarrassment, misunderstanding, or pressure).

    • Example 1: High school students surveyed with the question "Do you ever drink alcohol illegally?" answer "No" despite drinking because they fear judgment.

    • Example 2: Respondents asked "How many push-ups can you do in a minute?" respond with inflated figures like 5050 or 6060 instead of lower truthful counts like 33, 55, or 1010.

Sampling Bias and Incomplete Frames

  • Sampling Frame:

    • Definition: A list of all individuals in the population from which a sample is drawn.

  • Incomplete Frame:

    • Definition: A form of sampling bias that occurs when the sampling frame is missing some segment of the population.

  • Case Study 1: The 2016 Election Polls:

    • Prior to the 20162016 U.S. presidential election, polls showed Democratic candidate Hillary Clinton holding a 7%7\% lead over Republican candidate Donald Trump; however, Donald Trump won the election.

    • Pollsters constructed their sampling frame of "likely voters" largely using individuals who voted in the 20122012 presidential election.

    • Many voters who cast ballots for Barack Obama in 20122012 did not vote for Hillary Clinton in 20162016 (overcounting Democratic support in the polls).

    • Conversely, Republican voters who did not vote in 20122012 turned out to vote for Donald Trump in 20162016 (undercounting Republican support in the polls).

    • Because the sampling frame contained too many Democrats and not enough Republicans, pollsters overestimated support for Hillary Clinton. While Hillary Clinton won the popular vote by a small margin, the electoral result was much closer than pollsters predicted.

  • Case Study 2: University Housing Survey:

    • A university seeks to survey all students regarding campus services.

    • The sampling frame utilized consists solely of students currently enrolled in on-campus housing.

    • Because off-campus students are completely omitted from the list, the frame is incomplete, and results fail to represent the entire student body.

Interviewer Error

  • Definition: Results when the interviewer biases the response through their body language, tone, or attitude.

  • Mechanism: An interviewer's leading tone, body positioning, or explicit expectations push respondents away from giving truthful answers.

  • Example:

    • A health researcher surveys patients regarding exercise habits, asking: "How often do you exercise per week?"

    • The interviewer leans forward and states in an approving tone: "Most people I talk to say they exercise at least 3 times a week3\,\text{times a week}."

    • A participant who exercises only 1 time per week1\,\text{time per week} feels judged and responds: "Oh… about 3 times3\,\text{times}."

Questionnaire and Survey Design Errors

  • Questionnaire Design Error:

    • Definition: Results when the overall design or structure of the survey influences individuals to answer questions in a specific way.

  • Phrasing of Questions Error and Loaded Questions:

    • Phrasing of Questions Error: Results when the wording of questions influences responses. This may occur unintentionally.

    • Loaded Question: A question intentionally worded to elicit a specific desired response.

  • Leading Questions:

    • Definition: Questions structured to push respondents toward a specific answer.

    • Example: Asking "Don't you agree that the new park will greatly improve community life?" implies a positive outcome, coercing agreement.

  • Unbalanced Answer Choices:

    • Definition: Survey options that lack neutral or opposing options, skewing responses.

    • Example: Rating customer service with options "Excellent", "Very Good", "Good", and "Fair". The lack of options such as "Bad" or "Poor" forces results to skew positive.

  • Order Effect:

    • Definition: The sequence in which questions are presented alters respondent perceptions.

    • Example:

    • Version A (Biased Order): 1. How worried are you about rising crime in the city? 2. Do you feel safe walking in your neighborhood at night?

    • Version B (Neutral Order): 1. Do you feel safe walking in your neighborhood at night? 2. How worried are you about rising crime in the city?

    • Asking about rising crime first (Version A) makes respondents significantly more likely to report feeling unsafe in the second question.

  • Analysis of Question Wording (Police and Firefighter Retirement Example):

  

Firefighters battling a fire

  • Question 1: "Do you think police and fire fighters should be allowed to retire early? (Yes or No)"

    • Problem: Lacks essential context. Many answer "No", but would change to "Yes" if informed that the question applies to workers who are no longer physically capable of doing the job.

  • Question 2: "Do you think police and fire fighters should be forced to retire when they are physically unable to do the job? (Yes or No)"

    • Problem: Carries a harsh negative connotation due to the word "forced". Respondents feel uncomfortable forcing retirement on someone, driving higher "No" responses.

  • Question 3: "Do you think police and fire fighters should be allowed to continue working when they are physically unable to do the job? (Yes or No)"

    • Problem: Uses positive phrasing ("allowed to continue working"). Logically, a "No" to Question 3 is identical to a "Yes" to Question 2, yet respondents are far more likely to say "No" to forced retirement than "Yes" to allowing them to continue working.

  • Neutral Rewrite: "Do you think policy should require police officers and fire fighters to retire once they are no longer physically able to perform the duties of the job?"

Misrepresented Answers and Pollster Pressure

  • Misrepresented Answers:

    • Definition: Results when an individual does not give truthful information, either intentionally or unintentionally. Occurs when survey takers lie to avoid embarrassment or exaggerate the truth.

  • Pollster Pressure:

    • Definition: Occurs when people provide untruthful answers because they do not want to admit to unpopular opinions, taboo subjects, or bad behavior.

  • Classroom Survey Example:

    • A professor presents the following survey questions to a class:

    1. Do you regularly use illegal drugs? (Pressure answer: No)

    2. Do you consider yourself to be a racist? (Pressure answer: No)

    3. How often do you wash your hands? (Pressure answer: Very often)

    4. Are you a victim of domestic violence? (Pressure answer: No)

    5. Are you verbally abusive to your family or friends? (Pressure answer: No)

    • Even if the survey is completely anonymous, immense social pressure drives respondents to answer dishonestly to avoid social stigma or judgment.

Cause and Effect Errors

  • Cause and Effect Error:

    • Definition: Occurs when an observer assumes that one variable is the cause of another simply because the two variables demonstrate a statistical linkage or correlation through observation.

  • Confounding Variable (Lurking Variable):

    • Definition: A variable that was not considered in the study but exerts an influence on both the explanatory variable and the response variable.

  • Example 1: COVID-19 Mask Usage and Hospitalizations:

    • Observation: In 20202020, public mask wearing increased simultaneously with hospitalizations.

    • Fallacy: Concluding that mask wearing caused hospitalizations to increase, or vice versa.

    • Confounding Variable: The COVID-19 pandemic drove increases in both public mask wearing and hospitalizations.

  • Example 2: Gasoline Prices and Household Energy Usage:

    • Observation: During summer months, gasoline prices rise alongside residential energy consumption for home cooling.

    • Fallacy: Concluding that higher gasoline prices cause people to consume more home energy.

    • Confounding Variable: Time of year (summer). Summer travel demands increase gasoline costs, while summer heat forces increased air conditioning usage.

Key Summary

  • Sampling Error: Present in all sample-based studies. Tolerated in order to avoid conducting an expensive and time-consuming census.

  • Non-sampling Error: Must be strictly avoided, as it distorts data representativeness and leads directly to incorrect conclusions.