Survey Methodology, Response Effects, and Sampling Techniques
Survey Question Design and Response Scaling
Scale Length and Number of Response Options:
Rating scales can utilize varying numbers of response choices depending on research objectives:
2 options: Binary choice (e.g., "Strongly Disagree" or "Strongly Agree").
5 options: 5-point Likert scale ranging from 1 ("Strongly Disagree") to 5 ("Strongly Agree").
101 options: Continuous rating scale from 0 ("Strongly Disagree") to 100 ("Strongly Agree").
Empirical research indicates that providing 5 to 7 response options generally yields optimal psychometric reliability and measurement validity.
Numerical Scaling Effects and Anchor Labels:
The specific numerical values assigned to scale anchors significantly alter how respondents interpret question meaning, even when the underlying numerical range is equivalent.
Life Success Study Example ("How successful have you been in life?"):
When evaluated on a 0 to 10 scale, of respondents selected the highest rating ().
When evaluated on a -5 to +5 scale, of respondents selected the highest rating ().
Psychological mechanisms:
A numerical rating of represents the complete absence of a characteristic or quality.
Negative numerical values (e.g., ) denote the explicit presence of an opposing attribute or negative quality.
Subjective vs. Objective Labels:
Subjective scale example ("How often do you exercise each week?"): Bounded by anchors such as 1 ("Not at all") to 5 ("Very frequently").
Objective scale example: Bounded by concrete behavioral frequencies, such as 0 times per week, 1 time per week, 2 times per week, etc.
Selecting objective response categories requires careful alignment with actual population norms to prevent systematic behavioral report distortions.
Behavioral Frequency Reports (Schwarz et al., 1985):
A classic study by N. Schwarz, H. J. Hippler, B. Deutsch, and F. Strack (1985), published in Public Opinion Quarterly (), evaluated reported daily TV consumption across two different response option scales:
Low-Frequency Alternatives Scale:
Up to 1/2 hour:
1/2 hour to 1 hour:
1 hour to 1 1/2 hours:
1 1/2 hours to 2 hours:
2 hours to 2 1/2 hours:
More than 2 1/2 hours:
High-Frequency Alternatives Scale:
Up to 2 1/2 hours:
2 1/2 hours to 3 hours:
3 hours to 3 1/2 hours:
3 1/2 hours to 4 hours:
4 hours to 4 1/2 hours:
More than 4 1/2 hours:
Theoretical Implication: Only reported watching more than 2.5 hours of TV daily under the low-frequency scale, whereas reported watching more than 2.5 hours daily under the high-frequency scale. Respondents use the distribution of provided options as a frame of reference to infer normal behavioral baselines.

Non-Attitudes and the "Don't Know" Response Option
Non-Attitudes Definition and Prevalence:
Non-attitudes occur when respondents offer opinions on topics or issues about which they possess no actual knowledge, opinion, or background.
When asked about fictitious legislation (e.g., "Do you favor or oppose the Public Affairs Act of 1975?"), meta-analytic research demonstrates that up to of respondents will express an opinion despite the policy being completely fictitious.
Psychological Causes of Non-Attitudes:
Social desirability pressure to appear informed, intelligent, and opinionated.
Guessing behavior equivalent to a mental coin-flip.
Structural Variations in "Don't Know" Prompts:
Weak explicit filter: "… or don't you have an opinion on this?"
Moderate filter: "Do you have an opinion on this or not?"
Strong filter: "Have you thought / heard / read much about this issue?"

Methodological Debate: Should "Don't Know" Options Be Included?
Arguments Supporting Inclusion (Yes):
Uneducated, less knowledgeable, and less experienced individuals select "don't know" at significantly higher rates.
Prevents floating or unstable attitudes from contaminating datasets on volatile issues.
Respondents report higher satisfaction when explicit "don't know" choices are provided.
Arguments Opposing Inclusion (No):
Respondents who are initially forced or "pushed" past a "don't know" response provide stable responses that accurately predict subsequent voting behavior.
Election polling accuracy improves when "leaners" are explicitly included.
Measures of scale reliability and response order effects show no statistically significant differences whether "don't know" options are offered or omitted.
Question Order, Context, and Response Order Effects
Response Order Bias Mechanisms:
Primacy Effect: Participants preferentially select options presented early in a sequence. Driven by visual presentation formats and hypothesis confirmation cognitive bias.
Recency Effect: Participants preferentially select options presented late in a sequence. Driven by auditory/oral presentation formats and working memory capacity limits.
Empirical Demonstration (Schumann & Presser, 1981):
Evaluated responses to a survey question regarding U.S. oil reserve longevity across two option ordering conditions:
Option sequence A: "Plenty of oil in 25 years" option presented first; "used up in 15 years" option presented second.
Option sequence B: "Used up in 15 years" option presented first; "plenty of oil in 25 years" option presented second.
Results: Endorsing "plenty of oil" was when presented first, but increased to when presented last, demonstrating a strong recency effect.
Moderators of Response Order Sensitivity:
Response order bias is reduced among individuals with higher cognitive capability, advanced education, high GPA, high topic knowledge, or strong baseline attitudes.
Response order bias is magnified when survey items are lengthy, cognitively demanding, or complex.
Question Context and Order Effects:
Prior survey questions establish cognitive context and alter the accessibility or interpretation of subsequent items.
Stapel & Schwarz (1998) Colin Powell Experiment:
In 1995, General Colin Powell joined the Republican Party but publicly announced he would not run for U.S. President in 1996.
Inclusion condition: "General Colin L. Powell, the hero of the Gulf War, recently decided to become a member of a political party. Do you happen to know which party that is?" (Includes Powell within the Republican Party).
Exclusion condition: "General Colin L. Powell, the hero of the Gulf War, has recently been wooed by a political party to run as its candidate in the 1996 Presidential elections. He decided not to run. Do you happen to know which party it was that made Powell an offer he rejected?" (Excludes Powell from presidential candidacy for the Republican Party).
Results on 9-point political party evaluation scale:
Inclusion Condition Mean:
Control Condition (No Powell Question) Mean:
Exclusion Condition Mean:
Haddock & Carrick (1999) Royal Family Experiment:
Assessed favorability ratings (1 = very unfavorable to 9 = very favorable) for the Queen Mother and the Royal Family under two questionnaire designs:
Separate questionnaires: Evaluating the Queen Mother established her as part of the Royal Family, raising Royal Family evaluations ( vs. ) and yielding high inter-item correlation ( vs. ).
Combined questionnaire: Conversational norms led respondents to assume separate questions implied distinct categories, causing them to subtract the Queen Mother from their general evaluation of the Royal Family.
Cross-Group Comparative Validity Issues:
Comparing survey responses across demographic cohorts, cultures, eras, or languages presents measurement challenges due to varying conversational norms, cultural expectations, translation variations, and response styles.
Frequency Claims and Graphical Data Representation
Frequency Claims Overview:
Frequency claims quantify the baseline rate, degree, or prevalence of specific beliefs, attitudes, or behaviors across a target population.
Commonly presented in visual formats, including pie charts, bar graphs, and histograms.
Social Media Usage Trends (U.S. Adults, 2013–2025):
Long-term trend data reveals YouTube ( ) and Facebook ( ) maintain the highest overall adult usage rates, followed by Instagram ( ).
Tracks secondary platforms over time: Pinterest, TikTok, LinkedIn, WhatsApp, Snapchat, X (formerly Twitter), Reddit, BeReal, Nextdoor, Threads, Bluesky, and Truth Social.
Notes a survey methodology transition ("Change in survey mode") occurring around 2021–2022.

Sources of Political Belief Influence:
Personal Life Influence: Parents ( ), Teachers / Mentors ( ), Friends ( ), Extended Family ( ), Religious Leaders ( ), Siblings ( ).
Public Sector Influence: News Hosts ( ), Politicians I admire ( ), Writers / Authors ( ), Comedy News Hosts ( ), TV / Movie Actors ( ), Musicians ( ).

Local News Source Shifts (2018 vs. 2024 Comparison):
Friends, family and neighbors: in 2024 vs. in 2018.
TV news stations: in 2024 vs. in 2018.
Radio stations: in 2024 vs. in 2018.
Online forums or groups (e.g., Facebook groups): in 2024 vs. in 2018.
Local organizations (e.g., churches): in 2024 vs. in 2018.
Local govt. agencies and officials: in 2024 vs. in 2018.
Daily newspapers: in 2024 vs. in 2018.
Newsletters/Listservs: in 2024 vs. in 2018.
Non-daily newspapers: in 2024 vs. in 2018.
Other online-only sources: in 2024 vs. in 2018.

Distributional Properties, Skewness, and Central Tendency
Histograms for Continuous Data:
Histograms display frequency distributions for quantitative variables measured on ordinal, interval, or ratio scales.
Chipotle Meal Calorie Consumption Analysis:
Modal range: of customer meals cluster between (reflecting standard burritos and burrito bowls).
Secondary mode: A distinct secondary bump around stems from adding side items such as chips and guacamole.
Extreme values: Approximately of meals exceed the recommended daily allowance benchmark of .
Distribution type: Demonstrates positive skewness with a prolonged right-side tail.

The Normal Distribution (Bell Curve):
Represents a perfectly symmetrical, bell-shaped distribution that serves as the theoretical standard for parametric statistics.
Standard Empirical Rules illustrated via IQ Score Distribution:
Population Mean = .
of scores fall within (; on each side of the mean).
of scores fall within (; in each intermediate zone).
of scores fall between or between .
of scores fall below or above 145$.\n\n\n\n- Types of Distributional Skewness:\n - *Positively Skewed Distribution:* The tail extends toward the right (positive direction). Occurs when extreme outliers are high. Relative positioning of central tendency: \text{Mode} < \text{Median} < \text{Mean}.\n - *Negatively Skewed Distribution:* The tail extends toward the left (negative direction). Occurs when extreme outliers are low. Relative positioning of central tendency: \text{Mean} < \text{Median} < \text{Mode}.\n\n\n\n- Measures of Central Tendency and Skew Sensitivity:\n - *Mean:* Arithmetic average calculated by summing all observations and dividing by total count. Highly sensitive to extreme score values.\n - *Median:* Precise middle score dividing an ordered dataset into two equal halves.\n - *Mode:* Most frequently observed score in the data set.\n - *Impact of Skewness Example (Five Test Scores: 85, 87, 90, 90, 45):*\n - Total sum = 397\n - Mean = \frac{397}{5} = 79.4\n - Ordered array: 45, 85, 87, 90, 90\n - Median = 87\n - Mode = 90\n - The extreme low outlier score (4579.487) a superior indicator of central tendency.\n\n\n# Population Inferences, Sample Size, and Sampling Techniques\n\n- Population vs. Sample Definitions:\n - *Population:* The complete group of interest for a research study (e.g., all likely election voters, all TTU students).\n - *Sample:* The specific subset of individuals selected from the population who supply survey data.\n\n\n\n- Sample Size Simulation and Sampling Error:\n - Uniform Population Parameters (Range 0 to 32): Mean = 16.0016.009.52$.
Sampling distribution of sample means ( repetitions):
Sample Size : Mean of sample means = , Standard Deviation of sample means () = 3.03$.\n - Sample Size N = 2515.98SD1.91$.
Methodological Takeaway: Larger sample sizes yield estimates closer to the true population parameter with reduced standard error (). However, sample size expansion subject to diminishing returns; final sample size selection depends on population variance and allowable margin of error.

Probability vs. Nonprobability Sampling Methodologies:
Probability Sampling: Every member of the population possesses a known, non-zero probability of selection. Required for generalizing results to broader populations.
Simple Random Sampling: Every individual in the population has an equal chance of selection.
Stratified Random Sampling: The population is partitioned into distinct demographic strata (e.g., age, gender, education), and random samples are drawn independently from each stratum.
Cluster Sampling: Pre-existing clusters (e.g., schools, geographic regions) are identified and randomly sampled. Subsequently, all individuals within selected clusters are included in the sample.
Nonprobability Sampling: The selection probability of population members is unknown, restricting generalizability.


Sampling Frames and Historical Case Studies
Sampling Frame Definition:
The actual list or dataset of individuals from which the sample is selected (e.g., telephone directory, registered voter file).
Rarely aligns perfectly with the target theoretical population.
If the sampling frame systematically differs from the target population, sampling frame error compromises study validity.
Case Study: 1936 U.S. Presidential Election Polls:
Target Population: All likely U.S. voters in 1936.
Parameter of Interest: Percentage voting for Franklin D. Roosevelt vs. Alf Landon.
Literary Digest Poll:
Sample size: Distributed postcards drawn from telephone directories and luxury club membership lists; received responses.
Prediction: Alf Landon would win with to Roosevelt's .
George Gallup Poll:
Sample size: Surveyed individuals using quota-based statistical probability sampling techniques.
Prediction: Franklin D. Roosevelt would win with .
Actual Result: Franklin D. Roosevelt won the election with over of the popular vote.
Methodological Error: The Literary Digest sampling frame suffered severe coverage bias. Sampling telephone owners and club members during the Great Depression overrepresented wealthy voters who favored Landon, while excluding lower-income citizens who overwhelmingly supported Roosevelt.
Course Assignment: Survey Evaluation Pop Quiz
Pop Quiz Submission Requirements:
Take-home pop quiz assignment due on Canvas by 10/13.
Assignment Prompt and Evaluation Questions:
Complete any live survey (e.g., a customer satisfaction survey for a chain restaurant) and answer these two core analytical questions:
Question Formulation Evaluation: Evaluate how well the survey items were asked. Identify potential modifications that would improve the questions. If the questions were designed effectively, describe the explicit strategies the survey creators executed well.
Respondent Selection and Nonresponse Bias: Analyze key systematic differences between individuals who are likely to complete the survey and those who are unlikely to complete it. Explain how these nonresponse factors affect the validity of responses collected by the survey creators.