Survey Wording and Random Sampling in Research

Impact of Survey Wording on Opinions

• The specific phrasing of a question significantly influences how people respond and the opinions they express.
• Critical thinkers are advised to reflect on how question wording can sway public opinion.
• Table 1.1 Survey Wording Effects - Examples of how phrasing impacts approval:
  • "aid to the needy" garners more approval than "welfare."
  • "affirmative action" garners more approval than "preferential treatment."
  • "undocumented workers" garners more approval than "illegal aliens."
  • "gun safety laws" garners more approval than "gun control laws."
  • "revenue enhancers" garners more approval than "taxes."
  • "enhanced interrogation" garners more approval than "torture."
• Real-world illustration based on evolution beliefs:
  • When Americans were asked if they believed in evolution "as 'God's way of creating life,'" $78$ percent expressed belief.
  • However, when another group was asked simply if they believed in evolution (without religious context or reference to beginning of time), $68$ percent expressed belief.
  • This demonstrates a $10$ percentage point difference in expressed opinion purely due to the nuances in question wording.

Random Sampling for Accurate Generalization

• Bias in Everyday Thinking: Individuals often tend to generalize from observations, particularly vivid or striking cases, which can lead to faulty conclusions.
• Sampling Bias: This is the strong tendency to generalize from a limited number of vivid but ultimately unrepresentative cases, which is difficult to resist.
  • Example: An administrator evaluating a professor might be unduly influenced by the striking, negative comments of two angry students, even when a statistical summary shows numerous favorable evaluations. The vivid but biased sample can outweigh more representative data.
• Objective of Random Sampling: To obtain a representative sample that accurately reflects the opinions or characteristics of an entire population, especially when surveying the whole group is impractical.
• Definition of Random Sample: A sampling method where every single person or element within the entire population has an equal chance of being selected for inclusion in the sample group.
• Methods to Achieve a Representative Random Sample:
  • For instance, to gauge student opinion on a proposed tuition increase at a college, one could:
    • Assign a number to each student listed in a general student directory.
    • Utilize a random-number generator to select the participants for the survey.
• Limitations of Non-Random Methods: Simply distributing questionnaires to an entire group and relying on returns does not produce a random sample. Conscientious individuals who choose to return the questionnaire represent a self-selected group, not a random cross-section of the population.
• Importance of Sample Quality (Size vs. Representativeness):
  • While larger representative samples are generally preferable to smaller ones, representativeness is the critical factor.
  • A smaller sample of $100$ individuals that is truly representative of the population is superior to a larger sample of $500$ individuals that is unrepresentative.
  • It's impossible to correct or compensate for a non-representative sample simply by increasing the total number of people included in that flawed sample.
• Application in Political Polling:
  • Professional political pollsters employ random sampling extensively for national election surveys.
  • They typically sample around $1500$ individuals, ensuring these participants are randomly drawn from all geographical areas.
  • This rigorous method explains why polls like those found on websites, despite often having very large response numbers, frequently yield misleading results due to their inherent lack of random sampling.