COMG 102: Everyday Communications with Numbers - Surveys

Recap

• Cross-sectional vs. Longitudinal

  • Key differences: How many times are the data collected?

  • Types of longitudinal research

• Choosing a design depends upon available resources and research objectives

Agenda

• Survey question formats:

  • Open-ended

  • Dichotomous

  • Multiple choice

  • Likert

  • Semantic differential

• Survey wording:

  • Leading

  • Double-barreled

  • Negative

  • Double negative

• Survey methods: Pros and cons

• Guiding respondents: Funnel vs. inverted funnel

Surveys (introductory definitions)

• The most frequently used method to collect data

• A series of formatted questions given to a sample with the expectation that their responses will be returned somewhere between immediately or within a few days

• Questionnaire

  • The specific set of questions that respondents answer (i.e., “measurement items”)

• Answer quality reflects question quality: Surveys must be carefully designed to get the most out of them

Response rate and design considerations

• Response rate is important

• Surveys should be:

  • Accessible

  • Easy to understand

  • Easy to fill out

  • Short

• If survey response rate is too low…

• Is there a systematic difference between respondents and non-respondents?

Survey Questions

• Must be carefully developed

• 4 common types of information being gathered: 4

  1. Demographic data

  2. Knowledge of or beliefs about an issue

  3. Attitude toward the issue

  4. Behaviors

• Goal of survey questions: To clarify both the question and the response options as far as possible for respondents, without over doing it

Open-Ended Questions

• Allow respondents to answer in their own words (in blanks)
  • Limit response lengths
  • online: word limit, physical: space allotment
  Pro: Rich information you can’t get with closed-ended questions

• Con: Time-consuming to code and analyze

• Limits on response lengths (online: word limit; physical: space allotment)

Dichotomous Questions

• Force respondents to select one of two possible answers.

• Pro: Simplify data coding and analysis

• Con: Not every question can be answered with a “yes/no”

• What if there are more than two possible responses to a question?

Multiple Choice Questions

• Provide respondents with several possible answers

• Select one or more answers or rank order them

• Pro: More than two response options

• Con: All possible response options must be determined in advance

• Opportunity for filter questions that help narrow down specific responses and gather more targeted insights from participants.

Filter questions (Multiple Choice) example

• Example structure:

  • "If this home has an Internet connection, please continue with Question 6. If it has no Internet connection, please go to Question 18."

• Question: "Does this home have an Internet connection?"

  • Yes → continue with Question 6

  • No → go to Question 18

Likert (Type) Scales

• Ask respondents to mark their level of agreement with statements

• Scale: From “strongly disagree” to “strongly agree”

• Usually on a 5 or 7 -point scale

• Always presented as statements

• Pro: Easy to code and analyze the data

• Con: Difficult to match statement grammar with the response options

Semantic Differential Scales

• Present a topic, object, or concept followed by scales anchored at each end by words or phrases that have opposite meanings

• Respondents express their opinions of that topic by marking their positions between these word pairs

• Pro: Easy to code and analyze the data

• Con: Not always easy to find opposite words

Survey Wording

• If it can be misinterpreted, it will be

• Primarily happens when a question was poorly worded and/or was not pre-tested to see what misinterpretations are possible

• Common problems with wording: Leading, double-barreled, negative, double-negative

Common Problems with Wording — Leading Questions

• Lead the respondents to a particular answer

• Force the respondents into an assumption that may not be true

• Solution: Use neutral wording and pre-test to identify potential misinterpretations

Common Problems with Wording — Double-Barreled Questions

• Ask two questions simultaneously but allow for only one answer

• Solution: Split each question into two

Common Problems with Wording — Negative Worded Questions

• Whenever possible, avoid negative phrases/statements

• Often misunderstood by people reading in a hurry or misheard over the phone

• Solution: Remove the negative phrase

Common Problems with Wording — Double Negative Questions

• Include two negative words

• Can completely confuse or mislead respondents

• Solution: Split each question into two or reword each question

Survey Methods

• Various ways to run surveys

• Phone, mail, online, and face-to-face surveys

• Choosing a method

  • Available resources

  • The size of surveys

• There is no “perfect” method

• Understanding advantages and disadvantages is important

Survey Methods — Phone

• Pros:

  • Can reach out to a wide range of people

  • Can survey large samples in a short time

• Cons:

  • Phone surveys are generally negatively perceived by respondents – difficult to get response

  • Typically limited to a few short questions

Survey Methods — Mail

• Pros:

  • Gives respondents time to consider questions and the ability to answer questions in any order

  • Good for studies that involve those who do not have an Internet connection

  • May be perceived as more legitimate than phone surveys

• Cons:

  • Low response rate

  • No way of knowing who completed the survey

Survey Methods — Online (Internet)

• Pros:

  • Quick, flexible, and inexpensive

  • Asynchronous

  • Can present visuals

  • May elicit sensitive information that respondents would not provide face-to-face to an interviewer

  • Utilize survey websites/applications

  • Wide geographic and demographic coverage

Online (continued) — Cons

• Limited generalizability – who has access to wifi?

• Cannot control the survey display - different platforms may display the survey differently

• May not know who completed the survey

• Invitations can be overlooked or deleted as spam

Survey Methods — Face-to-face (interview)

• Pros:

  • Respondents are less likely to refuse a request

  • Opportunity to assess respondent’s nonverbal cues

  • Some control over timing and pacing of interview

  • Interviewers can clarify questions

• Cons:

  • Resource hungry

  • Respondents may not feel the survey is confidential

  • Interviewers may need additional training in interpersonal skills and cross-cultural communication

  • Not suitable when social distancing is required

Guiding Respondents — Funnel vs. Inverted Funnel

• Funnel format: Broad questions → Specific questions

• Inverted funnel format: Specific questions → Broad questions

Funnel vs. Inverted Funnel details

• Funnel format:

  • Start with open-ended questions then closed-ended questions

  • More general questions then specific questions

  • Useful when asking about a sensitive topic

  • Allows participant to express themselves

  • Initial temp check on how the participant feels about a topic

• Inverted funnel format:

  • Start with closed-ended questions then open-ended questions

  • More specific questions then general questions

  • Useful when participants may not be familiar with the topic

  • Useful when participants need to warm up to the topic

Summary

• Different formats of survey questions

• Common wording problems

• 4 survey methods: pros vs. cons

• Structures of survey: funnel vs. inverted funnel

• Surveys must be carefully designed!

• Low response rate leads to non-meaningful findings

• Critical evaluation is necessary