Survey Construct Validity

Construct Validity in Surveys

  • Surveys should measure what they intend to measure.
  • Responses should reflect true feelings.

Question Types and Construct Validity

  • Construct validity is not determined by question type (open-ended vs. forced choice).
  • Both types have advantages and disadvantages.
  • Critical factor: how questions are worded.

Framing

  • Framing refers to how the way a question is worded can affect someone's interpretation and response.
  • It's a subtle influence that the survey taker is usually unaware of.
  • Framing effects: question wording affects answers due to human psychology.
Leading Questions

  • Leading questions: imply the desired answer.

  • Issue: hard to know if the response reflects true feelings or agreement with the survey maker.

  • Example of a leading question:

"Taxpayers are paying billions (in bold) in tax dollars to welfare, unemployment, public housing, bilingual education, and even legal defensive costs. The problem is becoming worse. Do you personally approve of having your tax dollars spent to support illegal aliens?" (Disapprove, neutral, or approve.)

*   This question is clearly biased against the issue being asked.
*   The person who made the survey is against the topic, and wants the survey taker to also disagree with the position presented in the question.
*   The capitalization of billions is unnecessary as it makes the question more exaggerated.
*   The use of the term illegal aliens is also frowned upon since undocumented immigrants would be the more ethical term to implement.
  • Example of a leading question:

"Experts believe that all consumers should comparison shop. Do you agree?"

*   Leading because disagreeing implies disagreeing with experts.
*   A better approach would be to ask: "Do you think all consumers should comparison shop?"
  • If the survey taker is aware of the survey maker's angle, it's not a neutral survey.
Framing Effects
  • How a question is worded can affect responses, even if the choice consequences are the same.
  • Example:
    • Frozen yogurt: 20% fat versus 80% fat-free.
    • People prefer the "80% fat-free" label, even though it's the same as "20% fat."
  • Decision-making can be irrational, influenced by wording.
  • Example:
    • Register early to avoid a penalty (almost 100% registered early).
    • Register early to get a discount (65% registered early).
    • Same amount of money, but penalty avoidance is a stronger motivator.
  • Superficial aspects of questions can sway answers.

Complicated Questions

  • Avoid questions that complicate or make things difficult to process.
  • There is survey fatigue, so keep surveys smooth, understandable, and efficient.
Double-Barreled Questions
  • Two questions in one, but only allows for one answer.
  • Example:

"Do you agree that the second amendment to our United States constitution guarantees your individual right to own a gun and that the second amendment is just as important as your other constitutional rights? (Support, oppose, or no opinion.)"

*   A person may agree with one part, but disagree with the other.
*   This forces the survey taker into answering something they may not feel.
Negatively Worded Questions
  • Questions with negative wording are harder to process.
  • Example:

"Does it seem possible or does it not seem impossible to you that the Nazi extermination of the Jews never happened?"

*   Confusing wording led to an inaccurate result.
*   20% of Americans seemed to believe the Holocaust never occurred.
  • Simpler version:

"Does it seem possible to you that the Nazi extermination of the Jews never happened, or do you feel certain that it happened?"

*   Resulted in only 1% indicating the Holocaust didn't occur.
  • Example:

"Abortion should never be restricted" with options from disagree to agree.

*   Disagreeing is a double negative, it must be converted into a positive to understand how to answer.
  • Easier version:

"I favor strict restrictions on abortion" with disagreement/agreement options.

  • If unsure, give both versions and check if responses differ to see if wording matters.
  • Negatives are cognitively challenging.

Question Order

  • Order of questions can affect responses.
  • The same question can be answered differently based on its position in the survey.
  • Example: The Jealousy Survey
    • First Version: Start with less intense scenarios and increase to more intense (significant other is having sex with an attractive individual)
    • Second Version: Start with a significant other having sex with an attractive individual, and lower the level of intensity with each step after that.
    • Depending on where the survey taker receives the question, their opinion of the matter could differ, even if it's the same scenario.
  • To test the differences in question order, give the survey with both orderings to see if responses differ.

Shortcuts

  • People want to complete surveys quickly.
  • Sometimes, they will select whatever option is easiest to choose in order to finish the survey faster.
Response Set
  • Answering the same way to multiple questions in a row to save time.
  • Example: Agreeing (acquiescence) or disagreeing with multiple questions just to be agreeable or disagreeable.
  • Restaurant comment cards prompt people to answer with delightful which leads people to say answer everything is delightful in order to not hurt someone's feelings.
Fence Sitting
  • Choosing the neutral option to avoid taking a stand on a controversial question.
  • Example:

"Is there some kind of higher power in the universe", ranges from definitely no to definitely yes.

  • Choosing 4 (midpoint) means you don't have to be in either camp.
  • To solve this, remove the middle option, force the people to pick a side.
  • The disadvantage is that this takes away survey takers' ability to say they feel neutrally about the topic.