Survey Construct Validity and Question Design

Construct Validity of Surveys

  • Construct validity ensures surveys measure intended concepts and respondents answer truthfully.
  • Question type (open-ended vs. forced choice) doesn't determine construct validity.
  • Critical factor: Question wording.

Framing

  • Framing: How question wording subtly influences interpretation, similar to a picture frame.
  • Framing effects: Wording affects answers due to human psychology.

Leading Questions

  • Leading questions imply the desired answer, potentially skewing responses.
  • Example: A question that prefaces with negative statements about a topic before asking for approval.
  • Another Example: Citing experts to sway opinion.

Framing Effects

  • Framing: How choices are presented affects decisions, even with equal outcomes.
  • Example: "80% fat-free" vs. "20% fat."
  • Example: "Avoid a penalty" vs. "get a discount" (avoiding penalty more effective).
  • Superficial aspects of questions can sway people.

Complicated Questions

  • Avoid questions that complicate the survey.
  • Complicated questions make answering harder.

Double-Barreled Questions

  • Double-barreled questions combine two questions into one, limiting response options.
  • Example: A question asking about agreement with two distinct aspects of the second amendment.

Negatively Worded Questions

  • Negatively worded questions are harder to process than positive ones.
  • Example: Confusing wording regarding belief about the Holocaust.
  • Revised question: Only 1% indicated the Holocaust didn't occur.
  • Example: "Abortion should never be restricted" involves a double negative.
  • Alternative: "I favor strict restrictions on abortion."
  • If unsure, ask both versions and compare responses.
  • Multiple negative questions can exhaust survey takers.

Question Order

  • Question order affects responses; earlier questions influence later ones.
  • Example: Jealousy survey with scenarios in ascending or descending order of intensity.
  • Same question can elicit different responses based on its position in the survey.
  • Test different question orders to assess impact.

Shortcuts

  • Respondents may take shortcuts due to survey fatigue.

Response Sets

  • Response set: Answering multiple questions with the same response.
  • Acquiescence: Repeatedly agreeing due to it being the nicer answer.

Fence Sitting

  • Fence sitting: Choosing the neutral option to avoid taking a stand.
  • Example: A scale about a higher power with a neutral midpoint.
  • Solution: Remove the neutral option to force a choice.
  • Disadvantage: Eliminates the ability to express a truly neutral stance.