Chapter 6: Construct Validity of Surveys and Polls

What’s the difference between a survey vs a poll?

Surveys have multiple questions and a poll generally has just 1 questions

What are the 2 kinds of question formats? What are the pros and cons of each?

• Open-ended - allows for participants to respond in their own words

  • Pros: rich source of information; good if you’re looking for a number/quantitative response

  • Cons: answers are hard to analyze if they’re qualitative in nature

  • Ex. What do you think of SDSU? This is too broad and not a good question

• Forced-choice - gives participants answer options and participants can only choose from one of those options

  • Pros: easy data analysis

  • Cons: doesn’t give much information; doesn’t tell you the difference within categories (only between categories); trickier to write

  • Ex. Do you like SDSU? Yes or No - doesn’t say how much each person likes/dislikes SDSU

How do you write well worded questions?

• Keep them simple

• Avoid loaded/leading questions - guiding someone towards an answer

  • Ex. Using “slam” instead of “hit” when asking for the speed of a car that collided with another car

• Avoid double barreled questions - asking 2 questions at once

  • Ex. Do you enjoy swimming and wearing sunscreen?

• Avoid negations - negatively worded questions; makes it confusing

  • Ex. People who don’t drive with a suspended license should never be punished  disagree 1, 2, 3, 4, 5 agree - double negative, confusing

• Avoid floor & ceiling effects - restriction of range; participants either primarily reply on the low end of the scale (floor) or high end of the scale (ceiling)

  • Ex. Puppies are cute - you probably are going to get a majority of 4 and 5s

  • Variability allows us to predict behavior; ideally have wide gender and age range

• Order of questions - you probably don’t want to start with sensitive questions, instead build up to them; control by making different versions of survey

How do you encourage accurate responses?

As a researcher, be aware of response sets and participants trying look good

What are response sets? What are the different types?

• Response sets (non-differentiation) - participants have a pattern of responding; decreases construct validity

  • Acquiescence - yeah-saying a lot

  • Fence sitting - responding in the middle ground; have an equal number of points to prevent this; ie have 1 2 3 4 (no true middle) instead of 1 2 3 4 5

Ways for a participant to look a specific way:

• Social desirability/faking good - people exaggerate their positive behavior

• Faking bad - typically so they don’t have to do more; went to jail; not very common

Problems with Observations

• Participant reaction bias: participant expectancies, participant reactance, and evaluation apprehension

• Observer (experimenter) bias

What is participant reaction bias? What are the types?

• when participants are aware that their behaviors are being observed and thus adjust their behavior

• includes: participant expectancies, participant reactance, and evaluation apprehension

Define participant expectancies, participant reactance, and evaluation apprehension

• Participant expectancies - most common and damaging; participants act in a way they think researcher wants them to; demand characteristics (ie Weapons Effect - shock experiment with gun vs magazine); clue as to how to behave (Clever Hans)

• Participant reactance - participants act in the opposite way they think the researchers want them to

• Evaluation apprehension - the anxiety or worry people feel when they anticipate being judged or evaluated by others, potentially leading to inhibited behavior or reduced performance

observer (experimenter) bias

• experimenters make biased observations; experimenters treat Ss differently; observers see what they expect to see

• Ex. maze bright vs maze dull rates - researcher told RAs some rats were smart/bright and some were dumb/dull (they all had the same amount of intelligence); bright rats were faster because of experimenter’s bias

How do you avoid problems with observation?

Double-blind procedure - neither the participants nor the researcher knows who’s getting the placebo vs the drug; reduces bias