Surveys

what decisions need to be made to generate questions with high construct validity (operationalizing correctly)?

1. what question format to use

2. how should you phrase the questions

3. how can you get people to answer correctly

what question formats are available to use

• open ended questions

• close ended questions

open ended questions

allow people to respond to the question in their own ways

ex. of open ended questions

tell us about your views on legalizing marijuana

primary advantage of using open ended questions

allows people to tell you what is important to them (whether you thought of it or not)

disadvantages of open ended questions

• might never get to the stuff you care about; time consuming for participant and researcher

• Also, might be harder to get people to respond, and need to code all the responses

close ended questions

provide people with specific rating dimensions of interest

different types of close ended questions

1. forced choice

2. likert scale

3. semantic differential scale

forced choice question

close ended question

• “are you in favor of legalizing marijuana?” yes or no

likert scale question

close ended question

• ex. i favor legalizing marijuana —> 1: strongly disagree, 5: strongly agree

semantic differential scale

close ended question

• ex. legalizing marijuana is…

  • foolish 1 2 3 4 5 wise

  • uses adjectives instead of strongly agree/disagree

how should you phrase questions

• avoid

  • leading questions

  • double barreled questions

  • negatively worded questions

• think about question order

leading questions

makes one answer seem clearly better (or more correct) than the other 

ex. of leading questions

• “do you think its about time for marijuana to finally be legalized”

• “how bad of an effect do you think legalizing marijuana will have on families and children”

double barreled questions

asking two questions at once

double barreled question ex.

“do you think that legalizing marijuana will decrease the crime rate and lead to happier, healthier population?”

negatively worded questions

usign negations (not, never, wouldn’t, shouldn’t, etc.) makes questions more cognitively difficult

ex. of negatively worded questions

• “I do not believe that marijuana should not be marijuana not be legalized”

• “Marijuana should never be legal” disagree — agree

question order

responses on earlier questions can affect interpretation of later questions

question order ex.

1) are you in favor of legalizing heroin, crack, meth, and bath salts?

2) are you in favor of legalizing marijuana

maybe more likely to say yes to marijuana after reading about harder drugs in an earlier question

how can you get people to answer accurately

even when people answer accurately, they might be affected by response sets

response sets 

shortcuts

• acquiescence

• fence setting

Sometimes people worry about looking good than being accurate

acquiescence

yea-saying tendency (answering yes or agree to most questions without careful thought)

partial solution to acquiescence

include reverse scored items

reverse scored item

ex. if asking about self esteem, “I have a good sense of self esteem”, follow up question would be “i often feel like Im a bad person'“

• answer yes to the first one, then say no to the next one

fence sitting

staying close to the middle of the scale

fence sitting solution

even numbers of scale points (no mid point)

social desirability

concern over the impression one’s response might convey

solutions to social desirability

1. assure anonymity

2. give people a social desirability scale and account for scores in your analyses

3. include a few items to catch socially desirable responding

4. use surreptitious measures (makes it unclear to participant what you are measuring)