PSYC232 Week 2

PSYC232 WEEK 2 (LEC 1)

SURVEY DESIGN (RELIABILITY AND PARTICIPANT BIASES)



  • We use both of these in surveys to avoid ceiling and floor effects

  • I have a lot of respect for myself and would likely generate ceiling effects as it is too easy to score high.

  • ‘Drinking and driving is acceptable’ would likely generate floor effects as it is way too easy to say strongly disagree/score low.

  • Use strong question phrasing and scale anchors

to avoid ceiling and floor effects: “not at all”/“always” or ”strong disagree”/“strongly agree”

  • Use clear wording to ensure answers are directional

Response biases

Demand characteristics and social desirability

Participants respond in ways they

think

1. the researcher wants/hypothesizes

2. that are acceptable or desirable under sociocultural norms

 

Dealing with Demand/Desirability

  • “Double blind” measures, ensure confidentiality or anonymity wherever possible, emphasize there are no

  • right or wrong answers

  • Check data for external validity with other sources

  • Use methodologies that counter (or capitalize on) demand/desirability characteristics




Acquiescence Bias

People tend to agree more commonly and more strongly than they tend to disagree

  • Stronger in collectivistic than individualistic cultures

  • Smith & Fischer, 2008 “Negative” or “neutral” biases can appear (more rarely), particularly if suspicious of survey aims/researchers



Dealing with Acquiescence Bias

  • Use multiple items to average into one scale

  • Good reverse-wording avoids “not/never”

Extroversion/Introversion



Use multiple items to average into one scale. E.g. I talk to a lot of different people at parties and I keep in the background

  • Use multiple items to average into one scale

  • Use reverse wording (Try not to use negative wording!)





Bonus: You can also remove the midpoint of the scale to avoid “neutrality bias”, but then be extra careful about acquiescence because people tend to agree. Or that they just get annoyed because they can’t answer how they want.



  • People tend to agree more commonly and more strongly than they tend to disagree.

  • Stronger in collectivistic than individualistic cultures (smith and Fischer 2008)

  • Negative or neutral biases can appear (more rarely), particularly if suspicious of survey aims/researchers




Forming a scale

  • We average items that measure the same thing into one scale

  • After measurement, we check the consistency of our items that

we want to average together into a scale.



Cronbach’s alpha test in JAMOVI



Establishing item reliability

Cronbach’s Alpha test results:

Above .70 = acceptable

Above .80 = good

Above .90 = excellent






Priming

Exposure to a question/answering

a question influences answers and interpretation of subsequent questions



  • “Stereotype threat” when answering

questions about gender identity, ethnicity, age, socioeconomic status

Pennington, Heim, Levy, & Larkin, 2016



  • “Affective priming” when answering questions about values, morals or attitudes (e.g., “good/bad”, ”love/hate”)



Dealing with Priming

  • Move impactful questions to the end of the survey or have a “distraction task” in between scales

  • Randomize or counterbalance question order (this only averages the error across participants)
































Good questions adjust for psychological biases in the participant and in the survey

Avoid double-barrelled questions

“I regularly lack motivation and feel sad”



Avoid double-negatives

“Do you favour or oppose the law that prohibits the limiting of interest rates on student loans?”



Avoid emotive or loaded questions

“What is your favourite new feature introduced to Android this year?”



Avoid leading questions

“Given the recent corruption allegations in this sector, do you think the parliament should vote to limit funding until the allegations are resolved?



Avoid complex or ambiguous statements

PSYC232 WEEK 2 (LEC 2)

DERIVING A SCALE (PRINCIPLE COMPONENT ANALYSIS)



The Problem

There are so many questions you could ask, and so many ways to phrase each question



The Solution

A tool that identifies which of your questions group together

and 

Which of your questions give you unique information...




Objectives

◉ Describe the process of developing a scale



◉ Check for the assumptions in scale refinement



◉ Use Principal components analysis to refine items



◉ Identify the best items that make a scale



Developing a scale






















Starting out! (Making questions/items)

Designing questions for PCA



Generate as many questions/items as possible…

Follow your theory/operalization

‘Garbage in, garbage out’



Balance question wording and valence

Lecture on survey design



Select items appropriately for your aim and sample

Sample size > 200 and > 5 people per item



Using PCA

Looks at people's responses to the items and groups them to make the best summary of the items.

(the groups are called ‘components’)

This is a measurement tool not a psychological tool; look up ‘exploratory factor analysis’ for a psychological version of this tool

You can average rhe items in each component to make a reliable scale.



Principles components analysis

Identifies ways to summarize measures by looking at the patterns of data and fitting a series of lines (the ‘Principal components’)



You can average the items in each component to make a reliable scale.

Checking assumptions (avoid cheating)



Ways to cheat

  • Putting many unrelated items all together

  • Pairs of items are related, but not with anything else












































Assumptions










Interpret Components (what are the possible scales?)


















Interpretation

Use (1) statistical indicators in the scree plot and

(2) theoretical knowledge of components.



Look for the point before the drop to the flat part (the ‘elbow’ before the ‘plateau’)











And look for the final point above the ‘simulations’ line




















Description in the results



Does the grouping of items look like they have construct validity and internal validity?



Does it look like the components are grouped by something other than what you theorise?



Identify your component (find and label your scale)

Which items go with which components?

Item loadings

A number that goes from -1 to 1

Bigger numbers = this item is more like the whole component.





6 steps to report the PCA

  1. Examine the table of loadings from the PCA

  2. COnsider re-starting if items cross-load  or do not load.

  3. Convert to an APA table with item labels

  4. Identify the common theme of the components

  5. Average together the items to form a scale (remember to reverse-code items that have negative loadings)

  6. Report the descriptive statistics/reliability for your new scale.



Consider starting the PCA again, removing items that are ‘cross loaded’ and items that do not load any component.