L4 W3 Scale Construction Part 2; Construct an initial pool of items, response format, instructions.

issues at stage 5; two variations on traditional item format; big picture

what are the two most popular formats for items in person-based tests?

• dichotomous (true/ false); -dichotomous scales don’t hold up well to psychometric properties we’re going to examine tests with

• likert

what are examples of likert scales?

• agreement with declarative statements: true likert

• often referred as “continuous” measure

  • continuous scale: can respond on a continuum

  • ordinal measurement scale READING

• second example is expanded agreement scale

UNIMPORTANT JUST REVIEW what is an example of a likert scale

-statements are declarative statements

-bipolar in that people can agree or disagree or be neutral

what are some variations of likert scales (“likert-type”) scales”

• assessing intensity (or degree)

• assessing frequency

• unipolar vs bipolar scale REVIEW?

-want to verbally anchor each number, but are times you might not want to do so

-the first two are unipolar- the bottom end is never or not at all which is a lack of the attribute

-likert is originally bipolar

-some scales don’t have midpoints also

for the following questions, which scales would you match them to?

-first Q: second scale for frequency

-second Q: declarative statement- first

-third Q: extent- last one

what are some other issues with likert scales?

• should NOT have option for “not applicable”

  • -not applicable is not good when collecting data; if it’s not applicable then you don’t want to be including this in your measure

    -if you want a prefer not to answer for students from ethical considerations, they can skip the question and choose not to respond

• SHOULD NOT mix formats

• number of response options (ie. 1-5; 1-7; 1-9): up to you depends on construct

  • -if construct is normally distributed and the population will vary considerably, then 1-5 is good

    -if you think going to vary high or low, then 1-7 is good to get more variance on scale

    -if variance is restricted and everybody scores high (4-5), then correlation and reliability are affected, and want to get that variance; want to do 1-9; ie. if thinking about outcomes in the workplace being based on merit in a workplace, then most people will score 4/5

• do i always need a midpoint (ie. 1-6): up to you; depends on construct

  • -there are constructs where you want to force people to one side, then no midpoint- usually with bipolar type; scientists hate this, so sometimes we try not to not use a midpoint, but if we have good reasoning and can explain it, then we can use midpoint

    -if you only had dichotomous (yes/ no)- then everybody is scoring either 1 or 2, which limits variance

what are suggestions for writing likert-type items, and what are things we focused on in class?

• avoid statements that are likely to be endorsed by almost everyone or almost no one

  • kills the variance

• try to have an almost equal number of positive and negative-keyed items

what are some examples of negative/reverse-keying?

-you score low on one measure, and then high on the reverse measure and vice versa

-can also denote with a N for negative

what are double-barrelled items? what should you do about them?

• avoid them

• Limit each item to one idea, avoid complex/compound items which combine multiple ideas

• –Social media helps people connect and improves their mental health

• –I am satisfied with my job and the support I receive from my manager

• –I enjoy going to parties and meeting new people

what are some general scale instructions? why are they important?

• •General introductory instructions

• •Written to foster appropriate responding

• –Knowledge-based tests: maximal (optimal) performance

• –Person-based tests: “typical” performance

• Can help to reduce potential “response styles”

what are some examples of general scale instructions? and incorrect instructions?

-From first statement, expecting agreement statements, and how I’m feeling generally

-second: extent Q’s, and how I’m feeling right now

-social desirability response set removed with last statement in third Q

what are some issues run into with response sets/ styles?

• Tendency for an examinee to respond in a certain way regardless of item content

• a.   Acquiescence (yay saying)

• b.   Criticalness (nay saying)

• c.   Social Desirability

• d.   Dissimulation (faking ‘good’ or faking ‘bad’)

• e.   Random Responding

-a and b opposite sets of the same thing

-not demand characteristics- look at why a demand characteristic is and differ it

-not a big difference btwn social desirability and the faking good- dissimulation is conscious

-not an agreed upon spot check because its constantly changing- have to be up to speed, including for AI checks

for response sets/ styles, what are sources of SME and RME?

These can reduce the validity of your measure (a-d) a source of systematic measurement error: a.   Acquiescence 

• b.   Criticalness

• c.   Social Desirability

• d.   Dissimulation (faking ‘good’ or faking ‘bad’)

And reliability (e) a source of random measurement error:

• e.   Random Responding

what is acquiescence and criticalness

•Tendency to “agree/endorse” all items regardless of content

•Tendency to “disagree” with all items regardless of content

what are some solutions for acquiescence and criticalness?

•Balance the direction in which items are “keyed” i.e., half positive, half negative

•Positively keyed:  Item is written such that a person high on the construct should score high on the item

•Negatively (reverse) keyed: Item is written such that a person high on the construct would score low on the item

REVIEW AND UNDERSTAND elaborate on an example of how this solution works for acquiescence and criticalness

-does not just add “not” to positively keyed items; -otherwise reverse-keying taps the highs and lows of constructs

-Rosenberg global self-esteem (1965); review the ten items!

-Participants won’t see the R but we always indicate it in articles and in data

-the respond boxes are of the Q’s 1-4

-have you reverse-coded your reverse key items, which is what the first box does, hence why the second Q that scored 1 gets scored as a 5, to reflect a person high in self-esteem

-the reverse coding is neutralizing the confounds of acquiescence

-graph with leadership for Y, self esteem for X; first graph is true measure; second measure is if it’s tapping SE and also maybe acquiescence, then correlation is going to be lower btwn self-esteem and correlated items like leadership (ENSURE YOU UNDERSTAND THIS!)

Can’t we just drop people who score consistently at the high (or low) end for acquiescence and criticalness?

-don’t want to drop data

-people aren’t always going to score extreme highs or lows, so won’t always know who to drop accuratelywh

what is social desirability, and examples?

•Tendency to respond in a way that is socially desirable/acceptable

–May be conscious or unconscious

what are some potential solutions for social desirability?

• •Create items with neutral wording

  • hard to do is construct is naturally socially undesirable

• •Test instructions

  • –“be honest/candid”  “no answer is right or wrong”

  • –“anonymity”  “confidentiality”

• •Create items that are “non-transparent” (low face validity)

  • –E.g., “immigration laws in the US are too lax”

  • -third bullet is a NO

  • -face validity is just does the item look like what it’s measuring- so covert type items is what the third bullet is talking about

    -issue is what is it measuring, construct validity issues

• •In scale development, item analyses

  • –remove items that correlate more highly with a social desirability scale than with your scale

  • -best way is fourth bullet point; social desirability scale is in the next slide

what is the social desirability scale?

what is dissimulation, and how do you avoid this?

• •Consciously faking “good” (trying to look better than you are or faking “bad” (trying to look worse that you are)

• avoid:

  • •Test instructions  “be honest” “first reaction”

  • •Assure of anonymity and confidentiality

  • In applied settings, often give an ‘integrity’ test as well to gauge respondent ‘honesty.’

  • Or, in the instructions, state that the test can detect ‘faking’

• -can be related to social desirability, but is more conscious

what s random responding, and how do you avoid this?

• •Results from a lack of motivation or attention

• -this problem applies generally in studies, not in specific scales

• avoid:

• •Shorter test

• •Scale instructions:  “be careful”

• •For long tests, can add an “infrequency scale”

  • Items that would tend to be universally true or false

what is an example of an infrequency scale? what is an infrequency scale?

• Items that would tend to be universally true or false

• data would not be taken at face value if you respond incorrectly to the infrequency scale

what is a scenario format for item format?

• Present respondents with a scenario/vignette (vignette in sense its not likert)

  • • Several variations possible

• Depending on nature of the construct …

  • •May reduce social desirability bias

  • •May be most valid way to assess it

  • •But arguably trickier to develop, so need care

provide an example of a scale up to this fifth step

-internal consistency (alpha) reliability?

-ie. .7-.77 (subscale 1) and .71-.79 (subscale 2)

-want it to be higher?

Test-retest (2-3 week interval)

-.75 (subscale 1) and .77 (subscale 2)

provide another example of a scale up to this fifth step;

first pic:

-4 out of 12 items

-probably less internally consistent as well

-b/c 4 items and unidimensional, alpha may be higher, whereas the other scale had 4?

second pic:

-saying what you’re doing in each vignette, which helps reign you in as well