The Process of Scale Construction, Part 2

Most popular format for person based test?

Dichotomous (true/false)

likert

Likert Scale

Agreement with declarative statements

often referred to as a “continuous” measure

ordinal measurement scale

Variations of (“Likert-type” scales)

Assessing Intensity (or degree) - not at all, slightly, moderately etc

Assessing Frequency - never, rarely, sometimes etc

Unipolar vs. Bipolar Scale - very unfair, neutral, very fair

Most important thing to remember regarding Likert Scale

wording of items must fit anchors

e.g., asking for agreement but items relate to frequency → has to be the same

Should i have an option for “not applicable”?

NO

not good for when you want to develop a measure - you need as much data as possible - no missing data

Number of response options e.g., 1-5, 1-7, 1-9?

Up to you, depends on the construct

if you predict that the answers will answer higher, number of responses options will be higher - trying to get more variability - try to get much variance as you can

Do I always need a midpoint?

up to you, depends on construct

some constructs you want to force people to choose one side or the other

Suggestions for writing Likert-type Items

avoid statements that are likely to be endorsed by almost everyone or almost no one

try to have an almost equal number of positive and negative-keyed items

Avoid “double-barrelled” items

avoid complex/compound items which combine multiple ideas

e.g., social media helps people connect and improve their mental health

Response sets/styles

acquiescence

criticalness

social desirability

dissimulation (faking good or faking bad)

random responding

Source of response sets/styles

systematic measurement error

random measurement error

Systematic measurement error (a-d)

the person consistently responds in a certain way, which can distort their scores

reduces validity

Random measurement error (e)

the person answers randomly, making their scores unpredicitable

reduces reliability

Acquiescence

tendency to “agree/endorse” all items regardless of content

Criticalness

tendency to “disagree” with all items regardless of content

Solution for acquiescence and criticalness

balance the direction in which items are keyed (half positive, half negative

Positively keyed

item is written such that a person high on the construct should score high on the item

negatively (reverse) keyed

item is written such that a person high on the construct would score low on the item

example of reversed keyed items

5 → 1 and 1→5

how does reverse keying half the items help

allows to distinguish truly high SE people from those who are acquiscent

more construct valid

cant we drop people who score consistently at the high or low end?

no

their scores might be genuine

cant tell someone is responding honestly or just agreeing/disagreeing with everything based on their scores alone

Social desirability

tendency to respond in a way that is socially desirable/acceptable

conscious or unconscious

a source of measurement error → reduces validity

solutions to social desirability

items with neutral wording

test instructions - “be honest/candid”, “no right or wrong answers”

“anonymity”, “confidentiality”

create items that are non transparent (low face validity)

Dissimulation

consciously faking “good” (trying to look better than you are) or faking “bad” (trying to look worse that you are)

Solutions for dissimulation

test instructions “be honest” “first reaction”

Assure of anonymity and confidentiality

Random responding

results from lack of motivation or attention

Solution for random responding

shorter tests

scale instructions - “be careful”

for long tests, can add an “infrequency scale” - items that would tend to be universally true or false

Scenario format

present respondents with a scenario/vignette

how can scenario format help

depending on nature of the construct

may reduce social desirability bias

may be most valid way to assess it

but…. trickier to develop so need care

explicit self report measures

questionnaires where people directly report their own thoughts, feelings and/or behaviours

disadvantages of explicit self report measures

transparent - easily tell what the questions are measuring → susceptible to response styles or faking

assume participants are able and willing to report

Implicit measures

assess attitudes or traits that operate at a non conscious level → not realize we hold them

Implicit attitudes

less conscious and less controllable → may not be aware of them or able to change our responses

feels nervous around public speaking w out realizing they have negative association with it → automatic behaviour like avoiding eye contact

explicit attitudes

consciously held beliefs that are easier to control → influence behaviours we are monitoring

believes they dislike public speaking and reports “i hate giving presentations”

Construct-driven approach or rational approach

deductive

top down

start with theory, use it to define the construct, then develop items that measure it

criterion-driven approach or external approach

empirical approach

start with existing groups (people with and without depression)

give them a large, varied set of items

select items that best distinguish between the groups, based on their responses

start with data and observed differences rather than theory