11. Scale Development and Evaluation

From Item pool to scale development

Define (construct + domain) — Generate (broad item pool) — Review (expert + target users) — Pilot (Pretest + collect data) — Evaluate (structure, reliability, validity)

Three broad ways to build a test

1. Correspondence/ rational-content

2. Empirical/ criterion-keyed

3. Construct/ theoretical

Correspondence/ rational-content

All you need to do is look at the scale (Face value approach)

Selection rely on human judgment

Items are selected based on their content directly representing the behavior or attribute being measured.

Empirical/ criterion-keyed

Where you choose items based on purely on the data of how different groups answer them, without caring if the questions make logically sense.

E.g. a test for Depression

1. Two distinct group — one who have a specific trait (e.g. diagnosed with depression) + “control” group who doesn’t

2. Ask list of random questions of true/false statements e.g. “I like mechanics magasines”, “I prefer shower over baths”

3. You look at the data. If 90% of the depressed group answers “true” to “I prefer shower over baths” + 10% of the control group says “true”, you put that question on your final test about depression.

The data (90%) proves that the question successfully separates the two groups.

Construct/ theoretical

Items are selected based on a specific psychological theory.

It evaluates if the resulting test scores behave the way the theory predicts they should.

True or false, each strategy answers a different question about why an item belongs in the measure.

True

True or false, In the empirical strategy, evidence, not face appearance, determines whether an item earns its place.

True

What are empirical strategy

1. Start with a large pool of candidate items

2. Administer items to people who differ on a relevant external criterion

3. Retain items that predict the criterion (keep the items that work)

An item can be useful even when its relevance is not obvious.

True or false, In the empirical strategy, evidence, not face appearance, determines whether an item earns its place.

True

Empirical strategy — strengths and concerns

• A weak, unreliable criterion may select the wrong items

• Items may capitalize on characteristics of the development sample

• Items can distinguish groups for reasons unrelated to intended attribute

• Group differences also need interpretation

• Relationships may not generalize to other populations or setting

Construct strategy — theory tells us what the score should mean

1. Define a clear construct, specify boundaries/facets/intended population, items need to represent that domain

2. Multiple items act as clues for representing the construct

3. Use your established theory to predict what the final test scores should logically relate to and what they should not relate to.

True or false, theory, content judgment (generate items that represent the intended domain), empirical evidence (pretest and pilot the items) work together in modern scale development.

True

Scale development is a three-phase evidence cycle

Phase 1: Item development

A. Define domain(s) + generate items

B. Gather content evidence

Phase 2: Scale development

C. Pretest questions

D. Administer to an appropriate sample

E. Reduce or refine items

F. Extract latent factors

Phase 3: Scale evaluation

G. Test dimensionality

H. Test reliability

I. Build validity evidence

True or false, validity does not “start” at Step 9 (Step i); evidence begins with construct definition and accumulates across the process

True

Phase 1: Domain definition — define the purpose and domain before writing items

1. Purpose and population (context matters)

2. Does an adequate measure already exist? — if yes, why build another one? A new scale should solve a real measurement problem

3. State what the construct is + what it is not

4. If theory suggests facets, specify them, decide which content must be represented

Poor domain definition creates two risks

Construct underrepresentation

Construct-irrelevant content

What to do to generate a “good” item pool?

1. Deductive sources (start from established theory, literature, existing measures, construct definitions…and apply it to a specific situation to reach a certain conclusion)

2. Inductive sources (start with specific, raw observation…to build up to a general theory)

Combine les deux ensembles

3. Sample every important facet with multiple candidate items

What is content review (part of content evidence)

Ask experts whether the pool of items we’ve generated represent the construct.

Expert review:

• ask whether items are — relevant to construct, representative of domain.

• They can identify missing facets/contamination and redunctant content

Target-user review:

• ask whether items — make sense to target population, reflect lived experience, use understandable language, important experiences

Items may be added, removed or rewritten or reassigned to facets.

True or false, content evidence protect against both missing construct content and irrelevant construct content.

True

Difference between Pretesting/ cognitive interviews and Pilot administration

Pretesting: use small samples, ask how respondents understood + remembered + judged + selected their answers

Pilot administration: large sample, collect item-response data under realistic administration condition

True or false, do not use a large pilot to discover a wording problem that five cognitive interviews could have exposed first.

True

Pilot sample (what you need)

• People who ressemble the eventual test population (e.g. for a clinical scale, include relevant clinical respondents)

• Heterogeneity (diversity)

• Test environment must be realistic and match how the finale version will be delivered

• Total number required (N) depend on item distributions, factor structure)

What to keep, revise or drop items

Keep — content is relèvent, needed, responses show useful variability, item relates to intended score or factor

Revise — wording or response options, extreme/floor-ceiling effects, weak/inconsistent item relations, subgroup interpretation problems

Drop — irrelevant item, repeated evidence showing it contributes little to the intended construct.

Response distribution — item statistics

Does everyone almost choose the same option?

If yes, it cause extreme floor or ceiling effects

Item-total correlations — item statistics

Does the individual item move with the rest of the intended scale?

Inter-item redundancy — item statistics

Is the item adding new information or repeating another item on the test?

High similarity — impact internal consistency

Discrimination — item statistics

Does the item successfully distinguishes between respondents who have different meaning regarding the target trait or criterion?

IRT location and information — item statistics

Where along the alpha does the item work best? Does it add precision?

Exploratory factor analysis

A statistic tool used to explore/uncover structure, how many facets it suggested.

Use the (raw) data to determine the number of facets

EFA checks whether the items actually cluster into what factors or facets it propose

An item may need revision or removal if it does not align with factor

A blind analysis — it give an answer, but it’s not the final answer

If an item measure “time management” ends up grouping with a different facet — meaning there’s a problem with the item or theoretical definition.

Confirmatory factor analysis (CFA)

You test a proposed structure (pre-specified model) on a completely new separate dataset to see if it hold up. (Replication)

Ask e.g. check if items 1 to 5 measure factor one

It ask a direct: Does the proposed structure reproduce the observed item relationships well enough to be plausible?

Ensuring the scale measure the construct the same way across different populations

Differential item functioning (DIF) — ensuing item doesn’t favor or penalize a group over another (e.g. men and women)

Once the latent structure (facets) is clear enough, how you score your scale need to match with what your evidence supports.

Differentiate between one total score vs subscale scores vs weighed or unweighted scores

• One total score — using one total score if your data and factor analysis show that all items are sufficiently coherent.

Problem: not automatically meaningful

E.g. 20 items adding all number together report a single grand score

• Subscale scores — if distinct facets are supported, seperate scores may preserve useful information

Problem: still need adequate reliability and validity evidence

• Weighted or unweighted

  • Weighted — useful in context

  • Unweighted — e.g. giving every single item equal weight (1+1+1) or average item scores, perform well and easy to use

Reliability in scale development

Internal consistency: coherence among items

Stability/ reproducibility: if you’re measuring a stable trait such as a core personality, long-term academic habits, then you can use test-retest evidence

Inter-rater evidence — if multiple people judge or observe someone’s behavior

Alternate forms — if a test has multiple versions

True or false, a single coefficient cannot answer every reliability question

True

True or false, validity in a scale development is the point where multiple earlier sources of evidence are integrated into an argument.

True

True or false, scale development is a loop, not a straight line. Evidence can you send you back to earlier decisions.

True

Define — Generate — Review — Pretest — Pilot — Analyze — Revise then return to the construct.

Common traps of scale development projects that often get wrong

1. Write items before defining construct

2. Generate items from only one source

3. Keep or delete items from one cutoff (alpha or item-total correlation)

4. Confirm the structure in the same sample without replication

5. Treat validity as the last box to check