Testing the validity and reliability of measures in mental health research

What is reliability

Are measurements replicable and consistent

What is validity

Is it really measuring what it’s supposed to measure

Main psychometric properties of measures

Reliability

Validity

Feasibility and acceptability

Sensitivity to change/responsiveness

Appropriate scaling

Relevance

Inter-rater reliability

Agreement between 2+ raters/observers. Applies to rating scales where an independent interviewer/observer makes a judgement

Test-retest reliability

Replication of measurements over time to see how stable the responses are, circumstances of testing ideally constant.

Internal consistency

Do scores for individuals tend to be consistent, suggesting items are measuring the same thing? Applicable with a single underlying construct, not so much with a battery of items in a questionnaire.

Parallel forms reliability

Two versions of questionnaire developed: how well do results from them agree?

Split-half reliability

Whereby you randomly split an instrument that you wish to use as a single measure into items into two halves – how strongly do they correlate?

Face validity

Does it look as though it measures the right thing? May ask a range of people their views.

Content validity

Examines the extent to which the concepts of interest are comprehensively represented by the items in the questionnaire.

Criterion Validity

How far does the measure agree with other relevant measures or outcomes

Concurrent validity (criterion)

Agreement with other measures that have already been validated, usually using a “gold-standard” from the field

Predictive validity (criterion)

How far the measure predicts relevant outcomes

Construct Validity

A measure of how meaningful the measure is when it’s in practical use. Does it perform as you’d expect it to if it’s really measuring the intended construct?

Convergent validity

Does the measure have associations as expected with measures of related concepts, based on the strength of a relationship either positive or negative

Discriminant (divergent) validity

Are associations lacking with measures of unrelated concepts? Would be a weak relationship with an unrelated concept

Structural validity

Does a single main factor emerge from a scale, or does a scale consist of several different factors that measure related, but distinct concepts? Most often assessed with factor analysis