Exam 2- Psychometric Properties of Standardized Assessments

10 Steps to Select and Assessment Tool

1. Identify type of assessment tool

2. Identify intended purpose of the assessment tool

3. Determine frame of reference that guided creation of tool

4. Determine appropriate population

5. Determine environment protocols and/or client factors impacting set-up of environment

6. Gather necessary materials

7. Identify user qualifications

8. Review administration protocols

9. Review scoring protocols and usefulness of scores

10. Review Psychometric Properties:• Validity• Reliability

What is step 10 of selecting an assessment tool?

Review Psychometric Properties:• Validity• Reliability

What are assessment tools evaluating?

Latent constructs to generate numerical data

Latent Constructs

phenomena that are not easily observed

When talking about reliability and validity, are you talking about the data or the tool?

The data

3 Types of Psychometric Properties

Validity, Reliability, Responsiveness

Reliability

refers to the consistency of the results provided by the scores on an assessment,

Validity

how true the scores are to the underlying theory

and concepts that the assessment claims to measure

Responsiveness

ability of the scores on a measure to indicate change as a result of an intervention

Scales of Measurement

Nominal, Ordinal, Interval, Ratio

Nominal

- Categories or labels

- Frequency or proportion operations

- Ex: Male, Female, Non-binary/Non-conforming, & Transgender

Ordinal

- Rank-ordered categories

- Frequency or Proportion operations

- Ex: MMT scale, Likert Scales (Strong agree, agree, neutral, disagree, strongly disagree)

Interval

- Units have equal distance but there is no true 0

- Multiplication, Average operations

- Ex: Temperature; 24 hour clock time

Ratio

- Units have equal distance and there is a true 0

- All statistical/mathematical operations

- Ex: weight, height, range of motion

Correlation

degree of association between 2 random variables

Correlation Coefficient (r)

statistical indicator that quantifies the degree of the relationship

Direction of relationship for Correlation Coefficient

- 1.00 - +1.00

(r always falls in this range)

What does 0.00 mean

unrelated

What does -1.00 mean

negatively impacted

What does +1.00 mean

positively impacted

What does it mean if you are closer to -1 or +1

There is a stronger relationship

What are p values

significance or amount of possible error

T/F: Reliability & Validity is reported in correlation coefficients (0 to +1)

True

What are the 3 types of correlation coefficient reports?

- Pearson Product-Moment Correlation Coefficient

- Spearman's Rank Correlation Coefficient

- Intraclass Correlation

Standard Error of Measurement (Reliability)

variability in measurement

Reliability Coefficient (Reliability)

consistency & reproducibility of scores

5 Types of Reliability

- Standard of Error of Measurement (SEM)

- Test-Retest Reliability

- Parallel Forms of Equivalence

- Internal Consistency

- Stability of Scores

Standard Error of Measurement (SEM)

combo of actual score and the inherent testing error out of the OTs control

Test-Retest Reliability

measures whether change in individuals performance is the result of growth vs. external forces

Parallel Forms of Equivalence

Reliability of 2 different forms of the assessment tool to account for learning/memory

Internal Consistency

items measuring same concept or skill (reported as coefficient α)

- Not recommended to be the only thing OT looks at

Stability of Scores

the stability of scores based on the rater (Intrarater Reliability and Interater Reliability)

Intrarater Reliability

reliability among self

Interater Reliability

reliability between 2 separate examiners

Validity reworded

"is this a valid assessment of what we are saying we are assesing?"

Types of Validity

- Face Validity

- Content Validity

- Criterion-Related validity

- Concurrent Validity

- Predictive Validity

- Construct Validity

- Factorial Validity

Face Validity

outline purpose of the assessment tool

- not empirically tested

Content Validity

test items on assessment represent the construct being tested

Criterion Related Validity

The assessment is measuring same concept according to a "well-established external measure

Concurrent Validity

A form of Criterion Related Validity

- assessment measures concept similar to "gold standard"

Ex. the concurrent validity of scores on a measure is determine to be adequate when the correlation between the measure and another standard or legacy measure used in practice is high

Predictive Validity

A form of Criterion Related Validity

- results from assessment provide info on future performance (Ex. KELS predicts independence)

Construct Validity

Assessment is congruent w/ identified theory

- Includes convergent and divergent validity

Convergent Validity

measuring if 2 assessments measure the same concept

Ex. A very high correlation between two similar tests of a construct can mean that administration of both tests is not needed because they measure the same thing

Divergent Validity

show low correlation w/ other measures of dissimilar constructs

Ex. Scores on a measure of shoulder function will show low correlation w/ those on a measure of walking, an unrelated construct

Factorial Validity

different constructs measured by a single measure

Responsiveness reworded

indicates degree of change from interventions provided, looking at outcome

Types of Responsiveness

- Minimal Detectable Change

- Minimal Clinically Important Difference

- Floor & Ceiling Effects

- Sensitivity and Specificity

Minimal Detectable Change

changes in scores beyond measurement error that a test can detect

Ex. Stroke Impact Scale's ADL/IADL sub scale, the standard error of measurement is 6.3, thus, a change score of 6.3 or less from admission to discharge for a client w/ stroke would not qualify as a change because it is w/n the bounds of the test's measurement error

Minimal Clinically Important Difference

change in scores that are meaningful to person/family

Ex. MCID for the Stroke Impact Scale's Hand Function subscale is 17.8 points for people with chronic stroke. A score of 17.8 or greater is considered a clinically meaningful change for the client and would a desirable outcome after OT intervention

Floor and Ceiling Effects

cannot detect changes in scores at lower ability level (Floor) and upper ability level (Ceiling), 15-20% is acceptable

Example of Floor and Ceiling Effects

For Stroke Impact Scale's Hand Function sub scale, Duncan et al., reported floor effects in 40% of those w. moderate stroke and in 2% of those with mild stroke. The same study reported the reverse trend: ceiling effects in 15% of those w/ mild stroke and 5% of those with moderate stroke. Thus, one can interpret the floor effects for moderate stroke as unacceptable for this scale, and the scale should be used with caution for those w/ moderate stroke.

Sensitivity and Specificity

can detect impairment w/ high accuracy (sensitivity)

- can detect those w/o impairment (specificity)

Ex. Beck Depression

Ex. MMSE is used to detect cognitive impairment; for community-dwelling individuals, its sensitivity is reported to be 85% and its specificity is reported to be 86%.