Psychometrics Lecture #6

What is Validity?

The accuracy of the testing instrument

What does validity ask?

Does the test measure what construct it is designed to measure ?

What is the range of a validity coefficient?

0-1 ; The closer to 1, the more accurate the measure

What is construct validity?

Assesses the accuracy of a test to measure personality or psychological constructs and is the overarching validity that subsumes all other validities

What does it mean for a test to be robust in construct validity?

The test should be:

• Timeless

• Able to be used across groups

• Have clear objectives that allow it to measure a construct across time and populations

What must be true about test scores for construct validity?

The relationship entered into by scores on the test must be consistent with the theory

Which validities are subsumed under construct validity?

• Content validity

• Convergent validity

• Discriminant validity

• Criterion-related validity

• Incremental validity

• Ecological validity

What is content validity?

The degree to which the various items collectively cover the material the instrument is supposed to cover and how relevant the items are to the construct

What are 2 primary questions of content validity?

1. Does the test cover a representative sample of specifiied skills and knowledge?

2. Is test performance reasonably free from the influence of irrelevant variables

What assumption does content validity make?

That you have a good, detailed description of the content domain (not always possible)

Why is developing a relevant instrument important?

If the measure is content invalid, you may make erroneous and incorrect clinical judgements based on the measure

Why is content validity not stable?

New theories and discoveries may call into question a measure’s content

Why might a measure only be content valid under certain circumstances?

A measure may be good for measuring one aspect of a construct (ex. situational depression) but not another (trait depression)

OR

May only apply to certain populations

What are the 2 primary sources of content validity?

1. Experts

2. Focus groups

What are 2 problems w relying solely on experts?

1. They have their own biases and may view the construct through their own lens

2. They often lack lived experience with the construct

Why are focus groups useful for content validity?

They allow researchers to learn about the meaning of a construct from people who actually experience it

What advantages do focus groups provide?

• A deeper understanding of the construct

• Richer information

• Participants may feel more comfortable sharing in a group

• Capture how people actually expres distress

What was the lesson from the combats PTSD measure revised with focus groups?

Validity coefficients improved because the language became more consistent w the experiences of individuals who had actually been in combat

What are the drawbacks of focus groups?

• Some groups cannot participate effectively (dementia, schizophrenia)

• Rare constructs are difficult to study

• Facilitators may introduce bias

What happens after items are generated through experts and focus groups?

Multiple experts evaluate the scales and response options to improve clarity and precision

Does content validity have a validity coefficient?

No

Content validity is a theory-based judgement

What is criterion related validity?

The degree to which scores on an instrument accurately compare w a relevant criterion available

What is a criterion variable?

Any variable with a real-life outcome

What is predictive validity

When a test is used to predict a future real life outcome (criterion)

What assumption underlies predictive validity?

The measure can correctly predict something that theory suggests it should predict

How is predictive validity measured?

By correlating test scores w the criterion variable for each person

What validity coefficient is desired for predictive validity

Closer to 1.0

Ex of predictive validity?

SAT scores predicting 1st yr college gpa

What is concurrent validity?

when a new test is administered and criterion variable is also collected at the same time. we want to see if measurement is matched to a criteria at a present time

What is the goal of concurrent validity?

To determine whether the measurement matches a criterion in the present

Ex of concurrent validity

Administering a test and collecting a real life measure on the same day to see if they correlate

What is convergent validity?

The testing measurement is shown to be correlated w another measurement examining the same construct

What are the 2 ways convergent validity can be demonstrated?

1. Comparing 2 instruments measuring the same construct

2. Comparing the same construct using different measurement methods

Example of convergent validity using 2 instruments

Beck depression inventory & Hamilton depression inventory

Ex of convergent validity using different methods

Parent & teacher forms of the Child Behavior Checklist

What validity coefficient is desired for convergent validity?

A coefficient close to r=1

What is discriminant validty?

The testing measurement is shown to be minimally correlated w another measurement examining a dissimilar or opposite construct

What validity coefficient is desired for discriminant validity?

r=0

Why must unrelated constructs be identified beforehand?

To reduce error and demonstrate a lack of overlap btwn constructs

Ex of discriminant validity?

Happiness and sadness should diverge and have a correlation near 0

What is face validity

Whether the test loos valid to the examinee taking it

Is face validity considered a real variability ? Why?

no - it is subjective and impercise

Which test has low face validity?

Rorschach (ROR)

What are contrasted groups?

Selective very different groups to determine whether a test measures specific skills

Ex of contrasted groups

engineering majors vs English majors

What is the previously available test method?

using previous tests as the criterion to compare a new test

What is criterion contamination

When a rater knows a person’s test score and lets it influence their judgement

Why is criterion contamination a problem?

Diagnoses and ratings should be made blindly and not influenced by test scores

What is age differentiation?

Comparing scores across chronological ages to determine whether performance increases w age

What is physical evidence of behavior?

Using objective indicators, such as pedometers, to verify self-reported behaviors

What are real time observations?

Observing people in real world settings, although it is labor intensive

What are controlled stimuli?

Standardized situations that vary behavior and can be compared w self-report measures

Why must testing procedures be standardized?

Without standardized administration, validity cannot be established

Why must the criterion itself be valid and reliable

A criterion is meaningless unless it is a reliable and valid real world outcome

Why should the subject population be reviewed?

To ensure it is representative

Why is sample size important?

Small samples produce unstable validity coefficients

what does “never confuse the criterion with the predictor” mean?

Success on a criterion does not automatically imply success on the predictor. Causation cannot be assumed

What is restricted range?

Only part of the normal curve is represented, creating validity problems.

What does generalizability refer to ?

whether validity extends to different settings and circumstances

What are differential predictions?

Whether validity changes across age, gender, ethnicity, or other groups

What 3 factors impact validity coefficient size?

1. Association btwn constructs

2. skew of distribution

3. method variance

How does weak association between predictor and criterion affect vaidity?

It depresses the validity coefficient

How does skew impact validity?

It artificially restricts or depresses the validity coefficient

What is method variance

When a high validity coefficient is driven by a shared method (identical Likert scales) rather than the construct itself

How can social desirability affect validity?

it can falsely inflate validity coefficients because people respond in socially desirable ways

What is incremental validity?

The degree to which a psychometric assessment improves predictive ability beyond existing assessment methods

Existing methods beyond which a test must add prediction

Clinical interviews and behavioral observations

Does incremental validity have a validity coefficient?

No

How is incremental validity measured

Using R² (coefficient of determiniation)

What does R² represent

The amount of variance in the dependent variable explained by the independent variable

Why is incremental validity important clinically?

If testing does not improve diagnostic accuracy beyond intake interviews and observations, there is no reason to use it

What is ecological validity

How well test results generalize to real-life or real world settings

In what specialty is ecological validity important

Neuropsychology

What are the 2 ways to establish ecological validity according to Chaytor & Schmitter-Edgecombe (2003)?

1. Verisimilitude

2. Veridicality

What is Verisimilitude?

The extent to which a test simulates everyday activities

Examples of verisimilitude?

Writing a check, answering the phone, creating a grocery list,

What is veridicality?

The relationship between everyday activities and cognitive functioning skills

Example of veridicality?

The Texas Functional Living Scale (TFLS) should relate to the Problem Solving Inventory (PSI)

Can a measure be valid and not reliable

no

Can a measure be reliable and not valid?

Yes

What is the relationship between reliability and validity?

Every valid measure must be reliable, but not every reliable measure is valid

Clock example

A clock that is always 2 mins ahead is reliable (consistent) but not valid (accurate). It is consistently wrong

What validities fall under criterion related validity?

Predictive validity and concurrent validity

procedures to ensure validity

1. contrasted groups

2. criterion established by rater

3. real time observations

4. age differentiation

5. previously available tests

6. physical evidence of behavior

CCRAPP

cautions when interpreting validity

1. Differential prediction

2. changes in cause of relationships

3. restricted range

4. adequate sample size

5. what does criterion mean

6. subject population

7. Never confuse criterion/predictor

8. Generalizability

DCRAWSNG

Correlation coefficients for each validity

Face: none

Content: none (expert judgement)

Criterion: r

Construct: usually r

convergent: r (high)

Divergent: r (low)

Incremental: change in R²