5.3 validity

Validity

• most important concept in psychological measurement

• whether the test measures what we think it is measuring

we can have a reliable tool, but

• validity might be missing → the tool is measuring other constructs that we think it is measuring

important decisions such as diagnosis or placement in a given group based on 

• scores from psychometric tests

Validity can be an

• elusive, abstract concept

conceptualization of validity has

• changed over time & it keeps changing

Validity refers to the degree to which 

• evidence & theory support the interpretations of the test scores for the proposed uses of tests

Validity does not really refer to the test itself, but also

• scores that are derived from the test

Validity analysis should begin with an 

• explicit statement of the proposed interpretation of test scores

Validity of test scores: an example

Validity is a global judgment based on different types of

• evidence

Types of validity

• content validity

• factorial/structural validity

• criterion-related validity

Content validity

• does the content of the test reflects the conceptualization of the construct

factorial/structural validity

• is the structure of the construct as we had theorized

• unidimensional vs multidimensional.

criterion-related validity

• validity with respect to other external criteria.

  • concurrent or predictive

Content Validity refers to

• analysis of the relationship between the content of a test & the construct it is intended to measure

Content Validity includes

• themes, wording, & format of the items, tasks, or questions on a test

In order to be content validity

• content of the items needs to be relevant to the construct

• subjects performing tasks must be a good representation of the construct we intend to measure

• wording & format of the items must be appropriate (adapted to a certain level & abilities of the test-taker

• data must be content-based evidence

The evidence for content validity comes from:

• the logical relationships the creator of the test has established between construct & content.

• expert judgments of the relationship between different parts of the test & the construct.

Structural/Factorial Validity

• analysis of the internal structure of a test

• degree to which the relationship among test items & test components (dimensions) conform to the theoretical construct that we are intending to measure

After we analyze the test (using factor analysis), we should

• see how items that measure each dimension group together

Evidence for construct validity usually comes from applying

• factor analysis to the results of the test

Structural/Factorial Validity: Employee Engagement → example

Criterion Validity

• how the scores of the test relate to another established standard of comparison

  • concurrent validity

  • predictive validity

Concurrent validity

• refers to measures taken at the same time.

• We measure subjects (or obtain other type of information related to the construct) at the same time & we expect to see a relationship.

Predictive validity

• Refers to obtaining a measure at a later time to compare with our original measurement.

• We measure subjects at a later time & we expect to see that their scores in our test predict other indicators of the same construct.

The evidence for criterion related validity comes from

• how the scores of our test relate to the results of other tests (or other measures that have been validated) & our measure that is supposed to target the construct.

• how the scores of our test relate to the results of other measurements that are taken later in time (& that we believe are related to the target construct)

• in practise → we try to correlate the scores of our test with the scores of the criterion

New conceptualization of validity → according to Denny Borsboom & collaborators

• Current conceptualization has proven to be useless

• Do not believe that a test should be built & checked for validity afterwards

• Validity of the measure should start before the measure is even conceptualized

• Validity should be a quality that we put in an instrument, & not something to be discovered afterwards

According to Borsboom → a test is valid for measuring an attribute if

• the attribute exists & variations in the attribute causally produce variation in the measurement outcomes

  • he proposed to start test development by thinking about how a process that would lead to difference in test scores would look like

  • we should build a test based on this, how differences would manifest on the sample

  • We should focus on how differences manifest in people & not so much in theoretical definitions.

counterargument for Borsboom explanation of a test

• Theories exists only in some areas

• We cannot always build a test based on how different levels of the trait are manifested in different situations.

• Sometimes we can only build a test based on the available theory, administer it, & look for correlations with other measures etc.

Validity refers to the 

• extent to which a particular test score interpretation (or use) is justified

The prevailing approach to validity analysis considers

• content

• structural

• criterion-related validity

There is some opposition to this approach that claims that validity should precede

• test constructions