Chapter 6

Validity

is a judgment or estimate of how well a test measures what it purports to measure in a particular context

Content validity

This is a measure of validity based on an evaluation of the subjects, topics, or content covered by the items in the test.

Criterion-related validity

This is a measure of validity obtained by evaluating the relationship of scores obtained on the test to scores on other tests or measures

Construct validity

This is a measure of validity that is arrived at by executing a comprehensive analysis of

a. how scores on the test relate to other test scores and measures, and

b. how scores on the test can be understood within some theoretical framework for understanding the construct that the test was designed to measure.

trinitarian view

each of the three conceptions of validity provides evidence that, with other evidence, contributes to a judgment concerning the validity of a test

not

Trinitarian approaches to validity assessment are _____ mutually exclusive

ecological validity

refers to a judgment regarding how well a test measures what it purports to measure at the time and place that the variable being measured (typically a behavior, cognition, or emotion) is actually emitted

Face validity

relates more to what a test appears to measure to the person being tested than to what the test actually measures.

Content validity

describes a judgment of how adequately a test samples behavior representative of the universe of behavior that the test was designed to sample

test blueprint

a plan regarding the types of information to be covered by the items, the number of items tapping each area of coverage, the organization of the items in the test, and so forth

Criterion-related validity

is a judgment of how adequately a test score can be used to infer an individual’s most probable standing on some measure of interest—the measure of interest being the criterion

Concurrent validity
Predictive validity

Two types of validity evidence are subsumed under the heading criterion-related validity

Concurrent validity

is an index of the degree to which a test score is related to some criterion measure obtained at the same time (concurrently)

Predictive validity

is an index of the degree to which a test score predicts some criterion measure.

criterion

standard against which a test or a test score is evaluated.

Criterion contamination

is the term applied to a criterion measure that has been based, at least in part, on predictor measures.

concurrent validity

If test scores are obtained at about the same time as the criterion measures are obtained, measures of the relationship between the test scores and the criterion provide evidence of

predictive validity

Measures of the relationship between the test scores and a criterion measure obtained at a future time provide an indication of

base rate

is the extent to which a particular trait, behavior, characteristic, or attribute exists in the population (expressed as a proportion)

hit rate

may be defined as the proportion of people a test accurately identifies as possessing or exhibiting a particular trait, behavior, characteristic, or attribute.

miss rate

may be defined as the proportion of people the test fails to identify as having, or not having, a particular characteristic or attribute.

false positive

is a miss wherein the test predicted that the testtaker did possess the particular characteristic or attribute being measured when in fact the testtaker did not.

false negative

is a miss wherein the test predicted that the testtaker did not possess the particular characteristic or attribute being measured when the testtaker actually did.

the validity coefficient and expectancy data

Judgments of criterion-related validity, whether concurrent or predictive, are based on two types of statistical evidence

validity coefficient

is a correlation coefficient that provides a measure of the relationship between test scores and scores on the criterion measure.

Restricted range

happens when the sample used in a study is too limited compared to the population.

Inflation of range

happens when a sample has more variation than the population it represents.

Incremental validity

means how much a new predictor (test or measure) adds to the accuracy of predicting an outcome, beyond what is already predicted by other tests or predictors.

• If a new test gives additional useful information, it has incremental validity.

• If it doesn’t improve prediction, it doesn’t have incremental validity.

Construct validity

is a judgment about the appropriateness of inferences drawn from test scores regarding individual standings on a variable called a construct.

same construct

If all test items show significant, positive correlations with total test scores and if high scorers on the test tend to pass each item more than low scorers do, then each item is probably measuring the _______ as the total test.

convergent evidence

if scores on the test undergoing construct validation tend to correlate highly in the predicted direction with scores on older, more established, and already validated tests designed to measure the same (or a similar) construct

discriminant evidence

A validity coefficient showing little (a statistically insignificant) relationship between test scores and/or other variables with which scores on the test being construct-validated should not theoretically be correlated provides

multitrait-multimethod matrix

is the matrix or table that results from correlating variables (traits) within and between methods. Values for any number of traits (such as aggressiveness or extraversion) as obtained by various methods (such as behavioral observation or a personality test) are inserted into the table, and the resulting matrix of correlations provides insight with respect to both the convergent and the discriminant validity of the methods used.

Factor analysis

is a shorthand term for a class of mathematical procedures designed to identify factors or specific variables that are typically attributes, characteristics, or dimensions on which people may differ.

Exploratory factor analysis

typically entails “estimating, or extracting factors; deciding how many factors to retain; and rotating factors to an interpretable orientation”

confirmatory factor analysis

researchers test the degree to which a hypothetical model (which includes factors) fits the actual data.

factor loading

conveys information about the extent to which the factor determines the test score or scores.