Chapter 6 - Validity

validity

meaningfulness of a test score; what the test score truly means

validity

this term, as applied to a test, is a judgement or estimate of how well a test measures what it purports to measure in a particular context;

it is a judgement based on evidence about the appropriateness of inferences drawn from test scores

inference

is a logical result or deduction

validation

is the process of gathering and evaluating evidence about validity

validation studies

this is sometimes appropriate for test users to conduct with their own groups of testtakers

local validation studies

are absolutely necessary when the test user plans to alter in some way the format, instructions, language, or content of the test;

these would be necessary if a test user sought to use a test with a population of testtakers that differed in some significant way from the population on which the test was standardized

content validity

this is a measure of validity based on an evaluation of the subjects, topics, or content covered by the items in the test

criterion-related validity

this is a measure of validity obtained by evaluating the relationship of scores obtained on the test to scores on other tests or measures

construct validity

this is a measure of validity that is arrived at by executing a comprehensive analysis;

the “umbrella validity”

strategies

referred to as the many ways of approaching the process of test validation;

plan of attacks

ecological validity

refers to a judgement regarding how well a test measures what it purports to measure at the time and place that the variable being measured is actually emitted

face validity

what a test appears to measure to the person being tested than to what the test actually measures;

a judgment concerning how relevant the test items appear to be;

are frequently thought of from the view of the testtaker, not the test user

content validity

describes a judgment of how adequately a test samples behavior representative of the universe of behavior that the test was designed to sample

test blueprint

“structure” of the evaluation;

a plan regarding the types of information to be covered by the items, the number of items tapping each area of coverage, the organization of the items in the test, and so forth

behavioral observation

a technique frequently used in blueprinting

criterion-related validity

a judgment of how adequately a test score can be used to infer an individual’s most probable standing on some measure of interest

concurrent validity and predictive validity

what are the two types of validity evidence subsumed under criterion-related validity?

concurrent validity

is an index of the degree to which a test score is related to some criterion measure obtained at the same time

predictive validity

is an index of the degree to which a test score predicts some criterion measure

criterion

the standard against which a test or a test score is evaluated

relevant, valid, and uncontaminated

what are the ideal characteristics of a criterion?

criterion contamination

is the term applied to a criterion measure that has been based, at least in part, on predictor measures

base rate

is the extent to which a particular trait, behavior, characteristic, or attribute exists in the population (expressed as a proportion)

hit rate

defined as the proportion of people a test accurately identifies as possessing or exhibiting a particular trait, behavior, characteristic, or attribute

miss rate

defined as the proportion of people the test fails to identify as having, or not having, a particular characteristic or attribute

false positive

is a miss wherein the test predicted that the testtaker did possess the particular characteristic or attribute being measured when in fact the testtaker did not

false negative

is a miss wherein the test predicted that the testtaker did not possess the particular characteristic or attribute being measured when the testtaker actually did

validity coefficient

is a correlation coefficient that provides a measure of the relationship between test scores and scores on the criterion measure

incremental validity

defined as the degree to which an additional predictor explains something about the criterion measure that is not explained by predictors already in use

construct validity

is a judgment about the appropriateness of inferences drawn from test scores regarding individual standings on a variable

construct

is an informed, scientific idea developed or hypothesized to describe or explain behavior;

is unobservable presupposed (underlying) trait that a test developer may invoke to describe test behavior or criterion performance

homegeneity

refers to how uniform a test is in measuring a single concept

method of contrasted groups

a way of providing evidence for the validity of a test by demonstrating that scores on the test vary in a predictable way as a function of membership in some group

convergent evidence

may come not only from correlations with tests purpoting to measure an identical construct but also from correlations with measures purporting to measure related constructs

discriminant evidence

a validity coefficient showing little (a statistically insignificant) relationship between test scores and/or other variables with which scores on the test being construct-validated should not theoretically be correlated provides _________

multitrait-multimethod matrix

an experimental technique useful for examining both convergent and discriminant validity evidence;

is the matrix or table that results from correlating variables (traits) within and between methods

convergent validity

data indicating that a test measures the same construct as other tests purporting to measure the same construct

factor analysis

is a shorthand term for a class of mathematical procedures designed to identify factors or specific variables that are typically attributes, characteristics, or dimensions on which people may differ;

used to obtain both convergent and discriminant evidence of construct validity

exploratory factor analysis

typically entails “estimating, or extracting factors; deciding how many factors to retain; and rotating factors to an interpretable orientation”

confirmatory factor analysis

researchers test the degree to which a hypothetical model (which includes factors) fits the actual data

factor loading

a term commonly employed in factor analysis;

“a sort of metaphor; each test is thought of as a vehicle carrying a certain amount of one or more abilities”

bias

is a factor inherent in a test that systematically prevents accurate, impartial measurement;

it implies systematic variation

estimated true score transformations

a procedure helpful during test development;

represents one of many available post hoc remedies

rating

is a numerical or verbal judgment that places a person or an attribute along a continuum identified by a scale of numerical or word descriptors

rating scale

a scale of numerical or word descriptors

rating error

is a judgment resulting from the intentional or unintentional misuse of a rating scale

leniency error or generosity error

an error in rating that arises from the tendency on the part of the rater to be lenient (generous ) in scoring, marking, and/or grading

severity error

extreme error;

movie critics who pan just about everything they review may be guilty of this

central tendency error

the rater exhibits a general and systematic reluctance to giving ratings at either the positive or the negative extreme, as a result, the ratings would tend to cluster in the middle of the rating continuum

rankings

a procedure that requires the rater to measure individuals against one another instead of against an absolute scale;

one way to overcome the restriction-of-range errors (central tendency, leniency, severity errors)

halo effect

describes the fact that, for some raters, some ratees can do no wrong;

defined as a tendency to give a particular ratee a higher rating than he or she objectively deserves because of the rater’s failure to discriminate among conceptually distinct and potentially independent aspects of a ratee’s behavior

fairness

the extent to which a test is used in an impartial, just, and equitable way